Sample efficiency is one of the key factors when applying policy search to real-world problems.
In recent years, Bayesian Optimization (BO) has become prominent in the field of robotics due
to its sample efficiency and little prior knowledge needed. However, one drawback of BO is its
poor performance on high-dimensional search spaces as it focuses on global search. In the policy
search setting, local optimization is typically sufficient as initial policies are often available, e.g.,
via meta-learning, kinesthetic demonstrations or sim-to-real approaches. In this paper, we propose
to constrain the policy search space to a sublevel-set of the Bayesian surrogate model’s predictive
uncertainty. This simple yet effective way of constraining the policy update enables BO to scale to
high-dimensional spaces (>100) as well as reduces the risk of damaging the system. We demonstrate
the effectiveness of our approach on a wide range of problems, including a motor skills task, adapting
deep RL agents to new reward signals and a sim-to-real task for an inverted pendulum system.
Keywords: Local Bayesian Optimization, Policy Search, Robot Learning

In this paper, we propose a continuous finite-time-convergent active disturbance rejection control design scheme for a class of second order uncertain nonlinear systems. A continuous finite-time-convergent extended state observer is first developed to estimate the total disturbance which includes the nonlinear time-varying uncertain dynamics and the external disturbances. A continuous finite-time active disturbance rejection control design scheme is proposed by using saturation method. The key idea is using the estimate of the total disturbance to compensate the uncertainty such that the output tracks a given setpoint in finite time and stays at the setpoint afterwards, and all states of the closed-loop system remain uniformly bounded. By this dynamic compensation and saturation strategy, the finite-time stability of the error system is established under some weak restrictions on nonlinearities of the system. Experiments on a DC motor platform are carried out to illustrate the effectiveness of the proposed method.

This paper considers the problem of determining an optimal control action based on observed data. We formulate the problem assuming that the system can be modelled by a nonlinear state-space model, but where the model parameters, state and future disturbances are not known and are treated as random variables. Central to our formulation is that the joint distribution of these unknown objects is conditioned on the observed data. Crucially, as new measurements become available, this joint distribution continues to evolve so that control decisions are made accounting for uncertainty as evidenced in the data. The resulting problem is intractable which we obviate by providing approximations that result in finite dimensional deterministic optimisation problems. The proposed approach is demonstrated in simulation on a nonlinear system.

When learning policies for robot control, the required real-world data is typically prohibitively expensive to acquire, so learning in simulation is a popular strategy. Unfortunately, such polices are often not transferable to the real world due to a mismatch between the simulation and reality, called ‘reality gap’. Domain randomization methods tackle this problem by randomizing the physics simulator (source domain) during training according to a distribution over domain parameters in order to obtain more robust policies that are able to overcome the reality gap. Most domain randomization approaches sample the domain parameters from a fixed distribution. This solution is suboptimal in the context of sim-to-real transferability, since it yields policies that have been trained without explicitly optimizing for the reward on the real system (target domain). Additionally, a fixed distribution assumes there is prior knowledge about the uncertainty over the domain parameters. In this letter, we propose Bayesian Domain Randomization (BayRn), a black-box sim-to-real algorithm that solves tasks efficiently by adapting the domain parameter distribution during learning given sparse data from the real-world target domain. Bayesian Domain Randomization (BayRn) uses Bayesian optimization to search the space of source domain distribution parameters such that this leads to a policy which maximizes the real-word objective, allowing for adaptive distributions during policy optimization. We experimentally validate the proposed approach in sim-to-sim as well as in sim-to-real experiments, comparing against three baseline methods on two robotic tasks. Our results show that BayRn is able to perform sim-to-real transfer, while significantly reducing the required prior knowledge.

Artificial intelligence (AI) has an increasing influence in the manufacturing and processing industries. An increasingly interesting area of application is the design of controllers for technical systems through reinforcement learning. This allows the design of a controller with reduced effort for human experts, which is normally involved in the design process, for example by eliminating the determination of suitable controller parameters. In this paper a neural controller for a rotational inverted pendulum is developed by using artificial neural networks in combination with reinforcement learning. The rotational inverted pendulum is a classic example of a nonlinear and unstable system and has been little covered in previous publications on this subject. Furthermore, the behaviour of the neural controller is compared with a conventional controller when swinging up and balancing a rotational inverted pendulum.

At our University, the Dynamic Systems classes utilize a combination of the Quanser® Qube servo 2 hardware, the National Instruments® myRIO units, and Mathworks® MATLAB/Simulink software in its lab portions. The combination of the Qube, the myRIO, and MATLAB has been especially useful in systems control classes. But this usefulness has been limited to the Qubes two attachments: the inertial disk and the inverted pendulum. The purpose of this project was to extend the usefulness of the Quanser Qube servo 2. This is achieved by the development of new attachments for it that use its existing equipment in various and new ways with some new programming in MATLAB. The first set of new attachments are different sized inertial disks with different inertias. The next set uses two Qubes to setup a drive gear and belt drive between the two using different sized gears. Another new attachment is a four-bar linkage for use on a single Qube, and the last one being a customizable wind turbine attachment.

This paper will first present the background of the implementation of the Qube and the myRIO in the course and why this project was started. Next it will introduce each of the custom-made attachments or sets of attachments in their own section. Each section will describe the attachment and what can be done with them and how they can be implemented into the course.

This paper is dedicated to the experimental analysis of discrete-time differentiators implemented in closed-loop control systems. To this end, two laboratory setups, namely an electro-pneumatic system and a rotary inverted pendulum have been used to implement 25 different differentiators. Since the
selected laboratory setups behave differently in the case of dynamic response and noise characteristics, it is expected that the results remain valid for a wide range of control applications. The validity of several theoretical results, which have been already reported in the literature using mathematical analysis and
numerical simulations, has been investigated, and several comments are provided to allow one to select an appropriate differentiation scheme in practical closed-loop control systems.

The main goal of the paper is to test the Embedded Model Control (EMC) design and implementation on a typical underactuated apparatus, like the Furuta pendulum, by comparing experimental results with a Linear Quadratic Regulator (LQR). EMC can be considered as a disturbance rejection control strategy, since the state predictor is extended to explicitly include disturbance dynamics, in charge of predicting the uncertainty to be rejected by control law. Essential in EMC design is the separation between controllable and not controllable dynamics, a task which allows us to find the controllable channel of underactuated systems from the low-dimensional command to the whole system degrees of freedom (DF). Pursuing this objective, a rather generic method is shown, which is applicable to other underactuated systems. The result is a very simple controllable dynamics from the single pendulum command to pendulum DF arranged in a single series of controllable integrators. The neglected feedback channels, including the unstable gravity feedback, are treated as unknown thus posing a challenge to disturbance prediction and closed loop stability. Typical in EMC, closed loop eigenvalues are chosen to guarantee stability, a pre-requisite to performance. Experimental results point out effectiveness and advantage, with respect to LQR, of design and implementation under adverse conditions, due to a disturbance pulse, in which command saturates.

As deep learning becomes more prevalent for prediction and control of real physical systems, it is important that these overparameterized models are consistent with physically plausible dynamics. This elicits a problem with how much inductive bias to impose on the model through known physical parameters and principles to reduce complexity of the learning problem to give us more reliable predictions. Recent work employs discrete variational integrators parameterized as a neural network architecture to learn conservative Lagrangian systems. The learned model captures and enforces global energy preserving properties of the system from very few trajectories. However, most real systems are inherently non-conservative and, in practice, we would also like to apply actuation. In this paper we extend this paradigm to account for general forcing (e.g. control input and damping) via discrete d'Alembert's principle which may ultimately be used for control applications. We show that this forced variational integrator networks (FVIN) architecture allows us to accurately account for energy dissipation and external forcing while still capturing the true underlying energy-based passive dynamics. We show that in application this can result in highly-data efficient model-based control and can predict on real non-conservative systems.

The objective of this work is to design a fuzzy rule-based set point weighting mechanism for fuzzy PID (FPID) controller so that an overall improved closed-loop performance may be achieved for linear as well as nonlinear process models. Till date, tuning criteria for FPID controllers are not well defined. Trial-and-error approach is primarily adopted and it is quite time-consuming and does not always ensure improved overall closed-loop behaviour. Hence, to ascertain satisfactory closed-loop performance with an initially tuned fuzzy controller, a fuzzy rule-based set point weighting mechanism is reported here. The proposed scheme is capable of providing performance enhancement with instantaneous weighting factor calculated online for each instant based on the latest process operating conditions. The proposed methodology is capable of ascertaining acceptable performances during set point tracking as well as load recovery phases. Efficacy of the proposed scheme is verified for linear as well as nonlinear process models through simulation study along with real-time verification on servo position control in comparison with the others’ reported performance augmentation schemes as well as fuzzy sliding mode control.

Proximal and trust-region policy optimization methods (PPO and TRPO) belong to the standard reinforcement learning toolbox. Notably, PPO can be viewed as transforming the constrained TRPO problem into an unconstrained one, either via turning the constraint into a penalty or via objective clipping. In this chapter, an alternative problem reformulation is studied, where the information loss is bounded using a novel transformation of the KullbackLeibler (KL) divergence constraint. In contrast to PPO, the considered method does not require tuning of the regularization parameter, which is known to be hard due to its sensitivity to the reward scaling. The resulting algorithm, termed information-loss-bounded policy optimization (ILBPO), both enjoys the benefits of the first-order methods, being straightforward to implement using automatic differentiation, and maintains the advantages of the quasi-second order methods. It performs competitively in simulated OpenAI MuJoCo environments and achieves robust performance on a real robotic task of the Furuta pendulum swing-up and stabilization.

This paper deals with the practical implementation of implicit homogeneous controllers (IHCs) for linearized mechanical systems. The control design includes the methodology to get gains of the IHC based on the linearized approximation of the system. If the approximation error enforced by the linearization is vanishing with the state, locally measurable and bounded, the IHC can lead the state to the origin in finite-time. This IHC control allows accelerating the convergence rate of the states. A semi-explicit algorithm is provided to exert the digital implementation of the controller. The application of the bisection method estimates the controller gain ensuring the finite-time convergence of the state to the origin. A complementary analysis provides a simplified algorithm with a reduced number of computation stages but equally efficient gain estimation. The proposed IHC is applied to a rotary inverted pendulum QUBE Servo 2 platform of Quanser. The obtained results for the state convergence are compared with other classical feedback controllers to validate the effectiveness of the proposed scheme. The comparative analysis of the state convergence provides evidence of the faster convergence for the state trajectory to a zone centered on the origin with smaller hypervolume than the one gotten with the classical controllers.

The goal of this paper is to investigate if it is possible to upgrade a given linear controller to a sliding mode one with an improvement of the control performance. Starting from a given linear controller, a design procedure for a sliding mode control having better performance than the linear one, is proposed. If the system has disturbances, which is always the case in experiments, the upgraded sliding mode control brings also a better robustness with respect to the given linear robust controller. The main idea is to divide the state-space into two areas, introducing a design parameter which separates the area of the linear control from the area of the sliding mode control. Some issues related to the chattering reduction are discussed. The control scheme’s efficiency is demonstrated experimentally on a rotary inverted pendulum. The experimental results demonstrate the effectiveness of the obtained controls, and show an improvement with respect to the given linear proportional control.

@conference{srinivasa_2020,
title = {A Hardware Proof of Concept of Networked Adaptive Systems},
author = {Srinivasa, S.B.; Makam, R.; George, K.},
booktitle = {2020 6th International Conference on Control, Automation and Robotics (ICCAR)},
year = {2020},
institution = {PES University, India},
abstract = {A hardware proof of concept of adaptation over the internet to achieve stability and output tracking despite the presence of both packet dropouts and delays induced by the network is provided in this paper. Assuming that the parameters of the models of the physical systems are unknown, a modified model reference adaptive control law for such networked adaptive systems ensures that the outputs of these systems asymptotically track specified desired trajectories. Further, we demonstrate that the multiple models, switching, and tuning methodology improves the transient performance. The example systems considered here are the Linear Servo Base Unit and the Qube Servo. },
keywords = {adaptive control, discrete-time systems, communication network, linear system},
language = {English},
publisher = {IEEE},
isbn = {978-1-7281-6140-2}
}

A hardware proof of concept of adaptation over the internet to achieve stability and output tracking despite the presence of both packet dropouts and delays induced by the network is provided in this paper. Assuming that the parameters of the models of the physical systems are unknown, a modified model reference adaptive control law for such networked adaptive systems ensures that the outputs of these systems asymptotically track specified desired trajectories. Further, we demonstrate that the multiple models, switching, and tuning methodology improves the transient performance. The example systems considered here are the Linear Servo Base Unit and the Qube Servo.

It is observed that classical controller design techniques, i.e., particularly fixed type of PI/PD/PID analytical control approach, fails for the parametric uncertainty issues. However, the graphical approach for PID tuning and advanced control techniques such as H infinity, Quantitative feedback technique (QFT) have already been developed for handling parametric uncertainty, but these approaches are complex. Because of this, instead of using advanced controller techniques and graphical PID tuning approach, a variant of PID, i.e., a fixed type proportional derivative (PD) controller design is proposed for a single input single output plant (SISO) interval system with no zeros using Krishnamurthy's approach based on Routh criterion, Kharitonov's theorem and model order reduction approach based on Routh criterion. The beauty of the proposed approach is that the direct formulae is proposed for the tuning of the PD controller for an interval system. The proposed controller design gives the necessary and sufficient condition of stability and also the desired performance. The proposed approach is validated through simulation, and further experimental validation is carried out on the Servo system.

@article{muratore_2020,
title = {Bayesian Domain Randomization for Sim-to-Real Transfer},
author = {Muratore, F.; Eilers, C.; Gienger, M.; Peters, J.},
journal = {arXiv},
year = {2020},
institution = {Technical University Darmstadt, Germany; Honda Research Institute Europe, Germany},
abstract = {When learning policies for robot control, the real-world data required is typically prohibitively expensive to acquire, so learning in simulation is a popular strategy. Unfortunately, such polices are often not transferable to the real world due to a mismatch between the simulation and reality, called 'reality gap'. Domain randomization methods tackle this problem by randomizing the physics simulator (source domain) according to a distribution over domain parameters during training in order to obtain more robust policies that are able to overcome the reality gap. Most domain randomization approaches sample the domain parameters from a fixed distribution. This solution is suboptimal in the context of sim-to-real transferability, since it yields policies that have been trained without explicitly optimizing for the reward on the real system (target domain). Additionally, a fixed distribution assumes there is prior knowledge about the uncertainty over the domain parameters. Thus, we propose Bayesian Domain Randomization (BayRn), a black box sim-to-real algorithm that solves tasks efficiently by adapting the domain parameter distribution during learning by sampling the real-world target domain. BayRn utilizes Bayesian optimization to search the space of source domain distribution parameters which produce a policy that maximizes the real-word objective, allowing for adaptive distributions during policy optimization. We experimentally validate the proposed approach by comparing against two baseline methods on a nonlinear under-actuated swing-up task. Our results show that BayRn is capable to perform direct sim-to-real transfer, while significantly reducing the required prior knowledge. },
keywords = {machine learning},
language = {English}
}

When learning policies for robot control, the real-world data required is typically prohibitively expensive to acquire, so learning in simulation is a popular strategy. Unfortunately, such polices are often not transferable to the real world due to a mismatch between the simulation and reality, called 'reality gap'. Domain randomization methods tackle this problem by randomizing the physics simulator (source domain) according to a distribution over domain parameters during training in order to obtain more robust policies that are able to overcome the reality gap. Most domain randomization approaches sample the domain parameters from a fixed distribution. This solution is suboptimal in the context of sim-to-real transferability, since it yields policies that have been trained without explicitly optimizing for the reward on the real system (target domain). Additionally, a fixed distribution assumes there is prior knowledge about the uncertainty over the domain parameters. Thus, we propose Bayesian Domain Randomization (BayRn), a black box sim-to-real algorithm that solves tasks efficiently by adapting the domain parameter distribution during learning by sampling the real-world target domain. BayRn utilizes Bayesian optimization to search the space of source domain distribution parameters which produce a policy that maximizes the real-word objective, allowing for adaptive distributions during policy optimization. We experimentally validate the proposed approach by comparing against two baseline methods on a nonlinear under-actuated swing-up task. Our results show that BayRn is capable to perform direct sim-to-real transfer, while significantly reducing the required prior knowledge.

@article{frohlich_2020,
title = {Bayesian Optimization for Policy Search in High-Dimensional Systems via Automatic Domain Selection},
author = {Fröhlich, L.P.; Klenske, E.D.; Daniel, C.G.; Zeilinger, M.N.},
journal = {arXiv},
year = {2020},
institution = {Bosch Center for Artificial Intelligence, Germany; ETH Zurich, Switzerland},
abstract = {Bayesian Optimization (BO) is an effective method for optimizing expensive-to-evaluate black-box functions with a wide range of applications for example in robotics, system design and parameter optimization. However, scaling BO to problems with large input dimensions (>10) remains an open challenge. In this paper, we propose to leverage results from optimal control to scale BO to higher dimensional control tasks and to reduce the need for manually selecting the optimization domain. The contributions of this paper are twofold: 1) We show how we can make use of a learned dynamics model in combination with a model-based controller to simplify the BO problem by focusing onto the most relevant regions of the optimization domain. 2) Based on (1) we present a method to find an embedding in parameter space that reduces the effective dimensionality of the optimization problem. To evaluate the effectiveness of the proposed approach, we present an experimental evaluation on real hardware, as well as simulated tasks including a 48-dimensional policy for a quadcopter. },
language = {English}
}

Bayesian Optimization (BO) is an effective method for optimizing expensive-to-evaluate black-box functions with a wide range of applications for example in robotics, system design and parameter optimization. However, scaling BO to problems with large input dimensions (>10) remains an open challenge. In this paper, we propose to leverage results from optimal control to scale BO to higher dimensional control tasks and to reduce the need for manually selecting the optimization domain. The contributions of this paper are twofold: 1) We show how we can make use of a learned dynamics model in combination with a model-based controller to simplify the BO problem by focusing onto the most relevant regions of the optimization domain. 2) Based on (1) we present a method to find an embedding in parameter space that reduces the effective dimensionality of the optimization problem. To evaluate the effectiveness of the proposed approach, we present an experimental evaluation on real hardware, as well as simulated tasks including a 48-dimensional policy for a quadcopter.

@article{de-maity2_2020,
title = {Comparative Performance Study of Optimal Interval Type-2 Fuzzy PID Controllers with Practical System},
author = {Rani De (Maity), R.; Mudi, R.K.; Dey, C.},
journal = {International Journal of Computer Sciences and Engineering},
year = {2020},
month = {03},
volume = {8},
number = {3},
institution = {Jadavpur University, India; University of Calcutta, India},
abstract = {In this paper, the input and output scaling factors of the type-2 fuzzy PID Controller (IT2-FPID) are determined using three different optimization algorithms (Cuckoo search (CS), Particle swarm optimization (PSO), and Bee colony algorithm (BCA)) for a first-order integrating plus dead time (FOIPD) model. A comparative performance study is made for these three optimization algorithms in terms of various transient performance indices. The comparative analysis on the experimental results reveals that BCA based optimal IT2-FPID shows better performance on a simulation model whereas CS based optimal IT2-FPID is found to be superior for practical system over other algorithms. },
keywords = {Particle swarm optimization(PSO), Cuckoo search algorithm (CS), Bee colony algorithm(BCA), Interval type-2 fuzzy controller},
language = {English}
}

In this paper, the input and output scaling factors of the type-2 fuzzy PID Controller (IT2-FPID) are determined using three different optimization algorithms (Cuckoo search (CS), Particle swarm optimization (PSO), and Bee colony algorithm (BCA)) for a first-order integrating plus dead time (FOIPD) model. A comparative performance study is made for these three optimization algorithms in terms of various transient performance indices. The comparative analysis on the experimental results reveals that BCA based optimal IT2-FPID shows better performance on a simulation model whereas CS based optimal IT2-FPID is found to be superior for practical system over other algorithms.

In this brief, we study the cooperative output regulation of linear multiagent systems under switching communication topology and exosystem detectability constraints, and applications to the synchronization of networked motors. Compared to similar works in the literature, in this brief, we consider the problem scenario in which none of the agents can estimate the exosystem states from their individual measurements on any of the switching configurations of the system. In other words, no agent in the system can solve the output regulation problem independently. By devising a distributed observer on the exosystem states, and under intuitive assumptions on the combined detectability of the multiagent system and joint connectivity of the information graphs, we are able to solve the problem by a distributed feedback protocol. Finally, the developed theoretical results are applied to the synchronization problem of a group of motors in a time-varying communication network.

This paper proposes a design of a robust controller for a rotary motion control system that includes a PID controller, a disturbance observer, and a friction compensator. Friction force versus angular velocity has been measured, and viscous, Coulomb friction and stiction components have been identified. With nominal PID (proportional- integral-derivative) controller, we have observed adverse effects due to friction such as excessive steady-state errors, oscillations, and limit-cycles. By adding a friction model as an augmented nonlinear dynamics of a plant, we are able to conduct a simulation study of a motion control system that matches very well with experimental results. The disturbance observer (DOB) based on simple and effective robust control theory has been implemented to make the rotary motion control system “robust” against inertia/load variations, external torque disturbances, and some of friction forces. Further performance enhancement of the DOB-based robust motion control system has been achieved by adding the friction compensator and experimentally verified.

Recently, reinforcement learning has been applied to various fields and shows better performance than humans. In particular, it is attracting attention in the fields of smart factories and robotics that require automatic control without human intervention. In this paper, we try to allow multiple reinforcement learning agents to learn optimal control policy on their own IoT devices of the same type. There is no guarantee that the reinforcement learning agent that has learned the optimal control policy using one IoT device will perform optimal control of other IoT devices. Therefore, since reinforcement learning must be performed individually for each IoT device, it takes a lot of time and cost. To solve this problem, we propose a new method of federated reinforcement learning. In the proposed federated reinforcement learning, multiple agents have independent IoT devices, perform learning at the same time, and federate with each other to improve learning performance. Therefore, we apply a new gradient sharing method and transfer learning to reinforcement learning. In addition, Actor-Critic PPO, which shows good performance in reinforcement learning algorithms, is used. And, for smooth learning in the IoT environment where numerous devices exist, we propose an architecture based on Software-Defined Networking. Using multiple rotary inverted pendulum devices interconnected via a SDN, we demonstrate that the proposed federated reinforcement learning scheme can effectively facilitate the learning process for multiple IoT devices.

@conference{lim2_2020,
title = {Federated Reinforcement Learning for Controlling Multiple Rotary Inverted Pendulums in Edge Computing Environments},
author = {Lim, H.-K.; Kim, J.-B.; Kim, C.-M.; Hwang, G.-Y.; Choi, H. ; Han, Y.-H.},
booktitle = {2020 International Conference on Artificial Intelligence in Information and Communication (ICAIIC)},
year = {2020},
institution = {Korea University of Technology and Education, Korea},
abstract = {Reinforcement learning has recently been studied in various fields and also used to optimally control real devices (e.g., robotic arms). In this paper, we try to allow multiple reinforcement learning agents to learn optimal control policy on their own devices of the same type but with slightly different dynamics. For such multiple devices, there is no guarantee that an agent who interacts only with one device and learns the optimal control policy will also control another device well. Therefore, we may need to apply independent reinforcement learning to each device individually, which requires time-consuming effort. To solve this problem, we propose a new federated reinforcement learning architecture where each agent working on its independent device shares their learning experience with each other, and transfers a mature policy model parameters into other agents. We incorporate the Actor-Critic PPO algorithm into each agent in the proposed collaborative architecture, and propose an efficient procedure for the gradient sharing and the model transfer. We also use edge computing to solve network problems that occur when training multiple real devices at the same time. Using multiple rotary inverted pendulum devices, we demonstrate that the proposed federated reinforcement learning scheme can effectively facilitate the learning process for multiple devices, and that the learning speed can be faster if more agents are involved. },
keywords = {Actor-Critic PPO, Edge computing, Federated reinforcement learning, Multiple RIP control},
language = {English},
publisher = {IEEE},
isbn = {978-1-7281-4986-8}
}

Reinforcement learning has recently been studied in various fields and also used to optimally control real devices (e.g., robotic arms). In this paper, we try to allow multiple reinforcement learning agents to learn optimal control policy on their own devices of the same type but with slightly different dynamics. For such multiple devices, there is no guarantee that an agent who interacts only with one device and learns the optimal control policy will also control another device well. Therefore, we may need to apply independent reinforcement learning to each device individually, which requires time-consuming effort. To solve this problem, we propose a new federated reinforcement learning architecture where each agent working on its independent device shares their learning experience with each other, and transfers a mature policy model parameters into other agents. We incorporate the Actor-Critic PPO algorithm into each agent in the proposed collaborative architecture, and propose an efficient procedure for the gradient sharing and the model transfer. We also use edge computing to solve network problems that occur when training multiple real devices at the same time. Using multiple rotary inverted pendulum devices, we demonstrate that the proposed federated reinforcement learning scheme can effectively facilitate the learning process for multiple devices, and that the learning speed can be faster if more agents are involved.

@article{lim_20220,
title = {Federated Reinforcement Learning for Training Control Policies on Multiple IoT Devices},
author = {Lim, H.-K.; Kim, J.-B.; Heo, J.-S.; Han, Y.-H. },
journal = {Sensors},
year = {2020},
volume = {20},
number = {5},
institution = {Korea University of Technology and Education, Korea},
abstract = {Reinforcement learning has recently been studied in various fields and also used to optimally control IoT devices supporting the expansion of Internet connection beyond the usual standard devices. In this paper, we try to allow multiple reinforcement learning agents to learn optimal control policy on their own IoT devices of the same type but with slightly different dynamics. For such multiple IoT devices, there is no guarantee that an agent who interacts only with one IoT device and learns the optimal control policy will also control another IoT device well. Therefore, we may need to apply independent reinforcement learning to each IoT device individually, which requires a costly or time-consuming effort. To solve this problem, we propose a new federated reinforcement learning architecture where each agent working on its independent IoT device shares their learning experience (i.e., the gradient of loss function) with each other, and transfers a mature policy model parameters into other agents. They accelerate its learning process by using mature parameters. We incorporate the actor–critic proximal policy optimization (Actor–Critic PPO) algorithm into each agent in the proposed collaborative architecture and propose an efficient procedure for the gradient sharing and the model transfer. Using multiple rotary inverted pendulum devices interconnected via a network switch, we demonstrate that the proposed federated reinforcement learning scheme can effectively facilitate the learning process for multiple IoT devices and that the learning speed can be faster if more agents are involved. },
issn = {1424-8220},
keywords = { Actor–Critic PPO; federated reinforcement learning; multi-device control},
language = {English},
publisher = {MDPI AG}
}

Reinforcement learning has recently been studied in various fields and also used to optimally control IoT devices supporting the expansion of Internet connection beyond the usual standard devices. In this paper, we try to allow multiple reinforcement learning agents to learn optimal control policy on their own IoT devices of the same type but with slightly different dynamics. For such multiple IoT devices, there is no guarantee that an agent who interacts only with one IoT device and learns the optimal control policy will also control another IoT device well. Therefore, we may need to apply independent reinforcement learning to each IoT device individually, which requires a costly or time-consuming effort. To solve this problem, we propose a new federated reinforcement learning architecture where each agent working on its independent IoT device shares their learning experience (i.e., the gradient of loss function) with each other, and transfers a mature policy model parameters into other agents. They accelerate its learning process by using mature parameters. We incorporate the actor–critic proximal policy optimization (Actor–Critic PPO) algorithm into each agent in the proposed collaborative architecture and propose an efficient procedure for the gradient sharing and the model transfer. Using multiple rotary inverted pendulum devices interconnected via a network switch, we demonstrate that the proposed federated reinforcement learning scheme can effectively facilitate the learning process for multiple IoT devices and that the learning speed can be faster if more agents are involved.

@inbook{mani_2020,
title = {Implementation of ACO Tuned Modified PI-like Position and Speed Control of DC Motor: An Application to Electric Vehicle},
author = {Mani, G.},
booktitle = {Soft Computing for Problem Solving},
year = {2020},
institution = {Vellore Institute of Technology, India},
abstract = {The modified PI-like control scheme has been derived from classic Internal Model Control (IMC), having two tuning parameters namely controller gain Kc and reset time τI. The controller action resembles like PI control action with inherent reset windup protection and dead time compensation, except the inverse of the process model gain is set as controller gain and dominant time constant of the process model set as reset time. The controller structure is feasible enough to implement in hardware. When an accurate process model is not available, the metaheuristic algorithms like Ant Colony Optimization (ACO) is applied to find the optimal tuning parameters. The performance of proposed control scheme for both speed control and position tracking is compared and analyzed under different operating conditions with well-known Proportional-Velocity (PV) control scheme and Two-Degree-of-Freedom (2-DoF) PID structure. The experimental evaluations have been demonstrated via computer-interfaced QUBE-servo DC Motor through Hardware in Loop (HiL) implementation. },
keywords = {Modified PI-like, 2-DoF, QUBE-servo DC motor, IAE, ISE, ACO },
language = {English},
series = {Advances in Intelligent Systems and Computing},
publisher = {Springer Nature Switzerland},
isbn = {978-981-15-0035-0},
pages = {629-645}
}

The modified PI-like control scheme has been derived from classic Internal Model Control (IMC), having two tuning parameters namely controller gain Kc and reset time τI. The controller action resembles like PI control action with inherent reset windup protection and dead time compensation, except the inverse of the process model gain is set as controller gain and dominant time constant of the process model set as reset time. The controller structure is feasible enough to implement in hardware. When an accurate process model is not available, the metaheuristic algorithms like Ant Colony Optimization (ACO) is applied to find the optimal tuning parameters. The performance of proposed control scheme for both speed control and position tracking is compared and analyzed under different operating conditions with well-known Proportional-Velocity (PV) control scheme and Two-Degree-of-Freedom (2-DoF) PID structure. The experimental evaluations have been demonstrated via computer-interfaced QUBE-servo DC Motor through Hardware in Loop (HiL) implementation.

@article{tran_2019,
title = {Implementing Homomorphic Encryption Based Secure Feedback Control for Physical Systems},
author = {Tran, .; Farokhi, F.; Cantoni, M.; Shames, I.},
journal = {Control Engineering Practice},
year = {2020},
month = {04},
volume = {97},
institution = {University of Melbourne, Australia},
abstract = {This paper is about an encryption based approach to the secure implementation of feedback controllers for physical systems. Specifically, Paillier's homomorphic encryption is used in a custom digital implementation of a class of linear dynamic controllers, including static gain as a special case. The implementation is amenable to Field Programmable Gate Array (FPGA) realization. Experimental results, including timing analysis and resource usage characteristics for different encryption key lengths, are presented for the realization of a controller for an inverted pendulum; as this is an unstable plant, the control is necessarily fast. },
keywords = {secure control, homomorphic encryption, Paillier encryption, digital design, FPGA},
language = {English},
publisher = {Elsevier B.V.}
}

This paper is about an encryption based approach to the secure implementation of feedback controllers for physical systems. Specifically, Paillier's homomorphic encryption is used in a custom digital implementation of a class of linear dynamic controllers, including static gain as a special case. The implementation is amenable to Field Programmable Gate Array (FPGA) realization. Experimental results, including timing analysis and resource usage characteristics for different encryption key lengths, are presented for the realization of a controller for an inverted pendulum; as this is an unstable plant, the control is necessarily fast.

Abstract: Sensors and encoders had become an essential part of all the machines. Without these, a closedloop feedback system cannot be made. However, the system is difficult to be controlled precisely due to the external noise corruption in the signal received in sensors or encoders. A common solution to reduce the noise is employing a low pass filter. Although low pass filter can helped to reduce noise, but it has side effect on lowering the response time and therefore it may not so suitable in most of the application by only deploying low pass filter itself. A filtering module usually integrated with other modules to improve a system. One of the famous modules is Proportional-Integral-Derivative (PID) controller. The plant to be controlled in this module was a DC motor which can be regarded as an inductive load, therefore a proportional-derivative (PD) controller is desired. Therefore, the objectives of this paper are to find the suitable value of cut-off frequency for the low-pass filter and find the optimum value of kp and kd in the PD controller which satisfy the transient and stead-state performance. The response time varies with the cutoff frequency in the low-pass filter. Since the response time does not differ much in between cut-off
frequency value, therefore, the cut-off frequency value was set at 150 rad s-1 in which the noise signal is significantly eliminated. As for the PD controller, the observation result showed that when kp increased, the settling time of the system and the percentage of peak overshoot also increased. Meanwhile, the increasing of value of kd reduce the settling time and the percentage of overshoot. The optimum value for kp and kd were calculated using formula and the results are 5.3678 and 0.2094 respectively. Since the formula does not include the external disturbance into the calculation, therefore the actual response will be slightly different from the target response. Therefore, the understanding of property changes in kp and kd able to help in fine-funing for these two values. After fine-tuning, the value for kp and kd are 7.0 and 0.2094 respectively.

@article{de-maity_2020,
title = {Nature-inspired and hybrid optimization algorithms on interval Type-2 fuzzy controller for servo processes: a comparative performance study},
author = {De (Maity), R.R.; Mudi, R.K.; Dey, C.},
journal = {SN Applied Sciences},
year = {2020},
institution = {Jadavpur University, India; University of Calcutta, India},
abstract = {In this paper, performance evaluations of six well-known nature-inspired algorithms have been reported containing genetic algorithm, cuckoo search, particle swarm optimization, differential evolution, bee colony, and combined particle swarm optimization and differential evolution (CPSODE) algorithms. Based on these optimization algorithms, input and output scaling factors of an interval Type-2 fuzzy PID controller (IT2-FLC) are chosen for closed-loop servo tracking. Optimal values of the scaling factors are chosen by minimization of the objective function which is defined based on the closed-loop controller performance criteria. A detailed comparative analysis is reported based on the simulation and experimental results. Performance analysis reveals that improved performance, reliability, robustness, and lesser noise sensitivity are reported by IT2-FLC with the optimal values obtained by the hybrid algorithm CPSODE. },
keywords = {Genetic algorithm (GA), Particle swarm optimization (PSO), Differential evolution (DE), Cuckoo search (CS), Bee colony (BC), Combined particle swarm optimization and differential evolution (CPSODE) algorithms, Interval Type-2 fuzzy controller, Servo tracking process},
language = {English},
publisher = {Springer Nature Switzerland}
}

In this paper, performance evaluations of six well-known nature-inspired algorithms have been reported containing genetic algorithm, cuckoo search, particle swarm optimization, differential evolution, bee colony, and combined particle swarm optimization and differential evolution (CPSODE) algorithms. Based on these optimization algorithms, input and output scaling factors of an interval Type-2 fuzzy PID controller (IT2-FLC) are chosen for closed-loop servo tracking. Optimal values of the scaling factors are chosen by minimization of the objective function which is defined based on the closed-loop controller performance criteria. A detailed comparative analysis is reported based on the simulation and experimental results. Performance analysis reveals that improved performance, reliability, robustness, and lesser noise sensitivity are reported by IT2-FLC with the optimal values obtained by the hybrid algorithm CPSODE.

@conference{wisotzki_2020,
title = {On IMC-Based PID Tuning Using Gain-Integrator-Delay Dynamics},
author = {Wisotzki, S.; Brahma, S.; Ossareh, H.R.},
booktitle = {2020 IEEE Conference on Control Technology and Applications (CCTA)},
year = {2020},
institution = {Booz Allen Hamilton, USA; University of Vermont, USA},
abstract = {The Internal Model Control (IMC)-based PID tuning using integrator dynamics is an effective technique for tuning the boost pressure control system in a turbocharged gasoline engine, as investigated in a previous work by the authors. In that work, two IMC-based PID tuning approaches were delineated: one that involved a post hoc modification of initial design to achieve a desired closed-loop performance, and another that assigned a desired closed-loop bandwidth exactly. This paper extends the results of that work by introducing a third design approach employing the technique: one that assigns the desired phase margin exactly. Also, the viability of all three approaches for use with more general plants that are less similar to the boost system is investigated through Monte Carlo simulations. Finally, to illustrate their effectiveness in practice, all three design approaches are applied to a DC motor control problem. },
keywords = {Bandwidth, Tuning, Relays, Engines, DC motors, Gain, Monte Carlo methods},
language = {English},
publisher = {IEEE},
isbn = {978-1-7281-7141-8}
}

The Internal Model Control (IMC)-based PID tuning using integrator dynamics is an effective technique for tuning the boost pressure control system in a turbocharged gasoline engine, as investigated in a previous work by the authors. In that work, two IMC-based PID tuning approaches were delineated: one that involved a post hoc modification of initial design to achieve a desired closed-loop performance, and another that assigned a desired closed-loop bandwidth exactly. This paper extends the results of that work by introducing a third design approach employing the technique: one that assigns the desired phase margin exactly. Also, the viability of all three approaches for use with more general plants that are less similar to the boost system is investigated through Monte Carlo simulations. Finally, to illustrate their effectiveness in practice, all three design approaches are applied to a DC motor control problem.

@article{lee_2020,
title = {Performance Enhancement of Motion Control Systems Through Friction Identification and Compensation},
author = {Lee, H. S.; Jung, S.; Ryu, S.},
journal = {Journal of the Korean Society of Manufacturing Process Engineers},
year = {2020},
volume = {19},
number = {6},
institution = {Gyeongsang National University, Korea},
abstract = {This paper proposes a method for measuring friction forces and creating a friction model for a rotary motion control system as well as an autonomous vehicle testbed. The friction forces versus the velocity were measured, and the viscous friction, Coulomb friction, and stiction were identified. With a nominal PID (proportional-integral-derivative) controller, we observed the adverse effects due to friction, such as excessive steady-state errors, oscillations, and limit-cycles. By adding an adequate friction model as part of the augmented nonlinear dynamics of a plant, we were able to conduct a simulation study of a motion control system that well matched experimental results. We have observed that the implementation of a model-based friction compensator improves the overall performance of both motion control systems, i.e., the rotary motion control system and the Altino testbed for autonomous vehicle development. By utilizing a better simulation tool with an embedded friction model, we expect that the overall development time and cost can be reduced. },
issn = {1598-6721},
keywords = {Position Control; Motion Control System; Friction Modeling; Friction Compensation },
language = {English},
publisher = {Korea Institute of Science and Technology Information}
}

This paper proposes a method for measuring friction forces and creating a friction model for a rotary motion control system as well as an autonomous vehicle testbed. The friction forces versus the velocity were measured, and the viscous friction, Coulomb friction, and stiction were identified. With a nominal PID (proportional-integral-derivative) controller, we observed the adverse effects due to friction, such as excessive steady-state errors, oscillations, and limit-cycles. By adding an adequate friction model as part of the augmented nonlinear dynamics of a plant, we were able to conduct a simulation study of a motion control system that well matched experimental results. We have observed that the implementation of a model-based friction compensator improves the overall performance of both motion control systems, i.e., the rotary motion control system and the Altino testbed for autonomous vehicle development. By utilizing a better simulation tool with an embedded friction model, we expect that the overall development time and cost can be reduced.

In this paper, a self-tuning rule-based position control algorithm is proposed for DC motors with system parameter estimation using the recursive least squares method. First, a mathematical model of the angular position control of a DC motor was derived. Next, the timevarying parameters including the rotational inertia in the model were estimated using the RLS method along with multiple forgetting factors without prior knowledge of the system. Based on the derived model and the parameter estimation, a sliding mode control algorithm was designed by applying a self-tuning rule that enables the magnitude of the voltage input to be adaptively adjusted for improvement of the energy efficiency. The performance of the designed control algorithm was then experimentally evaluated under several different load conditions. Finally, the evaluation results show that the designed controller achieves a satisfactory capability for a DC motor to deal with both tracking accuracy and energy efficiency without prior knowledge of the system.

@conference{truchetta_2019,
title = {Robust Model-free Reinforcement Learning with Multi-objective Bayesian Optimization},
author = {Turchetta, M.; Krause, A.; Trimpe, S.},
booktitle = {IEEE International Conference on Robotics and Automation (ICRA)},
year = {2020},
institution = {ETH Zurich, Switzerland; Max Planck ETH Center for Learning Systems, Germany},
abstract = {In reinforcement learning (RL), an autonomous agent learns to perform complex tasks by maximizing an exogenous reward signal while interacting with its environment. In real-world applications, test conditions may differ substantially from the training scenario and, therefore, focusing on pure reward maximization during training may lead to poor results at test time. In these cases, it is important to trade-off between performance and robustness while learning a policy. While several results exist for robust, model-based RL, the model-free case has not been widely investigated. In this paper, we cast the robust, model-free RL problem as a multi-objective optimization problem. To quantify the robustness of a policy, we use delay margin and gain margin, two robustness indicators that are common in control theory. We show how these metrics can be estimated from data in the model-free setting. We use multi-objective Bayesian optimization (MOBO) to solve efficiently this expensive-to-evaluate, multi-objective optimization problem. We show the benefits of our robust formulation both in sim-to-real and pure hardware experiments to balance a Furuta pendulum. },
keywords = {robotics, artificial intelligence, machine learning},
language = {English}
}

In reinforcement learning (RL), an autonomous agent learns to perform complex tasks by maximizing an exogenous reward signal while interacting with its environment. In real-world applications, test conditions may differ substantially from the training scenario and, therefore, focusing on pure reward maximization during training may lead to poor results at test time. In these cases, it is important to trade-off between performance and robustness while learning a policy. While several results exist for robust, model-based RL, the model-free case has not been widely investigated. In this paper, we cast the robust, model-free RL problem as a multi-objective optimization problem. To quantify the robustness of a policy, we use delay margin and gain margin, two robustness indicators that are common in control theory. We show how these metrics can be estimated from data in the model-free setting. We use multi-objective Bayesian optimization (MOBO) to solve efficiently this expensive-to-evaluate, multi-objective optimization problem. We show the benefits of our robust formulation both in sim-to-real and pure hardware experiments to balance a Furuta pendulum.

@article{zhang2_2020,
title = {Robust Non-Minimal State Feedback Control for a Furuta Pendulum with Parametric Modelling Errors},
author = {Zhang, L.; Dixon, R.},
journal = { IEEE Transactions on Industrial Electronics},
year = {2020},
institution = {National University of Defense Technology, China; Birmingham University, United Kingdom},
abstract = {Non-minimal state feedback (NMSF) control, also known as proportional-integral-plus (PIP) control, together with anti-windup integrator, is applied, for the first time, to stabilize a Furuta pendulum (FP). The key elements in the non-minimal state space (NMSS) state transition matrix is obtained directly from the original minimal state space (MSS) model. The NMSS model is proved to be controllable if the controllability conditions are met. In order to demonstrate the potential robustness of NMSF control, the controller is designed based on the model with two deliberate plant parameter mismatches. The feedback gains are optimized in an eigenvalue assignment way by minimizing a robust cost function, which is related to the closed-loop state matrix's eigenvalues' sensitivities to the two mismatched plant parameters. The sum of the normalized eigenvalue sensitivities (NES) is defined as cost function. Multiple simulations with different plant parameters are implemented to evaluate the robustness. The optimized FP control system shows smaller envelope of multiple time responses compared to the one before optimization. Experimental results also confirm the optimized controller's improved robustness under parametric mismatches. },
issn = {0278-0046},
keywords = {Non-minimal state space, parametric modelling errors, robust optimization, Proportional-Integral-Plus control, Furuta pendulum},
language = {English},
publisher = {IEEE}
}

Non-minimal state feedback (NMSF) control, also known as proportional-integral-plus (PIP) control, together with anti-windup integrator, is applied, for the first time, to stabilize a Furuta pendulum (FP). The key elements in the non-minimal state space (NMSS) state transition matrix is obtained directly from the original minimal state space (MSS) model. The NMSS model is proved to be controllable if the controllability conditions are met. In order to demonstrate the potential robustness of NMSF control, the controller is designed based on the model with two deliberate plant parameter mismatches. The feedback gains are optimized in an eigenvalue assignment way by minimizing a robust cost function, which is related to the closed-loop state matrix's eigenvalues' sensitivities to the two mismatched plant parameters. The sum of the normalized eigenvalue sensitivities (NES) is defined as cost function. Multiple simulations with different plant parameters are implemented to evaluate the robustness. The optimized FP control system shows smaller envelope of multiple time responses compared to the one before optimization. Experimental results also confirm the optimized controller's improved robustness under parametric mismatches.

@article{zhang3_2020,
title = {Switching Stabilization for Nonlinear Networked Control Systems with Delays and Packet Losses},
author = {Zhang, Q.; Liu, B.},
journal = {Information Technology and Control},
year = {2020},
volume = {49},
number = {2},
institution = {Engineering Research Center for Metallurgical Automation and Measurement Technology of Ministry of Education, China; Hubei Province Key Laboratory of Systems Science in Metallurgical Process, China},
abstract = {This paper studies the stabilization problem for nonlinear NCSs(NNCSs) with bilateral network-induced random delay and packet dropout. T-S fuzzy model is employed to represent the nonlinear controlled plant. Based on the T-S model, a discrete-time fuzzy switched system model with uncertain parameters is established by means of the uncertain method and switching system method. Furthermore, the exponential stability condition for the state of the fuzzy switched system is obtained by using the combination of slow switching model-dependent average dwell time (MDADT) method and fast switching MDADT method. Finally, a series of rotary inverted pendulum experiments are provided to illustrates the effectiveness of the proposed method and prove that the proposed fuzzy controller based on T-S fuzzy model can balance the rotary inverted pendulum in a greater state range rather than the linear controller based on linearization },
keywords = {nonlinear networked control systems, T-S fuzzy model, switched system, mode-dependent average dwell time},
language = {English},
pages = {302-316}
}

This paper studies the stabilization problem for nonlinear NCSs(NNCSs) with bilateral network-induced random delay and packet dropout. T-S fuzzy model is employed to represent the nonlinear controlled plant. Based on the T-S model, a discrete-time fuzzy switched system model with uncertain parameters is established by means of the uncertain method and switching system method. Furthermore, the exponential stability condition for the state of the fuzzy switched system is obtained by using the combination of slow switching model-dependent average dwell time (MDADT) method and fast switching MDADT method. Finally, a series of rotary inverted pendulum experiments are provided to illustrates the effectiveness of the proposed method and prove that the proposed fuzzy controller based on T-S fuzzy model can balance the rotary inverted pendulum in a greater state range rather than the linear controller based on linearization

@article{leamy_2020,
title = {Two-scale command shaping for arresting motion in nonlinear systems},
author = {Leamy, M.J.; Alyukov, A.},
journal = {Nonlinear Dynamics},
year = {2020},
institution = {Georgia Institute of Technology, USA; South Ural State University, Russia},
abstract = {This paper presents a feedforward technique for arresting motion in nonlinear systems based on two-scale command shaping (TSCS). The advantages of the proposed technique arise from its feedforward nature and ease of implementation in linear and nonlinear systems. Using the TSCS strategy, the control input required to arrest motion is decomposed into two scales—the first arrests dynamics associated with the linear subproblem, while the second eliminates response from the nonlinearities. Using direct numerical integration, the method is assessed using a traditional Duffing system and multi-degree-of-freedom nonlinear systems. Experiments are conducted on a compound pendulum attached to a servomotor, documenting effective arrest of the system in close agreement with theoretical predictions. },
keywords = {Nonzero initial conditions, Motion arrest, Nonlinear systems, Perturbation methods, Input shaping, Two-scale command shaping},
language = {English},
publisher = {Springer Nature Switzerland}
}

This paper presents a feedforward technique for arresting motion in nonlinear systems based on two-scale command shaping (TSCS). The advantages of the proposed technique arise from its feedforward nature and ease of implementation in linear and nonlinear systems. Using the TSCS strategy, the control input required to arrest motion is decomposed into two scales—the first arrests dynamics associated with the linear subproblem, while the second eliminates response from the nonlinearities. Using direct numerical integration, the method is assessed using a traditional Duffing system and multi-degree-of-freedom nonlinear systems. Experiments are conducted on a compound pendulum attached to a servomotor, documenting effective arrest of the system in close agreement with theoretical predictions.

@conference{eilers_2020,
title = {Underactuated Waypoint Trajectory Optimization for Light Painting Photography},
author = {Eilers, C.; Eschmann, J.; Menzenbach, R.; Belousov, B.; Muratore, F.; Peters, J.},
booktitle = {2020 IEEE International Conference on Robotics and Automation (ICRA)},
year = {2020},
institution = {Technical University Darmstadt, Germany},
abstract = {Despite their abundance in robotics and nature, underactuated systems remain a challenge for control engineering. Trajectory optimization provides a generally applicable solution, however its efficiency strongly depends on the skill of the engineer to frame the problem in an optimizer-friendly way. This paper proposes a procedure that automates such problem reformulation for a class of tasks in which the desired trajectory is specified by a sequence of waypoints. The approach is based on introducing auxiliary optimization variables that represent waypoint activations. To validate the proposed method, a letter drawing task is set up where shapes traced by the tip of a rotary inverted pendulum are visualized using long exposure photography. },
language = {English},
publisher = {IEEE}
}

Despite their abundance in robotics and nature, underactuated systems remain a challenge for control engineering. Trajectory optimization provides a generally applicable solution, however its efficiency strongly depends on the skill of the engineer to frame the problem in an optimizer-friendly way. This paper proposes a procedure that automates such problem reformulation for a class of tasks in which the desired trajectory is specified by a sequence of waypoints. The approach is based on introducing auxiliary optimization variables that represent waypoint activations. To validate the proposed method, a letter drawing task is set up where shapes traced by the tip of a rotary inverted pendulum are visualized using long exposure photography.

This paper considers parameter estimation for nonlinear state-space models, which is an important but challenging problem. We address this challenge by employing a variational inference (VI) approach, which is a principled method that has deep connections to maximum likelihood estimation. This VI approach ultimately provides estimates of the model as solutions to an optimisation problem, which is deterministic, tractable and can be solved using standard optimisation tools. A specialisation of this approach for systems with additive Gaussian noise is also detailed. The proposed method is examined numerically on a range of simulation and real examples with a focus on robustness to parameter initialisations; we additionally perform favourable comparisons against state-of-the-art alternatives.

This paper considers the problem of computing Bayesian estimates of both states and model parameters for nonlinear state-space models. Generally, this problem does not have a tractable solution and approximations must be utilised. In this work, a variational approach is used to provide an assumed density which approximates the desired, intractable, distribution. The approach is deterministic and results in an optimisation problem of a standard form. Due to the parametrisation of the assumed density selected first- and second-order derivatives are readily available which allows for efficient solutions. The proposed method is compared against state-of-the-art Hamiltonian Monte Carlo in two numerical examples.

@article{jain_2020,
title = {Weighted Internal Model Control-Proportional Integral Derivative Control Scheme Via Fractional Gradient Descent Algorithm},
author = {Jain, S.; Hote, Y.V.},
journal = {Journal of Dynamic Systems, Measurement, and Control},
year = {2020},
month = {12},
volume = {142},
number = {2},
institution = {Indian Institute of Technology Roorkee , India},
abstract = {An adaptive controller design technique based on internal model control (IMC) scheme is proposed in this paper. Multiple IMC controllers having different values of filter time constants and exhibiting widely different performance are combined via weight update rule. The weight update rule, formulated via convex combination of integral and fractional order gradient descent algorithms, assigns time varying weights to individual candidate controllers to obtain an enhanced performance over the individual candidate controllers. The beauty of the proposed technique is that it employs the simplicity of one degree-of-freedom (1DOF) IMC structure to achieve an improved performance over existing 2DOF control schemes. The efficacy of the proposed technique is demonstrated via three illustrative examples and via experimental validation on the hardware setup of dc servosystem. An extensive comparative analysis in terms of simulation plots and performance indices offers a testimony to the effectiveness of the proposed scheme. },
issn = {0022-0434},
language = {English},
publisher = {ASME}
}

An adaptive controller design technique based on internal model control (IMC) scheme is proposed in this paper. Multiple IMC controllers having different values of filter time constants and exhibiting widely different performance are combined via weight update rule. The weight update rule, formulated via convex combination of integral and fractional order gradient descent algorithms, assigns time varying weights to individual candidate controllers to obtain an enhanced performance over the individual candidate controllers. The beauty of the proposed technique is that it employs the simplicity of one degree-of-freedom (1DOF) IMC structure to achieve an improved performance over existing 2DOF control schemes. The efficacy of the proposed technique is demonstrated via three illustrative examples and via experimental validation on the hardware setup of dc servosystem. An extensive comparative analysis in terms of simulation plots and performance indices offers a testimony to the effectiveness of the proposed scheme.

This work presents a miniaturized industrial plant built to portray a hydraulic and thermal system. The system was developed in a low-cost open-source platform for studies in the fields of automation, dynamic systems, and control. In order to test its capacity, different techniques of classic and modern control, estimation, modeling, and identification were applied. It should be noted that the work provides the ability to reproduction and continuous development of the plant, and also examples of educational applications, reducing the demand for laboratory equipment acquisition for teaching and research in Universities.

@article{rodriguez-cortes_2019,
title = {A swinging up controller for the Furuta pendulum based on the Total Energy Control System approach},
author = {Rodriguez-Cortes, H.},
journal = {Kybernetika},
year = {2019},
volume = {55},
number = {2},
institution = {CINVESTAV-IPN, Mexico},
abstract = {This paper considers the problem of swinging up the Furuta pendulum and proposes a new smooth nonlinear swing up controller based on the concept of energy. This new controller results from the Total Energy Control System (TECS) approach in conjunction with a linearizing feedback controller. The new controller commands to the desired reference the total energy rate of the Furuta pendulum; thus, the Furuta pendulum oscillates and reaches a neighborhood of its unstable configuration while the rotation of its base remains bounded. Once the Furuta pendulum configuration is in the neighborhood of its unstable equilibrium point, a linear controller stabilizes the unstable configuration of the Furuta pendulum. Real-time experiments are included to support the theoretical developments. },
keywords = {total energy control system, Furuta pendulum, swinging up control, real-time experiments},
language = {English},
publisher = {Institute of Information Theory and Automation AS CR},
pages = {402-421}
}

This paper considers the problem of swinging up the Furuta pendulum and proposes a new smooth nonlinear swing up controller based on the concept of energy. This new controller results from the Total Energy Control System (TECS) approach in conjunction with a linearizing feedback controller. The new controller commands to the desired reference the total energy rate of the Furuta pendulum; thus, the Furuta pendulum oscillates and reaches a neighborhood of its unstable configuration while the rotation of its base remains bounded. Once the Furuta pendulum configuration is in the neighborhood of its unstable equilibrium point, a linear controller
stabilizes the unstable configuration of the Furuta pendulum. Real-time experiments are included to support the theoretical developments.

@article{polzounov_2019,
title = {Blue River Controls: A toolkit for Reinforcement Learning Control Systems on Hardware},
author = {Polzounov, K.; Redden, L.; Sundar, R.},
year = {2019},
institution = {University of Calgary, Canada; Blue RiverTechnology, USA},
abstract = {Much of the recent success of Reinforcement learning (RL) has been made possible with training and testing tools like OpenAI Gym and Deepmind Control Suite [1, 2]. Unfortunately, tools for quickly testing and transferring high-frequency RL algorithms from simulation to real hardware environment remain mostly absent. We present Blue River Controls, a tool that allows to train and test reinforcement learning algorithms on real-world hardware. It features a simple interface based on OpenAI Gym, that works directly on both simulation and hardware. We use Quanser’s Qube Servo2-USB platform, an underactuated rotary pendulum as an initial testing device. We also provide tools to simplify training RL algorithms on other hardware. Several baselines, from both classical controllers and pretrained RL agents are included to compare performance across tasks. Blue River Controls is available at this https URL: github.com/BlueRiverTech/quanser-openai-driver },
language = {English}
}

Much of the recent success of Reinforcement learning (RL) has been made possible with training and testing tools like OpenAI Gym and Deepmind Control Suite [1, 2]. Unfortunately, tools for quickly testing and transferring high-frequency RL algorithms from simulation to real hardware environment remain mostly absent. We present Blue River Controls, a tool that allows to train and test reinforcement learning algorithms on real-world hardware. It features a simple interface based on OpenAI Gym, that works directly on both simulation and hardware. We use Quanser’s Qube Servo2-USB platform, an underactuated rotary pendulum as an initial testing device. We also provide tools to simplify training RL algorithms on other hardware. Several baselines, from both classical controllers and pretrained RL agents are included to compare performance across tasks. Blue River Controls is available at this https URL: github.com/BlueRiverTech/quanser-openai-driver

@article{lutter_2019,
title = {Deep Lagrangian Networks for end-to-end learning of energy-based control for under-actuated systems},
author = {Lutter, M.; Listmann, K.; Peters, J.},
year = {2019},
institution = {TU Darmstadt, Germany; ABB Corporate Research Center, Germany},
abstract = {Applying Deep Learning to control has a lot of potential for enabling the intelligent design of robot control laws. Unfortunately common deep learning approaches to control, such as deep reinforcement learning, require an unrealistic amount of interaction with the real system, do not yield any performance guarantees, and do not make good use of extensive insights from model-based control. In particular, common black-box approaches -- that abandon all insight from control -- are not suitable for complex robot systems. We propose a deep control approach as a bridge between the solid theoretical foundations of energy-based control and the flexibility of deep learning. To accomplish this goal, we extend Deep Lagrangian Networks (DeLaN) to not only adhere to Lagrangian Mechanics but also ensure conservation of energy and passivity of the learned representation. This novel extension is embedded within generic model-based control laws to enable energy control of under-actuated systems. The resulting DeLaN for energy control (DeLaN 4EC) is the first model learning approach using generic function approximation that is capable of learning energy control. DeLaN 4EC exhibits excellent real-time control on the physical Furuta Pendulum and learns to swing-up the pendulum while the control law using system identification does not. },
keywords = {robotics, machine learning, systems and control},
language = {English}
}

Applying Deep Learning to control has a lot of potential for enabling the intelligent design of robot control laws. Unfortunately common deep learning approaches to control, such as deep reinforcement learning, require an unrealistic amount of interaction with the real system, do not yield any performance guarantees, and do not make good use of extensive insights from model-based control. In particular, common black-box approaches -- that abandon all insight from control -- are not suitable for complex robot systems. We propose a deep control approach as a bridge between the solid theoretical foundations of energy-based control and the flexibility of deep learning. To accomplish this goal, we extend Deep Lagrangian Networks (DeLaN) to not only adhere to Lagrangian Mechanics but also ensure conservation of energy and passivity of the learned representation. This novel extension is embedded within generic model-based control laws to enable energy control of under-actuated systems. The resulting DeLaN for energy control (DeLaN 4EC) is the first model learning approach using generic function approximation that is capable of learning energy control. DeLaN 4EC exhibits excellent real-time control on the physical Furuta Pendulum and learns to swing-up the pendulum while the control law using system identification does not.

@conference{kim_2019,
title = {Deep Q-Network Based Rotary Inverted Pendulum System and Its Monitoring on the EdgeX Platform},
author = {Kim, J.-B.; Lim, H.-K.; Kwon, D.-H.; Kim, M.-S.; Hong, Y.-G.; Han, Y.-H. },
booktitle = {2019 International Conference on Artificial Intelligence in Information and Communication (ICAIIC)},
year = {2019},
institution = {Korea University of Technology and Education, Korea; Electronics and Telecommunications Research Institute, Korea},
abstract = {A rotary inverted pendulum is an unstable and highly nonlinear device and is used as a common model for engineering applications in linear and nonlinear control. In this study, we created a cyber physical system (CPS) to demonstrate that a deep reinforcement learning agent using a rotary inverted pendulum can successfully control a remotely located physical device. The device we created is composed of a cyber environment and physical environment using the Message Queuing Telemetry Transport (MQTT) protocol with an Ethernet connection to connect the cyber environment and the physical environment. The reinforcement learning agent controls the physical device, which is located remotely from the controller and a classical proportional integral derivative (PID) controller is utilized to implement imitation and reinforcement learning and facilitate the learning process. In addition, the control and monitoring system is built on the open source EdgeX platform, so that learning tasks performed near the source of data generation and real-time data emitted from the physical device can be observed while reinforcement learning is performed. From our CPS experimental system, we verify that a deep reinforcement learning agent can control a remotely located real-world device successfully. },
keywords = {EdgeX, Deep Q-Network (DQN), Deep Reinforcement Learning, Rotary Inverted Pendulum},
language = {English},
publisher = {IEEE},
isbn = {978-1-5386-7823-7 }
}

A rotary inverted pendulum is an unstable and highly nonlinear device and is used as a common model for engineering applications in linear and nonlinear control. In this study, we created a cyber physical system (CPS) to demonstrate that a deep reinforcement learning agent using a rotary inverted pendulum can successfully control a remotely located physical device. The device we created is composed of a cyber environment and physical environment using the Message Queuing Telemetry Transport (MQTT) protocol with an Ethernet connection to connect the cyber environment and the physical environment. The reinforcement learning agent controls the physical device, which is located remotely from the controller and a classical proportional integral derivative (PID) controller is utilized to implement imitation and reinforcement learning and facilitate the learning process. In addition, the control and monitoring system is built on the open source EdgeX platform, so that learning tasks performed near the source of data generation and real-time data emitted from the physical device can be observed while reinforcement learning is performed. From our CPS experimental system, we verify that a deep reinforcement learning agent can control a remotely located real-world device successfully.

@article{tran_2109,
title = {Digital Implementation of Homomorphically Encrypted Feedback Control for Cyber-Physical Systems},
author = {Tran, J.; Farokhi, F.; Cantoni, M.; Shames, I.},
year = {2019},
institution = {University of Melbourne, Australia},
abstract = {This paper is about an encryption based approach to the secure implementation of feedback controllers for cyberphysical systems. Specifically, Paillier’s homomorphic encryption is used in a custom digital implementation of a class of linear dynamic controllers, including static gain as a special case. The implementation is amenable to Field Programmable Gate Array (FPGA) realization. Experimental results, including timing analysis and resource usage characteristics for different encryption key lengths, are presented for the realization of a controller for an inverted pendulum; as this is an unstable plant, the control is necessarily fast. },
language = {English}
}

This paper is about an encryption based approach to the secure implementation of feedback controllers for cyberphysical systems. Specifically, Paillier’s homomorphic encryption is used in a custom digital implementation of a class of linear dynamic controllers, including static gain as a special case. The implementation is amenable to Field Programmable Gate Array (FPGA) realization. Experimental results, including timing analysis and resource usage characteristics for different encryption key lengths, are presented for the realization of a controller for an inverted pendulum; as this is an unstable plant, the control is necessarily fast.

@article{lee_2019,
title = {Enhancement of Energy-Based Swing-Up Controller via Entropy Search},
author = {Lee, C.S.; Chang, D.E.},
year = {2019},
institution = {KAIST, Korea},
abstract = {An energy based approach for stabilizing a mechanical system has offered a simple yet powerful control scheme. However, since it does not impose such strong constraints on parameter space of the controller, finding appropriate parameter values for an optimal controller is known to be hard. This paper intends to generate an optimal energy-based controller for swinging up a rotary inverted pendulum, also known as the Furuta pendulum, by applying the Bayesian optimization called Entropy Search. Simulations and experiments show that the optimal controller has an improved performance compared to a nominal controller for various initial conditions. },
language = {English}
}

An energy based approach for stabilizing a mechanical system has offered a simple yet powerful control scheme. However, since it does not impose such strong constraints on parameter space of the controller, finding appropriate parameter values for an optimal controller is known to be hard. This paper intends to generate an optimal energy-based controller for swinging up a rotary inverted pendulum, also known as the Furuta pendulum, by applying the Bayesian optimization called Entropy Search. Simulations and experiments show that the optimal controller has an improved performance compared to a nominal controller for various initial conditions.

@conference{alavi,
title = {Enhancing a Control Systems Design Course by Using Experiential Learning Model},
author = {Alavi, Z.; Meehan, K.},
booktitle = {2019 ASEE Annual Conference & Exposition},
year = {2019},
institution = {California State University, Chico},
abstract = { In this paper, authors present the outcomes of implementing an experiential learning model to explore innovative teaching pedagogy in CSU Chico EECE 482 Control Systems Design course. To reach this goal, multiple projects and demonstrations are designed to engage students of senior level Control Systems Design course in actual engineering problems to develop their practical engineering skills and to enhance students’ connection with junior level courses such as microcontroller programming. By utilizing the available resources within the College of Engineering at CSU Chico, multiple projects were developed for the Control Systems Design course. These projects benefited from existing laboratory spaces and equipment, which included a wind tunnel and a solar photovoltaic array at the Energy Systems Lab and a water flume at the Fluid and Mechanics Lab. The control systems were implemented using either the TM4C123GH6PM, the Tiva™ C Series microprocessor, which students use in the two required embedded systems courses in the curriculum, or an Arduino microcontroller. Prior to the assignment of the projects, the properties of several control algorithms were discussed during the course lectures. Hands-on demonstrations of the algorithms were performed using a Quanser QUBE Servo 2 inverted pendulum. To continue this learning, students were asked to form small teams and to select a project from a list provided by the course instructor. Examples of three projects will be described in this paper. In the first project, a controller was designed to optimize the performance of the wind turbine by maximizing the power delivered to the wind turbine load. The second project was an optimization of performance of a motorized solar photovoltaic panel where students designed a sun tracker, a proportional-integral-derivative (PID) controller to maximize the power generated by the solar panel by adjusting the angle of the panel with respect to the solar irradiance. The third project was a control system to fix the lateral position of a boat in the water flume while it was navigating along the flume to prevent contacts with the walls. Finally, a discussion on how the set of demonstrations and the design projects boosted students’ interest in the Control Systems Design course. An assessment has been conducted to evaluate the impact of the demonstrations, labs and project on the student learning outcomes and the depth of their understanding of key concepts in comparison to that obtained in the previous semesters. The assessment results show that the inclusion of the demonstrations, labs and design projects in the course resulted in increased mastery of control theory concepts through the visualization and hands-on experience and that students were better able to relate theory to practice. },
language = {English},
publisher = {ASEE}
}

In this paper, authors present the outcomes of implementing an experiential learning model to explore innovative teaching pedagogy in CSU Chico EECE 482 Control Systems Design course. To reach this goal, multiple projects and demonstrations are designed to engage students of senior level Control Systems Design course in actual engineering problems to develop their practical engineering skills and to enhance students’ connection with junior level courses such as microcontroller programming. By utilizing the available resources within the College of Engineering at CSU Chico, multiple projects were developed for the Control Systems Design course. These projects benefited from existing laboratory spaces and equipment, which included a wind tunnel and a solar photovoltaic array at the Energy Systems Lab and a water flume at the Fluid and Mechanics Lab. The control systems were implemented using either the TM4C123GH6PM, the Tiva™ C Series microprocessor, which students use in the two required embedded systems courses in the curriculum, or an Arduino microcontroller.

Prior to the assignment of the projects, the properties of several control algorithms were discussed during the course lectures. Hands-on demonstrations of the algorithms were performed using a Quanser QUBE Servo 2 inverted pendulum. To continue this learning, students were asked to form small teams and to select a project from a list provided by the course instructor. Examples of three projects will be described in this paper. In the first project, a controller was designed to optimize the performance of the wind turbine by maximizing the power delivered to the wind turbine load. The second project was an optimization of performance of a motorized solar photovoltaic panel where students designed a sun tracker, a proportional-integral-derivative (PID) controller to maximize the power generated by the solar panel by adjusting the angle of the panel with respect to the solar irradiance. The third project was a control system to fix the lateral position of a boat in the water flume while it was navigating along the flume to prevent contacts with the walls. Finally, a discussion on how the set of demonstrations and the design projects boosted students’ interest in the Control Systems Design course. An assessment has been conducted to evaluate the impact of the demonstrations, labs and project on the student learning outcomes and the depth of their understanding of key concepts in comparison to that obtained in the previous semesters. The assessment results show that the inclusion of the demonstrations, labs and design projects in the course resulted in increased mastery of control theory concepts through the visualization and hands-on experience and that students were better able to relate theory to practice.

@article{schluter_2019,
title = {Event-triggered Learning for Linear Quadratic Control},
author = {Schluter, H.; Solowjow, F.; Trimpe, S.},
year = {2019},
institution = {Max Planck Institute for Intelligent Systems, Germany},
abstract = {When models are inaccurate, performance of model-based control will degrade. For linear quadratic control, an event triggered learning framework is proposed that automatically detects inaccurate models and triggers learning of a new process model when needed. This is achieved by analyzing the probability distribution of the linear quadratic cost and designing a learning trigger that leverages Chernoff bounds. In particular, whenever empirically observed cost signals are located outside the derived confidence intervals, we can provably guarantee that this is with high probability due to a model mismatch. With the aid of numerical and hardware experiments, we demonstrate that the proposed bounds are tight and that the event-triggered learning algorithm effectively distinguishes between inaccurate models and probabilistic effects such as process noise. Thus, a structured approach is obtained that decides when model learning is beneficial. },
keywords = {Event-triggered Learning; Optimal Control; Statistical Learning; Stochastic Systems},
language = {English}
}

When models are inaccurate, performance of model-based control will degrade. For linear quadratic control, an event triggered learning framework is proposed that automatically detects inaccurate models and triggers learning of a new process model when needed. This is achieved by analyzing the probability distribution of the linear quadratic cost and designing a learning trigger that leverages Chernoff bounds. In particular, whenever
empirically observed cost signals are located outside the derived confidence intervals, we can provably guarantee that this is with high probability due to a model mismatch. With the aid of numerical and hardware experiments, we demonstrate that the proposed bounds are tight and that the event-triggered learning algorithm effectively distinguishes between inaccurate models and probabilistic effects such as process noise. Thus,
a structured approach is obtained that decides when model learning is beneficial.

@article{volpp_2019,
title = {Meta-Learning Acquisition Functions for Bayesian Optimization},
author = {Volpp, M.; Frohlich, L.; Doerr, A.; Hutter, F.; Daniel, C.},
year = {2019},
institution = {Bosch Center for Artificial Intelligence, Germany; Max Planck Institute for Intelligent Systems, Germany; University of Freiburg; Germany },
abstract = {Many practical applications of machine learning require data-efficient black-box function optimization, e.g., to identify hyperparameters or process settings. However, readily available algorithms are typically designed to be universal optimizers and are, thus, often suboptimal for specific tasks. We therefore propose a method to learn optimizers which are automatically adapted to a given class of objective functions, e.g., in the context of sim-to-real applications. Instead of learning optimization from scratch, the proposed approach is firmly based within the famous Bayesian optimization framework. Only the acquisition function (AF) is replaced by a learned neural network and therefore the resulting algorithm is still able to exploit the proven generalization capabilities of Gaussian processes. We present experiments on several simulated as well as on a sim-to-real transfer task. The results show that the learned optimizers (1) consistently perform better than or on-par with known AFs on general function classes and (2) can automatically identify structural properties of a function class using cheap simulations and transfer this knowledge to adapt rapidly to real hardware tasks, thereby significantly outperforming existing problem-agnostic AFs. },
language = {English}
}

Many practical applications of machine learning require data-efficient black-box function optimization, e.g., to identify hyperparameters or process settings. However, readily available algorithms are typically designed to be universal optimizers and are, thus, often suboptimal for specific tasks. We therefore propose a method to learn optimizers which are automatically adapted to a given class of objective functions, e.g., in the context of sim-to-real applications. Instead of learning optimization from scratch, the proposed approach is firmly based within the famous Bayesian optimization framework. Only the acquisition function (AF) is replaced by a learned neural network and therefore the resulting algorithm is still able to exploit the proven generalization capabilities of Gaussian processes. We present experiments on several simulated as well as on a sim-to-real transfer task. The results show that the learned optimizers (1) consistently perform better than or on-par with known AFs on general function classes and (2) can automatically identify structural properties of a function class using cheap simulations and transfer this knowledge to adapt rapidly to real hardware tasks, thereby significantly outperforming existing problem-agnostic AFs.

@article{kandala_2019,
title = {Pole Placement for Time-Delayed Systems Using Galerkin Approximations},
author = {Kandala, S.S.; Uchida, T.K.; Vyasarayani, C.P.},
journal = {Journal of Dynamic Systems, Measurement, and Control},
year = {2019},
month = {5},
volume = {141},
number = {5},
institution = {Indian Institute of Technology Hyderabad, India; Stanford University, USA},
abstract = {Many dynamic systems of practical interest have inherent time delays and thus are governed by delay differential equations (DDEs). Because DDEs are infinite dimensional, time-delayed systems may be difficult to stabilize using traditional controller design strategies. We apply the Galerkin approximation method using a new pseudo-inverse-based technique for embedding the boundary conditions, which results in a simpler mathematical derivation than has been presented previously. We then use the pole placement technique to design closed-loop feedback gains that stabilize time-delayed systems and verify our results through comparison to those reported in the literature. Finally, we perform experimental validation by applying our method to stabilize a rotary inverted pendulum system with inherent sensing delays as well as additional time delays that are introduced deliberately. The proposed approach is easily implemented and performs at least as well as existing methods. },
keywords = {Delay differential equations, Delays, Feedback, Galerkin method, Pole placement, Pendulums, Time delay systems},
language = {English},
publisher = {ASME}
}

Many dynamic systems of practical interest have inherent time delays and thus are governed by delay differential equations (DDEs). Because DDEs are infinite dimensional, time-delayed systems may be difficult to stabilize using traditional controller design strategies. We apply the Galerkin approximation method using a new pseudo-inverse-based technique for embedding the boundary conditions, which results in a simpler mathematical derivation than has been presented previously. We then use the pole placement technique to design closed-loop feedback gains that stabilize time-delayed systems and verify our results through comparison to those reported in the literature. Finally, we perform experimental validation by applying our method to stabilize a rotary inverted pendulum system with inherent sensing delays as well as additional time delays that are introduced deliberately. The proposed approach is easily implemented and performs at least as well as existing methods.

@article{umenberger_2019,
title = {Robust exploration in linear quadratic reinforcement learning},
author = {Umenberger, U.; Ferizbegovic, M.; Schön, T.B.; Hjalmarsson, H.},
year = {2019},
institution = {Uppsala University, Sweden; KTH, Sweden},
abstract = {This paper concerns the problem of learning control policies for an unknown linear dynamical system to minimize a quadratic cost function. We present a method, based on convex optimization, that accomplishes this task robustly: i.e., we minimize the worst-case cost, accounting for system uncertainty given the observed data. The method balances exploitation and exploration, exciting the system in such a way so as to reduce uncertainty in the model parameters to which the worst-case cost is most sensitive. Numerical simulations and application to a hardware-in-the-loop servo-mechanism demonstrate the approach, with appreciable performance and robustness gains over alternative methods observed in both. },
language = {English}
}

This paper concerns the problem of learning control policies for an unknown linear dynamical system to minimize a quadratic cost function. We present a method, based on convex optimization, that accomplishes this task robustly: i.e., we minimize the worst-case cost, accounting for system uncertainty given the observed data. The method balances exploitation and exploration, exciting the system in such a way so as to reduce uncertainty in the model parameters to which the worst-case cost is most sensitive. Numerical simulations and application to a hardware-in-the-loop servo-mechanism demonstrate the approach, with appreciable performance and robustness gains over alternative methods observed in both.