Approximate Dynamic Programming for Stochastic and Dynamic Decision Making

Companies operate in an increasingly volatile environment. Within their respective business models, decisions are often made under incomplete information. These decisions are further adapted based on newly revealed information over time, for example, when customers place new orders, when the production process gets disturbed, when congestion delays parcel delivery or when stock exchange rates promise future opportunities. The challenges of incomplete information and real-time adaptions of plans affect many business fields, for example, logistics, production, sales, or finance. Fortunately, companies gain access to vast amounts of data. To provide effective support for subsequent decision making, advanced mathematical models and methods are needed allowing for integration of data into real-time decision support systems.

The resulting problems are stochastic and dynamic decision processes and can be modeled as a Markov decision process (MDP). An MDP models a problem as a sequence of states connected by decisions and stochastic transitions. A solution of an MDP is a decision policy assigning a decision to every potential decision state. An optimal solution maximizes the expected reward. Small MDPs can be solved to optimality, for example, by means of dynamic programming. Dynamic programming uses recursion and operates on the Bellman Equation maximizing the sum of immediate and expected future reward in every state. However, the MDPs for the business models under consideration are generally complex, the accessible calculation time is limited, and an optimal solution is not possible due to the infamous Curses of Dimensionality.

Instead, methods of approximate dynamic programming (ADP) are applied. Simple ADP-methods straightforwardly mimic practical rules-of-thumb. Advanced ADP-methods approximate the second term of the Bellman Equation, the expected future reward, by means of simulation. Dependent on the type of method, the approximated reward is either discarded after the decision is made or stored for future use. In the latter case, the algorithm learns subsequently letting the results guide future simulations for a more accurate approximation. However, to store the results, aggregation is necessary.

In this course, we describe the process of approaching complex stochastic and dynamic decision problems with ADP-methods. We present the required steps from business problem over MDP to the ADP-solutions in detail and give an overview over the most prominent ADP-methods. We especially focus on offline learning methods known as value function approximation. The theoretical content of this course is accompanied by many illustrative examples from the field of logistics and by a serious gaming application.

This course spreads over 4 days. On the first day, complex business models are analyzed with respect to dynamism and stochasticity. Stochastic and dynamic decision making is introduced both conceptually and based on examples. On the second day, the MDP-modeling framework is defined and solving MDPs by means of dynamic programming is discussed, particularly, in context of the curses of dimensionality. Furthermore, modeling MDPs is practiced in a hands-on part. On the third day, a selection of advanced ADP-methods is presented. First, solution methods are introduced and classified. Then, the course focuses on simulation methods, namely, lookahead models and value function approximation as well as generic combinations of different ADP-methods. On the last day, the results are summarized, and the participants conceptually traverse the steps from problem to method in a case-study.

The course combines the fields of process modeling, optimization, simulation, and machine learning. It is aimed at PhD-students from various disciplines, including operations management, production and logistics, supply chain management, finance, marketing, and management science. The course fits well for PhD-students working in the field of data science. The elements of the course range from lectures on theory, to the discussion of best practices and case studies all accompanied by regular presentations of work done by the students.

Anmeldefrist: 29. August 2027

Syllabus

Sprache / Language

Englisch

Ort / Location

Otto-von-Guericke-Universität Magdeburg
Vilfredo-Pareto Building (G22), Room A225
39106 Magdeburg

Tickets

Noch 20 Plätze verfügbar.