Michael Kaisers, PhD

Scientific Staff Member
Intelligent and Autonomous Systems Group
Centrum Wiskunde & Informatica, Amsterdam

Curriculum Vitae

Michael Kaisers graduated from Maastricht University with a BSc in Knowledge Engineering in 2007 and a MSc in Artificial Intelligence in 2008. He earned the honor summa cum laude in both cases, while abbreviating the three-years bachelor's program to two years and complementing his master's program by an extra-curricular four-month research visit to Prof. dr. Simon Parsons at Brooklyn College, City University of New York.

In a nationwide competition, the Netherlands Organisation for Scientific Research (NWO) awarded him a TopTalent 2008 grant for his PhD research. In September 2008, he commenced his PhD position at Eindhoven University of Technology. From August 2009, the project continued at Maastricht University. He intensified his international research experience through a three-month research visit to Prof. dr. Michael Littman at Rutgers, State University of New Jersey, and published at various peer-reviewed workshops, conferences and journals.

From September 2010 to January 2012 he has chaired PhD Academy, which brings Maastricht PhD candidates together for social, educative and fun activities. He has also chaired the 4th Maastricht PhD conference (PhDC 2011) and the local organizing committee of the 9th European Workshop on Multi-agent Systems (EUMAS 2011). His dissertation coherently summarizes his PhD research and has been defended in December 2012.

Michael remained affiliated with Maastricht University as a visiting postdoctoral research fellow while he filled the position of Director of Competence Center Agent Core Technologies in the Distributed Artificial Intelligence Laboratory at the Technical University Berlin from March to September 2013. Since November 2013 he is pursuing his research at the Centrum Wiskunde & Informatica in Amsterdam within the Intelligent and Autonomous Systems group, where he built a team working on intelligent joint decision making with applications to energy systems.

The public defense of my dissertation took place on Monday, 17 December 2012 at 4pm.

Publications

Keywords

Artificial intelligence, multi-agent reinforcement learning, evolutionary game theory, dynamical systems

List of Journal Publications, Books and Chapters

2017
Tim Baarslag, Michael Kaisers, Enrico H Gerding, Catholijn M Jonker, and Jonathan Gratch. Computers That Negotiate on Our Behalf: Major Challenges for Self-sufficient, Self-directed, and Interdependent Negotiating Agents. In Autonomous Agents and Multiagent Systems: AAMAS 2017 Workshops, Visionary Papers, pages 143-163. Sao Paulo, Brazil, May 8-12, 2017. Revised Selected Papers, Springer International Publishing.
Pablo Hernandez-Leal and Michael Kaisers. Towards a fast detection of opponents in repeated stochastic games. In Autonomous Agents and Multiagent Systems: AAMAS 2017 Workshops, Best Papers, pages 239-257. Sao Paulo, Brazil, May 8-12, 2017. Revised Selected Papers, Springer International Publishing.
2016
Sherief Abdallah and Michael Kaisers. Addressing environment non-stationarity by repeating Q-learning updates. Journal of Machine Learning Research, 17.46, 1-31. 2016. [Open Access]
2015
Daan Bloembergen, Karl Tuyls, Daniel Hennes and Michael Kaisers (2015). Evolutionary Dynamics of Multi-Agent Learning: A Survey. Journal of Artificial Intelligence Research, 53, 659–697. [Open Access]
2013
Marco Lützenberger, Tobias Küster, Thomas Konnerth, Alexander Thiele, Nils Masuch, Axel Heß ler, Jan Keiser, Michael Burkhardt, Silvan Kaiser, Jakob Tonn, Michael Kaisers, and Sahin Albayrak. A Multi-agent Approach to Professional Software Engineering. In Proceedings of the first workshop on Engineering Multi-Agent Systems (EMAS), pages 158-177. Lecture Notes in Computer Science. Springer, 2013.
2012
Michael Kaisers. Learning against Learning - Evolutionary Dynamics of Reinforcement Learning Algorithms in Strategic Interactions. Doctoral dissertation, Maastricht University. 2012. [Dissertation, Propositions]
Haitham Bou Ammar, Karl Tuyls, and Michael Kaisers. Evolutionary Dynamics of Ant Colony Optimization. In Ingo J. Timm and Christian Guttmann, editors, Multiagent System Technologies. 10th German Conference, MATES 2012, pages 40-52. Lecture Notes in Computer Science, Vol. 7598. Springer, 2012. [Download]
Marcel Neumann, Karl Tuyls, and Michael Kaisers. Using Time as a Strategic Element in Continuous Double Auctions (Short Paper). In Ingo J. Timm and Christian Guttmann, editors, Multiagent System Technologies. 10th German Conference, MATES 2012, pages 106-115. Lecture Notes in Computer Science, Vol. 7598. Springer, 2012. [Download]
Michael Kaisers, Daan Bloembergen and Karl Tuyls. Multi-agent Learning and the Reinforcement Gradient. In Massimo Cossentino, Michael Kaisers, Karl Tuyls, and Gerhard Weiss, editors, Multi-Agent Systems. 9th European Workshop, EUMAS 2011, pages 145-159. Lecture Notes in Computer Science, Vol. 7541. Springer, 2012. [Download]
2010
Michael Kaisers and Karl Tuyls. Replicator Dynamics for Multi-agent Learning - An Orthogonal Approach. In Matthew E. Taylor and Karl Tuyls, editors, Adaptive and Learning Agents. Second Workshop, ALA 2009, pages 49-59. Lecture Notes in Computer Science, Vol. 5924. Springer, 2010. [Download]
2009
Marc Ponsen, Karl Tuyls, Michael Kaisers, and Jan Ramon. An evolutionary game-theoretic analysis of poker strategies. Entertainment Computing, 1(1):39-45, January 2009. [Download]
2007
Jaap H. van den Herik, Daniel Hennes, Michael Kaisers, Karl Tuyls, and Katja Verbeeck. Multi-agent learning dynamics: A survey. In Proceedings of the 11th International Workshop on Cooperative Information Agents (CIA 2007), pages 36-56. Lecture Notes in Computer Science, Vol. 4676. Springer, 2007. [Download]

List of Conference and Workshop Publications

2017
Tim Baarslag and Michael Kaisers. The Value of Information in Automated Negotiation: A Decision Model for Eliciting User Preferences. In Proc. of 16th Int. Conf. on Autonomous Agents and Multiagent Systems (AAMAS 2017). International Foundation for AAMAS, 2017.
Tim Baarslag, Michael Kaisers, Catholijn Jonker, Jonathan Gratch, and Enrico Gerding. When will negotiation agents be able to represent us? The challenges and opportunities for autonomous negotiators. In Proc. of the 26th Int. Joint Conf. on Artificial Intelligence (IJCAI), 2017. [Download]
Pablo Hernandez-Leal and Michael Kaisers. Learning against sequential opponents in repeated stochastic games. In Proc. of the 3rd Multidisciplinary Conference on Reinforcement Learning and Decision Making (RLDM), 2017.
Georgios Methenitis, Michael Kaisers and Han La Poutré. SLA–Mechanisms for Electricity Trading under Volatile Supply and Varying Criticality of Demand (Extended Abstract). In Proc. of 16th Int. Conf. on Autonomous Agents and Multiagent Systems (AAMAS 2017). International Foundation for AAMAS, 2017.
Tim Baarslag, Michael Kaisers, Enrico H Gerding, Catholijn M Jonker, and Jonathan Gratch. When will negotiation agents be able to represent us? The challenges and opportunities for autonomous negotiators. In Proc. of the Int. Workshop on Agent-Mediated Electronic Commerce and Trading Agents Design and Analysis (AMEC/TADA), 2017.
Pablo Hernandez-Leal and Michael Kaisers. Towards a fast detection of opponents in repeated stochastic games. In Proc. of the 1st Workshop on Transfer in Reinforcement Learning (TiRL), 2017.
Georgios Methenitis, Michael Kaisers, and Han La Poutre. SLA Mechanisms for Electricity Trading under Volatile Supply and Varying Criticality of Demand. In Proc. of the Int. Workshop on Agent- Mediated Electronic Commerce and Trading Agents Design and Analysis (AMEC/TADA), 2017.
2016
Felix Claessen, Michael Kaisers and Han La Poutré. Efficient balancing by effort-based activation of demand response services. In Innovative Smart Grid Technologies-Europe (ISGT Europe), IEEE, 2016. [Download]
Aliene van der Veen and Michael Kaisers. Computing the value of flexibility in electricity retail, ahead and balancing markets. In Innovative Smart Grid Technologies-Europe (ISGT Europe), IEEE, 2016.
Christian Gitte, Huiwen Xu, Fabian Rigoll, Joeri van Eekelen, and Michael Kaisers. Multi-Commodity Energy Management Applied to Micro CHPs and Electrical Heaters in Smart Buildings. In 5th D-A-CH+ Energy Informatics Conference in conjunction with 7th Symposium on Communications for Energy Systems (ComForEn), pages 74-80. Vol. 84, Österreichischer Verband für Elektrotechnik. [Link]
Georgios Methenitis, Michael Kaisers and Han La Poutré. Incentivizing Intelligent Customer Behavior in Smart-Grids: A Risk-Sharing Tariff & Optimal Strategies. In Proceedings of the 25th International Conference on Artificial Intelligence (IJCAI), 2016. AAAI Press. [Link]
Timon V. Kanters, Frans A. Oliehoek, Michael Kaisers, Stan R. van den Bosch, Joep Grispen, and Jeroen Hermans. Energy- and Cost-Efficient Pumping Station Control. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 2016. [Link]
2015
Georgios Methenitis, Michael Kaisers and Han La Poutré. A Multi-Scale Energy Demand Model suggests sharing Market Risks with Intelligent Energy Cooperatives. In Innovative Smart Grid Technologies-Asia (ISGT Asia), IEEE, 2015. [Link]
Felix Claessen, Bart Liefers, Michael Kaisers and Han La Poutré. The value of online information for demand response in Walrasian electricity markets. In Innovative Smart Grid Technologies-Asia (ISGT Asia), IEEE, 2015.
Sherief Abdallah and Michael Kaisers. Improving Multi-Agent Learners Using Less-biased Value Estimators. In Proc. of Int. Conf. on Web Intelligence and Intelligent Agent Technology (WI-IAT 2015). IEEE/WIC/ACM, 2015.
Tobias Linnenberg, Michael Kaisers and Alexander Fay. Open Energy Exchange - A new way of social networking. In Proc. of SmartER Europe 15, Essen, February 2015.
Tobias Linnenberg, Alexander Fay and Michael Kaisers. Bottom-up Demand Response by Following Local Energy Generation Voluntarily (Demonstration). In Proc. of the Twenty-Ninth AAAI Conference on Artificial Intelligence (AAAI 2015), pages 4282-4283, 2015.
2013
Sherief Abdallah and Michael Kaisers. Addressing the Policy-bias of Q-learning by Repeating Updates. In Proc. of 12th Int. Conf. on Autonomous Agents and Multiagent Systems (AAMAS 2013), pages 1045-1052. International Foundation for AAMAS, 2013. [Download]
2012
Daniel Hennes, Daan Bloembergen, Michael Kaisers, Karl Tuyls, and Simon Parsons. Evolutionary advantage of foresight in markets. In Proc. of the Genetic and Evolutionary Computation Conference (GECCO 2012), pages 943-949. Sheridan Printing, 2012. [Download]
Michael Wunder, John R. Yaros, Michael Littman, and Michael Kaisers. A Framework for Modeling Population Strategies by Depth of Reasoning. In Conitzer, Winikoff, Padgham, and van der Hoek, editors, Proc. of 11th Int. Conf. on Autonomous Agents and Multiagent Systems (AAMAS 2012), pages 1393-1394. International Foundation for AAMAS, 2012. [Download]
Michael Kaisers, Daan Bloembergen, and Karl Tuyls. A Common Gradient in Multi-agent Reinforcement Learning (Extended Abstract). In Conitzer, Winikoff, Padgham, and van der Hoek, editors, Proc. of 11th Int. Conf. on Autonomous Agents and Multiagent Systems (AAMAS 2012), pages 947-954. International Foundation for AAMAS, 2012. [Download]
2011
Sjriek Alers, Daan Bloembergen, Daniel Hennes, Steven de Jong, Michael Kaisers, Nyree Lemmens, Karl Tuyls, and Gerhard Weiss. Bee-inspired foraging in an embodied swarm (Demonstration). In Tumer, Yolum, Sonenberg, and Stone, editors, Proc. of 10th Int. Conf. on Autonomous Agents and Multiagent Systems (AAMAS 2011), pages 1311-1312. International Foundation for AAMAS, 2011. [Download]
Daan Bloembergen, Michael Kaisers, and Karl Tuyls. Empirical and Theoretical Support for Lenient Learning (Extended Abstract). In Tumer, Yolum, Sonenberg, and Stone, editors, Proc. of 10th Int. Conf. on Autonomous Agents and Multiagent Systems (AAMAS 2011), pages 1105-1106. International Foundation for AAMAS, 2011. [Download]
Michael Kaisers and Karl Tuyls. FAQ-Learning in Matrix Games: Demonstrating Convergence near Nash Equilibria, and Bifurcation of Attractors in the Battle of Sexes. In Workshop on Interactive Decision Theory and Game Theory (IDTGT 2011). Assoc. for the Advancement of Artif. Intel. (AAAI), 2011. [Download]
Daniel Mescheder, Karl Tuyls, and Michael Kaisers. Opponent Modeling with POMDPs. In Proc. of 23nd Belgium-Netherlands Conf. on Artificial Intelligence (BNAIC 2011), pages 152-159. KAHO Sint-Lieven, Gent, 2011. [Download]
Michael Wunder, Michael Kaisers, J.R. Yaros, and Michael Littman. Using iterated reasoning to predict opponent strategies. In Tumer, Yolum, Sonenberg, and Stone, editors, Proc. of 10th Int. Conf. on Autonomous Agents and Multiagent Systems (AAMAS 2011), pages 593-600. International Foundation for AAMAS, 2011. [Download]
2010
Daan Bloembergen, Michael Kaisers, and Karl Tuyls. A comparative study of multi-agent reinforcement learning dynamics. In Proc. of 22nd Belgium- Netherlands Conf. on Artificial Intelligence (BNAIC 2010), pages 11-18. University of Luxembourg, 2010. [Download]
Daan Bloembergen, Michael Kaisers, and Karl Tuyls. Lenient frequency adjusted Q-learning. In Proc. of 22nd Belgium-Netherlands Conf. on Artificial Intelligence (BNAIC 2010), pages 19-26. University of Luxembourg, 2010. [Download]
Michael Wunder, Michael Kaisers, Michael Littman, and John Robert Yaros. A Cognitive Hierarchy Model Applied to the Lemonade Game. In Workshop on Interactive Decision Theory and Game Theory (IDTGT 2010). Assoc. for the Advancement of Artif. Intel. (AAAI), 2010. [Download]
Michael Kaisers and Karl Tuyls. Frequency Adjusted Multi-agent Q-learning. In van der Hoek, Kamina, Lespérance, Luck, and Sen, editors, Proc. of 9th Intl. Conf. on Autonomous Agents and Multiagent Systems (AAMAS 2010), pages 309-315, 2010. [Download]
Daniel Hennes, Michael Kaisers, and Karl Tuyls. RESQ-learning in stochastic games. In Adaptive and Learning Agents (ALA 2010) Workshop, 2010. [Download]
2009
Michael Kaisers. Replicator Dynamics for Multi-agent Learning - An Orthogonal Approach. In Toon Calders, Karl Tuyls, and Mykola Pechenizkiy, editors, Proc. of the 21st Benelux Conference on Artificial Intelligence (BNAIC 2009), pages 113-120, Eindhoven, 2009. Eindhoven University of Technology. [Download]
Michael Kaisers, Karl Tuyls, and Simon Parsons. An Evolutionary Model of Multi-agent Learning with a Varying Exploration Rate (Extended Abstract). In Proc. of 8th Int. Conf. on Autonomous Agents and Multiagent Systems (AAMAS 2009), pages 1255-1256. International Foundation for AAMAS, 2009. [Download]
2008
Michael Kaisers, Karl Tuyls, Frank Thuijsman, and Simon Parsons. Auction Analysis by Normal Form Game Approximation. In Proc. of Int. Conf. on Web Intelligence and Intelligent Agent Technology (WI-IAT 2008), pages 447-450. IEEE/WIC/ACM, December 2008. [Download]
Michael Kaisers, Karl Tuyls, and Frank Thuijsman. Discovering the game in auctions. In Proc. of 20th Belgian-Netherlands Conference on Artificial Intelligence (BNAIC 2008), pages 113-120. University of Twente, 2008. [Download]

You can find partial lists of my publications on google scholar or DBLP.

Education

List of potential thesis topics

M.Sc.
Reinforcement learning and drifting reward distributions: Reinforcement learning is well-founded for multi-armed bandit problems, where reward distributions are stationary. In order to estimate the expected sum of possibly discounted future rewards, several rewards need to be aggregated. However, in sequential decision making (and similarly in multi-agent learning), these rewards are not actually IID - their distributions rather change (slowly). Theory of Monte Carlo Tree Search and Tree Learning Search largely ignores the drift. This thesis will look at ways to explicitly consider drift for reward estimation to improve learning speed and performance.
Planning in continuous domains: Tree Learning Search (TLS) is a variant of Monte-Carlo Tree Search (MCTS) for continuous state and action spaces. MCTS is a best-first search technique that estimates game tree node values based on the results from simulated gameplay. It replaces exhaustive search through the game tree with well founded sampling techniques and has been quite successful in games difficult for computer AI's like Go and Poker. However, MCTS can not deal with continuous action and state spaces and requires a priori discretization of both to be applicable. The recently developed idea of Tree Learning Search uses techniques from data stream mining to dynamically discretize both actions and states. Preliminary experiments have shown good performance in simple benchmarks, but an evaluation in realistic noisy environments is still outstanding. The application to a simple robotic system would be a plus.
Analyzing interactive learning: It is common to assume in our analysis of learning that players have to interact with each other. Let us turn that assumption upside down: How does the ability to choose your interaction partners influence what you learn from your interaction? This project can be performed empirically and analytically. The empirical approach requires implementing learning algorithms and running them in games with fixed and flexible interaction partners. For the analytical approach, both types of games need to be defined in terms of Game theory in order to establish a formal connection between them.
Tangible reinforcement learning: Find an application for reinforcement learning that exposes the learning progress in an enjoyable way. Many think that reinforcement learning is an abstract concept that is hard to grasp, but it doesn't need to be that way. There are applications like cart-pole balancing that make it quite tangible, or projects like RL-Viz that try to visualize learning progress more technically. Consider an iPhone app or a audioization or get even more creative. For a bachelor topic, this project should culminate in a software that allows to playfully explore the dynamics of reinforcement learning (e.g., by comparing behaviors of varying parameters). A master student could additionally build a hardware system.
Privacy through P2P sharing: Imagine you could connect to your friends by sharing information directly between your and their devices. This paradigm shift will re-introduce privacy, and is enabled by today's always-online rich clients like mobile phones. In addition, a user may synchronize all of his/her devices to maximize the availability of his/her online-ego. Investigate the technical opportunities and challenges of implementing a community using P2P architecture.
B.Sc.
You can approach me with your ideas, as long as they fit with my experience.

List of students I have been (co-) coaching

2012
Andreas ten Pas, M.Sc. Simulation Based Planning for Partially Observable Markov Decision Processes with Continuous Observation Spaces. August 2012.
Colin Schepers, M.Sc. Automatic Decomposition of Continuous Action and State Spaces in Simulation-Based Planning. July 2012.
Lukas Kirchhart, M.Sc. Reusing Knowledge in Tree Learning Search. July 2012.
2011
Marcel Neumann, B.Sc. Price Formation of Continuous Double Auction Agents using Time as a Strategic Element. June 2011. [Download]
Daniel Mescheder, B.Sc. POMDP Opponent Models for Best Response Behavior. June 2011. [Download]
2010
Franz Hahn, B.Sc. An Artificial Intelligence Look at Playing Risk. August 2010.
Daniel Claes, B.Sc. Balancing Anarchy and Central Control - Individual vs. Joint Action Reinforcement Learning. June 2010. [Download]
Daan Bloembergen, M.Sc. Analyzing Reinforcement Learning Algorithms using Evolutionary Game Theory. June 2010. [Download]

List of courses

2012
Introduction to Programming (lectures, TA),
University College Maastricht
Introduction to LaTeX (lecture, TA),
B. Sc. Knowledge Engineering and Computer Science, year 2, block 1
2011
Introduction to LaTeX (lecture, TA),
B. Sc. Knowledge Engineering and Computer Science, year 2, block 1
Theoretical computer science (TA),
B. Sc. Knowledge Engineering and Computer Science, year 2, block 4
2010
Object Oriented Modelling (TA, some lectures),
B. Sc. Knowledge Engineering and Computer Science, year 2, block 2
Introduction to LaTeX (lecture, TA),
B. Sc. Knowledge Engineering and Computer Science, year 2, block 1
Linear algebra (TA, some lectures),
B. Sc. Knowledge Engineering and Computer Science, year 1, block 4
Theoretical computer science (TA),
B. Sc. Knowledge Engineering and Computer Science, year 2, block 4

Contact

You can contact me in German, English, or Dutch for professional requests. I would also like to improve my Spanish and I'll appreciate opportunities to practice. [download vCard]

Facebook Profile
View Michael Kaisers's profile on LinkedIn

Michael Kaisers, PhD
Scientific Staff Member
Centrum Wiskunde & Informatica, Amsterdam
Tel.: +31 (0) 20 - 592 4035
E-mail:

Visiting address
Room M 3.59
Science Park 123
1098 XG Amsterdam

Postal address
P.O. Box 94079
1090 GB Amsterdam
The Netherlands