Michael Kaisers

Michael Kaisers graduated from Maastricht with a B. Sc. in Knowledge Engineering in 2007, and a M. Sc. in Artificial Intelligence in 2008. In both cases, he earned the honor summa cum laude, additionally abbreviating the three-years bachelor program to two years, and complementing his master program by an extra-curricular four-month research visit to Simon Parsons at Brooklyn College, New York City.

In a nationwide competition, the Netherlands Organization for Scientific Research (NWO) awarded him a TopTalent 2008 PhD grant for his proposal Multi-agent Learning in Auctions. In September 2008, he commenced his PhD position at Eindhoven University of Technology. Since August 2009, the project is continued at Maastricht University. He intensified his international research network through a three-month research visit to Michael Littman at Rutgers, State University of New Jersey, and gave presentations at various workshops and conferences.

From September 2010 to January 2012 he has chaired PhD Academy, which brings people together for social, educative and fun activities. He has also chaired the 4th Maastricht PhD conference (PhDC 2011) and the local organizing committee of the 9th European Workshop on Multi-agent Systems (EUMAS 2011). He is expected to obtain his PhD from Maastricht University in Summer 2012.

Project

Multi-agent Learning in Auctions: The design and analysis of markets and traders read more

Affiliations

Maastricht University

Netherlands Organisation for Scientific Research

PhD Academy (also on facebook)

Young Researchers Academy (YRA) [download]

Co-promotors

Karl Tuyls, Ph. D.

Simon Parsons, Ph. D.

Promotor

Prof. Dr. Gerhard Weiss

Research group

Swarmlab; Robots, Agents and Interaction (RAI)

Collaborations

Agents Lab, CUNY (Simon Parsons)

RL3, Rutgers (Michael Littman)

Project timeline, honours and awards

2012

Co-editor of the post-proceedings for EUMAS 2011

2011

EUMAS 2011 Local Organizing Chair

Tutorial on Multi-Agent Reinforcement Learning: A new framework for analyzing and improving Multi-Agent Reinforcement Learning. Held in conjunction with AAMAS 2011.

Chair of the 4th Maastricht PhD Conference (PhDC 2011)

2010

Chair of the board of PhD Academy

2009

Visiting research scholar: Collaborating with Michael Littman at Rutgers Laboratory for Real-Life Reinforcement Learning (RL3), New Jersey.

Nominated for the BNAIC 2009 best paper award.

BNAIC 2009 Local Organizing Committee

2008

Young Researchers Academy Member

TopTalent 2008 PhD scholarship, awarded by NWO Project commenced at Eindhoven University of Technology (TU/e), transferred to Maastricht University in August 2009.

Master thesis, summa cum laude: Games and Learning in Auctions. Partially conducted during the 4-month research visit to Simon Parsons, New York City. [Download]

2007

Visiting research scholar: Collaborating with Simon Parsons at Brooklyn College, City University of New York, New York City.

On 22. June 2007, Karl Tuyls suggests applying for a PhD position under his guidance.

Bachelor thesis, summa cum laude: Reinforcement Learning in Multi-agent Games. [Download]

I study how learning processes affect each other, i.e., how the changing behavior of one agent affects the learning goals of another agent. Nowadays, both people and machines are commonly organized in interactive networks; I focus on economic applications like auctions and stock exchanges. Artificial intelligence allows controlled experiments to investigate the stylized impact of learning parameters such as 'degree of exploration' or 'learning speed' on what is learned: Does the behavior converge to some equilibrium, does it cycle, or end in chaos? Can we prove it?

Theoretical advances have been largely exploiting the link between multi-agent reinforcement learning and evolutionary game theory. The contributions of my thesis can be summarized as follows:

1. I provide an in-depth analysis of the average behavior of Q-learning and its evolutionary dynamical model. Results show that the dynamical model features more rational learning trajectories than the average behavior of Q-learning. For that reason, I derive and propose the algorithm Frequency Adjusted Q-learning (FAQ-learning).
2. I provide a proof of convergence for FAQ-learning in two-action two-player games, constructed within the newly derived framework.
3. I extend the dynamical systems methodology to be applicable to more realistic problems. (1) The model of FAQ-learning is extended to cover time dependent exploration rates and multiple states. (2) A model of lenient FAQ-learning is derived.
4. I propose two new perspectives on multi-agent learning dynamics: (1) an orthogonal view complements existing analysis, especially to design time dependent parameters. (2) I unify the bipartite multi-agent learning literature, proving that multi-agent reinforcement learning implements on-policy stochastic gradient ascent.
5. Finally, the viability of this framework is demonstrated by analyzing meta-strategies in auctions and poker.

Keywords

Artificial intelligence, multi-agent reinforcement learning, evolutionary game theory, dynamical systems

Attracted research money

2011

  0.8k USD AAMAS 2011 travel scholarship

2010

  1.1k USD AAMAS 2010 travel scholarship

2009

  0.3k USD AAMAS 2009 travel scholarship

2008

160.0k Euro TopTalent PhD scholarship

  1.5k Euro Top 3% student scholarship (Master in Artificial Intelligence)

List of Journal Publications and Book Chapters

2010

Michael Kaisers and Karl Tuyls. Replicator Dynamics for Multi-agent Learning - An Orthogonal Approach. In Matthew E. Taylor and Karl Tuyls, editors, Adaptive and Learning Agents, LNAI, pages 49-59. Springer Berlin/Heidelberg, 2010. [Download]

2009

Marc Ponsen, Karl Tuyls, Michael Kaisers, and Jan Ramon. An evolutionary game-theoretic analysis of poker strategies. Entertainment Computing, 1(1):39-45, January 2009. [Download]

2007

Jaap H. van den Herik, Daniel Hennes, Michael Kaisers, Karl Tuyls, and Katja Verbeeck. Multi-agent learning dynamics: A survey. Cooperative Information Agents XI, 4676:36-56, 2007. [Download]

List of Conference and Workshop Publications

2012

Daniel Hennes, Daan Bloembergen, Michael Kaisers, Karl Tuyls, and Simon Parsons. Evolutionary advantage of foresight in markets. In Proc. of the Genetic and Evolutionary Computation Conference (GECCO-2012), pages ??. Sheridan Printing, 2012.

Michael Wunder, John R. Yaros, Michael Littman, and Michael Kaisers. A Framework for Modeling Population Strategies by Depth of Reasoning. In Conitzer, Winikoff, Padgham, and van der Hoek, editors, Proc. of 11th Int. Conf. on Autonomous Agents and Multiagent Systems (AAMAS 2012), pages ??. International Foundation for AAMAS, 2012.

Michael Kaisers, Daan Bloembergen, and Karl Tuyls. A Common Gradient in Multi-agent Reinforcement Learning (Extended Abstract). In Conitzer, Winikoff, Padgham, and van der Hoek, editors, Proc. of 11th Int. Conf. on Autonomous Agents and Multiagent Systems (AAMAS 2012), pages ??. International Foundation for AAMAS, 2012.

2011

Sjriek Alers, Daan Bloembergen, Daniel Hennes, Steven de Jong, Michael Kaisers, Nyree Lemmens, Karl Tuyls, and Gerhard Weiss. Bee-inspired foraging in an embodied swarm (Demonstration). In Tumer, Yolum, Sonenberg, and Stone, editors, Proc. of 10th Int. Conf. on Autonomous Agents and Multiagent Systems (AAMAS 2011), pages 1311-1312. International Foundation for AAMAS, 2011. [Download]

Daan Bloembergen, Michael Kaisers, and Karl Tuyls. Empirical and Theoretical Support for Lenient Learning (Extended Abstract). In Tumer, Yolum, Sonenberg, and Stone, editors, Proc. of 10th Int. Conf. on Autonomous Agents and Multiagent Systems (AAMAS 2011), pages 1105-1106. International Foundation for AAMAS, 2011. [Download]

Michael Kaisers and Karl Tuyls. FAQ-Learning in Matrix Games: Demonstrating Convergence near Nash Equilibria, and Bifurcation of Attractors in the Battle of Sexes. In Workshop on Interactive Decision Theory and Game Theory (IDTGT 2011). Assoc. for the Advancement of Artif. Intel. (AAAI), 2011. [Download]

Michael Kaisers and Karl Tuyls. Multi-agent Learning and the Reinforcement Gradient. In Proc. of 9th European Workshop on Multi-agent Systems (EUMAS 2011). Maastricht University, 2011. [Download]

Daniel Mescheder, Karl Tuyls, and Michael Kaisers. Opponent Modeling with POMDPs. In Proc. of 23nd Belgium-Netherlands Conf. on Artificial Intelligence (BNAIC 2011), pages 152-159. KAHO Sint-Lieven, Gent, 2011. [Download]

Michael Wunder, Michael Kaisers, J.R. Yaros, and Michael Littman. Using iterated reasoning to predict opponent strategies. In Tumer, Yolum, Sonenberg, and Stone, editors, Proc. of 10th Int. Conf. on Autonomous Agents and Multiagent Systems (AAMAS 2011), pages 593-600. International Foundation for AAMAS, 2011. [Download]

2010

Daan Bloembergen, Michael Kaisers, and Karl Tuyls. A comparative study of multi-agent reinforcement learning dynamics. In Proc. of 22nd Belgium- Netherlands Conf. on Artificial Intelligence (BNAIC 2010), pages 11-18. University of Luxembourg, 2010. [Download]

Daan Bloembergen, Michael Kaisers, and Karl Tuyls. Lenient frequency adjusted Q-learning. In Proc. of 22nd Belgium-Netherlands Conf. on Artificial Intelligence (BNAIC 2010), pages 19-26. University of Luxembourg, 2010. [Download]

Michael Wunder, Michael Kaisers, Michael Littman, and John Robert Yaros. A Cognitive Hierarchy Model Applied to the Lemonade Game. In Workshop on Interactive Decision Theory and Game Theory (IDTGT 2010). Assoc. for the Advancement of Artif. Intel. (AAAI), 2010. [Download]

Michael Kaisers and Karl Tuyls. Frequency Adjusted Multi-agent Q-learning. In van der Hoek, Kamina, Lespérance, Luck, and Sen, editors, Proc. of 9th Intl. Conf. on Autonomous Agents and Multiagent Systems (AAMAS 2010), pages 309-315, 2010. [Download]

Daniel Hennes, Michael Kaisers, and Karl Tuyls. RESQ-learning in stochastic games. In Adaptive and Learning Agents (ALA 2010) Workshop, 2010. [Download]

2009

Michael Kaisers. Replicator Dynamics for Multi-agent Learning - An Orthogonal Approach. In Toon Calders, Karl Tuyls, and Mykola Pechenizkiy, editors, Proc. of the 21st Benelux Conference on Artificial Intelligence (BNAIC 2009), pages 113-120, Eindhoven, 2009. Eindhoven University of Technology. [Download]

Michael Kaisers, Karl Tuyls, and Simon Parsons. An Evolutionary Model of Multi-agent Learning with a Varying Exploration Rate (Extended Abstract). In Proc. of 8th Int. Conf. on Autonomous Agents and Multiagent Systems (AAMAS 2009), pages 1255-1256. International Foundation for AAMAS, 2009. [Download]

2008

Michael Kaisers, Karl Tuyls, Frank Thuijsman, and Simon Parsons. Auction Analysis by Normal Form Game Approximation. In Proc. of Int. Conf. on Web Intelligence and Intelligent Agent Technology (WI-IAT 2008), pages 447-450. IEEE/WIC/ACM, December 2008. [Download]

Michael Kaisers, Karl Tuyls, and Frank Thuijsman. Discovering the game in auctions. In Proc. of 20th Belgian-Netherlands Conference on Artificial Intelligence (BNAIC 2008), pages 113-120. University of Twente, 2008. [Download]

You can download my personal bibtex database or find a partial list of my publications on DBLP.

A few selected videos will be uploaded to my YouTube Channel.

Word cloud of publication titles

List of potential thesis topics

M.Sc.

Reinforcement learning and drifting reward distributions: Reinforcement learning is well-founded for multi-armed bandit problems, where reward distributions are stationary. In order to estimate the expected sum of possibly discounted future rewards, several rewards need to be aggregated. However, in sequential decision making (and similarly in multi-agent learning), these rewards are not actually IID - their distributions rather change (slowly). Theory of Monte Carlo Tree Search and Tree Learning Search largely ignores the drift. This thesis will look at ways to explicitly consider drift for reward estimation to improve learning speed and performance.

Analyzing interactive learning: It is common to assume in our analysis of learning that players have to interact with each other. Let us turn that assumption upside down: How does the ability to choose your interaction partners influence what you learn from your interaction? This project can be performed empirically and analytically. The empirical approach requires implementing learning algorithms and running them in games with fixed and flexible interaction partners. For the analytical approach, both types of games need to be defined in terms of Game theory in order to establish a formal connection between them.

Tangible reinforcement learning: Find an application for reinforcement learning that exposes the learning progress in an enjoyable way. Many think that reinforcement learning is an abstract concept that is hard to grasp, but it doesn't need to be that way. There are applications like cart-pole balancing that make it quite tangible, or projects like RL-Viz that try to visualize learning progress more technically. Consider an iPhone app or a audioization or get even more creative. For a bachelor topic, this project should culminate in a software that allows to playfully explore the dynamics of reinforcement learning (e.g., by comparing behaviors of varying parameters). A master student could additionally build a hardware system.

B.Sc.

You can approach me with your ideas, as long as they fit with my experience.

List of students I have been (co-) coaching

2012

Lukas Kirchhart, M.Sc. [upcoming on reuse of knowledge in Tree Learning Search]. June 2012.

Colin Schepers, M.Sc. [upcoming on variants of Tree Learning Search]. June 2012.

Andreas ten Pas, M.Sc. [upcoming on Tree Learning Search in POMDPs]. June 2012.

2011

Marcel Neumann, B.Sc. Price Formation of Continuous Double Auction Agents using Time as a Strategic Element. June 2011. [Download]

Daniel Mescheder, B.Sc. POMDP Opponent Models for Best Response Behavior. June 2011. [Download]

2010

Franz Hahn, B.Sc. An Artificial Intelligence Look at Playing Risk. August 2010. [Download]

Daniel Claes, B.Sc. Balancing Anarchy and Central Control - Individual vs. Joint Action Reinforcement Learning. June 2010. [Download]

Daan Bloembergen, M.Sc. Analyzing Reinforcement Learning Algorithms using Evolutionary Game Theory. June 2010. [Download]

List of courses

2011

Introduction to LaTeX (lecture, TA),
B. Sc. Knowledge Engineering and Computer Science, year 2, block 1

Theoretical computer science (TA),
B. Sc. Knowledge Engineering and Computer Science, year 2, block 4

2010

Object Oriented Modelling (TA, some lectures),
B. Sc. Knowledge Engineering and Computer Science, year 2, block 2

Introduction to LaTeX (lecture, TA),
B. Sc. Knowledge Engineering and Computer Science, year 2, block 1

Linear algebra (TA, some lectures),
B. Sc. Knowledge Engineering and Computer Science, year 1, block 4

Theoretical computer science (TA),
B. Sc. Knowledge Engineering and Computer Science, year 2, block 4

List of related researchers

• Automated mechanism design: Sofia Ceppi, Peter McBurney
• Reinforcement learning: Eduardo Gomes, Daniel Hennes, Marc Ponsen, Ari Weinstein, Michael Wunder
• (Evolutionary) Game theory: Frank Thuijsman, Philippe Uyttendaele

My productivity toolbox

• Short emails with a self-containing subject line (e.g., "Verb | Topic") save other people some trouble
• Email signature: I prioritize short emails with a clear subject line, and process email in batches. Please know your Email etiquette.
• Inbox Zero by Merlin Mann to process email into GTD
• Getting Things Done (GTD) by David Allen for the project overview and next action lists
• Five types of actions: read, write, contact (email, call, meet), think (brainstorm, plan), do. Nothing else belongs on a todo list.
• Eisenhauer time matrix: important beats urgent
• Project milestones and rewards
• Contact is expensive: all contact needs a clear purpose and agenda. Try standing meetings or calls.
• The pomodoro technique to focus time and attention on execution of GTD
• Scrum and sprints to manage projects and responsibilities
• A calendar and paper diary to capture my progress in a timeless way.
• Track time activity and sleep using the Fitbit.
• Track time and attention performance using Rescue Time.

List of academic resources

• Vacancies UM
• Calls for proposals CORDIS
• Calls for papers WikiCFP
• Conferences Overview, gCal, AAMAS 2010, AAAI, GECCO 2010, ICML 2010, ECAI 2010, ECML 2010, BNAIC, IJCAI 2011
• Journals (EF/AI) ML (80.9/94.7), JMLR (87.1/94.6), AI (83.1/92.1), JAIR (68.1/86.9), AAMAS (24.7/41.7), EC (n/a), InTech Open Access (n/a)
• Contests RL competition, Trading competitions (TAC/CAT), Starcraft AI competition, The Lemonade Game
• Lifestyle guides Pocketmod, 43 folders, 4HWW/Timothy Ferriss, Reading guide
• Quality information Learn anything online, iTunes U, Wikipedia, Google
• Interesting reads Reinforcement Learning: An Introduction, Multiagent Systems, Networks, Crowds, and Markets, Predictably Irrational, PhD comics
• Applications MindMapping, Research article management, Publish or perish, Detexify, Word cloud, H-index, Erdos numebr
• Valorisation Maastricht Centre for Entrepreneurship, BioMedbooster, Service Science Factory

List of personal recommendations

• Events in Maastricht Maastrichtnet, VVV Maastricht, Jazz Maastricht, Landbouwbelang, PhD Academy (facebook), Pesthuys
• Salsa in Maastricht MSDV Let's Dance, Mo. Edd's Cafe, Tu. Clinique, Wed. DeKadans (ardesqo), Th. TopFit, Fr. StayOK, Su. summer parties @ Stein
• Paragliding XC planner, DHV (tests and second hand), Paragliding 365, Google wheather map, Wing infos @ Para 2000, Papillon Flugschule und Shop, Windfinder, Wetter online, USAF map, Buienradar, Windguru, Weather Spark, Forum, NOTAM, CTR Europe
• Social networks Facebook, LinkedIn, StudiVZ, Hyves
• Cooking Open Source Food, Order foor to your door
• Finding a room in NL Studentenkamers, Kamernet, Maastricht Housing (former Kamerburo), Housing Anywhere, Maaslandbeheer, Woonpunt/Woonplein
• Public transport in NL Bus schedule, Train schedule, International train schedule, Trains to/from Germany
• Airports nearby
  • MST Maastricht-Aachen (via bus, 0.5h)
  • EIN Eindhoven (via bus and Dutch railways, 2h)
  • BRU Bruessels (via Belgian and Dutch railways, Brussel-N-L/Bruxelles-Nat-Aero to Maastricht Station, 2h)
  • AMS Amsterdam (via Dutch railways, 2.5h)
  • DUS Duesseldorf (via German and Dutch railways, 3.5h)
• Flights QFly, Billigflieger, Expedia, Opodo
• Accomodation Hostelworld, Booking
• Maps Google Maps
• Administration Gemeente Maastricht, IND (residence permits)

One of my biggest passions besides my work and dancing is traveling. Luckily, I have been able to make a large number of trips in recent years for both business and pleasure. By now, I have visited 5 continents (Europe, Asia, Africa, Australia, North America), even though some only sparsely.

List of trips

This list excludes regular visits to Maastricht, Eindhoven, Duisburg, Groningen and Hamburg.

2012

Eifel, Germany (3 - 4 Mar.) visiting friends

2011

Ghent, Belgium (2 - 4 Nov.) Conf. BNAIC 2011

Berlin, Germany (28 Oct. - 1 Nov.) visiting friends

Munich, Germany (23 - 26 Sept.) visiting friends

Stubaital, Austria (19 - 23 Sept.) paragliding with friends

Walenstadt, Switzerland (9 - 11 Sept.) paragliding with friends

IJsselmeer, Netherlands (20 - 26 Aug.) sailing with family

Los Angeles, USA (13 - 17 Aug.) visiting colleagues

San Francisco, USA (5 - 13 Aug.) Twenty-Fifth Conference on Artificial Intelligence

Schleswig, Germany (16 - 23 July) vacation with friends

Calais, France (2 - 3 July) attending a wedding with friends

Dune du Pyla, France (11 - 25 June) paragliding

Cebu - Leyte - Bohol, Philippines (7 - 18 May) visiting family

Taipei, Taiwan (29 Apr. - 6 May) Int. Conf. on AAMAS 2011

Lund, Sweden and Copenhagen, Denmark (31 Mar. - 2 Apr.) visiting friends

Porto, Portugal (18 - 21 Mar.) visiting friends

Hamburg, Germany (19 - 21 Feb.) visiting friends

London, England (11 - 13 Feb.) visiting friends

Boston, USA (2 - 9 Jan.) NECSI Winter school on Complex Systems

2010

New York City, USA (29 Dec. - 2 Jan.) visiting colleagues and friends

Luson, Italy (2 - 10 Oct.) paragliding licence A acquired

Texel, The Netherlands (21 Aug.) day trip with friends

Wasserkuppe, Germany (7 - 14 Aug.) paragliding basic course with family

Rotterdam, The Netherlands (31 July) zomer carnaval day trip with friends

Münster, Germany (4 July) parachute jump day trip with family

Torronto, Canada (9 - 15 May) Int. Conf. on AAMAS 2010

New York City, USA (4 - 9 May) visiting colleagues and friends

Amsterdam, The Netherlands (30 Apr.) Queensday day trip with friends

Madrid, Spain (15 - 18 Jan.) visiting friends

2009

Berlin, Germany (30 Dec. - 2 Jan.) visiting friends

Eindhoven, The Netherlands (27 Oct. - 2 Nov.) Conf. BNAIC 2009

New Brunswick, NJ, USA (11 Oct. - 21 Dec.) visiting research scholar, regular visits to NYC

Trier, Germany (4 - 6 Sept.) visiting friends

All over Ireland (25 July - 9 Aug.) family vacation, bike trip

Niederrhein, Germany (3 - 5 July) family test bike trip

Hohes Venn, Belgium (12 - 15 June) family motorbike trip

All over Ethiopia (20 May - 7 June) visiting friends, round trip

Budapest, Hungary (9 - 16 May) Int. Conf. on AAMAS 2009

London, England (27 - 30 Mar.) visiting friends

Brussels, Belgium (20 - 21 Mar.) visiting friends

Amsterdam, The Netherlands (14 Mar.) day trip to meet with friends

Singapore (12 - 16 Jan.) visiting friends

2008

All over New Zealand (25 Dec. - 12 Jan.) road trip

Cairns to Brisbane, Australia (13 - 25 Dec.) road trip

Sydney, Australia (4 - 12 Dec.) Int. Conf. on IAT

Enschede, The Netherlands (29 - 31 Oct.) Conf. BNAIC 2008

All over Belgium/France (22 July - 10 Aug.) road trip

Estoril/Lisbon, Portugal (11 - 17 May) Int. Conf. on AAMAS 2008

Roermond, The Netherlands (5 May) liberation day trip with friends

Trier, Germany (2 - 4 May) visiting friends

Amsterdam, The Netherlands (30 April) visiting friends

Brussels, Belgium (28 - 29 April) visiting friends

Fuerteventura, Spain (15 - 20 Feb.) vacation with friends

2007

All over Puerto Rico (29 Dec. - 13 Jan.) road trip with friends

Boston, USA (18 - 20 Dec.) city tour with friends

Portland-Vancouver-San Francisco, USA/Canada (23 Oct. - 3 Nov.) road trip

Montreal, Canada (19 - 21 Oct.) city tour with friends

Kingston, Canada (10 - 19 Oct.) visiting friends

New York City, USA (13 Sept. - 24 Jan.) visiting research scholar

All over the north of The Netherlands (21 July - 6 Aug.) motorbike trip with friends

Spiekeroog, Germany (27 - 28 June) visiting family

2006

Gerlitzen, Austria (26 Dec. - 2 Jan.) ski vacation with friends

Taize, France (30 Sept. - 8 Oct.) retreat vacation with family

Berlin, Germany (14 - 17 Aug.) visiting friends

Waco, Texas (3 July - 6 Aug.) Baylor summer school on entrepreneurship

Karlsruhe, Germany (4 - 6 May) University promotion, visiting friends

Karlsruhe, Germany (2 - 6 Mar.) visiting friends

2005

Mallorca, Spain (28 July - 11 Aug.) vacation with friends

You can contact me in German, English, or Dutch for professional requests. I would also like to improve my Spanish and I'll appreciate opportunities to practice. [download vCard]

E-mail


Postal address
P.O. Box 616
6200 MD Maastricht
The Netherlands

Phone
+31-43-388-3919
+31-43-388-3916

Visiting address (Google Maps)
Sint Servaasklooster 39 (room 0.003)
6211 TE Maastricht
The Netherlands