{ "cells": [ { "cell_type": "markdown", "id": "c74ef5f0", "metadata": {}, "source": [ "# Building MDPs\n", "In Stormvogel, a **Markov Decision Process (MDP)** consists of:\n", "* states $S$,\n", "* actions $A$,\n", "* an initial state $s_0$,\n", "* a mapping from states to sets of *enabled actions*,\n", "* a successor distribution $P(s,a)$ for every state $s$ and every enabled action $a$, i.e., sets of transitions between states $s$ and $s'$, each annotated with an action and a probability.\n", "* state labels $L(s)$.\n", "\n", "\n", "Here we show how to construct a simple example mdp using the bird API and the model builder API.\n", "The idea is that you can choose to study (you will likely pass the exam but you have less free time) or not to study (you will have more free time but risk failing the exam)." ] }, { "cell_type": "markdown", "id": "046461f1", "metadata": {}, "source": [ "## The study dilemma\n", "This little MDP is supposed to help you decide whether you should stuy or not." ] }, { "cell_type": "markdown", "id": "8f844a74", "metadata": {}, "source": [ "### Bird API\n", "For MDPs, you specify the availaible actions in `available_actions`. An action here is simply a string. You specify the transition of a state-action pair in `delta`." ] }, { "cell_type": "code", "execution_count": 1, "id": "fab96dd0", "metadata": { "execution": { "iopub.execute_input": "2026-10-01T12:39:12.902080Z", "iopub.status.busy": "2026-10-01T12:39:12.901917Z", "iopub.status.idle": "2026-10-01T12:39:13.486168Z", "shell.execute_reply": "2026-10-01T12:39:13.485583Z" } }, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", " \n", " Network\n", " \n", " \n", " \n", " \n", " \n", "
\n", " \n", " \n", " \n", " \n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from stormvogel import *\n", "import stormvogel\n", "\n", "\n", "def available_actions(s):\n", " if s == \"init\":\n", " return [\"study\", \"don't study\"]\n", " else:\n", " return [\"\"]\n", "\n", "\n", "def delta(s, a):\n", " if a == \"study\":\n", " return [(1, \"did study\")]\n", " elif a == \"don't study\":\n", " return [(1, \"did not study\")]\n", " elif s == \"did study\":\n", " return [(9 / 10, \"pass test\"), (1 / 10, \"fail test\")]\n", " elif s == \"did not study\":\n", " return [(2 / 5, \"pass test\"), (3 / 5, \"fail test\")]\n", " else:\n", " return [(1, \"end\")]\n", "\n", "\n", "def labels(s):\n", " return s\n", "\n", "\n", "# For rewards, you have to provide a dict. This enables multiple reward models if you use a non-singleton list.\n", "def rewards(s: bird.BirdState):\n", " if s == \"pass test\":\n", " return {\"R\": 100}\n", " elif s == \"did not study\":\n", " return {\"R\": 15}\n", " else:\n", " return {\"R\": 0}\n", "\n", "\n", "bird_study = bird.build_bird(\n", " delta=delta,\n", " init=\"init\",\n", " available_actions=available_actions,\n", " labels=labels,\n", " modeltype=ModelType.MDP,\n", " rewards=rewards,\n", ")\n", "show(bird_study)" ] }, { "cell_type": "markdown", "id": "80dfb8cc", "metadata": {}, "source": [ "### Model API" ] }, { "cell_type": "code", "execution_count": 2, "id": "0eca2c32", "metadata": { "execution": { "iopub.execute_input": "2026-10-01T12:39:13.503571Z", "iopub.status.busy": "2026-10-01T12:39:13.503246Z", "iopub.status.idle": "2026-10-01T12:39:13.532876Z", "shell.execute_reply": "2026-10-01T12:39:13.532301Z" } }, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", " \n", " Network\n", " \n", " \n", " \n", " \n", " \n", "
\n", " \n", " \n", " \n", " \n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "\n", "mdp = stormvogel.model.new_mdp()\n", "\n", "init = mdp.initial_state\n", "study = mdp.action(\"study\")\n", "not_study = mdp.action(\"don't study\")\n", "\n", "did_study = mdp.new_state(\"did study\")\n", "did_not_study = mdp.new_state(\"did not study\")\n", "\n", "pass_test = mdp.new_state(\"pass test\")\n", "fail_test = mdp.new_state(\"fail test\")\n", "end = mdp.new_state(\"end\")\n", "\n", "init.set_choices(\n", " {\n", " study: [(1, did_study)],\n", " not_study: [(1, did_not_study)],\n", " }\n", ")\n", "\n", "did_study.set_choices([(9 / 10, pass_test), (1 / 10, fail_test)])\n", "did_not_study.set_choices([(4 / 10, pass_test), (6 / 10, fail_test)])\n", "\n", "pass_test.set_choices([(1, end)])\n", "fail_test.set_choices([(1, end)])\n", "\n", "mdp.add_self_loops()\n", "\n", "reward_model = mdp.new_reward_model(\"R\")\n", "reward_model.set_state_reward(pass_test, 100)\n", "reward_model.set_state_reward(fail_test, 0)\n", "reward_model.set_state_reward(did_not_study, 15)\n", "reward_model.set_unset_rewards(0)\n", "\n", "show(mdp)" ] }, { "cell_type": "markdown", "id": "261e1e23", "metadata": {}, "source": [ "## Grid model\n", "An MDP model that consists of a 3x3 grid. The direction to walk is chosen by an action." ] }, { "cell_type": "markdown", "id": "732762cc", "metadata": {}, "source": [ "### Bird API" ] }, { "cell_type": "code", "execution_count": 3, "id": "434a44e0", "metadata": { "execution": { "iopub.execute_input": "2026-10-01T12:39:13.551115Z", "iopub.status.busy": "2026-10-01T12:39:13.550921Z", "iopub.status.idle": "2026-10-01T12:39:13.579409Z", "shell.execute_reply": "2026-10-01T12:39:13.578849Z" } }, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", " \n", " Network\n", " \n", " \n", " \n", " \n", " \n", "
\n", " \n", " \n", " \n", " \n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "N = 3\n", "\n", "ACTION_SEMANTICS = {\"l\": (-1, 0), \"r\": (1, 0), \"u\": (0, -1), \"d\": (0, 1)}\n", "\n", "\n", "def available_actions(s):\n", " res = []\n", " if s[0] > 0:\n", " res.append(\"l\")\n", " if s[0] < N - 1:\n", " res.append(\"r\")\n", " if s[1] > 0:\n", " res.append(\"u\")\n", " if s[1] < N - 1:\n", " res.append(\"d\")\n", " return res\n", "\n", "\n", "def pairwise_plus(t1, t2):\n", " return (t1[0] + t2[0], t1[1] + t2[1])\n", "\n", "\n", "def delta(s, a):\n", " return [(1, pairwise_plus(s, ACTION_SEMANTICS[a]))]\n", "\n", "\n", "def labels(s):\n", " return [str(s)]\n", "\n", "\n", "m1 = bird.build_bird(\n", " init=(0, 0), available_actions=available_actions, labels=labels, delta=delta\n", ")\n", "show(m1)" ] }, { "cell_type": "markdown", "id": "28a39e1b", "metadata": {}, "source": [ "### Model API" ] }, { "cell_type": "code", "execution_count": 4, "id": "9f222b8c", "metadata": { "execution": { "iopub.execute_input": "2026-10-01T12:39:13.598571Z", "iopub.status.busy": "2026-10-01T12:39:13.598375Z", "iopub.status.idle": "2026-10-01T12:39:13.626818Z", "shell.execute_reply": "2026-10-01T12:39:13.626204Z" } }, "outputs": [ { "data": { "text/html": [ "\n", "\n", "\n", " \n", " Network\n", " \n", " \n", " \n", " \n", " \n", "
\n", " \n", " \n", " \n", " \n", "\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "grid_model = stormvogel.model.new_mdp(create_initial_state=False)\n", "\n", "for x in range(N):\n", " for y in range(N):\n", " labels = [f\"({x},{y})\"]\n", " if x == 0 and y == 0:\n", " labels.append(\"init\")\n", " grid_model.new_state(labels)\n", "\n", "for x in range(N):\n", " for y in range(N):\n", " state_tile_label = str((x, y)).replace(\" \", \"\")\n", " state = next(iter(grid_model.get_states_with_label(state_tile_label)))\n", " av = available_actions((x, y))\n", " for a in av:\n", " target_tile_label = str(pairwise_plus((x, y), ACTION_SEMANTICS[a])).replace(\n", " \" \", \"\"\n", " )\n", " target_state = next(\n", " iter(grid_model.get_states_with_label(target_tile_label))\n", " )\n", " state.add_choices([(grid_model.action(a), target_state)])\n", "show(grid_model)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.15" }, "widgets": { "application/vnd.jupyter.widget-state+json": { "state": { "161ac243935244d4bcdc4d312e070159": { "model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "1f9ad73ff96948d4bc72754a2d2f54b7": { "model_module": "@jupyter-widgets/output", "model_module_version": "1.0.0", "model_name": "OutputModel", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/output", "_model_module_version": "1.0.0", "_model_name": "OutputModel", "_view_count": null, "_view_module": "@jupyter-widgets/output", "_view_module_version": "1.0.0", "_view_name": "OutputView", "layout": "IPY_MODEL_6893a809f12846648b75ffe2c55f1f95", "msg_id": "", "outputs": [], "tabbable": null, "tooltip": null } }, "2d6d7cf2e729494d809005c34781e00d": { "model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "3999bf493dac45e4b1c3d2fd415d9ae8": { "model_module": "@jupyter-widgets/output", "model_module_version": "1.0.0", "model_name": "OutputModel", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/output", "_model_module_version": "1.0.0", "_model_name": "OutputModel", "_view_count": null, "_view_module": "@jupyter-widgets/output", "_view_module_version": "1.0.0", "_view_name": "OutputView", "layout": "IPY_MODEL_2d6d7cf2e729494d809005c34781e00d", "msg_id": "", "outputs": [], "tabbable": null, "tooltip": null } }, "425b7be0f7064dabafaa141c95b8cc23": { "model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "442f6196493e40baa1b22bec1c6942bd": { "model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "482a52348b1d4778a73f00665d4e4284": { "model_module": "@jupyter-widgets/output", "model_module_version": "1.0.0", "model_name": "OutputModel", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/output", "_model_module_version": "1.0.0", "_model_name": "OutputModel", "_view_count": null, "_view_module": "@jupyter-widgets/output", "_view_module_version": "1.0.0", "_view_name": "OutputView", "layout": "IPY_MODEL_e9f0a5715eba4f5786b1f1455b96cad7", "msg_id": "", "outputs": [], "tabbable": null, "tooltip": null } }, "5d1a6b9f98c649dbbf8c5c65870403d4": { "model_module": "@jupyter-widgets/output", "model_module_version": "1.0.0", "model_name": "OutputModel", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/output", "_model_module_version": "1.0.0", "_model_name": "OutputModel", "_view_count": null, "_view_module": "@jupyter-widgets/output", "_view_module_version": "1.0.0", "_view_name": "OutputView", "layout": "IPY_MODEL_425b7be0f7064dabafaa141c95b8cc23", "msg_id": "", "outputs": [], "tabbable": null, "tooltip": null } }, "5dab9e881793430da46c77c1416ba44c": { "model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "6893a809f12846648b75ffe2c55f1f95": { "model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "6dd30c786e3b427592b8c6495b3e15dd": { "model_module": "@jupyter-widgets/output", "model_module_version": "1.0.0", "model_name": "OutputModel", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/output", "_model_module_version": "1.0.0", "_model_name": "OutputModel", "_view_count": null, "_view_module": "@jupyter-widgets/output", "_view_module_version": "1.0.0", "_view_name": "OutputView", "layout": "IPY_MODEL_9aa4f64b1c92411791adbd5e473b1d4f", "msg_id": "", "outputs": [], "tabbable": null, "tooltip": null } }, "85d6ff078851463c9a5b51a535e9bfc7": { "model_module": "@jupyter-widgets/output", "model_module_version": "1.0.0", "model_name": "OutputModel", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/output", "_model_module_version": "1.0.0", "_model_name": "OutputModel", "_view_count": null, "_view_module": "@jupyter-widgets/output", "_view_module_version": "1.0.0", "_view_name": "OutputView", "layout": "IPY_MODEL_161ac243935244d4bcdc4d312e070159", "msg_id": "", "outputs": [], "tabbable": null, "tooltip": null } }, "9aa4f64b1c92411791adbd5e473b1d4f": { "model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } }, "c56e63fe34f643df90ed476dd85c6a03": { "model_module": "@jupyter-widgets/output", "model_module_version": "1.0.0", "model_name": "OutputModel", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/output", "_model_module_version": "1.0.0", "_model_name": "OutputModel", "_view_count": null, "_view_module": "@jupyter-widgets/output", "_view_module_version": "1.0.0", "_view_name": "OutputView", "layout": "IPY_MODEL_442f6196493e40baa1b22bec1c6942bd", "msg_id": "", "outputs": [], "tabbable": null, "tooltip": null } }, "e0e879d1b4a145f898ba5e5e7cc4efcb": { "model_module": "@jupyter-widgets/output", "model_module_version": "1.0.0", "model_name": "OutputModel", "state": { "_dom_classes": [], "_model_module": "@jupyter-widgets/output", "_model_module_version": "1.0.0", "_model_name": "OutputModel", "_view_count": null, "_view_module": "@jupyter-widgets/output", "_view_module_version": "1.0.0", "_view_name": "OutputView", "layout": "IPY_MODEL_5dab9e881793430da46c77c1416ba44c", "msg_id": "", "outputs": [], "tabbable": null, "tooltip": null } }, "e9f0a5715eba4f5786b1f1455b96cad7": { "model_module": "@jupyter-widgets/base", "model_module_version": "2.0.0", "model_name": "LayoutModel", "state": { "_model_module": "@jupyter-widgets/base", "_model_module_version": "2.0.0", "_model_name": "LayoutModel", "_view_count": null, "_view_module": "@jupyter-widgets/base", "_view_module_version": "2.0.0", "_view_name": "LayoutView", "align_content": null, "align_items": null, "align_self": null, "border_bottom": null, "border_left": null, "border_right": null, "border_top": null, "bottom": null, "display": null, "flex": null, "flex_flow": null, "grid_area": null, "grid_auto_columns": null, "grid_auto_flow": null, "grid_auto_rows": null, "grid_column": null, "grid_gap": null, "grid_row": null, "grid_template_areas": null, "grid_template_columns": null, "grid_template_rows": null, "height": null, "justify_content": null, "justify_items": null, "left": null, "margin": null, "max_height": null, "max_width": null, "min_height": null, "min_width": null, "object_fit": null, "object_position": null, "order": null, "overflow": null, "padding": null, "right": null, "top": null, "visibility": null, "width": null } } }, "version_major": 2, "version_minor": 0 } } }, "nbformat": 4, "nbformat_minor": 5 }