{
"cells": [
{
"cell_type": "markdown",
"id": "1c3f63f4",
"metadata": {},
"source": [
"# Gymnasium sampling\n",
"For the Frozenlake, Cliffwalking, and Taxi models, we were able to access the internal state of the gym environments to convert it to an accurate stormvogel model. However, this is not the case for arbitrary gym environments, hence we can use sampling to get an approximation of the gym envrironment in stormvogel. In this notebook we give some example usages of gymnasium sampling from `stormvogel.extensions`. Note that sampling is actually quite fast, but the visualization gets slow quickly when the amount of states increases.\n",
"\n",
"* `sample_gym` samples a gym environment and gives the result as 5 defaultdicts and an integer.\n",
"* `sample_to_stormvogel` converts such a sample to a stormvogel model.\n",
"* `sample_gym_to_stormvogel` combines the two for convenience."
]
},
{
"cell_type": "markdown",
"id": "23a59e1d",
"metadata": {},
"source": [
"## FrozenLake\n",
"Since FrozenLake does not have too many states, and it is fully observable, we are actually very likely to get the correct model if we use enough samples. If you lower the sample rate, you will observe that at some point, transitions and states will disappear. You can also enable is_slippery. You will then get an approximation of the FrozenLake with slipping ice."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "bd527441",
"metadata": {
"execution": {
"iopub.execute_input": "2026-07-01T08:30:15.862176Z",
"iopub.status.busy": "2026-07-01T08:30:15.861982Z",
"iopub.status.idle": "2026-07-01T08:30:16.607259Z",
"shell.execute_reply": "2026-07-01T08:30:16.606702Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"ModelType.MDP model with 19 states, 50 choices, and 18 distinct labels.\n"
]
},
{
"data": {
"text/html": [
"\n",
"\n",
"\n",
" \n",
" Network\n",
" \n",
" \n",
" \n",
" \n",
" \n",
" \n",
" \n",
" \n",
" \n",
" \n",
"\n"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
""
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from stormvogel import *\n",
"from stormvogel.extensions.gym_sampling import sample_gym_to_stormvogel\n",
"import gymnasium as gym\n",
"\n",
"env = gym.make(\"FrozenLake-v1\", render_mode=\"rgb_array\", is_slippery=False)\n",
"model = sample_gym_to_stormvogel(env, no_samples=200)\n",
"print(model.summary())\n",
"show(model)"
]
},
{
"cell_type": "markdown",
"id": "99b0b17f",
"metadata": {},
"source": [
"## Blackjack\n",
"Blackjack is not fully observable, hence states that are not identical in the gymnasium model are merged in the sample model."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "cf1238d8",
"metadata": {
"execution": {
"iopub.execute_input": "2026-07-01T08:30:16.616447Z",
"iopub.status.busy": "2026-07-01T08:30:16.616009Z",
"iopub.status.idle": "2026-07-01T08:30:16.658285Z",
"shell.execute_reply": "2026-07-01T08:30:16.657765Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"ModelType.MDP model with 99 states, 103 choices, and 60 distinct labels.\n"
]
},
{
"data": {
"text/html": [
"\n",
"\n",
"\n",
" \n",
" Network\n",
" \n",
" \n",
" \n",
" \n",
" \n",
" \n",
" \n",
" \n",
" \n",
" \n",
"\n"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
""
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import gymnasium as gym\n",
"from stormvogel import *\n",
"\n",
"env = gym.make(\"Blackjack-v1\", render_mode=\"rgb_array\")\n",
"model = sample_gym_to_stormvogel(env, no_samples=50)\n",
"print(model.summary())\n",
"show(model)"
]
},
{
"cell_type": "markdown",
"id": "352c118f",
"metadata": {},
"source": [
"## Acrobot\n",
"We can even sample continuous environments and treat them like MDPs. In this particular case, all numbers are rounded to 1 decimal (in `convert_obs`). The more accurate you want to be, the more states are required!"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "92cfe6c3",
"metadata": {
"execution": {
"iopub.execute_input": "2026-07-01T08:30:16.667576Z",
"iopub.status.busy": "2026-07-01T08:30:16.667334Z",
"iopub.status.idle": "2026-07-01T08:30:16.722655Z",
"shell.execute_reply": "2026-07-01T08:30:16.722103Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"ModelType.MDP model with 52 states, 48 choices, and 52 distinct labels.\n"
]
},
{
"data": {
"text/html": [
"\n",
"\n",
"\n",
" \n",
" Network\n",
" \n",
" \n",
" \n",
" \n",
" \n",
" \n",
" \n",
" \n",
" \n",
" \n",
"\n"
],
"text/plain": [
""
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
""
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"env = gym.make(\"Acrobot-v1\", render_mode=\"rgb_array\")\n",
"\n",
"\n",
"def convert_obs(xs):\n",
" return tuple([round(float(x), 1) for x in xs])\n",
"\n",
"\n",
"model = sample_gym_to_stormvogel(\n",
" env, no_samples=10, sample_length=5, convert_obs=convert_obs, max_size=10000\n",
")\n",
"print(model.summary())\n",
"show(model)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.13.14"
},
"widgets": {
"application/vnd.jupyter.widget-state+json": {
"state": {
"029d07f8c857456e987dcb493ebd9da1": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "2.0.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "2.0.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "2.0.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border_bottom": null,
"border_left": null,
"border_right": null,
"border_top": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"3f9a754201144dbd853acb382a27cebc": {
"model_module": "@jupyter-widgets/output",
"model_module_version": "1.0.0",
"model_name": "OutputModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/output",
"_model_module_version": "1.0.0",
"_model_name": "OutputModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/output",
"_view_module_version": "1.0.0",
"_view_name": "OutputView",
"layout": "IPY_MODEL_535c8915af8e4b0ea864d00571970fea",
"msg_id": "",
"outputs": [],
"tabbable": null,
"tooltip": null
}
},
"4d822e186c5842649caa96ee83026671": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "2.0.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "2.0.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "2.0.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border_bottom": null,
"border_left": null,
"border_right": null,
"border_top": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"535c8915af8e4b0ea864d00571970fea": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "2.0.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "2.0.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "2.0.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border_bottom": null,
"border_left": null,
"border_right": null,
"border_top": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"66ddd23bf81d4046b510b9e32d856dbf": {
"model_module": "@jupyter-widgets/output",
"model_module_version": "1.0.0",
"model_name": "OutputModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/output",
"_model_module_version": "1.0.0",
"_model_name": "OutputModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/output",
"_view_module_version": "1.0.0",
"_view_name": "OutputView",
"layout": "IPY_MODEL_029d07f8c857456e987dcb493ebd9da1",
"msg_id": "",
"outputs": [],
"tabbable": null,
"tooltip": null
}
},
"6d9e68a1afe546288ba8b7bdf5553663": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "2.0.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "2.0.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "2.0.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border_bottom": null,
"border_left": null,
"border_right": null,
"border_top": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"95b7654c788c43749a71c9a4a91b75d2": {
"model_module": "@jupyter-widgets/output",
"model_module_version": "1.0.0",
"model_name": "OutputModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/output",
"_model_module_version": "1.0.0",
"_model_name": "OutputModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/output",
"_view_module_version": "1.0.0",
"_view_name": "OutputView",
"layout": "IPY_MODEL_4d822e186c5842649caa96ee83026671",
"msg_id": "",
"outputs": [],
"tabbable": null,
"tooltip": null
}
},
"97b38e1161e44981b1c3363077472ffd": {
"model_module": "@jupyter-widgets/output",
"model_module_version": "1.0.0",
"model_name": "OutputModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/output",
"_model_module_version": "1.0.0",
"_model_name": "OutputModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/output",
"_view_module_version": "1.0.0",
"_view_name": "OutputView",
"layout": "IPY_MODEL_ff9f1aa29a0c4ca3b8aab31dfe034f33",
"msg_id": "",
"outputs": [],
"tabbable": null,
"tooltip": null
}
},
"b7e7190bfe484bc5a5501c2cdf59cf43": {
"model_module": "@jupyter-widgets/output",
"model_module_version": "1.0.0",
"model_name": "OutputModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/output",
"_model_module_version": "1.0.0",
"_model_name": "OutputModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/output",
"_view_module_version": "1.0.0",
"_view_name": "OutputView",
"layout": "IPY_MODEL_6d9e68a1afe546288ba8b7bdf5553663",
"msg_id": "",
"outputs": [],
"tabbable": null,
"tooltip": null
}
},
"bd31fe54f0bd44ed88d7b5a02a5ffeb4": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "2.0.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "2.0.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "2.0.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border_bottom": null,
"border_left": null,
"border_right": null,
"border_top": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
},
"ead57b54622846b0be194b2ecd5a93c1": {
"model_module": "@jupyter-widgets/output",
"model_module_version": "1.0.0",
"model_name": "OutputModel",
"state": {
"_dom_classes": [],
"_model_module": "@jupyter-widgets/output",
"_model_module_version": "1.0.0",
"_model_name": "OutputModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/output",
"_view_module_version": "1.0.0",
"_view_name": "OutputView",
"layout": "IPY_MODEL_bd31fe54f0bd44ed88d7b5a02a5ffeb4",
"msg_id": "",
"outputs": [],
"tabbable": null,
"tooltip": null
}
},
"ff9f1aa29a0c4ca3b8aab31dfe034f33": {
"model_module": "@jupyter-widgets/base",
"model_module_version": "2.0.0",
"model_name": "LayoutModel",
"state": {
"_model_module": "@jupyter-widgets/base",
"_model_module_version": "2.0.0",
"_model_name": "LayoutModel",
"_view_count": null,
"_view_module": "@jupyter-widgets/base",
"_view_module_version": "2.0.0",
"_view_name": "LayoutView",
"align_content": null,
"align_items": null,
"align_self": null,
"border_bottom": null,
"border_left": null,
"border_right": null,
"border_top": null,
"bottom": null,
"display": null,
"flex": null,
"flex_flow": null,
"grid_area": null,
"grid_auto_columns": null,
"grid_auto_flow": null,
"grid_auto_rows": null,
"grid_column": null,
"grid_gap": null,
"grid_row": null,
"grid_template_areas": null,
"grid_template_columns": null,
"grid_template_rows": null,
"height": null,
"justify_content": null,
"justify_items": null,
"left": null,
"margin": null,
"max_height": null,
"max_width": null,
"min_height": null,
"min_width": null,
"object_fit": null,
"object_position": null,
"order": null,
"overflow": null,
"padding": null,
"right": null,
"top": null,
"visibility": null,
"width": null
}
}
},
"version_major": 2,
"version_minor": 0
}
}
},
"nbformat": 4,
"nbformat_minor": 5
}