stormvogel.teaching.policy_to_pmc

Transformation: resolve POMDP/MDP nondeterminism with a parametric policy.

Functions

_sanitize(→ str)	Return a valid sympy/Python identifier derived from name.
_fresh(→ str)	Return desired if not in taken, else append a numeric suffix.
policy_to_pmc(→ stormvogel.model.model.Model)	Resolve POMDP/MDP nondeterminism with a parametric observation policy.

Module Contents

stormvogel.teaching.policy_to_pmc._sanitize(name: str) → str

Return a valid sympy/Python identifier derived from name.

stormvogel.teaching.policy_to_pmc._fresh(desired: str, taken: set[str]) → str

Return desired if not in taken, else append a numeric suffix.

stormvogel.teaching.policy_to_pmc.policy_to_pmc(model: stormvogel.model.model.Model) → stormvogel.model.model.Model

Resolve POMDP/MDP nondeterminism with a parametric observation policy.

For each observation o (POMDP) or state s (MDP) with k ≥ 2 available actions a₁ … aₖ, fresh parameters y_{o,a} (or y_{s,a}) are introduced, representing the probability of choosing action a. The resulting pMC transition from state s to s’ is:

\[\sum_{a} y_{\text{key},a} \cdot P(s, a, s')\]

where key is the observation (POMDP) or state (MDP). States with only one available action are copied without introducing a parameter.

Existing model parameters (e.g. transition probabilities in a pMDP) are declared on the new model and preserved in the combined expressions. New policy parameter names are chosen to be disjoint from all pre-existing names in the model.

Note

The constraint \(\sum_a y_{\text{key},a} = 1\) is not enforced in the model structure and must hold for any concrete policy.

Parameters:

model – An MDP, pMDP, POMDP, or MA with deterministic state observations.

Returns:

A new parametric DTMC.

Raises:

ValueError – If the model type is not supported, or if a POMDP state has a stochastic (distribution-valued) observation.