`neurospatial.behavior.reward`¶

reward ¶

Reward field generation for reinforcement learning applications.

This module provides functions for creating reward fields from spatial regions and goal locations, for reinforcement learning (RL) models of spatial navigation. A reward field can serve as the potential of potential-based reward shaping (Ng et al., 1999).

Imports

from neurospatial.behavior.reward import goal_reward_field, region_reward_field

Or via behavior package:

from neurospatial.behavior import goal_reward_field, region_reward_field

Classes¶

Functions¶

region_reward_field ¶

region_reward_field(env: Environment, *, region_name: str, reward_value: float = 1.0, decay: Literal['constant', 'linear', 'gaussian'] = 'constant', bandwidth: float | None = None, max_distance: float | None = None) -> NDArray[np.float64]

Generate reward field from a named region.

Creates a spatial reward field based on a region defined in the environment. Supports different decay profiles for reward shaping in reinforcement learning.

Parameters:

Name	Type	Description	Default
`env`	`Environment`	Environment with regions defined.	required
`region_name`	`str`	Name of region in env.regions to use as reward source.	required
`reward_value`	`float`	Maximum reward value in the region.	`1.0`
`decay`	`('constant', 'linear', 'gaussian')`	Reward decay profile: - 'constant': Binary reward (reward_value inside, 0 outside) - 'linear': Linear decay from region boundary - 'gaussian': Smooth Gaussian falloff from region	`'constant'`
`bandwidth`	`float`	Gaussian smoothing bandwidth, in the environment's distance units. Required for, and only accepted with, `decay='gaussian'`.	`None`
`max_distance`	`float`	Geodesic distance from the region, in the environment's distance units, at which the linear reward reaches 0. Only accepted with `decay='linear'`. Default: the distance of the farthest bin that can reach the region, so every reachable bin outside the region gets a nonzero gradient. That default depends on the arena size: the same distance gets a different reward in a larger arena. Pass `max_distance` to compare reward fields across arenas.	`None`

Returns:

Name	Type	Description
`reward`	`(NDArray[float64], shape(n_bins))`	Reward field with specified decay profile.

Raises:

Type	Description
`RegionNotFoundError`	If region_name is not found in env.regions (a `KeyError`).
`ValueError`	If `decay` is unknown; if `bandwidth` is missing for `decay='gaussian'`; if `bandwidth` or `max_distance` is given for a decay that does not use it; or if either is not finite and positive.

Notes

Decay Types:

Constant: Binary reward useful for sparse reward RL tasks. Creates clear goal regions with no reward shaping.
Linear: reward_value * max(0, 1 - d / max_distance), where d is the geodesic distance from the bin centre to the nearest region bin centre: the length of the shortest path between two positions that stays inside the environment (it goes around walls and never leaves the environment, and on a track it runs along the track's centerline). Bins that cannot reach the region (another connected component) get 0.
Gaussian: Smooth falloff using Gaussian kernel smoothing. After smoothing, rescales so maximum within the region equals reward_value. This preserves the intended reward magnitude while providing smooth gradients for policy learning.

Reward Shaping: A reward field can serve as the potential Phi(s) of potential-based reward shaping [1]_, which adds gamma * Phi(s') - Phi(s) to the task reward and leaves the optimal policy unchanged. Adding the field itself to the task reward is not potential-based and can change the optimal policy. Gaussian decay provides the smoothest gradients; validate any shaped agent against a sparse-reward baseline.

Examples:

>>> import numpy as np
>>> from neurospatial import Environment
>>> from neurospatial.behavior.reward import region_reward_field
>>> from shapely.geometry import Point
>>> # Create environment
>>> positions = np.random.default_rng(0).normal(0, 50, (1000, 2))
>>> env = Environment.from_samples(positions, bin_size=5.0)
>>> # Add goal region (use polygon, not point)
>>> goal_polygon = Point([0.0, 0.0]).buffer(10.0)
>>> from neurospatial.regions import Regions
>>> regions = Regions(env.regions.values())
>>> _ = regions.add("goal", polygon=goal_polygon)
>>> env = env.with_regions(regions)
>>> # Binary reward (sparse RL)
>>> reward_binary = region_reward_field(env, region_name="goal", decay="constant")
>>> # Smooth Gaussian reward shaping
>>> reward_smooth = region_reward_field(
...     env, region_name="goal", decay="gaussian", bandwidth=10.0
... )
>>> # Linear decay that reaches 0 at 30 units from the region
>>> reward_linear = region_reward_field(
...     env, region_name="goal", decay="linear", max_distance=30.0
... )
>>> float(reward_linear.max()), float(reward_linear.min())
(1.0, 0.0)

See Also

goal_reward_field : Distance-based rewards from goal bins neurospatial.ops.distance_field : Bin-graph distance maps neurospatial.ops.regions_to_mask : Convert regions to binary masks

References

.. [1] Ng, A. Y., Harada, D., & Russell, S. (1999). Policy invariance under reward transformations: Theory and application to reward shaping. ICML.

Source code in src/neurospatial/behavior/reward.py

def region_reward_field(
    env: Environment,
    *,
    region_name: str,
    reward_value: float = 1.0,
    decay: Literal["constant", "linear", "gaussian"] = "constant",
    bandwidth: float | None = None,
    max_distance: float | None = None,
) -> NDArray[np.float64]:
    """Generate reward field from a named region.

    Creates a spatial reward field based on a region defined in the environment.
    Supports different decay profiles for reward shaping in reinforcement learning.

    Parameters
    ----------
    env : Environment
        Environment with regions defined.
    region_name : str
        Name of region in env.regions to use as reward source.
    reward_value : float, default=1.0
        Maximum reward value in the region.
    decay : {'constant', 'linear', 'gaussian'}, default='constant'
        Reward decay profile:
        - 'constant': Binary reward (reward_value inside, 0 outside)
        - 'linear': Linear decay from region boundary
        - 'gaussian': Smooth Gaussian falloff from region
    bandwidth : float, optional
        Gaussian smoothing bandwidth, in the environment's distance units.
        Required for, and only accepted with, ``decay='gaussian'``.
    max_distance : float, optional
        Geodesic distance from the region, in the environment's distance
        units, at which the linear reward reaches 0. Only accepted with
        ``decay='linear'``. Default: the distance of the farthest bin that
        can reach the region, so every reachable bin outside the region gets
        a nonzero gradient. That default depends on the arena size: the same
        distance gets a different reward in a larger arena. Pass
        ``max_distance`` to compare reward fields across arenas.

    Returns
    -------
    reward : NDArray[np.float64], shape (n_bins,)
        Reward field with specified decay profile.

    Raises
    ------
    RegionNotFoundError
        If region_name is not found in env.regions (a ``KeyError``).
    ValueError
        If ``decay`` is unknown; if ``bandwidth`` is missing for
        ``decay='gaussian'``; if ``bandwidth`` or ``max_distance`` is given
        for a decay that does not use it; or if either is not finite and
        positive.

    Notes
    -----
    **Decay Types**:

    - **Constant**: Binary reward useful for sparse reward RL tasks.
      Creates clear goal regions with no reward shaping.

    - **Linear**: ``reward_value * max(0, 1 - d / max_distance)``, where
      ``d`` is the geodesic distance from the bin centre to the nearest region
      bin centre: the length of the shortest path between two positions that
      stays inside the environment (it goes around walls and never leaves the
      environment, and on a track it runs along the track's centerline). Bins
      that cannot reach the region (another connected component) get 0.

    - **Gaussian**: Smooth falloff using Gaussian kernel smoothing.
      After smoothing, rescales so maximum *within the region* equals
      reward_value. This preserves the intended reward magnitude while
      providing smooth gradients for policy learning.

    **Reward Shaping**: A reward field can serve as the potential
    ``Phi(s)`` of potential-based reward shaping [1]_, which adds
    ``gamma * Phi(s') - Phi(s)`` to the task reward and leaves the optimal
    policy unchanged. Adding the field itself to the task reward is not
    potential-based and can change the optimal policy. Gaussian decay
    provides the smoothest gradients; validate any shaped agent against a
    sparse-reward baseline.

    Examples
    --------
    >>> import numpy as np
    >>> from neurospatial import Environment
    >>> from neurospatial.behavior.reward import region_reward_field
    >>> from shapely.geometry import Point
    >>> # Create environment
    >>> positions = np.random.default_rng(0).normal(0, 50, (1000, 2))
    >>> env = Environment.from_samples(positions, bin_size=5.0)
    >>> # Add goal region (use polygon, not point)
    >>> goal_polygon = Point([0.0, 0.0]).buffer(10.0)
    >>> from neurospatial.regions import Regions
    >>> regions = Regions(env.regions.values())
    >>> _ = regions.add("goal", polygon=goal_polygon)
    >>> env = env.with_regions(regions)
    >>> # Binary reward (sparse RL)
    >>> reward_binary = region_reward_field(env, region_name="goal", decay="constant")
    >>> # Smooth Gaussian reward shaping
    >>> reward_smooth = region_reward_field(
    ...     env, region_name="goal", decay="gaussian", bandwidth=10.0
    ... )
    >>> # Linear decay that reaches 0 at 30 units from the region
    >>> reward_linear = region_reward_field(
    ...     env, region_name="goal", decay="linear", max_distance=30.0
    ... )
    >>> float(reward_linear.max()), float(reward_linear.min())
    (1.0, 0.0)

    See Also
    --------
    goal_reward_field : Distance-based rewards from goal bins
    neurospatial.ops.distance_field : Bin-graph distance maps
    neurospatial.ops.regions_to_mask : Convert regions to binary masks

    References
    ----------
    .. [1] Ng, A. Y., Harada, D., & Russell, S. (1999). Policy invariance
           under reward transformations: Theory and application to reward
           shaping. ICML.
    """
    if decay not in ("constant", "linear", "gaussian"):
        raise ValueError(
            f"Invalid decay type: {decay!r}. "
            "Must be one of: 'constant', 'linear', 'gaussian'."
        )
    _reject_unused_parameter("bandwidth", bandwidth, decay, uses="gaussian")
    _reject_unused_parameter("max_distance", max_distance, decay, uses="linear")
    validate_finite_scalar(
        bandwidth, name="bandwidth", positive=True, none_allowed=True
    )
    validate_finite_scalar(
        max_distance, name="max_distance", positive=True, none_allowed=True
    )
    region_mask = _require_region_mask(
        env, region_name, operation="region_reward_field()", argument="region_name"
    )

    if decay == "constant":
        # Binary reward: reward_value inside, 0 outside
        reward = np.where(region_mask, reward_value, 0.0)

    elif decay == "linear":
        # Geodesic distance to the nearest region bin: 0 inside the region,
        # inf for bins that cannot reach it. A shortest path into the region
        # enters through a boundary bin, so this is the distance from the
        # region boundary.
        from neurospatial.behavior._environment_paths import environment_paths

        distances = environment_paths(env, call="region_reward_field").from_bins(
            np.flatnonzero(region_mask)
        )
        reachable = np.isfinite(distances) & ~region_mask
        reward = np.where(region_mask, reward_value, 0.0)
        if reachable.any():
            decay_length = (
                distances[reachable].max() if max_distance is None else max_distance
            )
            reward[reachable] = reward_value * np.maximum(
                0.0, 1.0 - distances[reachable] / decay_length
            )

    elif decay == "gaussian":
        # Gaussian falloff using smoothing
        if bandwidth is None:
            raise ValueError(
                "bandwidth parameter is required for decay='gaussian'. "
                "Provide a smoothing bandwidth in the same units as your environment."
            )

        # Create indicator field (1 inside region, 0 outside)
        indicator = np.where(region_mask, 1.0, 0.0)

        # Smooth the indicator field. env.smooth is a linear operator, so under
        # truncation it can leave tolerance-level negatives at far bins; a reward
        # field is non-negative, so floor it (the dense clipped kernel did this
        # implicitly). Positivity is the consumer's job for the linear smoother.
        smoothed = np.maximum(
            np.asarray(env.smooth(indicator, bandwidth=bandwidth, mode="average")),
            0.0,
        )

        # Scale by the maximum inside the region (not the global maximum) so
        # the peak reward in the region equals reward_value. The region has at
        # least one bin and each bin keeps positive weight on itself, so this
        # maximum is positive.
        reward = (smoothed / smoothed[region_mask].max()) * reward_value

    return reward

goal_reward_field ¶

goal_reward_field(env: Environment[CartesianSystem], goal_bins: int | list[int] | NDArray[int_], *, decay: Literal['linear', 'exponential', 'inverse'] = 'exponential', scale: float = 1.0, max_distance: float | None = None) -> NDArray[np.float64]

Generate distance-based reward field from goal bins.

Creates a reward field that decays with distance from goal locations. Common for goal-directed navigation tasks and reward shaping in RL.