shapiq_games.ImageClassifier

class shapiq_games.ImageClassifier(image, model='vit_9_patches', *, n_superpixels=14, regions=None, fill=None, mask_strategy=None, class_index=None, batch_size=16, device='cpu', revision=None, normalize=True, verbose=False)[source]

Bases: RegionGame

The image classification game: the probability of a class when only some regions are visible.

The players are regions of the image, given by regions for every model:

  • for the vision transformers [Dosovitskiy et al., 2021] ("vit_9_patches", "vit_16_patches", "vit_36_patches", "vit_144_patches"), square groups of the model’s 12 x 12 patches;

  • for DINOv2 [Oquab et al., 2024] ("dinov2_16_patches", "dinov2_20_patches", "dinov2_25_patches": a 4 x 4, 5 x 4, or 5 x 5 grid), rectangular groups of the model’s 16 x 16 patch tokens;

  • for ResNet-18 [He et al., 2016] ("resnet_18") or a custom classifier, SLIC superpixels [Achanta et al., 2012] or your own regions.

How absent players are removed:

  • DINOv2 drops their patch tokens from the sequence (the present tokens keep their position embeddings), and only so: its checkpoint has no trained mask token.

  • The vision transformers remove them in token space by default: their patch tokens are masked (mask_strategy="mask", the default: the content is replaced by a zero mask token, the position embedding is kept) or dropped ("remove"). With a fill they remove them in image space instead.

  • In image space (fill; always for ResNet-18 and custom classifiers): their pixels are replaced by the image’s mean color (the default), gray, black, a blurred copy, or any image of the same shape, and the model sees the filled image.

ResNet-18 and DINOv2 see a 224 x 224 center crop, so the game explains that crop: image is what the model sees and every region is visible to it. The explained class defaults to the class predicted on the full image.

To look at the game, masked_image() shows what the model sees for a coalition, attribution_map() spreads first-order values over the pixels for a heatmap, and player_images() returns one image per player, e.g. for shapiq.plot.si_graph_plot().

Variables:

  • image – The explained RGB image (for ResNet-18 and DINOv2, its 224 x 224 crop).

  • regions – The player of every pixel, of shape (height, width), numbered from 0.

  • mask_strategy – How a transformer removes players in token space ("mask" or "remove"; always "remove" for DINOv2), or None in image space.

  • class_index – The explained class.

  • class_name – The name of the explained class, if the model provides class names.

  • empty_value – The probability of the class with every region removed, before centering.

  • original_model_output – The probability of the class on the full image.

Examples

>>> image = np.random.default_rng(0).integers(0, 255, (64, 64, 3), dtype=np.uint8)
>>> def classifier(images):  # (batch, height, width, 3) -> (batch, n_classes)
...     brightness = images.mean(axis=(1, 2, 3)) / 255
...     return np.stack([brightness, 1 - brightness], axis=1)
>>> game = ImageClassifier(image, model=classifier, n_superpixels=8)
>>> game.n_players
8
>>> from shapiq_games.vision import grid_regions
>>> grid = ImageClassifier(image, classifier, regions=grid_regions(64, 64, 2, 2), fill="blur")
>>> grid.n_players, grid.masked_image([1, 0, 0, 1]).shape
(4, (64, 64, 3))

The vision transformers mask or drop patch tokens, or remove pixels with a fill; DINOv2 drops tokens:

>>> game = ImageClassifier(image, "vit_16_patches")
>>> game = ImageClassifier(image, "vit_16_patches", mask_strategy="remove")
>>> game = ImageClassifier(image, "vit_16_patches", fill="blur")
>>> game = ImageClassifier(image, "dinov2_20_patches")

Initialize the image classification game.

Parameters:

  • image (ndarray | str | Path) – An RGB image array of shape (height, width, 3) or the path of an image. Float images with values in [0, 1] (e.g. from matplotlib.pyplot.imread) are scaled to 0, ..., 255.

  • model (ImageModel | Callable[[ndarray], ndarray]) – A builtin model name or a classifier mapping a batch of images of shape (batch, height, width, 3) to class probabilities of shape (batch, n_classes). Defaults to "vit_9_patches".

  • n_superpixels (int) – The number of SLIC superpixels for "resnet_18" and custom classifiers when no regions are given. Defaults to 14.

  • regions (ndarray | None) – Your own players for "resnet_18" and custom classifiers: the player of every pixel, of shape (height, width), numbered 0, ..., n - 1 (see grid_regions()). For ResNet-18 they refer to its 224 x 224 crop.

  • fill (Fill | ndarray | None) – Remove players in image space, replacing their pixels with "mean" (the image’s mean color), "gray", "black", "blur", or an image of the same shape. ResNet-18 and custom classifiers always do ("mean" by default); the vision transformers only when a fill is given; DINOv2 never.

  • mask_strategy (MaskStrategy | None) – How a vision transformer removes players in token space: "mask" (None, the default) or "remove" (see above). DINOv2 only drops tokens (None or "remove").

  • class_index (int | None) – The explained class, or None for the class predicted on the image.

  • batch_size (int) – The number of masked images per forward pass. The builtin models pad smaller batches to this size, so that a value does not depend on the batch. Defaults to 16.

  • device (str) – The torch device of the builtin models. Defaults to "cpu".

  • revision (str | None) – The Hugging Face revision (branch, tag, or commit) of the vision transformer or DINOv2. None loads the default branch. ResNet-18 uses pinned torchvision weights and takes no revision.

  • normalize (bool) – Whether to center the game such that the value of the empty coalition is zero. Defaults to True.

  • verbose (bool) – Whether to show a progress bar when evaluating the game.

Raises:

ValueError – If the model is unknown; a revision is given for a model that is not a Hugging Face model (vision transformer or DINOv2); regions are given for a transformer (its players are its patch grid); mask_strategy is given for a model without tokens, or together with a fill; DINOv2 is asked to mask or to fill; the regions, the fill or the mask strategy are invalid; or class_index is out of range.

attribution_map(values)

Spread one value per player over the player’s pixels, e.g. for a heatmap.

Parameters:

values (InteractionValues | ndarray) – Interaction values (their first-order values are used) or one value per player.

Return type:

ndarray

Returns:

The value of every pixel, of shape (height, width).

masked_image(coalition)

Return the image with the players outside coalition removed.

In image space this is the image the model sees (up to rounding, for models that fill normalized pixels). In token space the model removes the players inside the network, so their regions are shown in gray.

Parameters:

coalition (ArrayLike) – The players, as a boolean or 0/1 vector of length n_players.

Return type:

ndarray

Returns:

The image as a uint8 array of the image’s shape.

player_images(*, fade=0.75)

Return one image per player: its bounding box, with other pixels faded to white.

The images can be passed to shapiq.plot.si_graph_plot() as feature_image_patches.

Parameters:

fade (float) – How strongly pixels outside the player’s region are faded, from 0 (not at all) to 1 (white). Defaults to 0.75.

Return type:

list[Image]

Returns:

The images, in player order.

value_function(coalitions)

Return the model output with only each coalition’s regions present.

Return type:

GenericAlias[floating]

game_id

A unique identifier for the game, based on its class name and hash.

game_values

The mapping for coalitions to game values without normalization applied.

image
n_players

The number of players in the game.

normalization_value

The value which is used to normalize (center) the game values such that the value for the empty coalition is zero. If this is zero, the game values are not normalized.

player_name_lookup
regions