pyoe.metrics package

Submodules

pyoe.metrics.metrics module

class pyoe.metrics.metrics.DriftDelayMetric(dataloader: Dataloader, model=LinearRegression(), n_threshold: int = 30, warning_scale: int = 2, drift_scale: int = 3, **kwargs)

Bases: MetricTemplate

This metric measures the drift delay of the model. The drift delay is defined as the average number of samples between a detected drift point and the nearest true drift point to it.

The metric uses the DDM algorithm to detect drift points. The DDM algorithm is a statistical method that detects drift points by comparing the predicted value with the true value. When the difference between the predicted value and the true value exceeds a certain extent on continuous n_threshold points, the model is considered to have drifted.

measure(ground_truth: list[int] | None = None) → float | list[int]

This function calculates the drift delay of the model. If the ground_truth is not provided, it returns the index of the detected drift points. Otherwise, it returns the average drift delay of the model.

Parameters:

ground_truth (list[int], optional) – the ground truth drift points. If it is provided, the function will return the average drift delay of the model. Defaults to None.

Returns:

the average drift delay of the model or the index

of the detected drift.

Return type:

out (float | list[int])

class pyoe.metrics.metrics.EffectivenessMetric(dataloader: Dataloader, model: ModelTemplate, **kwargs)

Bases: MetricTemplate

measure(**kwargs) → float

This function calculates the effectiveness of the model. We do this by calculating the average loss of the model on the whole dataset.

Parameters:

**kwargs – any arguments that you want to pass.

Returns:

the average loss of the model on the whole dataset.

Return type:

out (float)

class pyoe.metrics.metrics.MetricTemplate(dataloader: Dataloader, model: ModelTemplate, **kwargs)

Bases: object

This is a template for metric function. We implemented some common metric functions and you can also implement your own metric function using this template.

abstract measure(**kwargs)

This function should be implemented in the subclass. By calling this function, you can get the evaluation result of the model.

Parameters:

**kwargs – any arguments that you want to pass.

Module contents