benchmark.assess

pywicta.benchmark.assess.normalize_array(input_array)[source]

Normalize the given image such that its pixels value fit between 0.0 and 1.0.

It applies

\[\text{normalize}(\boldsymbol{S}) = \frac{ \boldsymbol{S} - \text{min}(\boldsymbol{S}) }{ \text{max}(\boldsymbol{S}) - \text{min}(\boldsymbol{S}) }\]

where \(\boldsymbol{S}\) is the input array (an image).

Parameters:image (Numpy array) – The image to normalize (whatever its shape)
Returns:The normalized version of the input image (keeping the same dimension and shape)
Return type:Numpy array
pywicta.benchmark.assess.metric_mse(input_img, output_image, reference_image, **kwargs)[source]

Compute the score of output_image regarding reference_image with the Mean-Squared Error (MSE) metric.

It applies

\[\text{MSE}(\hat{\boldsymbol{S}}, \boldsymbol{S}^*) = \left\langle \left( \hat{\boldsymbol{S}} - \boldsymbol{S}^* \right)^{\circ 2} \right\rangle\]

with:

  • \(\hat{\boldsymbol{S}}\) the algorithm’s output image (i.e. the cleaned image);
  • \(\boldsymbol{S}^*\) the reference image (i.e. the clean image);
  • \(\langle \boldsymbol{S} \rangle\) the average of matrix \(\boldsymbol{S}\);
  • \(\boldsymbol{S}^{\circ 2}\) the Hadamar power (i.e. the element wise square) of matrix \(\boldsymbol{S}\).

See http://scikit-image.org/docs/dev/api/skimage.measure.html#compare-mse for more information.

Note

This function is not well-suited to high dynamic range images handled with this project (errors are correlated with energy levels).

Parameters:
  • input_img (2D ndarray) – The RAW original image.
  • output_image (2D ndarray) – The cleaned image returned by the image cleanning algorithm to assess.
  • reference_image (2D ndarray) – The actual clean image (the best result that can be expected for the image cleaning algorithm).
  • kwargs (dict) – Additional options.
Returns:

The score of the image cleaning algorithm for the given image.
Return type:

float
pywicta.benchmark.assess.metric_nrmse(input_img, output_image, reference_image, **kwargs)[source]

Compute the score of output_image regarding reference_image with the Normalized Root Mean-Squared Error (NRMSE) metric.

It applies

\[\text{NRMSE}(\hat{\boldsymbol{S}}, \boldsymbol{S}^*) = \frac{\sqrt{\text{MSE}}}{\sqrt{ \left\langle \hat{\boldsymbol{S}} \circ \boldsymbol{S}^* \right\rangle }}\]

with:

  • \(\hat{\boldsymbol{S}}\) the algorithm’s output image (i.e. the cleaned image);
  • \(\boldsymbol{S}^*\) the reference image (i.e. the clean image);
  • \(\langle \boldsymbol{S} \rangle\) the average of matrix \(\boldsymbol{S}\);
  • \(\circ\) the Hadamar product (i.e. the element wise product operator).

See http://scikit-image.org/docs/dev/api/skimage.measure.html#compare-nrmse and https://en.wikipedia.org/wiki/Root-mean-square_deviation for more information.

Parameters:
  • input_img (2D ndarray) – The RAW original image.
  • output_image (2D ndarray) – The cleaned image returned by the image cleanning algorithm to assess.
  • reference_image (2D ndarray) – The actual clean image (the best result that can be expected for the image cleaning algorithm).
  • kwargs (dict) – Additional options.
Returns:

The score of the image cleaning algorithm for the given image.
Return type:

float
pywicta.benchmark.assess.metric1(input_img, output_image, reference_image, **kwargs)[source]

Compute the score of output_image regarding reference_image with a (unusually) normalized version of the Root Mean-Squared Error (RMSE) metric.

It applies

\[\text{uNRMSE}(\hat{\boldsymbol{S}}, \boldsymbol{S}^*) = \left\langle \left( \left( \hat{\boldsymbol{S}}_n - \boldsymbol{S}^*_n \right)^{\circ 2} \right)^{\circ \frac{1}{2}} \right\rangle\]

with:

  • \(\hat{\boldsymbol{S}}_n\) the algorithm’s normalized output image (i.e. the cleaned image), (using normalize_array());
  • \(\boldsymbol{S}^*_n\) the normalized reference image (i.e. the clean image) (using normalize_array());
  • \(\langle \boldsymbol{S} \rangle\) the average of matrix \(\boldsymbol{S}\);
  • \(\boldsymbol{S}^{\circ 2}\) the Hadamar power (i.e. the element wise square) of matrix \(\boldsymbol{S}\).

Note

This function is not robust to noise on extreme values.

Parameters:
  • input_img (2D ndarray) – The RAW original image.
  • output_image (2D ndarray) – The cleaned image returned by the image cleanning algorithm to assess.
  • reference_image (2D ndarray) – The actual clean image (the best result that can be expected for the image cleaning algorithm).
  • kwargs (dict) – Additional options.
Returns:

The score of the image cleaning algorithm for the given image.
Return type:

float
pywicta.benchmark.assess.metric2(input_img, output_image, reference_image, **kwargs)[source]

Compute the score of output_image regarding reference_image with the \(\mathcal{E}_{\text{shape}}\) metric.

It applies

\[f(\hat{\boldsymbol{S}}, \boldsymbol{S}^*) = \left\langle \text{abs} \left( \frac{\hat{\boldsymbol{S}}}{\sum_i \hat{\boldsymbol{S}}_i} - \frac{\boldsymbol{S}^*}{\sum_i \boldsymbol{S}^*_i} \right) \right\rangle\]

with:

  • \(\hat{\boldsymbol{S}}\) the algorithm’s output image (i.e. the cleaned image);
  • \(\boldsymbol{S}^*\) the reference image (i.e. the clean image);
  • \(\langle \boldsymbol{S} \rangle\) the average of matrix \(\boldsymbol{S}\).
Parameters:
  • input_img (2D ndarray) – The RAW original image.
  • output_image (2D ndarray) – The cleaned image returned by the image cleanning algorithm to assess.
  • reference_image (2D ndarray) – The actual clean image (the best result that can be expected for the image cleaning algorithm).
  • kwargs (dict) – Additional options.
Returns:

The score of the image cleaning algorithm for the given image.
Return type:

float
pywicta.benchmark.assess.metric3(input_img, output_image, reference_image, **kwargs)[source]

Compute the score of output_image regarding reference_image with the \(\mathcal{E}^+_{\text{energy}}\) (a.k.a. relative total counts difference) metric.

It applies

\[f(\hat{\boldsymbol{S}}, \boldsymbol{S}^*) = \frac{ \text{abs} \left( \sum_i \hat{\boldsymbol{S}}_i - \sum_i \boldsymbol{S}^*_i \right) }{ \sum_i \boldsymbol{S}^*_i }\]

with \(\hat{\boldsymbol{S}}\) the algorithm’s output image (i.e. the cleaned image) and \(\boldsymbol{S}^*\) the reference image (i.e. the clean image).

Parameters:
  • input_img (2D ndarray) – The RAW original image.
  • output_image (2D ndarray) – The cleaned image returned by the image cleanning algorithm to assess.
  • reference_image (2D ndarray) – The actual clean image (the best result that can be expected for the image cleaning algorithm).
  • kwargs (dict) – Additional options.
Returns:

The score of the image cleaning algorithm for the given image.
Return type:

float
pywicta.benchmark.assess.metric4(input_img, output_image, reference_image, **kwargs)[source]

Compute the score of output_image regarding reference_image with the \(\mathcal{E}_{\text{energy}}\) (a.k.a. signed relative total counts difference) metric.

It applies

\[f(\hat{\boldsymbol{S}}, \boldsymbol{S}^*) = \frac{ \sum_i \hat{\boldsymbol{S}}_i - \sum_i \boldsymbol{S}^*_i }{ \sum_i \boldsymbol{S}^*_i }\]

with \(\hat{\boldsymbol{S}}\) the algorithm’s output image (i.e. the cleaned image) and \(\boldsymbol{S}^*\) the reference image (i.e. the clean image).

Parameters:
  • input_img (2D ndarray) – The RAW original image.
  • output_image (2D ndarray) – The cleaned image returned by the image cleanning algorithm to assess.
  • reference_image (2D ndarray) – The actual clean image (the best result that can be expected for the image cleaning algorithm).
  • kwargs (dict) – Additional options.
Returns:

The score of the image cleaning algorithm for the given image.
Return type:

float
pywicta.benchmark.assess.metric_ssim(input_img, output_image, reference_image, **kwargs)[source]

Compute the score of output_image regarding reference_image with the Structural Similarity Index Measure (SSIM) metric.

See [1], [2], [3] and [4] for more information.

The SSIM index is calculated on various windows of an image. The measure between two windows \(x\) and \(y\) of common size \(N.N\) is:

\[\hbox{SSIM}(x,y) = \frac{(2\mu_x\mu_y + c_1)(2\sigma_{xy} + c_2)}{(\mu_x^2 + \mu_y^2 + c_1)(\sigma_x^2 + \sigma_y^2 + c_2)}\]

with:

  • \(\scriptstyle\mu_x\) the average of \(\scriptstyle x\);
  • \(\scriptstyle\mu_y\) the average of \(\scriptstyle y\);
  • \(\scriptstyle\sigma_x^2\) the variance of \(\scriptstyle x\);
  • \(\scriptstyle\sigma_y^2\) the variance of \(\scriptstyle y\);
  • \(\scriptstyle \sigma_{xy}\) the covariance of \(\scriptstyle x\) and \(\scriptstyle y\);
  • \(\scriptstyle c_1 = (k_1L)^2\), \(\scriptstyle c_2 = (k_2L)^2\) two variables to stabilize the division with weak denominator;
  • \(\scriptstyle L\) the dynamic range of the pixel-values (typically this is \(\scriptstyle 2^{\#bits\ per\ pixel}-1\));
  • \(\scriptstyle k_1 = 0.01\) and \(\scriptstyle k_2 = 0.03\) by default.

The SSIM index satisfies the condition of symmetry:

\[\text{SSIM}(x, y) = \text{SSIM}(y, x)\]
Parameters:
  • input_img (2D ndarray) – The RAW original image.
  • output_image (2D ndarray) – The cleaned image returned by the image cleanning algorithm to assess.
  • reference_image (2D ndarray) – The actual clean image (the best result that can be expected for the image cleaning algorithm).
  • kwargs (dict) – Additional options.
Returns:

The score of the image cleaning algorithm for the given image.
Return type:

float

References

[1]Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004). Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 13, 600-612. https://ece.uwaterloo.ca/~z70wang/publications/ssim.pdf, DOI:10.1.1.11.2477
[2]Avanaki, A. N. (2009). Exact global histogram specification optimized for structural similarity. Optical Review, 16, 613-621. http://arxiv.org/abs/0901.0065, DOI:10.1007/s10043-009-0119-z
[3]http://scikit-image.org/docs/dev/api/skimage.measure.html#compare-ssim
[4]https://en.wikipedia.org/wiki/Structural_similarity
pywicta.benchmark.assess.metric_psnr(input_img, output_image, reference_image, **kwargs)[source]

Compute the score of output_image regarding reference_image with the Peak Signal-to-Noise Ratio (PSNR) metric.

See [5] and [6] for more information.

Parameters:
  • input_img (2D ndarray) – The RAW original image.
  • output_image (2D ndarray) – The cleaned image returned by the image cleanning algorithm to assess.
  • reference_image (2D ndarray) – The actual clean image (the best result that can be expected for the image cleaning algorithm).
  • kwargs (dict) – Additional options.
Returns:

The score of the image cleaning algorithm for the given image.
Return type:

float

References

[5]http://scikit-image.org/docs/dev/api/skimage.measure.html#skimage.measure.compare_psnr
[6]https://en.wikipedia.org/wiki/Peak_signal-to-noise_ratio
pywicta.benchmark.assess.metric_delta_psi(input_img, output_image, reference_image, geom, **kwargs)[source]

Compute the score of output_image regarding reference_image with the following relative psi parameters (relative difference of shower angle between the cleaned image and the reference image).

Parameters:
  • input_img (2D ndarray) – The RAW original image.
  • output_image (2D ndarray) – The cleaned image returned by the image cleanning algorithm to assess.
  • reference_image (2D ndarray) – The actual clean image (the best result that can be expected for the image cleaning algorithm).
  • kwargs (dict) – Additional options.
Returns:

The score of the image cleaning algorithm for the given image.
Return type:

float
pywicta.benchmark.assess.metric_hillas_delta(input_img, output_image, reference_image, geom, **kwargs)[source]

Compute the score of output_image regarding reference_image with the following relative Hillas parameters:

  • \(\Delta_{\text{size}} = \text{reference_image}_{\text{size}} - \text{output_image}_{\text{size_out}}\)
  • \(\Delta_{\text{cen_x}} = \text{reference_image}_{\text{cen_x}} - \text{output_image}_{\text{cen_x_out}}\)
  • \(\Delta_{\text{cen_y}} = \text{reference_image}_{\text{cen_y}} - \text{output_image}_{\text{cen_y_out}}\)
  • \(\Delta_{\text{length}} = \text{reference_image}_{\text{length}} - \text{output_image}_{\text{length_out}}\)
  • \(\Delta_{\text{width}} = \text{reference_image}_{\text{width}} - \text{output_image}_{\text{width_out}}\)
  • \(\Delta_{\text{r}} = \text{reference_image}_{\text{r}} - \text{output_image}_{\text{r_out}}\)
  • \(\Delta_{\text{phi}} = \text{reference_image}_{\text{phi}} - \text{output_image}_{\text{phi_out}}\)
  • \(\Delta_{\text{psi}} = \text{reference_image}_{\text{psi}} - \text{output_image}_{\text{psi_out}}\)
  • \(\Delta_{\text{miss}} = \text{reference_image}_{\text{miss}} - \text{output_image}_{\text{miss_out}}\)

See http://adsabs.harvard.edu/abs/1989ApJ…342..379W for more details about Hillas parameters.

Parameters:
  • input_img (2D ndarray) – The RAW original image.
  • output_image (2D ndarray) – The cleaned image returned by the image cleanning algorithm to assess.
  • reference_image (2D ndarray) – The actual clean image (the best result that can be expected for the image cleaning algorithm).
  • kwargs (dict) – Additional options.
Returns:

The score of the image cleaning algorithm for the given image.
Return type:

namedtuple
pywicta.benchmark.assess.metric_hillas_delta2(input_img, output_image, reference_image, geom, **kwargs)[source]

Compute the score of output_image regarding reference_image with the Hillas parameters.

It works exactly like metric_hillas_delta() except that isolated pixels are removed from the reference_image before the evaluation (using pywi.processing.filtering.pixel_clusters.filter_pixels_clusters()).

Parameters:
  • input_img (2D ndarray) – The RAW original image.
  • output_image (2D ndarray) – The cleaned image returned by the image cleanning algorithm to assess.
  • reference_image (2D ndarray) – The actual clean image (the best result that can be expected for the image cleaning algorithm).
  • kwargs (dict) – Additional options.
Returns:

The score of the image cleaning algorithm for the given image.
Return type:

namedtuple
pywicta.benchmark.assess.assess_image_cleaning(input_img, output_img, reference_img, benchmark_method, **kwargs)[source]

Compute the score of output_image regarding reference_image with the benchmark_method metrics:

Parameters:
  • input_img (2D ndarray) – The RAW original image.
  • output_img (2D ndarray) – The cleaned image returned by the image cleanning algorithm to assess.
  • reference_img (2D ndarray) – The actual clean image (the best result that can be expected for the image cleaning algorithm).
  • kwargs (dict) – Additional options.
Returns:

The score(s) of the image cleaning algorithm for the given image.
Return type:

tuple of float numbers