How to inspect the filtered points during pre-processing

[RunzeZhang123]

Currently there's no function that compares the pre-processed data with original data, and directly show the filtering effect. Add a function to visualize the removed points could be helpful.

[ma-sadeghi]

Can you explain more?

[RunzeZhang123]

For sure. The idea is pretty simple - to visualize what points got dropped during the pre-processing process. Might be beneficial for users to see a comparison between filtered data and original data directly, so that they can judge if the process makes sense. Currently, we have the residual plot but that might not be very informative for people to judge if the filtering makes sense for their data

[ma-sadeghi]

They can already do that; Do you think this is sufficient?

import numpy as np
import autoeis as ae
import matplotlib.pyplot as plt

freq, Z = ae.io.load_test_dataset()
freq2, Z2 = ae.utils.preprocess_impedance_data(freq, Z)

freq_removed = freq[~np.isin(freq, freq2)]
Z_removed = Z[~np.isin(freq, freq2)]
print(f"Removed points: {list(zip(freq_removed, Z_removed))}")

fig, ax = plt.subplots()
ae.visualization.plot_nyquist(Z, fmt="o", label="original", ax=ax)
ae.visualization.plot_nyquist(Z2, fmt="x", label="preprocessed", ax=ax)

Removed points: [(200019.48, (130.4171-34.680012j)), (149716.77, (134.15704-27.274202j)), (112070.29, (136.40355-21.577759j)), (83886.695, (137.78223-17.275707j)), (62792.953, (138.55463-14.341043j)), (47011.707, (139.11638-12.488245j)), (35185.535, (139.57651-11.539886j))]

PS. In this particular case, since the filtered points are from the high frequency region, they don't show on the Nyquist plot.