mantispy.pp.feature_select

mantispy.pp.feature_select(adata, operations=('drop_degenerate', 'blocklist', 'drop_na_columns', 'variance_threshold', 'correlation_threshold'), min_variance=1e-06, freq_cut=0.05, unique_cut=0.01, corr_threshold=0.9, corr_method='pearson', corr_window=None, corr_stride=None, na_cutoff=0.05, outlier_cutoff=500.0, blocklist='default', noise_removal_perturb_groups='Metadata_Perturbation', noise_removal_stdev_cutoff=0.8, corr_absolute=False, corr_iterative=False, decorrelate=False, decorr_threshold=0.99, decorr_method='pearson', key_added='selected', copy=False, *, noise_removal_cutoff_mode='absolute', subset=False)

Flag the features worth keeping.

Parameters:

adata AnnData

Object to select features on. Usually well-level profiles.

operations Sequence[str] (default: ('drop_degenerate', 'blocklist', 'drop_na_columns', 'variance_threshold', 'correlation_threshold'))

Which operations to run, from OPERATIONS, in the order they run. The default, DEFAULT_OPERATIONS, holds pycytominer’s usual operations plus drop_degenerate, which removes nothing from an object normalize() did not flag, ordered cheapest first so correlation_threshold runs last on what is left. Passing pycytominer’s order, ["variance_threshold", "correlation_threshold", "drop_na_columns", "blocklist"], gives pycytominer’s selection. Without drop_outliers, a warning names the kept features whose values sit more than outlier_cutoff robust standard deviations from their own median.

min_variance float (default: 1e-06)

variance_threshold: keep features with variance above this.

freq_cut float (default: 0.05)

frequency_threshold: drop a feature when the count of its second most common value divided by the count of its most common is below this. Either this rule or unique_cut drops a feature.

unique_cut float (default: 0.01)

frequency_threshold: drop a feature when its share of distinct values is below this.

corr_threshold float (default: 0.9)

correlation_threshold: drop one member of every pair correlated above this.

corr_method str (default: 'pearson')

correlation_threshold: "pearson" or "spearman".

corr_window int | None (default: None)

correlation_threshold: None runs the exact pass (the default, matching pycytominer). An int switches to the two-pass fast path (prune redundancy within name-sorted windows of that size, then run the exact pass on the survivors); 500 is a good default, several times faster on large screens. It keeps a different set of features (a different member of each correlated group) but preserves the information and the downstream signal.

corr_stride int | None (default: None)

correlation_threshold: step between windows, default half the window.

corr_absolute bool (default: False)

correlation_threshold: threshold |r| rather than the signed correlation, so a strongly anti-correlated pair is also reduced. Off by default, matching pycytominer.

corr_iterative bool (default: False)

correlation_threshold: drop the most-connected feature one at a time until no pair is left, keeping at least as many features as the single pass. Off by default. With corr_absolute this approximates cytominer’s R path (caret::findCorrelation).

decorrelate bool (default: False)

Run an extra, experimental redundancy step after the operations, on the features they keep. Unlike correlation_threshold it removes features that are a linear combination of several others, not just pairwise duplicates, by a rank-revealing QR. Off by default, and not part of pycytominer.

decorr_threshold float (default: 0.99)

decorrelate: drop a feature once its multiple correlation with the kept set reaches this. The default 0.99 removes only near-collinear features, so the kept set spans almost the same space; lower it towards corr_threshold for a smaller, more aggressive set.

decorr_method str (default: 'pearson')

decorrelate: "pearson" or "spearman".

na_cutoff float (default: 0.05)

drop_na_columns: drop features missing in more than this fraction of rows.

outlier_cutoff float (default: 500.0)

drop_outliers: drop features whose absolute value exceeds this.

blocklist str | Sequence[str] (default: 'default')

blocklist: "default" for the bundled list, or explicit names. Matched against the current names and against var["original_name"], so it works either side of standardize_feature_names(). A name without an object prefix, as cp_measure writes them, is matched on its parsed group, feature, channels and scale, so the default list also applies to cp_measure features.

noise_removal_perturb_groups str (default: 'Metadata_Perturbation')

noise_removal: obs column grouping replicates.

noise_removal_stdev_cutoff float (default: 0.8)

noise_removal: drop features whose within-group standard deviation, averaged over groups, is above this. An absolute threshold on the scale normalize() left the values on, so it is only meaningful next to the normalization that produced them; pycytominer’s default assumes whole-plate standardization.

key_added str (default: 'selected')

Name of the boolean var column to write.

copy bool (default: False)

Return a modified copy instead of mutating in place.

noise_removal_cutoff_mode str (default: 'absolute')

noise_removal: what noise_removal_stdev_cutoff is compared against. "absolute" (the default) keeps pycytominer’s behaviour, comparing the raw within-group spread and so tied to the normalization scale. "ratio" compares the within-group spread to the feature’s total spread, and "quantile" reads the cutoff as a quantile of the observed within-group spreads; both keep a useful subset under the control-based or MAD normalization the package recommends, where the absolute cutoff keeps everything or nothing.

subset bool (default: False)

Inplace subset to the selected features, as well as flagging them in var[key_added]. With copy=True the returned copy is the subset.

Return type:

AnnData | None

Returns:

None, or the modified copy. Writes var[key_added] and a per-operation count of removals to uns["mantispy"]["feature_select"], each count being what that operation removed from the features the earlier ones kept. Nothing is dropped unless subset=True; subset_features() does it afterwards.

Raises:

• ValueError – If operations names an operation that is not in OPERATIONS.

• KeyError – If noise_removal is requested but noise_removal_perturb_groups is not an obs column.

Notes

drop_degenerate runs first: a feature normalize() could not scale can hold values large enough to decide the correlation ranking of every feature it is compared with. The other operations then run in the order operations lists them, each judging only the features the earlier ones kept, as pycytominer does. The counts in uns["mantispy"]["feature_select"] therefore do not overlap, and they sum to the number of features removed. Order matters only for correlation_threshold, which ranks a feature against the other features still present; every other operation judges each feature on its own values. So the default order, which runs the correlation last, can keep a different member of a correlated pair than pycytominer’s order, which runs it before drop_na_columns and blocklist; it is also much faster on data with many mostly-missing features. decorrelate runs last, on the features the operations kept.

correlation_threshold is the most expensive operation. pycytominer uses pandas.DataFrame.corr, one Cython pass per column pair. Here the pairs come from chunked matrix products over blocks of columns, so no n_vars ** 2 array is held in memory.