MicroLIA.optimization

Created on Sat Feb 25 10:39:23 2023

@author: daniel

Module Contents

Classes

objective_xgb	Optimization objective function for the tree-based XGBoost classifier.
objective_nn	Optimization objective function for the scikit-learn implementatin of the
objective_rf	Optimization objective function for the scikit-learn implementatin of the
ObjectiveOneClassSVM	Optimization objective function for the scikit-learn implementation of the

Functions

hyper_opt([data_x, data_y, clf, n_iter, opt_cv, ...])	Optimizes hyperparameters using a k-fold cross validation splitting strategy.
borutashap_opt(data_x, data_y[, boruta_trials, model, ...])	Applies a combination of the Boruta algorithm and
boruta_opt(data_x, data_y)	Applies the Boruta algorithm (Kursa & Rudnicki 2011) to identify features
impute_missing_values(data[, imputer, strategy, k, ...])	Impute missing values in the input data array using various imputation strategies.
Strawman_imputation(data)	Perform Strawman imputation, a time-efficient algorithm

class MicroLIA.optimization.objective_xgb(data_x, data_y, limit_search=False, opt_cv=3, eval_metric='f1', min_gamma=0)

Bases: object

Optimization objective function for the tree-based XGBoost classifier. The Optuna software for hyperparameter optimization was published in 2019 by Akiba et al. Paper: https://arxiv.org/abs/1907.10902

Note

If opt_cv is between 0 and 1, a pruning procedure will be initiliazed (a procedure incompatible with cross-validation), so as to speed up the XGB optimization. A random validation data will be generated according to this ratio, which will replace the cross-validation method used by default. It will prune according to the f1-score of the validation data, which would be 10% of the training data if opt_cv=0.1, for example. Need more testing to make this more cross-validation routine more robust. Recommend to set opt_cv > 1.

Parameters:

• data_x (ndarray) – 2D array of size (n x m), where n is the number of samples, and m the number of features.

• data_y (ndarray, str) – 1D array containing the corresponing labels.

• limit_search (bool) – If True, the search space for the parameters will be expanded, as there are some hyperparameters that can range from 0 to inf. Defaults to False.

• opt_cv (int) – Cross-validations to perform when assesing the performance at each hyperparameter optimization trial. For example, if cv=3, then each optimization trial will be assessed according to the 3-fold cross validation accuracy. If this is between 0 and 1, this value would be used as the ratio of the validation to training data. Defaults to 3. Can be set to None in which case 5-fold cross-validation is employed. Cannot be 1.

• eval_metric (str) – The evaluation metric when evaluating the validation data, used when opt_cv is less than 1. Defaults to “f1”. For all options see eval_metric from: https://xgboost.readthedocs.io/en/latest/parameter.html#metrics

• min_gamma (float) – Controls the optimization of the gamma Tree Booster hyperparameter. The gamma parameter is the lowest loss reduction needed on a tree leaf node in order to partition again. The algorithm’s level of conservatism increases with gamma, therefore it acts as a regularizer. By default, during the optimization routine will consider a miminum value of 0 when tuning the gamma parameter, unless this min_gamma input is set. This parameter determines the lowest gamma value the optimizer should consider. Must be less than 5. Defaults to 0. Consider increasing this to ~1 if the optimized models are overfitting.

Returns:

The cross-validation accuracy (if opt_cv is greater than 1, 1 would be single instance accuracy) or, if opt_cv is between 0 and 1, the validation accuracy according to the corresponding test size.

__call__(trial)

class MicroLIA.optimization.objective_nn(data_x, data_y, opt_cv)

Bases: object

Optimization objective function for the scikit-learn implementatin of the MLP classifier. The Optuna software for hyperparameter optimization was published in 2019 by Akiba et al. Paper: https://arxiv.org/abs/1907.10902.

The total number of hidden layers to test is limited to 10, with 10-100 possible number of neurons in each.

Parameters:

• data_x (ndarray) – 2D array of size (n x m), where n is the number of samples, and m the number of features.

• data_y (ndarray, str) – 1D array containing the corresponing labels.

• opt_cv (int) – Cross-validations to perform when assesing the performance at each hyperparameter optimization trial. For example, if cv=3, then each optimization trial will be assessed according to the 3-fold cross validation accuracy.

Returns:

The performance metric, determined using the cross-fold validation method.

__call__(trial)

class MicroLIA.optimization.objective_rf(data_x, data_y, opt_cv)

Bases: object

Optimization objective function for the scikit-learn implementatin of the Random Forest classifier. The Optuna software for hyperparameter optimization was published in 2019 by Akiba et al. Paper: https://arxiv.org/abs/1907.10902

Parameters:

• data_x (ndarray) – 2D array of size (n x m), where n is the number of samples, and m the number of features.

• data_y (ndarray, str) – 1D array containing the corresponing labels.

• opt_cv (int) – Cross-validations to perform when assesing the performance at each hyperparameter optimization trial. For example, if cv=3, then each optimization trial will be assessed according to the 3-fold cross validation accuracy.

Returns:

The performance metric, determined using the cross-fold validation method.

__call__(trial)

class MicroLIA.optimization.ObjectiveOneClassSVM(data_x, data_y, opt_cv)

Bases: object

Optimization objective function for the scikit-learn implementation of the One-Class SVM. The Optuna software for hyperparameter optimization was published in 2019 by Akiba et al. Paper: https://arxiv.org/abs/1907.10902

Parameters:

• data_x (ndarray) – 2D array of size (n x m), where n is the number of samples, and m is the number of features.

• data_y (ndarray, str) – 1D array containing the corresponding labels.

• opt_cv (int) – Cross-validations to perform when assessing the performance at each hyperparameter optimization trial. For example, if cv=3, then each optimization trial will be assessed according to the 3-fold cross-validation accuracy.

Returns:

The performance metric, determined using the cross-fold validation method.

__call__(trial)

Define the optimization objective function for the One-Class SVM.

Parameters:

trial (optuna.trial.Trial) – A Trial object containing the current state of the optimization.

Returns:

The performance metric, determined using cross-validation.

Return type:

float

MicroLIA.optimization.hyper_opt(data_x=None, data_y=None, clf='rf', n_iter=25, opt_cv=None, balance=True, limit_search=True, min_gamma=0, return_study=True)

Optimizes hyperparameters using a k-fold cross validation splitting strategy. If save_study=True, the Optuna study object will be the third output. This object can be used for various analysis, including optimization visualizations. See: https://optuna.readthedocs.io/en/stable/reference/generated/optuna.study.Study.html

Example

The function will create the classification engine and optimize the hyperparameters using an iterative approach:

>>> model, params = hyper_opt(data_x, data_y, clf='rf')

The first output is our optimal classifier, and will be used to make predictions:

>>> prediction = model.predict(new_data)

The second output of the optimize function is the dictionary containing the hyperparameter combination that yielded the highest mean accuracy.

If save_study = True, the Optuna study object will also be returned as the third output. This can be used to plot the optimization results, see: https://optuna.readthedocs.io/en/latest/tutorial/10_key_features/005_visualization.html#sphx-glr-tutorial-10-key-features-005-visualization-py

>>> from optuna.visualization.matplotlib import plot_contour
>>>
>>> model, params, study = hyper_opt(data_x, data_y, clf='rf', save_study=True)
>>> plot_contour(study)

Parameters:

• data_x (ndarray) – 2D array of size (n x m), where n is the number of samples, and m the number of features.

• data_y (ndarray, str) – 1D array containing the corresponing labels.

• clf (str) – The machine learning classifier to optimize. Can either be ‘rf’ for Random Forest, ‘nn’ for Neural Network, or ‘xgb’ for eXtreme Gradient Boosting.

• n_iter (int, optional) – The maximum number of iterations to perform during the hyperparameter search. Defaults to 25.

• opt_cv (int) – Cross-validations to perform when assesing the performance at each hyperparameter optimization trial. For example, if cv=3, then each optimization trial will be assessed according to the 3-fold cross validation accuracy. If clf=’xgb’ and this value is set between 0 and 1, this sets the size of the validation data, which will be chosen randomly each trial. This is used to enable an early stopping callback which is not possible with the cross-validation method. Defaults to 10.

• balance (bool, optional) – If True, a weights array will be calculated and used when fitting the classifier. This can improve classification when classes are imbalanced. This is only applied if the classification is a binary task. Defaults to True.

• limit_search (bool) – If True the optimization search spaces will be limited, for quicker computation. Defaults to True.

• min_gamma (float) – Controls the optimization of the gamma Tree Booster hyperparameter, applicable if clf=’xgb’. The gamma parameter is the lowest loss reduction needed on a tree leaf node in order to partition again. The algorithm’s level of conservatism increases with gamma, therefore it acts as a regularizer. By default, during the optimization routine will consider a miminum value of 0 when tuning the gamma parameter, unless this min_gamma input is set. This parameter determines the lowest gamma value the optimizer should consider. Must be less than 5. Defaults to 0. Consider increasing this to ~1 if the optimized models are overfitting.

• return_study (bool, optional) – If True the Optuna study object will be returned. This can be used to review the method attributes, such as optimization plots. Defaults to True.

Returns:

The first output is the classifier with the optimal hyperparameters. Second output is a dictionary containing the optimal hyperparameters. If save_study=True, the Optuna study object will be the third output.

MicroLIA.optimization.borutashap_opt(data_x, data_y, boruta_trials=50, model='rf', importance_type='gain')

Applies a combination of the Boruta algorithm and Shapley values, a method developed by Eoghan Keany (2020).

See: https://doi.org/10.5281/zenodo.4247618

Parameters:

• data_x (ndarray) – 2D array of size (n x m), where n is the number of samples, and m the number of features.

• data_y (ndarray, str) – 1D array containing the corresponing labels.

• boruta_trials (int) – The number of trials to run. A larger number is better as the distribution will be more robust to random fluctuations. Defaults to 50.

• model (str) – The ensemble method to use when fitting and calculating the feature importance metric. Only two options are currently supported, ‘rf’ for Random Forest and ‘xgb’ for Extreme Gradient Boosting. Defaults to ‘rf’.

• importance_type (str) – The feature importance type to use, only applicable when using clf=’xgb’. The options include “gain”, “weight”, “cover”, “total_gain” or “total_cover”. Defaults to ‘gain’.

Returns:

First output is a 1D array containing the indices of the selected features. These indices can then be used to select the columns in the data_x array. Second output is the feature selection object, which contains feature selection history information and visualization options.

MicroLIA.optimization.boruta_opt(data_x, data_y)

Applies the Boruta algorithm (Kursa & Rudnicki 2011) to identify features that perform worse than random.

See: https://arxiv.org/pdf/1106.5112.pdf

Parameters:

• data_x (ndarray) – 2D array of size (n x m), where n is the number of samples, and m the number of features.

• data_y (ndarray, str) – 1D array containing the corresponing labels.

Returns:

1D array containing the indices of the selected features. This can then be used to index the columns in the data_x array.

MicroLIA.optimization.impute_missing_values(data, imputer=None, strategy='knn', k=3, constant_value=0, nan_threshold=0.5)

Impute missing values in the input data array using various imputation strategies. By default the imputer will be created and returned, unless the imputer argument is set, in which case only the transformed data is output.

The function first identifies the columns with mostly missing values based on the nan_threshold parameter. It replaces the values in those columns with zeros before performing the imputation step using the selected imputation strategy. This way, the ignored columns will have zeros and won’t be taken into account during imputation. The resulting imputed_data will have all columns preserved, with missing values filled according to the chosen imputation strategy. This is required because the imputation techniques employed remove columns that have too many nans!

Note

As the KNN imputation method bundles neighbors according to their eucledian distance, it is sensitive to outliers. Furthermore, it can also yield weak predictions if the training features are heaviliy correlated. Tang & Ishwaran 2017 reported that if there is low to medium correlation in the dataset, Random Forest imputation algorithms perform better than KNN imputation

Parameters:

• data (ndarray) – Input data array with missing values.

• imputer (optional) – A KNNImputer class instance, configured using sklearn.impute.KNNImputer. Defaults to None, in which case the transformation is created using the data itself.

• strategy (str, optional) – Imputation strategy to use. Defaults to ‘knn’. - ‘mean’: Fill missing values with the mean of the non-missing values in the same column. - ‘median’: Fill missing values with the median of the non-missing values in the same column. - ‘mode’: Fill missing values with the mode (most frequent value) of the non-missing values in the same column. - ‘constant’: Fill missing values with a constant value provided by the user. - ‘knn’: Fill missing values using k-Nearest Neighbor imputation.

• k (int, optional) – Number of nearest neighbors to consider for k-Nearest Neighbor imputation. Only applicable if the imputation strategy is set to ‘knn’. Defaults to 3.

• constant_value (float or int, optional) – Constant value to use for constant imputation. Only applicable if the imputation strategy is set to ‘constant’. Defaults to 0.

• nan_threshold (float) – Columns with nan values greater than this ratio will be zeroed out before the imputation. Defualts to 0.5.

Returns:

The first output is the data array with with the missing values filled in. The second output is the KNN Imputer that should be used to transform new data, prior to predictions.

MicroLIA.optimization.Strawman_imputation(data)

Perform Strawman imputation, a time-efficient algorithm in which missing data values are replaced with the median value of the entire, non-NaN sample. If the data is a hot-encoded boolean (as the RF does not allow True or False), then the instance that is used the most will be computed as the median.

This is the baseline algorithm used by (Tang & Ishwaran 2017). See: https://arxiv.org/pdf/1701.05305.pdf

Note

This function assumes each row corresponds to one sample, and that missing values are masked as either NaN or inf.

Parameters:

data (ndarray) – 1D array if single parameter is input. If data is 2-dimensional, the medians will be calculated using the non-missing values in each corresponding column.

Returns:

The data array with the missing values filled in.