scikit-learn : Data Preprocessing I - Missing / categorical data

In real-world samples, it is not uncommon that there are missing one or more values such as the blank spaces in our data table.

Quite a few computational tools, however, are unable to handle such missing values and might produce unpredictable results. So, before we proceed with further analyses, it is critical that we take care of those missing values.

To get a better feel for the problem, let's create a simple example using CSV file:

The StringIO() function allows us to read the string assigned to csv_data into a pandas DataFrame via the read_csv() function as if it was a regular CSV file on our hard drive.

Note that the two missing cells were replaced by NaN.

We can use isnull() method to check whether a cell contains a numeric value ( False ) or if data is missing ( True ):

For a larger DataFrame, we may want to use the sum() method which returns the number of missing values per column:

Note that we can always access the underlying NumPy array of the DataFrame via the values attribute before we feed it into a scikit-learn estimator:

So, while scikit-learn was developed for working with NumPy arrays, it can sometimes be more convenient to preprocess data using pandas' DataFrame.

We can remove the corresponding features (columns) or samples (rows) from the dataset.

The rows with missing values can be dropped via the pandas.DataFrame.dropna() method:

We can drop columns that have at least one NaN in any row by setting the axis argument to 1:

where axis : {0 or 'index', 1 or 'columns'}.

The dropna() method has several additional parameters:

The removal of missing data appears to be a convenient approach, however, it also comes with certain disadvantages:

1. There are chances that we may end up removing too many, which will make our analysis not reliable.
2. By eliminating too many feature columns, we may run the risk of losing valuable information for our classifier.

In the next section, we will look into interpolation techniques which one of the most commonly used alternatives for dealing with missing data.

Mean imputation is a method replacing the missing values with the mean value of the entire feature column. While this method maintains the sample size and is easy to use, the variability in the data is reduced, so the standard deviations and the variance estimates tend to be underestimated.

We'll use the sklearn.preprocessing.Imputer class:

Here, we replaced each NaN value by the corresponding mean from each feature column.

We can use the row means if we change the setting axis=0 to axis=1:

As for the strategy parameter, there are other options such as median or most_frequent.

The Imputer class we used in the previous section belongs to the so-called transformer classes in scikit-learn that are used for data transformation.

There are two methods of estimators:

1. fit:
   This method is used to learn the parameters from the training data.
2. transform:
   transform() method uses those parameters to transform the data.

The diagram below shows how a transformer fitted on the training data is used to transform a training dataset as well as a new test dataset:

In supervised learning tasks, we additionally provide the class labels for fitting the model, which can then be used to make predictions about new data samples via the predict() method:

picture source : Python machine learning by Sebastian Raschka

Not all data has numerical values. Here are examples of categorical data:

1. The blood type of a person: A, B, AB or O.
2. The state that a resident of the United States lives in.
3. T-shirt size. XL > L > M
4. T-shirt color.

Even among categorical data, we may want to distinguish further between nominal and ordinal which can be sorted or ordered features. So, T-shirt size can be an ordinal feature, because we can define an order XL > L > M.

Let's create a new categorical data frame:

As we can see from the output, the DataFrame contains a nominal feature (color), an ordinal feature (size) as well as a numerical feature (price) column. In the last column, the class labels are created.

In order for out learning algorithm to interpret the ordinal features correctly, we should convert the categorical string values into integers.

However, since there is no convenient function that can automatically derive the correct order of the labels of our size feature, we have to define the mapping manually.

Let's assume that we know the difference between features such as XL = L + 1 = M + 2.

If we want to transform the integer values back to the original string representation, we can simply define a reverse-mapping dictionary "inv_size_mapping" that can then be used via the pandas' map method on the transformed feature column similar to the "size_mapping" dictionary that we used previously:

Usually, class labels are required to be encoded as integer values.

While most estimators for classification in scikit-learn convert class labels to integers internally, as a good practice, we may want to provide class labels as integer arrays to avoid any issues.

To encode the class labels, we can use an approach similar to the mapping of ordinal features discussed in the previous section.

Since class labels are not ordinal, it doesn't matter which integer number we assign to a particular string-label.

So, we can simply enumerate the class labels starting at 0:

Now we want mapping dictionary to transform the class labels into integers:

As we did for "size" in the previous section, we can reverse the key-value pairs in the mapping dictionary as follows to map the converted class labels back to the original string representation:

Though we encoded the class labels manually, luckily, there is a convenient LabelEncoder class directly implemented in scikit-learn to achieve the same:

Note that the fit_transform() method is just a shortcut for calling fit and transform separately, and we can use the inverse_transform() method to transform the integer class labels back into their original string representation:

So far, we used a simple dictionary-mapping approach to convert the ordinal size feature into integers.

Because scikit-learn's estimators treat class labels without any order, we used the convenient LabelEncoder class to encode the string labels into integers.

We can use a similar approach to transform the nominal color column of our dataset as well:

We may want to create a new dummy feature for each unique value in the nominal feature column. In other words, we would convert the color feature into three new features: blue, green, and red. Binary values can then be used to indicate the particular color of a sample; for example, a blue sample can be encoded as blue=1, green=0, red=0. This technique is called one-hot encoding.

In order to perform this transformation, we can use the scikit-learn.preprocessingOneHotEncoder:

Note that when we initialized the OneHotEncoder, we defined the column position of the variable that we want to transform via the categorical_features parameter which is the first column in the feature matrix X.

The OneHotEncoder, by default, returns a sparse matrix when we use the transform() method, and we converted the sparse matrix representation into a regular (dense) NumPy array for the purposes of visualization via the toarray() method. Sparse matrices are simply a more efficient way of storing large datasets, and one that is supported by many scikit-learn functions. It is especially useful if it contains a lot of zeros.

If we want to omit the toarray() step, we may initialize the encoder as OneHotEncoder(...,sparse=False) to return a regular NumPy array:

Another way which is more convenient is to create those dummy features via one-hot encoding is to use the pandas.get_dummies() method. Applied on a DataFrame, the get_dummies() method will only convert string columns and leave all other columns unchanged:

Source is available from bogotobogo-Machine-Learning .

Next:

Machine Learning with scikit-learn

Machine learning algorithms and concepts

Artificial Neural Networks (ANN)