{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Preprocessing\n",
"\n",
"Preprocessing (cleaning and transforming) your data is utterly essential to ML. Some would say that after defining interesting problems to solve (which isn't a modeling step), preprocessing (which isn't a modeling step) is the next most important part of ML projects. So I wanted to have a little bit on preprocessing on the website. But preprocessing is a MASSIVE topic, so this is only a small taste of what you can do. As always, the official docs are useful!\n",
"\n",
"Here is a great video on the topic from a core sk-learn dev that I highly recommend:\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Imputation / Missing Values\n",
"\n",
"_We talked [about imputation a bit before](../03/05c_missingdata) in the context of `pandas`. [These slides](https://github.com/matthewbrems/ODSC-missing-data-may-18/blob/master/Analysis%20with%20Missing%20Data.pdf) on missing data are quite good! [This article](https://www.geeksforgeeks.org/working-with-missing-data-in-pandas/) has examples too._\n",
"\n",
"Before modeling, you have to decide how to deal with missing values. You can \n",
"1. Drop observations with any missing values, \n",
"2. Impute missing values (mean, median, mode, interpolation, deduction, mean-of-group, etc), \n",
"3. Or model the missing values explicitly (e.g. in a regression, as an incremental intercept but with no impact on the slope). \n",
"\n",
"What's the right choice? It depends. On the data, the domain, the question, and economic theory. My choices change from project to project. You might use a combination of these!\n",
"\n",
"**You should focus on the whys and hows of dealing with missing data rather than the mechanics of the code. (You can look up mechanics later.)** You should have some livecoding from the prior lecture showing imputation in `pandas`.\n",
"\n",
"`sklearn` comes with an `impute` class described in the [official docs](https://scikit-learn.org/stable/modules/impute.html)\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[[nan 0. 3.]\n",
" [ 3. 7. 9.]\n",
" [ 3. 5. 2.]\n",
" [ 4. nan 6.]\n",
" [ 8. 8. 1.]] \n",
"\n"
]
},
{
"data": {
"text/plain": [
"array([[4.5, 0. , 3. ],\n",
" [3. , 7. , 9. ],\n",
" [3. , 5. , 2. ],\n",
" [4. , 5. , 6. ],\n",
" [8. , 8. , 1. ]])"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# silly data\n",
"import numpy as np\n",
"X = np.array([[ np.nan, 0, 3 ],\n",
" [ 3, 7, 9 ],\n",
" [ 3, 5, 2 ],\n",
" [ 4, np.nan, 6 ],\n",
" [ 8, 8, 1 ]])\n",
"print(X,'\\n')\n",
"\n",
"# it's this easy:\n",
"from sklearn.impute import SimpleImputer\n",
"imp = SimpleImputer(strategy='mean')\n",
"imp.fit_transform(X) \n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`imp.fit_transform(X)` is the combination of `imp.fit(X)` and `imp.transform(X)`. \n",
"\n",
"If you have a train/test split, you shouldn't use `fit_transform`. Instead, use `imp.fit(X_train)` to get the means in the training sample and `imp.transform(X_test)` to apply those to the test data."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Standardization\n",
"\n",
"Effectively, this means that **continuous** variables should have a mean of 0 and a variance of 1.\n",
"\n",
"The [`sklearn` documentation](https://scikit-learn.org/stable/modules/preprocessing.html#standardization-or-mean-removal-and-variance-scaling) on this is quite good.\n",
"\n",
"> Standardization of datasets is a **common requirement for many machine learning estimators** implemented in scikit-learn; they might behave badly if the individual features do not more or less look like standard normally distributed data: Gaussian with zero mean and unit variance.\n",
"\n",
"Why does this matter? \"If a feature has a variance that is orders of magnitude larger than others, it might dominate the objective function and make the estimator unable to learn from other features correctly as expected.\"\n",
"\n",
"**In other words: STANDARDIZATION WILL IMPROVE YOUR PREDICTIONS.**\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`sklearn` can scale variables in many ways. Some alternative transforms are faster and some transform non-normal distributions into proto-normal distributions (which can improve the efficacy of many models).\n",
"\n",
"Visit (you guessed it!) [the documentation](https://scikit-learn.org/stable/modules/preprocessing.html#standardization-or-mean-removal-and-variance-scaling) for more.\n",
"\n",
"Here is a simple example using `preprocessing.StandardScaler`. "
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
" X_scaled\n",
" ---------------------------------------- \n",
" [[ 0. -1.22474487 1.33630621]\n",
" [ 1.22474487 0. -0.26726124]\n",
" [-1.22474487 1.22474487 -1.06904497]] \n",
"\n",
" Mean of each var:\n",
" ---------------------------------------- \n",
" [0. 0. 0.] \n",
"\n",
" STD of each var:\n",
" ---------------------------------------- \n",
" [1. 1. 1.] \n",
"\n"
]
}
],
"source": [
"# a very simple example\n",
"from sklearn.preprocessing import StandardScaler \n",
"import numpy as np\n",
"\n",
"X_train = np.array([[ 1., -1., 2.],\n",
" [ 2., 0., 0.],\n",
" [ 0., 1., -1.]])\n",
"\n",
"scaler = StandardScaler() \n",
"X_scaled = scaler.fit_transform(X_train)\n",
"\n",
"print(' X_scaled\\n', '-'*40,'\\n',X_scaled,'\\n')\n",
"print(' Mean of each var:\\n','-'*40,'\\n',X_scaled.mean(axis=0),'\\n')\n",
"print(' STD of each var:\\n', '-'*40,'\\n',X_scaled.std(axis=0),'\\n')\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.13"
}
},
"nbformat": 4,
"nbformat_minor": 4
}