{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Interpreting regression coefficients\n", "\n", "Revisiting the regression objectives: After this page, \n", "\n", "
\n", "
1. You can interpret the mechanical meaning of the coefficients for
2. \n", "
\n", "
• continuous variables
• \n", "
• categorical a.k.a qualitative variables with two or more values (aka \"dummy\", \"binary\", and \"categorical\" variables)
• \n", "
• interaction terms between two X variables changes interpretation
• \n", "
• variables in models with other controls included (including categorical variables)
• \n", "
\n", "
\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "{tip}\n", "Regressions of y on $N$ different variables take the form\n", "\n", "$$\n", "y = a+b_1*X_1+b_2*X_2+...+b_N*X_N+u\n", "$$\n", "\n", "The _generic_ interpretation of any of the $b$ coefficients is a sentence in three parts: \n", "1. \"A **1 unit increase** in $X_i$... \n", "1. ...**is associated** with a $b_i$ change in $y$,...\n", "1. **holding all other X constant**.\"\n", "\n", "\n", "## If X is a continuous variable \n", "\n", "| If the model is.................. | then $\\beta$ means (approx. in log cases) |\n", "| :--- | :--- |\n", "| $y=a+\\beta X$ | If $X \\uparrow$ 1 unit, then $y \\uparrow$ by $\\beta$ units |\n", "| $\\log y=a+\\beta X$ | If $X \\uparrow$ 1 unit, then $y \\uparrow$ by about $100*\\beta$%.
_(Exact: $100*(\\exp(\\beta)-1)$)_ |\n", "| $y=a+\\beta \\log X$ | If $X \\uparrow$ 1%, then $y \\uparrow$ by about $\\beta / 100$ units |\n", "| $\\log y=a+\\beta \\log X$ | If $X \\uparrow$ 1%, then $y \\uparrow$ by $\\beta$% |\n", "\n", "{note}\n", "This table should help you see why log transformations are useful: They model _**proportional**_ relationships. That is, instead of focusing on 1 unit changes in X and Y (i.e. \"linear\" changes), they model percent changes in X and/or Y!\n", "\n", "\n", "## If X is a binary variable\n", "\n", "\n", "This is a categorical or qualitative variable with two values (aka \"dummy\"). E.g. gender in Census data, and \"ideal\" above. \n", "\n", "Usually, we encode one value as zero, and the other as one before we include it in the regression. This makes interpretation simple, as it just follows from the previous table, since a \"1 unit change in X\" simply means changing from the baseline group encoded as zero to the other group encoded as one.\n", "\n", "| If the model is.................. | then $\\beta$ means |\n", "| :--- | :--- |\n", "| $y=a+\\beta X$ | $y$ is $\\beta$ units higher for cases when $X=1$ than when $X=0$. |\n", "| $\\log y=a+\\beta X$ | $y$ is about $100*\\beta$ % higher for cases when $X=1$ than when $X=0$. |" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## If X is a categorical variable\n", "\n", "Suppose X has three categories, and let's just call them 0, 1, and 2. To run this regression, first create two variables: $X_1$ and $X_2$, which are binary variables indicating if an observation's value of X equals the subscript. So: \n", "\n", "| If X (original variable) is | Then $X_1=$ | and $X_2=$ |\n", "| :--- | :--- | :--- |\n", "| 0 | 0 | 0 |\n", "| 1 | 1 | 0 |\n", "| 2 | 0 | 1 |\n", "\n", "Then, we run a regression of $y$ on $X_1$ and $X_2$. The way we interpret the coefficients is:\n", "\n", "| If the model is.................. | $a$ means | then $\\beta_1$ means | then $\\beta_2$ means | \n", "| :--- | :--- | :--- | :--- |\n", "| $y=a+\\beta_1 X_1 +\\beta_2 X_2$ | the average value of $y$ is $a$ for group 0 (because $X_1=X_2=0$ if $X=0$) | $y$ is $\\beta_1$ units higher on average for cases when $X=1$ than when $X=0$. | $y$ is $\\beta_2$ units higher on average for cases when $X=2$ than when $X=0$. |\n", "| $\\log y=a+\\beta_1 X_1 +\\beta_2 X_2$ | the average value of $\\log y$ is $a$ for group 0 (because $X_1=X_2=0$ if $X=0$) | $y$ is about $100*\\beta_1$ % higher on average for cases when $X=1$ than when $X=0$. | $y$ is about $100*\\beta_2$ % higher on average for cases when $X=2$ than when $X=0$. | \n", "\n", "{tip}\n", "The interpretation of $\\beta_{oneLevelOfACategoricalVariable}$ is the same as a binary variable (use the table above depending on if the model is using y or $\\log y$), **except that it is capturing the jump from the \"omitted group\" (X=0 above) to whichever level that particular $\\beta$ captures.**\n", "\n", "\n", "---\n", "\n", "Students often get confused by this at first, so let's do an example.\n", "\n", "Suppose we model the price of a diamond as a function of its cut and nothing else. [This is close to what we did previously](02b_mechanics.html#including-categorical-variables). This ends up looking like\n", "\n", "$$\n", "\\log(\\text{price})=\n", " \\begin{cases}\n", " a, & \\text{if cut is Ideal} \\\\\n", " a +\\beta_{Premium}, & \\text{if cut is Premium} \\\\\n", " a +\\beta_{Very Good}, & \\text{if cut is Very Good} \\\\\n", " a +\\beta_{Good}, & \\text{if cut is Good} \\\\\n", " a +\\beta_{Fair}, & \\text{if cut is Fair} \n", " \\end{cases} \n", "$$\n", "\n", "To do this, you take the cut variable (cut={Fair,Good,Very Good,Premium,Ideal}) and create a dummy variable for \"Fair\", a dummy variable for \"Good\", a dummy variable for \"Very Good\", and a dummy variable for \"Premium\". (But not for \"Ideal\"!) The \"statsmodels formula\" approach to specifying the regression does this step automatically for you!\n", "\n", "So now, your model can be rewritten in one line and used in a regression as\n", "\n", "$$\n", "\\log(\\text{price}) = a + \\beta_{Premium}*X_{Premium} + \\beta_{Very Good}*X_{Very Good} + \\beta_{Good}*X_{Good} + \\beta_{Fair}*X_{Fair} + u\n", "$$\n", "\n", "And we interpret these like this:\n", "\n", "| $\\beta$..... | means | or |\n", "| :--- | :--- | :--- |\n", "| $\\beta_{Premium}$ | The average log(price) for Premium diamonds is $100*\\beta_{Premium}$% higher than Ideal diamonds. | $avg_{Premium}-avg_{Ideal}$ |\n", "| $\\beta_{Very Good}$ | The average log(price) for Very Good diamonds is $100*\\beta_{Very Good}$% higher than Ideal diamonds. | $avg_{Very Good}-avg_{Ideal}$ |\n", "| $\\beta_{Good}$ | The average log(price) for Good diamonds is $100*\\beta_{Good}$% higher than Ideal diamonds. | $avg_{Good}-avg_{Ideal}$ |\n", "| $\\beta_{Fair}$ | The average log(price) for Fair diamonds is $100*\\beta_{Fair}$% higher than Ideal diamonds. | $avg_{Fair}-avg_{Ideal}$ |\n", "\n", "So if we run this regression, we get these coefficients:" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "tags": [ "hide-input" ] }, "outputs": [], "source": [ "# load some data to practice regressions\n", "import seaborn as sns\n", "import numpy as np\n", "from statsmodels.formula.api import ols as sm_ols # need this\n", "\n", "diamonds = sns.load_dataset('diamonds')\n", "\n", "# this alteration is not strictly necessary to practice a regression\n", "# but we use this in livecoding\n", "diamonds2 = (diamonds.query('carat < 2.5') # censor/remove outliers\n", " .assign(lprice = np.log(diamonds['price'])) # log transform price\n", " .assign(lcarat = np.log(diamonds['carat'])) # log transform carats\n", " .assign(ideal = diamonds['cut'] == 'Ideal') \n", " \n", " # some regression packages want you to explicitly provide \n", " # a variable for the constant\n", " .assign(const = 1) \n", " ) \n" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Intercept 7.636921\n", "C(cut)[T.Premium] 0.307769\n", "C(cut)[T.Very Good] 0.158754\n", "C(cut)[T.Good] 0.199155\n", "C(cut)[T.Fair] 0.431911\n", "dtype: float64\n" ] } ], "source": [ "print(sm_ols('lprice ~ C(cut)', data=diamonds2).fit().params)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**THE MAIN THING TO REMEMBER IS THAT $\\beta_{value}$ COMPARES THAT $value$ TO THE _OMITTED_ CATEGORY!**\n", "\n", "So, $\\beta_{Good}=0.199$ implies that \"Good\" cut diamonds are about 20% more expensive than \"Ideal\" diamonds. (Weird?)\n", "\n", "If we add $\\alpha$ (the average log price of \"Ideal\" diamonds) to $\\beta_{Good}$, we get $\\beta_{Good}+\\alpha=0.199+7.637=7.83$. This should be the average log price of \"Good\" diamonds.\n", "\n", "Let's check:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "cut\n", "Ideal 7.636921\n", "Premium 7.944690\n", "Very Good 7.795675\n", "Good 7.836076\n", "Fair 8.068832\n", "Name: lprice, dtype: float64" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "diamonds2.groupby('cut')['lprice'].mean() # avg lprice by cut" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## If X is an interaction term\n", "\n", "[Previously,](02b_mechanics.html#including-interaction-terms) we estimated \n", "\n", "$$\n", "\\log(\\text{price})= 8.2+ 1.53 *\\log(\\text{carat}) + 0.33* \\text{Ideal} + 0.18* \\log(\\text{carat})\\cdot \\text{Ideal}\n", "$$\n", "\n", "There are two natural questions:\n", "1. What is the impact of X on y? (Specifically, _what is the total impact of diamond size on price?_)\n", "1. What is does the interaction term's coefficient mean? \n", "\n", "**To find the answer to Q1, to figure out the relationship of X on y, take the derivative of X w.r.t. y:**\n", "- Relationship of size on price: $1.53 + 0.18*Ideal$. \n", " - A 1% increase in size is associated with a 1.53% higher price **for non ideal diamonds**\n", " - A 1% increase in size is associated with a 1.71% higher price **for ideal diamonds**\n", "- Relationship of cut on price: $0.33 + 0.18*log(carat)$. \n", " - For 1 carat diamonds ($log(1)=0$), ideal diamonds are 33% more expensive than non-ideal diamonds\n", " - For 2 carat diamonds ($log(2)=0.693$), ideal diamonds are 45% more expensive than non-ideal diamonds\n", "\n", "**Q2: How do you interpret $\\beta_3=0.18$? I recommend revisiting that link above too, but I'll summarize as:**\n", "1. $\\beta_3 \\neq 0$ implies that the relationship between **carat size** and price is different for ideal and non-ideal diamonds. \n", " - Mathematically: $1\\% \\uparrow$ in carat $\\rightarrow$ price increases by 1.53% for non-ideal but 1.71% for ideal\n", " - Graphically, the difference in the slope of those carat-price relationships for ideal/non-ideal diamonds is $\\beta_3$\n", " - _Economically, you might say that the value of a larger ring is even more valuable for better cut diamonds_\n", "1. $\\beta_3 \\neq 0$ implies that the relationship between **cut quality** and price is different for diamonds of different sizes. \n", " - Mathematically: 1 carat diamonds that are ideal are 33% more expensive than non-ideal diamonds, but 2 carat ideal diamonds are 45% more expensive than non-ideal diamonds\n", " - Graphically, the difference in the slope of those cut quality-price relationships for small/large diamonds is $\\beta_3$\n", " - _Economically, you might say that the value of a better cut is even more valuable for larger diamonds_\n", "\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## If other controls are included\n", "\n", "{tip}\n", "1. Always keep the \"holding all other controls constant\" mantra in mind!\n", "1. In reality, independent variables in X often move together. So the marginal effect of X (i.e. $\\beta$) is **not** the same as the total effect of X.\n", "\n", "\n", "If you have many control variables (up to N controls):\n", "\n", "$$\n", "y = a +\\beta_0 X_0 + \\beta_1 X_1+ ...+\\beta_N X_N+ u\n", "$$\n", "\n", "$\\beta_1$ estimates the expected change in Y for a 1 unit increase in $X_1$ (as we covered above), **holding all other controls constant!**\n", "\n", "As a hypothetical illustration, if Y = number of tackles by a football player in a year, $W$ is weight, and $H$ is height, suppose we estimate that\n", "\n", "$$\n", "y = a +\\hat{0.5} W + \\hat{-0.1} H\n", "$$\n", "\n", "How do you interpret $\\hat{\\beta_2} < 0$ on H? " ] }, { "cell_type": "markdown", "metadata": { "tags": [ "hide-input" ] }, "source": [ "- This regression implies that, **for a given weight (holding weight fixed),** taller players average fewer tackles. In other words, skinny football players get fewer tackles. \n", "- This regression does NOT imply that taller players average fewer tackles. In the real world, what we call \"independent variables\" in the regression often change together. Taller players are likely to be heavier. So if a 1 inch increase typically comes with a weight gain, the _total impact_ of height on tackles (i.e. _not_ holding all other factors constant) will include the estimated impact of weight, which $\\hat{\\beta_2} < 0$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## If other categorical controls are included\n", "\n", "Suppose you estimate $profits=a+b*investment+c*X+u$, and you want to focus on $b$ to capture how investments translate to profits. You've added some control variables X, but you're still worried that this regression will get the relationship wrong because different industries have different profit margins for reasons that have nothing to do with investment levels.\n", "\n", "In other words, you want to \"control for industry\". So you estimate $profits=a+b*investment+c*X+d*C(gsector)+u$, by including the firm's industry as a categorical control.\n", "\n", "{note}\n", "When you add industry to a regression as a categorical variable, it is called including \"industry **fixed effects**\".\n", "\n", "\n", "What does $b$ mean now? Well, the lessons above apply: It is the relationship between investment and profits, but now we capture the average profit level (across all firms in an industry over all years of our sample). So we are holding the industry profit level (again, over the whole time period of analysis) fixed.\n", "{epigraph}\n", "Intuitively, you can think of $b$ in this model as \"comparing firms in the same industry\" or \"controlling for industry factors\". \n", "\n", "\n", "This should go a decent way towards solving your worry above. \n", "\n", "---\n", "\n", "Similarly, you might be worried that some years are at high points in the business cycle, and these years have concurrently high investment and profits simply because of the business cycle. This would cause $b$ to be positive _even if investment does not lead to profits._\n", "\n", "So you might estimate $profits=a+b*investment+c*X+d*C(year)+u$. This is often referred to as \"year fixed effects\", and it means that your estimate of $b$ removes the impact of years, and presumably, the business cycle.\n", "{epigraph}\n", "Intuitively, you can think of $b$ in this model as \"comparing firms in the same year\" or \"controlling for time trends in profits\".\n", "\n", "\n", "---\n", "\n", "Here is an example to see how categorical controls can interact with \"normal\" continuous variables.\n", "\n", "Remember our weird result earlier? That better cut diamonds had lower average prices?\n", "\n", "The answer to that puzzle is pretty simple: Better cut diamonds tend to be smaller, and size is the most important aspect of diamond price. You can click to show the model results below. \n", "\n", "By adding carat size back to our model, we get the sensible result, that going from an Ideal cut to a Fair cut diamond (a big downgrade), as long as we compare similarly sized diamonds (\"control for diamond size\"), is associated with a 31% decrease in price. " ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "tags": [ "hide-output" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " OLS Regression Results \n", "==============================================================================\n", "Dep. Variable: lprice R-squared: 0.937\n", "Model: OLS Adj. R-squared: 0.937\n", "Method: Least Squares F-statistic: 1.613e+05\n", "Date: Thu, 25 Mar 2021 Prob (F-statistic): 0.00\n", "Time: 12:49:44 Log-Likelihood: -2389.9\n", "No. Observations: 53797 AIC: 4792.\n", "Df Residuals: 53791 BIC: 4845.\n", "Df Model: 5 \n", "Covariance Type: nonrobust \n", "=======================================================================================\n", " coef std err t P>|t| [0.025 0.975]\n", "---------------------------------------------------------------------------------------\n", "Intercept 8.5209 0.002 4281.488 0.000 8.517 8.525\n", "C(cut)[T.Premium] -0.0790 0.003 -28.249 0.000 -0.084 -0.074\n", "C(cut)[T.Very Good] -0.0770 0.003 -26.656 0.000 -0.083 -0.071\n", "C(cut)[T.Good] -0.1543 0.004 -38.311 0.000 -0.162 -0.146\n", "C(cut)[T.Fair] -0.3111 0.007 -46.838 0.000 -0.324 -0.298\n", "lcarat 1.7014 0.002 889.548 0.000 1.698 1.705\n", "==============================================================================\n", "Omnibus: 792.280 Durbin-Watson: 1.261\n", "Prob(Omnibus): 0.000 Jarque-Bera (JB): 1178.654\n", "Skew: 0.168 Prob(JB): 1.14e-256\n", "Kurtosis: 3.643 Cond. No. 7.20\n", "==============================================================================\n", "\n", "Notes:\n", " Standard Errors assume that the covariance matrix of the errors is correctly specified.\n" ] } ], "source": [ "print(sm_ols('lprice ~ lcarat + C(cut)', data=diamonds2).fit().summary())" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Comparing the size of two coefficients\n", "\n", "[Earlier](02b_mechanics.html#our-first-regression-with-statsmodels), we estimated that $\\log price = \\hat{8.41} + \\hat{1.69} \\log carat + \\hat{0.10} ideal$.\n", "\n", "So... I have questions:\n", "1. Does that mean that the size of the diamond ($\\log carat$) has a 17 times larger impact than the cut ($ideal$) in terms of price impact? \n", "2. How do we compare those magnitudes?\n", "3. More generally, how do we compare the magnitudes of any 2 control variables?\n", "\n", "To which, I'd say that how \"big\" a coefficient is depends on the variable!\n", "- For some variables, an increase of 1 unit is common (e.g. our $ideal$ dummy is one 40% of the time)\n", "- For some variables, an increase of 1 unit is rare (e.g. $cash/assets$)\n", "- **$\\rightarrow$ the meaning of the coefficient's magnitude _depends on the corresponding variable's variation!_**\n", "- $\\rightarrow$ so change variables so that a \"1 unit increase\" implies the same amount of movement\n", "\n", "{admonition} A great trick for comparing coefficient size\n", "**Scale continuous variables by their standard deviation!**\n", "\n", "\n", "{warning}\n", "_Only scale continuous variables! Don't scale dummy variables or categorical variables!_\n", "\n", "\n", "Here is that solution in action:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Divide lcarat by its std dev:\n", "\n", "Intercept 8.418208\n", "ideal[T.True] 0.100013\n", "lcarat 0.985628\n", "dtype: float64\n", "\n", "\n", "The original reg:\n", "\n", "Intercept 8.418208\n", "ideal[T.True] 0.100013\n", "lcarat 1.696259\n", "dtype: float64\n" ] } ], "source": [ "standardize = lambda x:x/x.std() # normalize(df['x']) will divide all 'x' by the std deviation of 'x'\n", "\n", "print(\"Divide lcarat by its std dev:\\n\")\n", "print(sm_ols('lprice ~ lcarat + ideal', \n", " # for **just** this regression, divide \n", " data=diamonds2.assign(lcarat = standardize(diamonds2['lcarat'])) \n", " # this doesn't change the diamonds2 data permanently, so the next time you call on\n", " # diamonds2, you can use lcarat as if nothing changed. if you want to repeat this\n", " # a bunch, you might instead create and save a permanent variable called \"lcarat_std\"\n", " # where \"_std\" indicates that you divided it by the std dev.\n", " ).fit().params)\n", "\n", "print(\"\\n\\nThe original reg:\\n\")\n", "print(sm_ols('lprice ~ lcarat + ideal',data=diamonds2 ).fit().params)\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**So a 1 standard deviation increase in $\\log carat$ is associated with a 98% increase in price. Compared to $ideal$, we can say that a reasonable variation in carat size is associated with a price increase about 10 times larger than the impact of cut, not 17 times larger.**" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Also, notice that the new coefficient (0.98) is about 58% of the original coefficient (1.69). \n", "\n", "{dropdown} **Q: Why is it 58% of the previous coefficient?**\n", "\n", "A: Because the standard deviation of $\\log carat$ is 0.58!\n", "\n", "This works because, if \"$std$\" stands for the standard deviation of $\\log carat$ each of these steps is valid and doesn't change the estimation:\n", "\n", "$$\n", "\\log price & = \\hat{8.41} + \\hat{1.69} \\log carat + \\hat{0.10} ideal \\\\\n", " & = \\hat{8.41} + \\hat{1.69} \\frac{std}{std} \\log carat + \\hat{0.10} ideal \\\\\n", " & = \\hat{8.41} + (\\hat{1.69}*std) \\frac{\\log carat}{std} + \\hat{0.10} ideal \n", "$$\n", "\n", "So what that last line shows is that if we divide the variable by its standard deviation, the coefficient will change by an offsetting amount.\n", "\n", "If a variable has a small standard deviation (e.g. 0.10), dividing the variable by 0.10 before running the regression will reduce the coefficient by 90%. \n", "\n", "" ] } ], "metadata": { "celltoolbar": "Tags", "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.13" } }, "nbformat": 4, "nbformat_minor": 4 }