What fraction of what you labeled positive are truly positive?_ | Binary variables | \n", "| **\"Accuracy\"** | _What fraction of predictions made are correct?_ | Binary variables | \n", "| False Positive Rate (FPR) = 1-TNR | _What fraction of the true negatives do you call positive?_ | Binary variables | \n", "| False Negative Rate (FNR) = 1-TPR | _What fraction of the true positives do you call negative?_ | Binary variables | \n", "| \"F-score\" or \"F1\" | _The harmonic mean of recall and precision. Intuition: Are you detecting positive cases, and correctly?_ | Binary variables | \n", "\n", "Here is a great trick from [Sean Taylor](https://twitter.com/seanjtaylor/status/1385776334048088065?s=27) for remembering which errors are type 1 and type 2 (II):\n", "\n", "```{image} img/type1and2errors.jpg\n", ":alt: Error_types\n", ":width: 400px\n", "```\n", "\n", "

Which measurement should you use?

\n", "\n", "It depends! If you have a continuous variable, use R2, MAE, or RMSE.\n", "\n", "If you have a classification problem (the \"binary variable\" rows in the table above), you have to think about your problem. [I REALLY like the discussion on this page from DS100.](https://www.textbook.ds100.org/ch/24/classification_sensitivity_specificity.html)\n", "- In medical tests more broadly, false negatives are really bad - if someone has cancer, don't tell them they are cancer free. Thus, medical tests tend to minimize false negatives.\n", "- Minimizing false negatives is the same as maximizing the true positives, aka the \"sensitivity\" of a test, since $FNR=1-TPR$. Better covid tests have higher specificity, because if someone has covid, we want to know that so we can require a quarantine.\n", "- In the legal field, false positives (imprisoning an innocent person) are considered worse than false negatives.\n", "- Identifying terrorists? You might want to maximize the detection rate (\"recall\"). Of course, simply saying \"everyone is a terrorist\" is guaranteed to generate a recall rate of 100%! But this rule would have a precision of about 0%! So you also should think about \"precision\" too. Maybe F1 is the metric for you. \n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.12" } }, "nbformat": 4, "nbformat_minor": 4 }