{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Classification: Logistic Regression\n",
    "\n",
    "\n",
    "**Prerequisites**\n",
    "\n",
    "- [Regression]\n",
    "**Outcomes**\n",
    "\n",
    "- Understand what problems classification solves \n",
    "- Understand the logistic regression model \n",
    "- Evaluate classification models using a variety of metrics  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Outline\n",
    "\n",
    "\n",
    "  - [Example: Logistic Regression](#Example:-Logistic-Regression)  \n",
    "  - [Model Evaluation](#Model-Evaluation)  \n",
    "  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "# Uncomment following line to install on colab\n",
    "#! pip install qeds fiona geopandas xgboost gensim folium pyLDAvis descartes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction to Classification\n",
    "\n",
    "We now move from regression to the second main branch of machine learning:\n",
    "classification.\n",
    "\n",
    "Recall that the regression problem mapped a set of\n",
    "feature variables to a continuous target.\n",
    "\n",
    "Classification is similar to regression, but instead of predicting a continuous\n",
    "target, classification algorithms attempt to apply one (or more) of a discrete\n",
    "number of labels or classes to each observation.\n",
    "\n",
    "Another perspective is that for regression, the targets are usually\n",
    "continuous-valued, while in classification, the targets are categorical.\n",
    "\n",
    "Common examples of classification problems are\n",
    "\n",
    "- Labeling emails as spam or not spam  \n",
    "- Person identification in a photo  \n",
    "- Speech recognition  \n",
    "- Whether or not a country is or will be in a recession  \n",
    "- Loan default.\n",
    "\n",
    "\n",
    "Classification can also be applied in settings where the target isn’t naturally\n",
    "categorical.\n",
    "\n",
    "For example, suppose we want to predict whether the unemployment rate for a state\n",
    "will be low ($ <3\\% $), medium ($ \\in [3\\%, 5\\%] $), or high ($ >5\\% $)\n",
    "but don’t care about the actual number.\n",
    "\n",
    "Most economic problems are posed in continuous terms, so it may take some creativity\n",
    "to determine the optimal way to categorize a target variable so\n",
    "classification algorithms can be applied.\n",
    "\n",
    "As many problems can be posed either as classification or regression, many\n",
    "machine learning algorithms have variants that perform regression or\n",
    "classification tasks.\n",
    "\n",
    "Throughout this lecture, we will revisit some of the algorithms from the\n",
    "[regression](https://datascience.quantecon.org/regression.html) lecture and discuss how they can be applied in\n",
    "classification settings.\n",
    "\n",
    "As we have already seen relatives of these algorithms, this lecture will be\n",
    "lighter on exposition and then build up to an application."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "import datetime\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "#import pandas_datareader.data as web\n",
    "\n",
    "from sklearn import (\n",
    "    linear_model, metrics, pipeline, preprocessing, model_selection\n",
    ")\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "# activate plot theme\n",
    "#import qeds\n",
    "#qeds.themes.mpl_style();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example: Logistic Regression\n",
    "\n",
    "We have actually already encountered a classification algorithm.\n",
    "\n",
    "In the [recidivism](https://datascience.quantecon.org/recidivism.html) example, we attempted to predict whether\n",
    "or not an individual would commit another crime by using a combination of the\n",
    "assigned COMPAS score and the individual’s gender or race.\n",
    "\n",
    "In that example, we used a *logistic regression* model, which is a close\n",
    "relative of the linear regression model from the [regression](https://datascience.quantecon.org/regression.html) section.\n",
    "\n",
    "The logistic regression model for predicting the likelihood of recidivism using\n",
    "the `COMPAS` score as the single feature is written\n",
    "\n",
    "$p(\\text{recid}) $ = L(latent variables)\n",
    "\n",
    "$$\n",
    "p(\\text{recid}) =\\beta_0 + \\beta_1 \\text{COMPAS} + \\epsilon\n",
    "$$\n",
    "\n",
    "$$\n",
    "p(\\text{recid}) = L(X=\\beta_0 + \\beta_1 \\text{COMPAS} + \\epsilon)\n",
    "$$\n",
    "\n",
    "\n",
    "\n",
    "where $ L $ is the *logistic function*: $ L(x) = \\frac{1}{1 + e^{-x}} $.\n",
    "\n",
    "To get some intuition for this function, let’s plot it below.\n",
    "\n",
    "probit model\n",
    "logit model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "value = -1.4297 + (0.2645*0)\n",
    "value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "decile_score=100000000000\n",
    "value =-1.3992 + 0.2638*decile_score\n",
    "value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    " p(recid) = L(-1.3992 + 0.2638 decile_score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prob = 1/(1+np.exp(-value))\n",
    "prob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "x = np.linspace(-5, 5, 100)\n",
    "y = 1/(1+np.exp(-x))\n",
    "plt.plot(x, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that for all values of $ x $, the value of the logistic function is\n",
    "always between 0 and 1.\n",
    "\n",
    "This is perfect for binary classification problems that need to\n",
    "output the probability of one of the two labels.\n",
    "\n",
    "Let’s load up the recidivism data and fit the logistic regression model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('compas-scores-two-years.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = df[[\"two_year_recid\"]] \n",
    "np.mean(y) ## 1,0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "0.03"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "linear_model.LogisticRegression()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "linear_model.LogisticRegression()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df[[\"decile_score\"]]\n",
    "y = df[\"two_year_recid\"]\n",
    "\n",
    "X_train1, X_test1, y_train1, y_test1 = model_selection.train_test_split(X, y, test_size=0.2)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_test1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "logistic_model1 = linear_model.LogisticRegression()\n",
    "logistic_model1.fit(X_train1, y_train1)\n",
    "\n",
    "beta_0 = logistic_model1.intercept_[0]\n",
    "beta_1 = logistic_model1.coef_[0][0]\n",
    "\n",
    "print(f\"Fit model: p(recid) = L({beta_0:.4f} + {beta_1:.4f} decile_score)\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x=-1.3766 + (0.2597*9)\n",
    "y = 1/(1+np.exp(-x))\n",
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "logistic_model1.coef_[0][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df[[\"decile_score\",\"priors_count\",\"sex\",'age']]\n",
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sex = {'Male': 1,'Female': 0} #data dict \n",
    "\n",
    "X.sex = [sex[item] for item in X.sex]\n",
    "print(X)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From these coefficients, we see that an increase in the `decile_score` leads\n",
    "to an increase in the predicted probability of recidivism.\n",
    "\n",
    "Suppose we choose to classify any model output greater than 0.5 as “at risk of\n",
    "recidivism”.\n",
    "\n",
    "Then, the positive coefficient on `decile_score` means that there is some cutoff score above which all individuals will be labeled as high-risk.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualization: Decision Boundaries\n",
    "\n",
    "With just one feature that has a positive coefficient, the model’s predictions\n",
    "will always have this cutoff structure.\n",
    "\n",
    "Let’s add a second feature the model: the age of the individual."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X1 = X[[\"decile_score\", \"age\"]]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = model_selection.train_test_split(X1,y,test_size=0.20)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "logistic_age_model = linear_model.LogisticRegression(solver=\"lbfgs\")\n",
    "logistic_age_model.fit(X_train, y_train)\n",
    "\n",
    "beta_0 = logistic_age_model.intercept_[0]\n",
    "beta_1, beta_2 = logistic_age_model.coef_[0]\n",
    "\n",
    "print(f\"Fit model: p(recid) = L({beta_0:.4f} + {beta_1:.4f} decile_score + {beta_2:.4f} age)\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "beta2=logistic_age_model.coef_[0][0]\n",
    "beta2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df[[\"decile_score\", \"age\",\"sex\"]]\n",
    "sex = {'Male': 1,'Female': 0}\n",
    "\n",
    "X.sex = [sex[item] for item in X.sex]\n",
    "\n",
    "X_train, X_test, y_train, y_test = model_selection.train_test_split(\n",
    "    X, y, test_size=0.20, random_state=100\n",
    ")\n",
    "\n",
    "logistic_age_sex_model = linear_model.LogisticRegression(solver=\"lbfgs\")\n",
    "logistic_age_sex_model.fit(X_train, y_train)\n",
    "\n",
    "beta_0 = logistic_age_sex_model.intercept_[0]\n",
    "beta_1, beta_2,beta_3 = logistic_age_sex_model.coef_[0]\n",
    "\n",
    "print(f\"Fit model: p(recid) = L({beta_0:.4f} + {beta_1:.4f} decile_score + {beta_2:.4f} age+{beta_3:.4f} sex)\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "logistic_pred =logistic_age_sex_model.predict(X_test)\n",
    "logistic_proba =logistic_age_sex_model.predict_proba(X_test)\n",
    "logistic_result=pd.DataFrame({'predicted': logistic_pred,\n",
    "                                  'actual':y_test,\n",
    "                                 'p[0]':[p[0] for p in logistic_proba ],\n",
    "                                'p[1]':[p[1] for p in logistic_proba ]     \n",
    "                                  })"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "logistic_result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Fit model: p(recid) = L(-1.4443 + 0.2746 decile_score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we see that an increase in the `decile_score` still leads to an increase in\n",
    "the predicted probability of recidivism, while older individuals are slightly\n",
    "less likely to commit crime again.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Evaluation\n",
    "\n",
    "Before we get too far into additional classification algorithms, let’s take a\n",
    "step back and think about how to evaluate the performance of a classification\n",
    "model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Accuracy\n",
    "\n",
    "Perhaps the most intuitive classification metric is *accuracy*, which is the\n",
    "fraction of correct predictions.\n",
    "\n",
    "For a scikit-learn classifier, this can be computed using the `score` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_acc = logistic_model1.score(X_train1, y_train1)\n",
    "test_acc = logistic_model1.score(X_test1, y_test1)\n",
    "\n",
    "train_acc, test_acc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "train_acc = logistic_age_model.score(X_train, y_train)\n",
    "test_acc = logistic_age_model.score(X_test, y_test)\n",
    "\n",
    "train_acc, test_acc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_acc = logistic_age_sex_model.score(X_train, y_train)\n",
    "test_acc = logistic_age_sex_model.score(X_test, y_test)\n",
    "\n",
    "train_acc, test_acc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# probability"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When the testing accuracy is similar to or higher than the training\n",
    "accuracy (as it is here), the model might be underfitting.\n",
    "Thus, we should consider either using a more powerful model or adding additional\n",
    "features.\n",
    "\n",
    "In many contexts, this would be an appropriate way to evaluate a model, but in\n",
    "others, this is insufficient.\n",
    "\n",
    "For example, suppose we want to use a classification model to predict the\n",
    "likelihood of someone having a rare, but serious health condition.\n",
    "\n",
    "If the condition is very rare (say it appears in 0.01% of the population), then\n",
    "a model that always predicts false would have 99.99% accuracy, but the false\n",
    "negatives could have large consequences."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Precision and Recall\n",
    "\n",
    "In order to capture situations like that, data scientists often use two other\n",
    "very common metrics:\n",
    "\n",
    "- *Precision*: The number of true positives over the number of positive\n",
    "  predictions. Precision tells us how often the model was correct when it\n",
    "  predicted true.  \n",
    "- *Recall*: The number of true positives over the number of actual positives.\n",
    "  Recall answers the question, “What fraction of the positives did we get\n",
    "  correct?”  \n",
    "\n",
    "\n",
    "In the rare health condition example, you may prefer\n",
    "a model with high recall (never misses an at-risk patient), even if the\n",
    "precision is a bit low (sometimes you have false positives).\n",
    "\n",
    "On the other hand, if your algorithm filters spam emails out of an inbox,\n",
    "you may prefer a model with high precision so that when an email is\n",
    "classified as spam, it is very likely to actually be spam (i.e. non-spam\n",
    "messages don’t get sent to spam folder).\n",
    "\n",
    "In many settings, both precision and recall are equally important and a\n",
    "compound metric known as the F1-score is used:\n",
    "\n",
    "$$\n",
    "F1 = 2 \\frac{\\text{precision} \\cdot \\text{recall}}{\\text{precision} + \\text{recall}}\n",
    "$$\n",
    "\n",
    "The F1 score is bounded between 0 and 1. It will only achieve a value of 1 if\n",
    "both precision and recall are exactly 1.\n",
    "\n",
    "We can have scikit-learn produce a textual report with precision and recall.\n",
    "\n",
    "Scikit-learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hide-output": false
   },
   "outputs": [],
   "source": [
    "report = metrics.classification_report(\n",
    "    y_test1, logistic_model1.predict(X_test1),\n",
    "    target_names=[\"no recid\", \"recid\"]\n",
    ")\n",
    "print(report)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "report = metrics.classification_report(\n",
    "    y_test, logistic_age_sex_model.predict(X_test),\n",
    "    target_names=[\"no recid\", \"recid\"]\n",
    ")\n",
    "print(report)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "date": 1596739269.8065376,
  "filename": "classification.rst",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "next_doc": {
   "link": "working_with_text",
   "title": "Working with Text"
  },
  "prev_doc": {
   "link": "maps",
   "title": "Mapping in Python"
  },
  "title": "Classification"
 },
 "nbformat": 4,
 "nbformat_minor": 2
}