{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Test3: (40 marks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Your ID: 6204641010 Phatcharida Amornborirak"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this excercise, we aimed at the case of customers default payments in Taiwan and compares the predictive accuracy of probability of default among data mining methods. From the perspective of risk management, the result of predictive accuracy of the estimated probability of default will be more valuable than the binary result of classification - credible or not credible clients. With the real probability of default as the response variable (Y), and the predictive probability of default as the independent variable (X), this research employed a binary variable, default payment (Yes = 1, No = 0), as the response variable. This study reviewed the literature and used the following 23 variables as explanatory variables: \\\\\n",
    "\n",
    "X1: Amount of the given credit (NT dollar): it includes both the individual consumer credit and his/her family (supplementary) credit. \\\\\n",
    "\n",
    "X2: Gender (1 = male; 2 = female). \\\\\n",
    "\n",
    "X3: Education (1 = graduate school; 2 = university; 3 = high school; 4 = others). \\\\\n",
    "\n",
    "X4: Marital status (1 = married; 2 = single; 3 = others). \\\\\n",
    "\n",
    "X5: Age (year). \\\\\n",
    "\n",
    "X6 - X11: History of past payment. We tracked the past monthly payment records (from April to September, 2005) as follows: X6 = the repayment status in September, 2005; X7 = the repayment status in August, 2005; . . .;X11 = the repayment status in April, 2005. The measurement scale for the repayment status is: -1 = pay duly; 1 = payment delay for one month; 2 = payment delay for two months; . . .; 8 = payment delay for eight months; 9 = payment delay for nine months and above. \\\\\n",
    "\n",
    "X12-X17: Amount of bill statement (NT dollar). X12 = amount of bill statement in September, 2005; X13 = amount of bill statement in August, 2005; . . .; X17 = amount of bill statement in April, 2005. \\\\\n",
    "\n",
    "X18-X23: Amount of previous payment (NT dollar). X18 = amount paid in September, 2005; X19 = amount paid in August, 2005; . . .;X23 = amount paid in April, 2005. \\\\\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Learning the Logistic Regression, KNN, in Python and Scikit-Learn  \n",
    "\n",
    "There are 4 sub-question:\n",
    "\n",
    "1.1 Create the table to report the proportion of case of customers default payments in Taiwan separated by gender, education, and marital status. What is/are the interesting result/s you can draw from this table?[10 Points].\\\\"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import datetime\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "#import pandas_datareader.data as web\n",
    "\n",
    "from sklearn import (\n",
    "    linear_model, metrics, pipeline, preprocessing, model_selection\n",
    ")\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>X1</th>\n",
       "      <th>X2</th>\n",
       "      <th>X3</th>\n",
       "      <th>X4</th>\n",
       "      <th>X5</th>\n",
       "      <th>X6</th>\n",
       "      <th>X7</th>\n",
       "      <th>X8</th>\n",
       "      <th>X9</th>\n",
       "      <th>...</th>\n",
       "      <th>X15</th>\n",
       "      <th>X16</th>\n",
       "      <th>X17</th>\n",
       "      <th>X18</th>\n",
       "      <th>X19</th>\n",
       "      <th>X20</th>\n",
       "      <th>X21</th>\n",
       "      <th>X22</th>\n",
       "      <th>X23</th>\n",
       "      <th>Y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>20000</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>24</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>689</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>120000</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>26</td>\n",
       "      <td>-1</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>3272</td>\n",
       "      <td>3455</td>\n",
       "      <td>3261</td>\n",
       "      <td>0</td>\n",
       "      <td>1000</td>\n",
       "      <td>1000</td>\n",
       "      <td>1000</td>\n",
       "      <td>0</td>\n",
       "      <td>2000</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>90000</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>34</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>14331</td>\n",
       "      <td>14948</td>\n",
       "      <td>15549</td>\n",
       "      <td>1518</td>\n",
       "      <td>1500</td>\n",
       "      <td>1000</td>\n",
       "      <td>1000</td>\n",
       "      <td>1000</td>\n",
       "      <td>5000</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>50000</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>37</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>28314</td>\n",
       "      <td>28959</td>\n",
       "      <td>29547</td>\n",
       "      <td>2000</td>\n",
       "      <td>2019</td>\n",
       "      <td>1200</td>\n",
       "      <td>1100</td>\n",
       "      <td>1069</td>\n",
       "      <td>1000</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>50000</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>57</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>20940</td>\n",
       "      <td>19146</td>\n",
       "      <td>19131</td>\n",
       "      <td>2000</td>\n",
       "      <td>36681</td>\n",
       "      <td>10000</td>\n",
       "      <td>9000</td>\n",
       "      <td>689</td>\n",
       "      <td>679</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29995</th>\n",
       "      <td>29996</td>\n",
       "      <td>220000</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>39</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>88004</td>\n",
       "      <td>31237</td>\n",
       "      <td>15980</td>\n",
       "      <td>8500</td>\n",
       "      <td>20000</td>\n",
       "      <td>5003</td>\n",
       "      <td>3047</td>\n",
       "      <td>5000</td>\n",
       "      <td>1000</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29996</th>\n",
       "      <td>29997</td>\n",
       "      <td>150000</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>43</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>...</td>\n",
       "      <td>8979</td>\n",
       "      <td>5190</td>\n",
       "      <td>0</td>\n",
       "      <td>1837</td>\n",
       "      <td>3526</td>\n",
       "      <td>8998</td>\n",
       "      <td>129</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29997</th>\n",
       "      <td>29998</td>\n",
       "      <td>30000</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>37</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>-1</td>\n",
       "      <td>...</td>\n",
       "      <td>20878</td>\n",
       "      <td>20582</td>\n",
       "      <td>19357</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>22000</td>\n",
       "      <td>4200</td>\n",
       "      <td>2000</td>\n",
       "      <td>3100</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29998</th>\n",
       "      <td>29999</td>\n",
       "      <td>80000</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>41</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>52774</td>\n",
       "      <td>11855</td>\n",
       "      <td>48944</td>\n",
       "      <td>85900</td>\n",
       "      <td>3409</td>\n",
       "      <td>1178</td>\n",
       "      <td>1926</td>\n",
       "      <td>52964</td>\n",
       "      <td>1804</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29999</th>\n",
       "      <td>30000</td>\n",
       "      <td>50000</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>46</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>36535</td>\n",
       "      <td>32428</td>\n",
       "      <td>15313</td>\n",
       "      <td>2078</td>\n",
       "      <td>1800</td>\n",
       "      <td>1430</td>\n",
       "      <td>1000</td>\n",
       "      <td>1000</td>\n",
       "      <td>1000</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>30000 rows × 25 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Unnamed: 0      X1  X2  X3  X4  X5  X6  X7  X8  X9  ...    X15    X16  \\\n",
       "0               1   20000   2   2   1  24   2   2  -1  -1  ...      0      0   \n",
       "1               2  120000   2   2   2  26  -1   2   0   0  ...   3272   3455   \n",
       "2               3   90000   2   2   2  34   0   0   0   0  ...  14331  14948   \n",
       "3               4   50000   2   2   1  37   0   0   0   0  ...  28314  28959   \n",
       "4               5   50000   1   2   1  57  -1   0  -1   0  ...  20940  19146   \n",
       "...           ...     ...  ..  ..  ..  ..  ..  ..  ..  ..  ...    ...    ...   \n",
       "29995       29996  220000   1   3   1  39   0   0   0   0  ...  88004  31237   \n",
       "29996       29997  150000   1   3   2  43  -1  -1  -1  -1  ...   8979   5190   \n",
       "29997       29998   30000   1   2   2  37   4   3   2  -1  ...  20878  20582   \n",
       "29998       29999   80000   1   3   1  41   1  -1   0   0  ...  52774  11855   \n",
       "29999       30000   50000   1   2   1  46   0   0   0   0  ...  36535  32428   \n",
       "\n",
       "         X17    X18    X19    X20   X21    X22   X23  Y  \n",
       "0          0      0    689      0     0      0     0  1  \n",
       "1       3261      0   1000   1000  1000      0  2000  1  \n",
       "2      15549   1518   1500   1000  1000   1000  5000  0  \n",
       "3      29547   2000   2019   1200  1100   1069  1000  0  \n",
       "4      19131   2000  36681  10000  9000    689   679  0  \n",
       "...      ...    ...    ...    ...   ...    ...   ... ..  \n",
       "29995  15980   8500  20000   5003  3047   5000  1000  0  \n",
       "29996      0   1837   3526   8998   129      0     0  0  \n",
       "29997  19357      0      0  22000  4200   2000  3100  1  \n",
       "29998  48944  85900   3409   1178  1926  52964  1804  1  \n",
       "29999  15313   2078   1800   1430  1000   1000  1000  1  \n",
       "\n",
       "[30000 rows x 25 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_excel('default of credit card clients.xls')\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>X1</th>\n",
       "      <th>X2</th>\n",
       "      <th>X3</th>\n",
       "      <th>X4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>20000</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>120000</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>90000</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>50000</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>50000</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29995</th>\n",
       "      <td>220000</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29996</th>\n",
       "      <td>150000</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29997</th>\n",
       "      <td>30000</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29998</th>\n",
       "      <td>80000</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29999</th>\n",
       "      <td>50000</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>30000 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           X1  X2  X3  X4\n",
       "0       20000   2   2   1\n",
       "1      120000   2   2   2\n",
       "2       90000   2   2   2\n",
       "3       50000   2   2   1\n",
       "4       50000   1   2   1\n",
       "...       ...  ..  ..  ..\n",
       "29995  220000   1   3   1\n",
       "29996  150000   1   3   2\n",
       "29997   30000   1   2   2\n",
       "29998   80000   1   3   1\n",
       "29999   50000   1   2   1\n",
       "\n",
       "[30000 rows x 4 columns]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1 = df[['X1','X2','X3', 'X4']]\n",
    "df1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "#From the result, the gender in the table seems to be quite distributed both in male and female\n",
    "#The education mostly is 2 which is University level on the upper part and some High school level in the lower part\n",
    "#The maritial status are only seen by 1 and 2 which is single and married.\n",
    "#The amount of credit is quite distribited as we can't see the pattern from only looking into this table."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, partition the data into training and test sets using 80 $\\%$ to be the train data. The model will be fit to the training data and evaluated on the test set. \n",
    "\n",
    "1.2 Use the Logistic regression to train the model with all features. Then, use the test data to conduct the confusion matrix. Interpret the results carefully [10 Points]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Unnamed: 0',\n",
       " 'X1',\n",
       " 'X2',\n",
       " 'X3',\n",
       " 'X4',\n",
       " 'X5',\n",
       " 'X6',\n",
       " 'X7',\n",
       " 'X8',\n",
       " 'X9',\n",
       " 'X10',\n",
       " 'X11',\n",
       " 'X12',\n",
       " 'X13',\n",
       " 'X14',\n",
       " 'X15',\n",
       " 'X16',\n",
       " 'X17',\n",
       " 'X18',\n",
       " 'X19',\n",
       " 'X20',\n",
       " 'X21',\n",
       " 'X22',\n",
       " 'X23',\n",
       " 'Y']"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df[['X2','X3', 'X4', 'X5', 'X6', 'X7', 'X8', 'X9', 'X10', 'X11', 'X12', 'X13', 'X14', 'X15', 'X16', 'X17', 'X18', 'X19', \\\n",
    "        'X20', 'X21', 'X22', 'X23',]]\n",
    "y = df[\"X1\"]\n",
    "\n",
    "X_train1, X_test1, y_train1, y_test1 = model_selection.train_test_split(X, y, test_size=0.2, random_state = 999)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\linear_model\\_logistic.py:814: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  n_iter_i = _check_optimize_result(\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "LogisticRegression()"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "logistic_model = linear_model.LogisticRegression()\n",
    "logistic_model.fit(X_train1, y_train1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.08054166666666666, 0.0765)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_acc = logistic_model.score(X_train1, y_train1)\n",
    "test_acc = logistic_model.score(X_test1, y_test1)\n",
    "\n",
    "train_acc, test_acc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#The accuracy of train data is higher than test data, but the result is very low in both."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "Number of classes, 71, does not match size of target_names, 22. Try specifying the labels parameter",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[1;32m~\\AppData\\Local\\Temp\\ipykernel_8872\\1740372991.py\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m report = metrics.classification_report(\n\u001b[0m\u001b[0;32m      2\u001b[0m     \u001b[0my_test1\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mlogistic_model\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mpredict\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mX_test1\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      3\u001b[0m     target_names= ['X2','X3', 'X4', 'X5', 'X6', 'X7', 'X8', 'X9', 'X10', 'X11', 'X12', 'X13', 'X14', 'X15', 'X16', 'X17', \\\n\u001b[0;32m      4\u001b[0m                   'X18', 'X19', 'X20', 'X21', 'X22', 'X23']\n\u001b[0;32m      5\u001b[0m )\n",
      "\u001b[1;32mC:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\metrics\\_classification.py\u001b[0m in \u001b[0;36mclassification_report\u001b[1;34m(y_true, y_pred, labels, target_names, sample_weight, digits, output_dict, zero_division)\u001b[0m\n\u001b[0;32m   2130\u001b[0m             )\n\u001b[0;32m   2131\u001b[0m         \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 2132\u001b[1;33m             raise ValueError(\n\u001b[0m\u001b[0;32m   2133\u001b[0m                 \u001b[1;34m\"Number of classes, {0}, does not match size of \"\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   2134\u001b[0m                 \u001b[1;34m\"target_names, {1}. Try specifying the labels \"\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mValueError\u001b[0m: Number of classes, 71, does not match size of target_names, 22. Try specifying the labels parameter"
     ]
    }
   ],
   "source": [
    "report = metrics.classification_report(\n",
    "    y_test1, logistic_model.predict(X_test1),\n",
    "    target_names= ['X2','X3', 'X4', 'X5', 'X6', 'X7', 'X8', 'X9', 'X10', 'X11', 'X12', 'X13', 'X14', 'X15', 'X16', 'X17', \\\n",
    "                  'X18', 'X19', 'X20', 'X21', 'X22', 'X23']\n",
    ")\n",
    "print(report)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1.3 Use the KNN to train the model with all features (you should convert the features to be the standarized variables first). Then, use the test data to conduct the confusion matrix. Interpret the results carefully [10 Points]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "from pathlib import Path\n",
    "\n",
    "import pandas as pd\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.neural_network import MLPClassifier\n",
    "\n",
    "from sklearn import metrics #evaluation model.\n",
    "from dmba import classificationSummary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\neural_network\\_multilayer_perceptron.py:549: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "  self.n_iter_ = _check_optimize_result(\"lbfgs\", opt_res, self.max_iter)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Intercepts\n",
      "[array([0.08700564, 0.00896977, 0.25940628]), array([ 0.50388308, -1.34950912,  1.99069438,  1.69110543,  0.09522247,\n",
      "        2.2403276 ,  1.03348093,  0.72033312,  1.87141249,  0.83904904,\n",
      "        1.2610428 ,  0.55135622,  1.16368003,  1.01355828,  1.0097891 ,\n",
      "        1.8696823 ,  1.28530141,  0.82936358,  1.53325018,  0.21492019,\n",
      "        2.23733874,  1.32162504,  0.64451447,  1.26010546,  0.96776646,\n",
      "        0.54083864,  0.84945958,  0.29896506,  0.86997446,  0.55755479,\n",
      "        1.07707221,  0.39298369,  0.35078823, -1.18704694,  0.21830163,\n",
      "        0.17973617,  0.36337527,  1.83209428, -0.39085213,  0.072447  ,\n",
      "        0.11391585,  0.44071511, -0.29696227, -0.02132018, -0.34389163,\n",
      "       -0.2687085 ,  0.32296931, -0.23511355, -0.43765482, -0.48291675,\n",
      "       -0.50843245,  1.2756548 , -0.92182297, -0.88386787, -1.03822287,\n",
      "       -1.14710817, -0.98551629, -1.12141119, -1.23168177, -1.13539896,\n",
      "       -1.15080179, -1.06619031, -1.08898683, -1.02533651, -0.93170265,\n",
      "       -1.10333577, -1.00051719, -1.34255935, -1.16955582, -1.13908171,\n",
      "       -1.16679123, -1.12192257, -1.06974545, -1.15425056, -1.09476265,\n",
      "       -1.19276753, -1.13643654, -1.23556961, -1.24617414, -1.27502365,\n",
      "       -1.20980622])]\n",
      "Weights\n",
      "[array([[-0.05476927,  0.12148816, -0.26828429],\n",
      "       [-0.12282898, -0.15468414, -0.21722487],\n",
      "       [-0.18620009, -0.07163593, -0.04631795],\n",
      "       [-0.1863089 , -0.35325132,  0.38161624],\n",
      "       [-0.1693761 ,  0.25473355, -0.26815672],\n",
      "       [ 0.10904775, -0.04410695,  0.01795779],\n",
      "       [-0.19036204, -0.16938856,  0.15487579],\n",
      "       [ 0.27765047, -0.10441609,  0.0932906 ],\n",
      "       [ 0.22600905,  0.22392872, -0.24990926],\n",
      "       [-0.24803072, -0.18559341,  0.19863792],\n",
      "       [-0.54308506,  1.25965602,  0.52304768],\n",
      "       [-0.98776249,  0.28611659,  0.22385608],\n",
      "       [ 0.18061218,  0.729446  , -0.05676491],\n",
      "       [ 0.27785931, -0.87157968,  0.32124701],\n",
      "       [ 0.74489132, -0.14439447,  0.27756372],\n",
      "       [ 0.63361214,  0.7782607 , -0.02560277],\n",
      "       [ 0.81024612, -1.51347058, -0.10611512],\n",
      "       [ 1.3321709 ,  0.2793382 ,  0.06964115],\n",
      "       [ 0.33367768, -1.70499664,  0.35954772],\n",
      "       [ 0.5206874 , -1.14734659,  0.83403093],\n",
      "       [ 0.20217665, -0.37185795,  0.54181666],\n",
      "       [-0.73286902, -0.20144294,  0.31414562]]), array([[ 6.16684405e-01, -3.01277423e-01,  7.84169136e-01,\n",
      "         8.09714029e-01,  6.05904702e-01, -4.42491026e-01,\n",
      "        -1.72499730e-01,  1.42591952e-01, -1.33280108e-01,\n",
      "        -3.93172087e-03,  3.61974136e-01,  2.60539255e-01,\n",
      "         2.34154498e-01, -2.75494078e-02,  3.73868663e-01,\n",
      "        -8.13177504e-02,  1.40376990e-01,  2.68550093e-01,\n",
      "         1.72923161e-01,  5.25914577e-01,  2.62030486e-01,\n",
      "         5.19478738e-02,  3.22561260e-01,  2.75162776e-01,\n",
      "         2.00738340e-01,  2.87794929e-01,  3.54528108e-01,\n",
      "         4.23668233e-01,  3.40032083e-01,  1.79630569e-01,\n",
      "        -1.90510531e-01,  2.48196950e-01,  3.67348359e-01,\n",
      "        -3.30035147e-01,  3.33903235e-01,  4.45365631e-01,\n",
      "         3.36890335e-01, -4.28300705e-01,  1.29164112e-01,\n",
      "         3.77970103e-01,  4.48685820e-01,  2.97262104e-01,\n",
      "         4.75600474e-02,  3.97123305e-01,  2.06820917e-01,\n",
      "         1.43553375e-02,  7.55372750e-02,  2.52915542e-01,\n",
      "         2.34919406e-01,  2.36825397e-01,  1.01276998e-03,\n",
      "         3.74699047e-01, -3.15073979e-01, -3.47641504e-01,\n",
      "        -3.81079811e-01, -2.76403325e-01, -3.19887899e-01,\n",
      "        -3.12779674e-01, -3.00770813e-01, -3.63740181e-01,\n",
      "        -3.82714115e-01, -2.39205913e-01, -1.69936000e-01,\n",
      "        -2.90422693e-01, -4.46760030e-01, -2.97782251e-01,\n",
      "        -4.29845967e-01, -3.36844828e-01, -3.85187787e-01,\n",
      "        -2.93147049e-01, -3.81224542e-01, -3.57111591e-01,\n",
      "        -4.39205106e-01, -4.72632833e-01, -4.82299306e-01,\n",
      "        -4.78248906e-01, -4.30738830e-01, -4.60984256e-01,\n",
      "        -3.20319868e-01, -4.14270153e-01, -5.10821834e-01],\n",
      "       [ 1.19778453e+00, -7.79064864e-01,  1.62095580e+00,\n",
      "         1.69318025e+00,  6.81477476e-01,  1.66038299e+00,\n",
      "         1.78308770e+00,  1.96227064e+00,  1.27385627e+00,\n",
      "         1.33008553e+00,  1.30131611e+00,  1.58492609e+00,\n",
      "         7.33305086e-01,  1.09747666e+00,  1.18439359e+00,\n",
      "         5.46829629e-01,  3.16839776e-01,  7.13487942e-01,\n",
      "         3.26814515e-01,  6.53517017e-01, -1.93909424e-02,\n",
      "         3.14302438e-01,  5.82953799e-01,  3.17017216e-01,\n",
      "         5.39357478e-01,  3.69786956e-01,  3.75027365e-01,\n",
      "         3.78993708e-01,  4.24228939e-01,  4.26042061e-01,\n",
      "         1.37406886e-01,  4.16259227e-01,  4.23787436e-01,\n",
      "        -9.99658795e-01,  1.84008119e-01,  2.40527916e-01,\n",
      "         1.91049330e-01, -4.38931728e-01, -2.99521475e-01,\n",
      "         1.98792256e-01,  2.80750676e-01,  9.51036203e-02,\n",
      "        -1.65753811e-01,  2.52850694e-01, -2.63024841e-01,\n",
      "        -9.88187689e-03, -2.52982904e-01, -3.10738862e-01,\n",
      "        -9.96573422e-02, -1.51557585e-01, -3.22315742e-01,\n",
      "         5.68891817e-02, -6.78286817e-01, -6.50800550e-01,\n",
      "        -8.77018413e-01, -8.82778687e-01, -6.45446595e-01,\n",
      "        -7.25555102e-01, -7.61437849e-01, -5.77998154e-01,\n",
      "        -7.89692839e-01, -7.26866937e-01, -7.63279564e-01,\n",
      "        -8.20347459e-01, -8.79635377e-01, -8.44449887e-01,\n",
      "        -9.16694504e-01, -7.89118425e-01, -7.61396956e-01,\n",
      "        -9.47819378e-01, -1.03744020e+00, -7.99443956e-01,\n",
      "        -8.17969053e-01, -7.25425375e-01, -9.02565492e-01,\n",
      "        -7.69961924e-01, -7.32853611e-01, -8.16863231e-01,\n",
      "        -7.64185235e-01, -9.18677407e-01, -8.52410363e-01],\n",
      "       [ 1.00408202e-01, -5.24766989e-01, -5.25518317e-01,\n",
      "        -5.10737708e-01,  1.97388581e-01,  5.22147308e-01,\n",
      "         1.47952034e-01, -3.38542656e-02,  2.75382312e-01,\n",
      "         3.20173374e-01,  1.47583776e-01,  1.17807808e-01,\n",
      "         4.18198988e-01,  4.69743836e-01,  1.86948206e-01,\n",
      "         4.41549727e-01,  5.88810151e-01,  4.24341718e-01,\n",
      "         6.90756903e-01,  1.31708251e-01,  5.06369048e-01,\n",
      "         6.51396266e-01,  5.51207194e-01,  5.90796463e-01,\n",
      "         6.02050210e-01,  5.01897963e-01,  5.60896646e-01,\n",
      "         3.07618419e-01,  4.59811437e-01,  5.23746537e-01,\n",
      "         8.92149049e-01,  4.25845173e-01,  4.69502389e-01,\n",
      "        -5.51055109e-01,  3.64407059e-01,  4.43213539e-01,\n",
      "         4.17644057e-01,  1.08450822e+00,  2.95009070e-02,\n",
      "         3.84814316e-01,  3.11189837e-01,  5.12178392e-01,\n",
      "         1.21524295e-01,  4.76250951e-01,  1.64044903e-01,\n",
      "         1.52759767e-01,  5.29543335e-01,  5.22730331e-02,\n",
      "         1.67982451e-01,  1.79343226e-01,  8.87399937e-02,\n",
      "         6.22590484e-01, -4.39609390e-01, -4.08281899e-01,\n",
      "        -4.50331723e-01, -5.47390386e-01, -3.10253652e-01,\n",
      "        -4.53448872e-01, -4.74550953e-01, -4.29874970e-01,\n",
      "        -4.56540497e-01, -4.34366797e-01, -4.79828401e-01,\n",
      "        -5.21709493e-01, -5.97776491e-01, -5.68497633e-01,\n",
      "        -6.37596026e-01, -4.77933699e-01, -6.20924644e-01,\n",
      "        -5.19831563e-01, -5.63401250e-01, -4.58479553e-01,\n",
      "        -4.80793985e-01, -5.95631980e-01, -5.79288831e-01,\n",
      "        -5.85856805e-01, -5.61577195e-01, -5.62495803e-01,\n",
      "        -6.08888819e-01, -5.23313565e-01, -4.77322816e-01]])]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>X1</th>\n",
       "      <th>X2</th>\n",
       "      <th>X3</th>\n",
       "      <th>X4</th>\n",
       "      <th>X5</th>\n",
       "      <th>X6</th>\n",
       "      <th>X7</th>\n",
       "      <th>X8</th>\n",
       "      <th>X9</th>\n",
       "      <th>...</th>\n",
       "      <th>700000</th>\n",
       "      <th>710000</th>\n",
       "      <th>720000</th>\n",
       "      <th>730000</th>\n",
       "      <th>740000</th>\n",
       "      <th>750000</th>\n",
       "      <th>760000</th>\n",
       "      <th>780000</th>\n",
       "      <th>800000</th>\n",
       "      <th>1000000</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>20000</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>24</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000207</td>\n",
       "      <td>0.000210</td>\n",
       "      <td>0.000188</td>\n",
       "      <td>0.000170</td>\n",
       "      <td>0.000175</td>\n",
       "      <td>0.000197</td>\n",
       "      <td>0.000164</td>\n",
       "      <td>0.000163</td>\n",
       "      <td>0.000148</td>\n",
       "      <td>0.000177</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>120000</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>26</td>\n",
       "      <td>-1</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000125</td>\n",
       "      <td>0.000116</td>\n",
       "      <td>0.000101</td>\n",
       "      <td>0.000091</td>\n",
       "      <td>0.000093</td>\n",
       "      <td>0.000110</td>\n",
       "      <td>0.000089</td>\n",
       "      <td>0.000102</td>\n",
       "      <td>0.000084</td>\n",
       "      <td>0.000091</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>90000</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>34</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000207</td>\n",
       "      <td>0.000210</td>\n",
       "      <td>0.000188</td>\n",
       "      <td>0.000170</td>\n",
       "      <td>0.000175</td>\n",
       "      <td>0.000197</td>\n",
       "      <td>0.000164</td>\n",
       "      <td>0.000163</td>\n",
       "      <td>0.000148</td>\n",
       "      <td>0.000177</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>50000</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>37</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000207</td>\n",
       "      <td>0.000210</td>\n",
       "      <td>0.000188</td>\n",
       "      <td>0.000170</td>\n",
       "      <td>0.000175</td>\n",
       "      <td>0.000197</td>\n",
       "      <td>0.000164</td>\n",
       "      <td>0.000163</td>\n",
       "      <td>0.000148</td>\n",
       "      <td>0.000177</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>50000</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>57</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000125</td>\n",
       "      <td>0.000116</td>\n",
       "      <td>0.000101</td>\n",
       "      <td>0.000091</td>\n",
       "      <td>0.000093</td>\n",
       "      <td>0.000110</td>\n",
       "      <td>0.000089</td>\n",
       "      <td>0.000102</td>\n",
       "      <td>0.000084</td>\n",
       "      <td>0.000091</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29995</th>\n",
       "      <td>29996</td>\n",
       "      <td>220000</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>39</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000207</td>\n",
       "      <td>0.000210</td>\n",
       "      <td>0.000188</td>\n",
       "      <td>0.000170</td>\n",
       "      <td>0.000175</td>\n",
       "      <td>0.000197</td>\n",
       "      <td>0.000164</td>\n",
       "      <td>0.000163</td>\n",
       "      <td>0.000148</td>\n",
       "      <td>0.000177</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29996</th>\n",
       "      <td>29997</td>\n",
       "      <td>150000</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>43</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000596</td>\n",
       "      <td>0.000566</td>\n",
       "      <td>0.000448</td>\n",
       "      <td>0.000479</td>\n",
       "      <td>0.000433</td>\n",
       "      <td>0.000492</td>\n",
       "      <td>0.000432</td>\n",
       "      <td>0.000470</td>\n",
       "      <td>0.000453</td>\n",
       "      <td>0.000459</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29997</th>\n",
       "      <td>29998</td>\n",
       "      <td>30000</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>37</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>-1</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000596</td>\n",
       "      <td>0.000566</td>\n",
       "      <td>0.000448</td>\n",
       "      <td>0.000479</td>\n",
       "      <td>0.000433</td>\n",
       "      <td>0.000492</td>\n",
       "      <td>0.000432</td>\n",
       "      <td>0.000470</td>\n",
       "      <td>0.000453</td>\n",
       "      <td>0.000459</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29998</th>\n",
       "      <td>29999</td>\n",
       "      <td>80000</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>41</td>\n",
       "      <td>1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000596</td>\n",
       "      <td>0.000566</td>\n",
       "      <td>0.000448</td>\n",
       "      <td>0.000479</td>\n",
       "      <td>0.000433</td>\n",
       "      <td>0.000492</td>\n",
       "      <td>0.000432</td>\n",
       "      <td>0.000470</td>\n",
       "      <td>0.000453</td>\n",
       "      <td>0.000459</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29999</th>\n",
       "      <td>30000</td>\n",
       "      <td>50000</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>46</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000207</td>\n",
       "      <td>0.000210</td>\n",
       "      <td>0.000188</td>\n",
       "      <td>0.000170</td>\n",
       "      <td>0.000175</td>\n",
       "      <td>0.000197</td>\n",
       "      <td>0.000164</td>\n",
       "      <td>0.000163</td>\n",
       "      <td>0.000148</td>\n",
       "      <td>0.000177</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>30000 rows × 106 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Unnamed: 0      X1  X2  X3  X4  X5  X6  X7  X8  X9  ...    700000  \\\n",
       "0               1   20000   2   2   1  24   2   2  -1  -1  ...  0.000207   \n",
       "1               2  120000   2   2   2  26  -1   2   0   0  ...  0.000125   \n",
       "2               3   90000   2   2   2  34   0   0   0   0  ...  0.000207   \n",
       "3               4   50000   2   2   1  37   0   0   0   0  ...  0.000207   \n",
       "4               5   50000   1   2   1  57  -1   0  -1   0  ...  0.000125   \n",
       "...           ...     ...  ..  ..  ..  ..  ..  ..  ..  ..  ...       ...   \n",
       "29995       29996  220000   1   3   1  39   0   0   0   0  ...  0.000207   \n",
       "29996       29997  150000   1   3   2  43  -1  -1  -1  -1  ...  0.000596   \n",
       "29997       29998   30000   1   2   2  37   4   3   2  -1  ...  0.000596   \n",
       "29998       29999   80000   1   3   1  41   1  -1   0   0  ...  0.000596   \n",
       "29999       30000   50000   1   2   1  46   0   0   0   0  ...  0.000207   \n",
       "\n",
       "         710000    720000    730000    740000    750000    760000    780000  \\\n",
       "0      0.000210  0.000188  0.000170  0.000175  0.000197  0.000164  0.000163   \n",
       "1      0.000116  0.000101  0.000091  0.000093  0.000110  0.000089  0.000102   \n",
       "2      0.000210  0.000188  0.000170  0.000175  0.000197  0.000164  0.000163   \n",
       "3      0.000210  0.000188  0.000170  0.000175  0.000197  0.000164  0.000163   \n",
       "4      0.000116  0.000101  0.000091  0.000093  0.000110  0.000089  0.000102   \n",
       "...         ...       ...       ...       ...       ...       ...       ...   \n",
       "29995  0.000210  0.000188  0.000170  0.000175  0.000197  0.000164  0.000163   \n",
       "29996  0.000566  0.000448  0.000479  0.000433  0.000492  0.000432  0.000470   \n",
       "29997  0.000566  0.000448  0.000479  0.000433  0.000492  0.000432  0.000470   \n",
       "29998  0.000566  0.000448  0.000479  0.000433  0.000492  0.000432  0.000470   \n",
       "29999  0.000210  0.000188  0.000170  0.000175  0.000197  0.000164  0.000163   \n",
       "\n",
       "         800000   1000000  \n",
       "0      0.000148  0.000177  \n",
       "1      0.000084  0.000091  \n",
       "2      0.000148  0.000177  \n",
       "3      0.000148  0.000177  \n",
       "4      0.000084  0.000091  \n",
       "...         ...       ...  \n",
       "29995  0.000148  0.000177  \n",
       "29996  0.000453  0.000459  \n",
       "29997  0.000453  0.000459  \n",
       "29998  0.000453  0.000459  \n",
       "29999  0.000148  0.000177  \n",
       "\n",
       "[30000 rows x 106 columns]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictors = ['X2','X3', 'X4', 'X5', 'X6', 'X7', 'X8', 'X9', 'X10', 'X11', 'X12', 'X13', 'X14', 'X15', 'X16', 'X17', \\\n",
    "              'X18', 'X19', 'X20', 'X21', 'X22', 'X23']\n",
    "outcome = 'X1'\n",
    "\n",
    "X = df[predictors]\n",
    "y = df[outcome]\n",
    "classes = sorted(y.unique())\n",
    "\n",
    "clf = MLPClassifier(hidden_layer_sizes=(3), activation='logistic', solver='lbfgs', random_state=1)\n",
    "clf.fit(X, y)\n",
    "clf.predict(X)\n",
    "\n",
    "# Network structure\n",
    "print('Intercepts')\n",
    "print(clf.intercepts_)\n",
    "\n",
    "print('Weights')\n",
    "print(clf.coefs_)\n",
    "\n",
    "# Prediction\n",
    "pd.concat([\n",
    "    df,\n",
    "    pd.DataFrame(clf.predict_proba(X), columns=classes)\n",
    "], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "       10000       0.00      0.00      0.00       493\n",
      "       16000       0.00      0.00      0.00         2\n",
      "       20000       0.12      0.00      0.00      1976\n",
      "       30000       0.00      0.00      0.00      1610\n",
      "       40000       0.00      0.00      0.00       230\n",
      "       50000       0.14      0.88      0.24      3365\n",
      "       60000       0.00      0.00      0.00       825\n",
      "       70000       0.00      0.00      0.00       731\n",
      "       80000       0.00      0.00      0.00      1567\n",
      "       90000       0.00      0.00      0.00       651\n",
      "      100000       0.00      0.00      0.00      1048\n",
      "      110000       0.00      0.00      0.00       588\n",
      "      120000       0.00      0.00      0.00       726\n",
      "      130000       0.00      0.00      0.00       729\n",
      "      140000       0.00      0.00      0.00       749\n",
      "      150000       0.00      0.00      0.00      1110\n",
      "      160000       0.00      0.00      0.00       694\n",
      "      170000       0.00      0.00      0.00       532\n",
      "      180000       0.00      0.00      0.00       995\n",
      "      190000       0.00      0.00      0.00       229\n",
      "      200000       0.08      0.33      0.13      1528\n",
      "      210000       0.00      0.00      0.00       730\n",
      "      220000       0.00      0.00      0.00       469\n",
      "      230000       0.00      0.00      0.00       737\n",
      "      240000       0.00      0.00      0.00       619\n",
      "      250000       0.00      0.00      0.00       350\n",
      "      260000       0.00      0.00      0.00       521\n",
      "      270000       0.00      0.00      0.00       238\n",
      "      280000       0.00      0.00      0.00       493\n",
      "      290000       0.00      0.00      0.00       348\n",
      "      300000       0.00      0.00      0.00       554\n",
      "      310000       0.00      0.00      0.00       272\n",
      "      320000       0.00      0.00      0.00       312\n",
      "      327680       0.00      0.00      0.00         1\n",
      "      330000       0.00      0.00      0.00       173\n",
      "      340000       0.00      0.00      0.00       217\n",
      "      350000       0.00      0.00      0.00       231\n",
      "      360000       0.10      0.21      0.13       881\n",
      "      370000       0.00      0.00      0.00        71\n",
      "      380000       0.00      0.00      0.00       156\n",
      "      390000       0.00      0.00      0.00       174\n",
      "      400000       0.00      0.00      0.00       271\n",
      "      410000       0.00      0.00      0.00        78\n",
      "      420000       0.00      0.00      0.00       168\n",
      "      430000       0.00      0.00      0.00        83\n",
      "      440000       0.00      0.00      0.00        83\n",
      "      450000       0.00      0.00      0.00       161\n",
      "      460000       0.00      0.00      0.00        80\n",
      "      470000       0.00      0.00      0.00        80\n",
      "      480000       0.00      0.00      0.00        79\n",
      "      490000       0.00      0.00      0.00        64\n",
      "      500000       0.00      0.00      0.00       722\n",
      "      510000       0.00      0.00      0.00        19\n",
      "      520000       0.00      0.00      0.00        20\n",
      "      530000       0.00      0.00      0.00        10\n",
      "      540000       0.00      0.00      0.00         6\n",
      "      550000       0.00      0.00      0.00        21\n",
      "      560000       0.00      0.00      0.00        10\n",
      "      570000       0.00      0.00      0.00         8\n",
      "      580000       0.00      0.00      0.00        11\n",
      "      590000       0.00      0.00      0.00         6\n",
      "      600000       0.00      0.00      0.00        16\n",
      "      610000       0.00      0.00      0.00        11\n",
      "      620000       0.00      0.00      0.00         9\n",
      "      630000       0.00      0.00      0.00         7\n",
      "      640000       0.00      0.00      0.00         7\n",
      "      650000       0.00      0.00      0.00         3\n",
      "      660000       0.00      0.00      0.00         3\n",
      "      670000       0.00      0.00      0.00         3\n",
      "      680000       0.00      0.00      0.00         4\n",
      "      690000       0.00      0.00      0.00         1\n",
      "      700000       0.00      0.00      0.00         8\n",
      "      710000       0.00      0.00      0.00         6\n",
      "      720000       0.00      0.00      0.00         3\n",
      "      730000       0.00      0.00      0.00         2\n",
      "      740000       0.00      0.00      0.00         2\n",
      "      750000       0.00      0.00      0.00         4\n",
      "      760000       0.00      0.00      0.00         1\n",
      "      780000       0.00      0.00      0.00         2\n",
      "      800000       0.00      0.00      0.00         2\n",
      "     1000000       0.00      0.00      0.00         1\n",
      "\n",
      "    accuracy                           0.12     30000\n",
      "   macro avg       0.01      0.02      0.01     30000\n",
      "weighted avg       0.03      0.12      0.04     30000\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\metrics\\_classification.py:1318: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\metrics\\_classification.py:1318: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\metrics\\_classification.py:1318: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.\n",
      "  _warn_prf(average, modifier, msg_start, len(result))\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report,confusion_matrix\n",
    "\n",
    "print(classification_report(y,clf.predict(X)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Confusion Matrix (Accuracy 0.1214)\n",
      "\n",
      "        Prediction\n",
      " Actual   10000   16000   20000   30000   40000   50000   60000   70000   80000   90000  100000  110000  120000  130000  140000  150000  160000  170000  180000  190000  200000  210000  220000  230000  240000  250000  260000  270000  280000  290000  300000  310000  320000  327680  330000  340000  350000  360000  370000  380000  390000  400000  410000  420000  430000  440000  450000  460000  470000  480000  490000  500000  510000  520000  530000  540000  550000  560000  570000  580000  590000  600000  610000  620000  630000  640000  650000  660000  670000  680000  690000  700000  710000  720000  730000  740000  750000  760000  780000  800000 1000000\n",
      "  10000       0       0       0       0       0     416       0       0       0       0       0       0       0       0       0       0       0       0       0       0      60       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0      17       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      "  16000       0       0       0       0       0       2       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      "  20000       0       0       3       0       0    1695       0       0       0       0       0       0       0       0       0       0       0       0       0       0     212       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0      66       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      "  30000       0       0       2       0       0    1375       0       0       0       0       0       0       0       0       0       0       0       0       0       0     188       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0      45       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      "  40000       0       0       1       0       0     196       0       0       0       0       0       0       0       0       0       0       0       0       0       0      26       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       7       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      "  50000       0       0       2       0       0    2953       0       0       0       0       0       0       0       0       0       0       0       0       0       0     296       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0     114       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      "  60000       0       0       0       0       0     721       0       0       0       0       0       0       0       0       0       0       0       0       0       0      88       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0      16       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      "  70000       0       0       1       0       0     682       0       0       0       0       0       0       0       0       0       0       0       0       0       0      43       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       5       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      "  80000       0       0       2       0       0    1256       0       0       0       0       0       0       0       0       0       0       0       0       0       0     235       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0      74       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      "  90000       0       0       0       0       0     538       0       0       0       0       0       0       0       0       0       0       0       0       0       0      84       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0      29       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 100000       0       0       2       0       0     839       0       0       0       0       0       0       0       0       0       0       0       0       0       0     161       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0      46       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 110000       0       0       0       0       0     522       0       0       0       0       0       0       0       0       0       0       0       0       0       0      50       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0      16       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 120000       0       0       1       0       0     504       0       0       0       0       0       0       0       0       0       0       0       0       0       0     170       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0      51       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 130000       0       0       0       0       0     553       0       0       0       0       0       0       0       0       0       0       0       0       0       0     125       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0      51       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 140000       0       0       0       0       0     585       0       0       0       0       0       0       0       0       0       0       0       0       0       0     127       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0      37       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 150000       0       0       0       0       0     746       0       0       0       0       0       0       0       0       0       0       0       0       0       0     269       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0      95       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 160000       0       0       0       0       0     409       0       0       0       0       0       0       0       0       0       0       0       0       0       0     220       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0      65       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 170000       0       0       1       0       0     361       0       0       0       0       0       0       0       0       0       0       0       0       0       0     134       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0      36       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 180000       0       0       1       0       0     622       0       0       0       0       0       0       0       0       0       0       0       0       0       0     283       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0      89       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 190000       0       0       0       0       0     184       0       0       0       0       0       0       0       0       0       0       0       0       0       0      35       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0      10       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 200000       0       0       2       0       0     891       0       0       0       0       0       0       0       0       0       0       0       0       0       0     503       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0     132       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 210000       0       0       0       0       0     427       0       0       0       0       0       0       0       0       0       0       0       0       0       0     225       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0      78       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 220000       0       0       0       0       0     312       0       0       0       0       0       0       0       0       0       0       0       0       0       0     127       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0      30       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 230000       0       0       1       0       0     430       0       0       0       0       0       0       0       0       0       0       0       0       0       0     250       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0      56       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 240000       0       0       1       0       0     397       0       0       0       0       0       0       0       0       0       0       0       0       0       0     165       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0      56       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 250000       0       0       0       0       0     215       0       0       0       0       0       0       0       0       0       0       0       0       0       0     110       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0      25       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 260000       0       0       0       0       0     318       0       0       0       0       0       0       0       0       0       0       0       0       0       0     164       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0      39       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 270000       0       0       0       0       0     157       0       0       0       0       0       0       0       0       0       0       0       0       0       0      68       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0      13       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 280000       0       0       0       0       0     305       0       0       0       0       0       0       0       0       0       0       0       0       0       0     153       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0      35       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 290000       0       0       0       0       0     224       0       0       0       0       0       0       0       0       0       0       0       0       0       0      95       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0      29       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 300000       0       0       1       0       0     293       0       0       0       0       0       0       0       0       0       0       0       0       0       0     177       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0      83       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 310000       0       0       0       0       0     184       0       0       0       0       0       0       0       0       0       0       0       0       0       0      70       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0      18       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 320000       0       0       0       0       0     192       0       0       0       0       0       0       0       0       0       0       0       0       0       0     103       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0      17       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 327680       0       0       0       0       0       1       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 330000       0       0       0       0       0     110       0       0       0       0       0       0       0       0       0       0       0       0       0       0      49       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0      14       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 340000       0       0       0       0       0     131       0       0       0       0       0       0       0       0       0       0       0       0       0       0      68       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0      18       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 350000       0       0       1       0       0     138       0       0       0       0       0       0       0       0       0       0       0       0       0       0      75       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0      17       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 360000       0       0       1       0       0     395       0       0       0       0       0       0       0       0       0       0       0       0       0       0     302       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0     183       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 370000       0       0       0       0       0      48       0       0       0       0       0       0       0       0       0       0       0       0       0       0      15       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       8       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 380000       0       0       1       0       0     102       0       0       0       0       0       0       0       0       0       0       0       0       0       0      38       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0      15       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 390000       0       0       0       0       0     117       0       0       0       0       0       0       0       0       0       0       0       0       0       0      43       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0      14       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 400000       0       0       0       0       0     137       0       0       0       0       0       0       0       0       0       0       0       0       0       0     114       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0      20       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 410000       0       0       0       0       0      50       0       0       0       0       0       0       0       0       0       0       0       0       0       0      20       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       8       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 420000       0       0       0       0       0     113       0       0       0       0       0       0       0       0       0       0       0       0       0       0      45       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0      10       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 430000       0       0       0       0       0      46       0       0       0       0       0       0       0       0       0       0       0       0       0       0      31       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       6       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 440000       0       0       0       0       0      59       0       0       0       0       0       0       0       0       0       0       0       0       0       0      19       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       5       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 450000       0       0       0       0       0      72       0       0       0       0       0       0       0       0       0       0       0       0       0       0      64       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0      25       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 460000       0       0       0       0       0      49       0       0       0       0       0       0       0       0       0       0       0       0       0       0      27       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       4       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 470000       0       0       0       0       0      60       0       0       0       0       0       0       0       0       0       0       0       0       0       0      17       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       3       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 480000       0       0       0       0       0      48       0       0       0       0       0       0       0       0       0       0       0       0       0       0      29       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       2       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 490000       0       0       0       0       0      41       0       0       0       0       0       0       0       0       0       0       0       0       0       0      20       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       3       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 500000       0       0       1       0       0     380       0       0       0       0       0       0       0       0       0       0       0       0       0       0     295       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0      46       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 510000       0       0       0       0       0      12       0       0       0       0       0       0       0       0       0       0       0       0       0       0       7       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 520000       0       0       0       0       0      13       0       0       0       0       0       0       0       0       0       0       0       0       0       0       5       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       2       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 530000       0       0       0       0       0       8       0       0       0       0       0       0       0       0       0       0       0       0       0       0       2       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 540000       0       0       0       0       0       2       0       0       0       0       0       0       0       0       0       0       0       0       0       0       3       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       1       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 550000       0       0       0       0       0      19       0       0       0       0       0       0       0       0       0       0       0       0       0       0       2       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 560000       0       0       0       0       0       6       0       0       0       0       0       0       0       0       0       0       0       0       0       0       4       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 570000       0       0       0       0       0       6       0       0       0       0       0       0       0       0       0       0       0       0       0       0       1       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       1       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 580000       0       0       0       0       0      11       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 590000       0       0       0       0       0       3       0       0       0       0       0       0       0       0       0       0       0       0       0       0       3       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 600000       0       0       1       0       0      10       0       0       0       0       0       0       0       0       0       0       0       0       0       0       5       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 610000       0       0       0       0       0       7       0       0       0       0       0       0       0       0       0       0       0       0       0       0       4       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 620000       0       0       0       0       0       5       0       0       0       0       0       0       0       0       0       0       0       0       0       0       4       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 630000       0       0       0       0       0       4       0       0       0       0       0       0       0       0       0       0       0       0       0       0       1       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       2       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 640000       0       0       0       0       0       5       0       0       0       0       0       0       0       0       0       0       0       0       0       0       2       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 650000       0       0       0       0       0       2       0       0       0       0       0       0       0       0       0       0       0       0       0       0       1       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 660000       0       0       0       0       0       3       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 670000       0       0       0       0       0       3       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 680000       0       0       0       0       0       4       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 690000       0       0       0       0       0       1       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 700000       0       0       0       0       0       5       0       0       0       0       0       0       0       0       0       0       0       0       0       0       2       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       1       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 710000       0       0       0       0       0       4       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       2       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 720000       0       0       0       0       0       2       0       0       0       0       0       0       0       0       0       0       0       0       0       0       1       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 730000       0       0       0       0       0       1       0       0       0       0       0       0       0       0       0       0       0       0       0       0       1       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 740000       0       0       0       0       0       1       0       0       0       0       0       0       0       0       0       0       0       0       0       0       1       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 750000       0       0       0       0       0       1       0       0       0       0       0       0       0       0       0       0       0       0       0       0       3       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 760000       0       0       0       0       0       1       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 780000       0       0       0       0       0       1       0       0       0       0       0       0       0       0       0       0       0       0       0       0       1       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      " 800000       0       0       0       0       0       1       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       1       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n",
      "1000000       0       0       0       0       0       1       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0       0\n"
     ]
    }
   ],
   "source": [
    "classificationSummary(y, clf.predict(X), class_names=classes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    " 1.4 Compare the predictive performance of two models. Which one you select? And Why? [10 Points]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#With the result from KNN's accuracy is 12% and logistic is which is higher than logistic method as the accuracy rate is only 8.05%\n",
    "#Select KNN would be better as the accuracy rate is higher than the logistic method."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}