第07章 特征选择.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 第七章 特征选择\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 二轮总结笔记\n",
    "\n",
    "\n",
    "### 一、特征的评价准则\n",
    "\n",
    "#### 1. 可分性判据的要求\n",
    "\n",
    "##### (1) 错误率单调性\n",
    "\n",
    "判据与错误率（或错误率的**上界**）有单调关系。\n",
    "\n",
    "##### (2) 可加性\n",
    "\n",
    "当特征**独立**时，判据对特征应该具有可加性，即\n",
    "\n",
    "$$\n",
    "J_{ij}(x_1,x_2,\\cdots,x_d)=\\sum_{k=1}^dJ_{ij}(x_k)\n",
    "$$\n",
    "\n",
    "##### (3) 度量特征\n",
    "\n",
    "$$\\begin{aligned}\n",
    "J_{ij} &> 0,\\quad 当i\\neq j时\\\\\n",
    "J_{ij} &= 0,\\quad 当i=j时\\\\\n",
    "J_{ij} &= J_{ji}\n",
    "\\end{aligned}$$\n",
    "\n",
    "##### (4) 特征单调性\n",
    "\n",
    "加入新的特征不会使判据减小，即\n",
    "\n",
    "$$\n",
    "J_{ij}(x_1,x_2,\\cdots,x_d) \\leq J_{ij}(x_1,x_2,\\cdots,x_d,x_{d+1})\n",
    "$$\n",
    "\n",
    "#### 2. 基于类内类间距离的可分性判据\n",
    "\n",
    "##### (1) 常用基于类内类间距离的可分性判据\n",
    "\n",
    "$$\\begin{aligned}\n",
    "J_1 &= \\text{tr}(\\mathbf{S}_\\text{w}+\\mathbf{S}_\\text{b}),\\quad J_1为各类之间的平均平方距离 \\\\\n",
    "J_2 &= \\text{tr}(\\mathbf{S}_\\text{w}^{-1}\\mathbf{S}_\\text{b}) \\\\\n",
    "J_3 &= \\ln\\frac{|\\mathbf{S}_\\text{b}|}{|\\mathbf{S}_\\text{w}|} \\\\\n",
    "J_4 &= \\frac{\\text{tr}\\mathbf{S}_\\text{b}}{\\text{tr}\\mathbf{S}_\\text{w}} \\\\\n",
    "J_5 &= \\frac{|\\mathbf{S}_\\text{b}-\\mathbf{S}_\\text{w}|}{|\\mathbf{S}_\\text{w}|}\n",
    "\\end{aligned}$$\n",
    "\n",
    "其中，$\\mathbf{S}_\\text{w}$和$\\mathbf{S}_\\text{b}$的估计值为\n",
    "\n",
    "$$\\begin{aligned}\n",
    "\\tilde{S}_\\text{w} &= \\sum_{i=1}^cP_i\\frac{1}{n_i}\\sum_{k=1}^{n_i}(\\mathbf{x}_k^{(i)}-\\mathbf{m}_i)(\\mathbf{x}_k^{(i)}-\\mathbf{m}_i)^\\text{T} \\\\\n",
    "\\tilde{S}_\\text{b} &= \\sum_{i=1}^cP_i(\\mathbf{m}_i-\\mathbf{m})(\\mathbf{m}_i-\\mathbf{m})^\\text{T}\n",
    "\\end{aligned}$$\n",
    "\n",
    "其中\n",
    "\n",
    "$$\\begin{aligned}\n",
    "\\mathbf{m}_i &= \\frac{1}{n_i}\\sum_{k=1}^{n_i}\\mathbf{x}_k^{(i)}\\\\\n",
    "\\mathbf{m} &= \\sum_{i=1}^{c}P_i\\mathbf{m}_i\\\\\n",
    "\\end{aligned}$$\n",
    "\n",
    "##### (2) 性质\n",
    "\n",
    "1. 很难从理论上建立起与分类错误率的联系。\n",
    "2. 两类样本分布有重叠时，不能反映重叠的情况。\n",
    "3. 各类样本分布协方差差别不大时，效果较好。\n",
    "\n",
    "#### 3. 基于概率分布的可分性判据\n",
    "\n",
    "##### (1) 概率距离度量的条件\n",
    "\n",
    "1. $J_P\\geqslant0$。\n",
    "2. 当两类完全不交叠（$p(\\mathbf{x}|\\omega_1)$和$p(\\mathbf{x}|\\omega_2)$完全不交叠）时$J_P$取最大值。\n",
    "3. 当两类完全交叠（$p(\\mathbf{x}|\\omega_1)=p(\\mathbf{x}|\\omega_2)$）时$J_P=0$.\n",
    "\n",
    "##### (2) 常用的概率距离度量\n",
    "\n",
    "1. Bhattacharyya距离：\n",
    "\n",
    "$$\n",
    "J_\\text{B}=-\\ln\\int[p(\\mathbf{x}|\\omega_1)p(\\mathbf{x}|\\omega_2)]^\\frac{1}{2}\\text{d}\\mathbf{x}\n",
    "$$\n",
    "\n",
    "理论错误率与Bhattacharyya距离的关系为\n",
    "\n",
    "$$\n",
    "P_\\text{e} \\leqslant [P(\\omega_1)P(\\omega_2)]^\\frac{1}{2}\\exp\\{-J_\\text{B}\\}\n",
    "$$\n",
    "\n",
    "2. Chernoff界限：\n",
    "\n",
    "$$\n",
    "J_\\text{C} = -\\ln\\int p^s(\\mathbf{x}|\\omega_1)p^{1-s}(\\mathbf{x}|\\omega_2)\\text{d}\\mathbf{x},\\quad s\\in[0,1]\n",
    "$$\n",
    "\n",
    "当$s=0.5$时，Chernoff界限与Bhattacharyya距离相同。\n",
    "\n",
    "3. 散度：\n",
    "\n",
    "$$\n",
    "J_\\text{D}=\\int [p(\\mathbf{x}|\\omega_1)-p(\\mathbf{x}|\\omega_2)]\\ln\\frac{p(\\mathbf{x}|\\omega_1)}{p(\\mathbf{x}|\\omega_2)}\\text{d}\\mathbf{x}\n",
    "$$\n",
    "\n",
    "当两类样本都服从**正态分布**且**协方差矩阵相等**的情况下，散度为两类均值间的**Mahalanobis距离**，即\n",
    "\n",
    "$$\n",
    "J_\\text{D} = 8J_\\text{B} = (\\boldsymbol{\\mu}_1-\\boldsymbol{\\mu}_2)^\\text{T}\\boldsymbol{\\Sigma}^{-1}(\\boldsymbol{\\mu}_1-\\boldsymbol{\\mu}_2)\n",
    "$$\n",
    "\n",
    "4. 概率相关性判据：\n",
    "\n",
    "把上面三种判据的$p(\\mathbf{x}|\\omega_1)$换成$p(\\mathbf{x}|\\omega_i)$，$p(\\mathbf{x}|\\omega_2)$换成$p(\\mathbf{x})$。\n",
    "\n",
    "#### 4. 基于熵的可分性判据\n",
    "\n",
    "##### (1) 常用熵度量\n",
    "\n",
    "1. Shanno熵：\n",
    "\n",
    "$$\n",
    "H = -\\sum_{i=1}^cP(\\omega_i|\\mathbf{x})\\log_2P(\\omega_i|\\mathbf{x})\n",
    "$$\n",
    "\n",
    "2. 平方熵：\n",
    "\n",
    "$$\n",
    "H=2\\left(1-\\sum_{i=1}^cP^2(\\omega_i|\\mathbf{x})\\right)\n",
    "$$\n",
    "\n",
    "##### (2) 基于熵的可分性判据\n",
    "\n",
    "$$\n",
    "J_\\text{E}=\\int H(\\mathbf{x})p(\\mathbf{x})\\text{d}\\mathbf{x}\n",
    "$$\n",
    "\n",
    "##### (3) 性质\n",
    "\n",
    "$J_\\text{E}$越小，可分性越好。\n",
    "\n",
    "#### 5. 利用统计检验作为可分性判据\n",
    "\n",
    "##### (1) 统计检验的基本概念\n",
    "\n",
    "1. 空假设（null hypothesis）：假设两类样本在所研究特征上不存在显著差异。\n",
    "2. 备择假设（alternative hypothesis）：假设两类样本在所研究特征上存在显著差异。\n",
    "3. 空分布：空假设下统计量取值的分布。\n",
    "\n",
    "##### (2) $t$-检验\n",
    "\n",
    "1. 条件：\n",
    "\n",
    "两类样本都服从**正态分布**且**方差相同**。\n",
    "\n",
    "2. 统计量：\n",
    "\n",
    "假设两类样本分别有$m$个和$n$个，在考察特征上的值分别为$\\{x_i|i=1,\\cdots,n_1\\}$和$\\{y_i|i=1,\\cdots,n_2\\}$。其中$x_i\\sim N(\\mu_x,\\sigma^2)$，$y_i\\sim N(\\mu_y,\\sigma^2)$。\n",
    "\n",
    "则统计量为\n",
    "\n",
    "$$\n",
    "t=\\frac{\\bar{x}-\\bar{y}}{\\displaystyle s_\\text{P}\\sqrt{\\frac{1}{n_1}+\\frac{1}{n_2}}},\\quad t\\sim t(n_1+n_2-2)\n",
    "$$\n",
    "\n",
    "其中$s_\\text{p}^2$是总体样本方差\n",
    "\n",
    "$$\n",
    "s_\\text{p}^2=\\frac{(n_1-1)S_x^2+(n_2-1)S_y^2}{n_1+n_2-2}\n",
    "$$\n",
    "\n",
    "3. 空假设和备择假设：\n",
    "\n",
    "+ 双边$t$-检验：空假设为两类样本在考察特征上分布相同，即$\\mu_x=\\mu_y$。备择假设为$\\mu_x\\neq\\mu_y$。\n",
    "+ 单边$t$-检验：空假设为$\\mu_x\\leqslant\\mu_y$。备择假设为$\\mu_x>\\mu_y$。\n",
    "\n",
    "4. 拒绝域：\n",
    "\n",
    "+ 双边$t$-检验：$\\displaystyle |t|\\geqslant t_\\frac{\\alpha}{2}(m+n-2)$\n",
    "+ 单边$t$-检验：$\\displaystyle t\\geqslant t_\\alpha(m+n-2)$\n",
    "\n",
    "##### (3) Wilcoxon秩和检验（Mann-Whitney U检验）\n",
    "\n",
    "1. 步骤：\n",
    "\n",
    "+ 两类样本混合到一起，所有样本按照所考察的特征从小到大排序。\n",
    "+ 两类样本的**秩和**即为所得排序**序号之和**（如果特征取值相等，则并列采用中间的序号）。\n",
    "+ 考察两类样本秩和的差异。\n",
    "\n",
    "2. 统计量：\n",
    "\n",
    "某一类的秩和，比如第一类秩和$T_1$。\n",
    "\n",
    "3. 统计量$T_1$的分布：\n",
    "\n",
    "当$n_1$和$n_2$较大（比如都大于10）时，$T_1$近似服从正态分布，即\n",
    "\n",
    "$$\\begin{aligned}\n",
    "T_1 &\\sim N(\\mu_1,\\sigma_1^2)\\\\\n",
    "\\mu_1 &= \\frac{n_1(n_1+n_2+1)}{2} \\\\\n",
    "\\sigma_1^2 &= \\frac{n_1n_2(n_1+n_2+1)}{12}\n",
    "\\end{aligned}$$\n",
    "\n",
    "##### (4) 性质\n",
    "\n",
    "1. 样本服从正态分布情况下$t$-检验比秩和检验敏感性更好，但秩和检验没有对样本分布做假设，适用范围广。\n",
    "2. $t$-检验只检验分布均值，秩和检验同时受到**分布均值**和**分布形状**的影响。\n",
    "3. 统计检验方法通常只能检验单个特征，然后对特征进行排序，也叫**过滤法**（filtering method）。\n",
    "4. 过滤准则与分类准则不一定有很好的联系。\n",
    "\n",
    "\n",
    "### 二、特征选择的最优算法\n",
    "\n",
    "#### 1. 思想\n",
    "\n",
    "分支界定法（branch and bound）。\n",
    "\n",
    "#### 2. 要求\n",
    "\n",
    "1. $D$个特征中选择$d$个最优的特征。\n",
    "2. 特征增多时判据值不会减小。\n",
    "\n",
    "#### 3. 实现方法\n",
    "\n",
    "##### (1) 树的形状\n",
    "\n",
    "1. 根节点为第$0$级，共$D-d$级。\n",
    "2. 每一级的节点在父节点基础上去掉一个特征，不出现相同组合的树枝和叶节点。\n",
    "3. 同一层中按照去掉**单个**特征后的准则函数值对结点进行排序，准则函数损失量最大的在左边。\n",
    "\n",
    "##### (2) 搜索策略\n",
    "\n",
    "1. 从右侧节点开始搜索，到达叶节点时更新准则函数值界限$B$。\n",
    "2. 搜索回溯中，遇到某一节点函数准则值小于$B$，则剪枝。\n",
    "\n",
    "#### 4. 性质\n",
    "\n",
    "$d$为$D$的一半时，分支界定法比穷举法节省的计算量最大。\n",
    "\n",
    "\n",
    "### 三、特征选择的次优算法\n",
    "\n",
    "#### 1. 单独最优特征的组合\n",
    "\n",
    "##### (1) 思想\n",
    "\n",
    "选取单个特征可分性判据值最大的$d$个特征。\n",
    "\n",
    "##### (2) 性质\n",
    "\n",
    "特征间**独立**并且所采用的判据时每个特征上的判据**之和**或**之积**时，这种方法选出来的是最优特征。\n",
    "\n",
    "#### 2. 顺序前进法（sequential forward selection，SFS）\n",
    "\n",
    "##### (1) 思想\n",
    "\n",
    "自底向上：第一个特征选取单独最优的特征，后面每一个特征都选择与已经入选的特征组合起来最优的特征。\n",
    "\n",
    "##### (2) 性质\n",
    "\n",
    "1. 计算量比单独最优特征组合大。\n",
    "2. 特征入选后无法剔除。\n",
    "\n",
    "#### 3. 顺序后退法（sequential backward selection，SBS）\n",
    "\n",
    "##### (1) 思想\n",
    "\n",
    "自顶向下：从所有特征开始逐一剔除特征，每次剔除使得剩余特征最优。\n",
    "\n",
    "##### (2) 性质\n",
    "\n",
    "1. 计算量比顺序前进法大。\n",
    "2. 特征剔除后无法加入。\n",
    "\n",
    "#### 4. 增$l$减$r$法\n",
    "\n",
    "##### (1) 思想\n",
    "\n",
    "1. 自底向上：$l>r$，先增再减。\n",
    "2. 自顶向下：$l<r$，先减再增。\n",
    "\n",
    "##### (2) 性质\n",
    "\n",
    "即能考虑到特征间的相关性又能保持适当的计算量。\n",
    "\n",
    "\n",
    "### 四、特征选择的遗传算法\n",
    "\n",
    "#### 1. 步骤\n",
    "\n",
    "1. 初始化，随机产生包含$L$条不同染色体的种群。每个染色体有长度为$D$的$0/1$字符串构成，$0$代表特征没有被选中，$1$代表特征被选中。染色体中有$d$个$1$。\n",
    "2. 计算每一条染色体的适应度$f(m)$，可以用可分性判据作为适应度。\n",
    "3. 按照选择概率$p(f(m))$对种群的染色体进行采样，并繁殖出下一代染色体。繁殖有两个基本操作：重组（也称交叉）（recombination（crossover））和突变（mutation）\n",
    "4. 反复迭代直到终止条件，适应度最大的染色体即为最优解。终止条件一般为某条染色体的适应度达到设定阈值。\n",
    "\n",
    "#### 2. 性质\n",
    "\n",
    "特征维数很高且对特征间的关系缺乏认识时，遗传算法效果好。\n",
    "\n",
    "\n",
    "### 五、以分类性能为准则的特征选择方法\n",
    "\n",
    "#### 1. 包裹法和过滤法\n",
    "\n",
    "##### (1) 包裹法\n",
    "\n",
    "利用分类器进行特征选择的方法叫**包裹法**（wrapper），即用所有特征设计分类器，考察每个特征在分类器中的贡献，剔除贡献小的特征。\n",
    "\n",
    "##### (2) 过滤法\n",
    "\n",
    "利用单独的可分性准则来选择特征再进行分类的方法叫**过滤法**。\n",
    "\n",
    "#### 2. 包裹法算法\n",
    "\n",
    "1. 算法开始前，确定选择特征的递归策略，例如每次选择（R-SVM）或剔除（SVM-RFE）特征总数的一个比例（如一半），或规定一个逐级减小的特征数目序列。\n",
    "2. 用当前所有候选特征训练**线性**SVM。\n",
    "3. 评估特征对分类贡献，按照贡献大小排序。\n",
    "4. 根据递归策略中的特征数目选择（R-SVM）或剔除（SVM-RFE）特征，接下来用新的特征重新递归训练线性SVM，直至达到规定的特征数目。\n",
    "\n",
    "#### 3. 递归支持向量机（R-SVM：recursive SVM）\n",
    "\n",
    "##### (1) 思想\n",
    "\n",
    "寻找使两类样本在线性SVM的输出上分离最开的特征。\n",
    "\n",
    "##### (2) 算法\n",
    "\n",
    "1. 每个特征的贡献是\n",
    "\n",
    "$$\n",
    "s_j = w_j(m_j^+-m_j^-),\\quad j=1,2,\\cdots,d\n",
    "$$\n",
    "\n",
    "其中，$m_j^+$和$m_j^-$分别是两类样本均值中第$j$维特征上的均值，$w_j$是权向量$\\mathbf{w}$的第$j$个分量\n",
    "\n",
    "$$\n",
    "\\mathbf{w} = \\sum_{i=1}^n\\alpha_iy_i\\mathbf{x}_i\n",
    "$$\n",
    "\n",
    "2. 特征排序后**选择**特征。\n",
    "\n",
    "##### (3) 性质\n",
    "\n",
    "不但取决于每个特征在线性分类器中对应的权重，而且考虑到两类样本在各个特征上均值的差别。\n",
    "\n",
    "#### 4. 支持向量机递归特征剔除（SVM-RFE：SVM recursive feature elimination）\n",
    "\n",
    "##### (1) 思想\n",
    "\n",
    "采用灵敏度的方法来推导各个特征在SVM分类器中的贡献，把SVM输出与正确类别标号的平均平方误差作为损失函数\n",
    "\n",
    "$$\n",
    "J = \\sum_{i=1}^{n_1+n_2}||\\mathbf{w}\\cdot\\mathbf{x}_i-y_i||^2\n",
    "$$\n",
    "\n",
    "##### (2) 算法\n",
    "\n",
    "1. 每个特征的贡献是\n",
    "\n",
    "$$\n",
    "s_j = w_j^2\n",
    "$$\n",
    "\n",
    "2. 特征排序后**剔除**特征。\n",
    "\n",
    "#### 5. 两类方法对比 \n",
    "\n",
    "1. R-SVM和SVM-RFE分类性能基本相同。\n",
    "2. R-SVM在选择特征的**稳定性**和在对未来样本的**推广能力**方面有优势，尤其是训练样本中存在较大的**噪声**和**野值**时优势明显。\n",
    "\n",
    "#### 6. 推广到非线性核\n",
    "\n",
    "特征$k$在非线性SVM中的贡献为\n",
    "\n",
    "$$\n",
    "DQ(k) = \\frac{1}{2}\\sum_{i,j=1}^n\\alpha_i\\alpha_jy_iy_j[K(\\mathbf{x}_i,\\mathbf{x}_j) - K(\\mathbf{x}_i^{(-k)},\\mathbf{x}_j^{(-k)})]\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## 一轮学习笔记（包含代码实现）\n",
    "\n",
    "这一章的目的是根据现有样本的特征，在其中提取出对样本分类有显著影响的特征。\n",
    "\n",
    "前几节介绍了判断某个特征是否对分类有显著影响的数学方法。这里不再赘述了。\n",
    "\n",
    "下面是书中介绍的特征选择的算法。\n",
    "\n",
    "### 一.特征选择的最优算法\n",
    "\n",
    "这个算法是基于深度优先搜索的选择特征算法，通过选取最大化准则函数值进行剪枝，这样可以大大减少搜索次数。但是算法题刷多了就会明白，就算剪枝剪得再多，搜索的复杂度还是指数级别的，还是非常的慢。书上也说了这个算法很慢，通常也没有人会用。\n",
    "\n",
    "### 二、特征选择的次优算法\n",
    "\n",
    "书上介绍了四种算法：单独最优特征的组合、顺序前进法、顺序后退法、增l减r法（l-r法）。四种算法思想都很简单，\n",
    "\n",
    "### 三、特征选择的遗传算法\n",
    "\n",
    "遗传算法是通过模拟生物染色体遗传进化来实现特征选择，在特征维数很高的时候效果很好。这个算法应用非常广，但是书上也只是简单介绍了一下。\n",
    "\n",
    "### 四、以分类性能为准则的特征选择方法\n",
    "\n",
    "这个方法的思路就是用分类器进行特征选择，书上介绍了R-SVM和SVM-RFE方法。\n",
    "\n",
    "这两个方法是利用了支持向量机在样本维数很高并且样本较少的情况下表现很好的特性，通过SVM训练之后的权值数据来判断特征对分类贡献大小。\n",
    "\n",
    "具体的方法如下：\n",
    "\n",
    "1. 用**线性**支持向量机对样本数据进行训练，得到对应对偶问题的$\\boldsymbol{\\alpha}$\n",
    "2. 通过$\\boldsymbol{\\alpha}$算出原始问题的系数$\\mathbf{w}$\n",
    "\n",
    "$$\n",
    "\\mathbf{w} = \\sum_{i=1}^N\\alpha_iy_i\\mathbf{x}_i\n",
    "$$\n",
    "\n",
    "说明一下，sklearn中的线性支持向量机LinearSVC中有直接算出$\\mathbf{w}$的API，不用先计算出$\\boldsymbol{\\alpha}$，下面的代码中我会直接用这个API。\n",
    "\n",
    "3. 计算每个特征对分类的贡献。对R-SVM和SVM-RFE来说，分类的贡献公式分别是：\n",
    "\n",
    "$$\\begin{align}\n",
    "s_j^{\\text{R-SVM}} &= w_j(m_j^+ - m_j^-) \\\\\n",
    "s_j^{\\text{SVM-RFE}} &= w_j^2 \\quad j = 1,\\cdots,d\n",
    "\\end{align}$$\n",
    "\n",
    "4. 接下来就根据不同特征的贡献选择特征\n",
    "\n",
    "实现代码如下：\n",
    "\n",
    "首先导包并导入数据（数据用之前用过的手写数字识别）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "训练集大小如下：\n",
      "(800, 400)\n",
      "(800,)\n",
      "测试集大小如下：\n",
      "(200, 400)\n",
      "(200,)\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from sklearn.svm import LinearSVC\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X = np.load(\"./data/digit_0_1_X.npy\")\n",
    "y = np.load(\"./data/digit_0_1_y.npy\")\n",
    "y = y.reshape(y.shape[0])\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X[:1000], y[:1000], test_size=0.2)\n",
    "\n",
    "print(\"训练集大小如下：\")\n",
    "print(X_train.shape)\n",
    "print(y_train.shape)\n",
    "print(\"测试集大小如下：\")\n",
    "print(X_test.shape)\n",
    "print(y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来用线性支持向量机训练上述数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "训练后的w大小为： (400,)\n",
      "在测试集上正确率为： 0.995\n"
     ]
    }
   ],
   "source": [
    "linear_SVC = LinearSVC()\n",
    "\n",
    "linear_SVC.fit(X_train, y_train)\n",
    "\n",
    "w = linear_SVC.coef_.reshape(X_train[0].shape[0])\n",
    "\n",
    "print(\"训练后的w大小为：\", w.shape)\n",
    "\n",
    "print(\"在测试集上正确率为：\", linear_SVC.score(X_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来计算不同特征的贡献。书上说R-SVM更好，所以下面计算R-SVM的贡献。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "s的大小为： (400,)\n",
      "s的最大值为： 0.00724647354728087 \n",
      "s的最小值为： -0.01821647548504886\n"
     ]
    }
   ],
   "source": [
    "m_1 = np.lib.average(X_train[:50], 0)\n",
    "m_2 = np.lib.average(X_train[50:], 0)\n",
    "\n",
    "s = w * (m_1 - m_2)\n",
    "\n",
    "print(\"s的大小为：\", s.shape)\n",
    "print(\"s的最大值为：\", np.max(s), \"\\ns的最小值为：\", np.min(s))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面删掉贡献值最小的200个特征特征（总共有400个特征），再重新训练测试正确率"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "删除后的训练集大小： (800, 200)\n",
      "删除后的测试集大小： (200, 200)\n",
      "删除两百个特征后的正确率： 0.995\n"
     ]
    }
   ],
   "source": [
    "# 统计贡献值最小的两百个特征\n",
    "# 下面的这段代码可能不太容易看懂，不懂也没关系，只需要知道作用就行了\n",
    "s_temp = np.concatenate((s.reshape(s.shape[0], 1), np.arange(0, s.shape[0]).reshape(s.shape[0], 1)), 1)\n",
    "s_temp.sort(0)\n",
    "index_to_delete = []\n",
    "for row in s_temp:\n",
    "    index_to_delete.append(int(row[1]))\n",
    "\n",
    "# 删掉贡献值最小的两百个特征\n",
    "X_train = np.delete(X_train, index_to_delete[:200], axis=1)\n",
    "X_test = np.delete(X_test, index_to_delete[:200], axis=1)\n",
    "\n",
    "print(\"删除后的训练集大小：\", X_train.shape)\n",
    "print(\"删除后的测试集大小：\", X_test.shape)\n",
    "\n",
    "linear_SVC.fit(X_train, y_train)\n",
    "\n",
    "print(\"删除两百个特征后的正确率：\", linear_SVC.score(X_test, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "很明显删除了一半的特征之后正确率还是没有变化，这说明我们使用的特征选择方法是很有效的。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}