diff --git a/README.md b/README.md index d3263200..3a748b69 100644 --- a/README.md +++ b/README.md @@ -75,14 +75,101 @@ Excerpts from the [Foreword](./docs/foreword_ro.pdf) and [Preface](./docs/prefac ## FAQ +### General Questions + +- [Why do you and other people sometimes implement machine learning algorithms from scratch?](./faq/implementing-from-scratch.md) +- [What learning path/discipline in data science I should focus on?](./faq/data-science-career.md) +- [At what point should one start contributing to open source?](./faq/open-source.md) +- [How important do you think having a mentor is to the learning process?](./faq/mentor.md) +- [Where are the best online communities centered around data science/machine learning or python?](./faq/ml-python-communities.md) + + +### Questions about ML Concepts + +##### Tree models + +- [How does the random forest model work? How is it different from bagging and boosting in ensemble models?](./faq/bagging-boosting-rf.md) +- [What are the disadvantages of using classic decision tree algorithm for large dataset?](./faq/decision-tree-disadvantages.md) +- [Why are implementations of decision tree algorithms usually binary, and what are the advantages of the different impurity metrics?](./faq/decision-tree-binary.md) + +##### About the Machine Learning Field + +- [What are the different fields of study in data mining?](./faq/datamining-overview.md) +- [What are differences in research nature between the two fields: machine learning & data mining?](./faq/datamining-vs-ml.md) +- [How do I know if the problem is solvable through machine learning?](./faq/ml-solvable.md) +- [What are the origins of machine learning?](./faq/ml-origins.md) +- [How was classification, as a learning machine, developed?](./faq/classifier-history.md) +- [Which machine learning algorithms can be considered as among the best?](./faq/best-ml-algo.md) +- [What are the broad categories of classifiers?](./faq/classifier-categories.md) + +##### Model evaluation + +- [Is it always better to have the largest possible number of folds when performing cross validation?](./faq/number-of-kfolds.md) +- [When training an SVM classifier, is it better to have a large or small number of support vectors?](./faq/num-support-vectors.md) +- [How do I evaluate a model?](./faq/evaluate-a-model.md) +- [What is the best validation metric for multi-class classification?](./faq/multiclass-metric.md) +- [What factors should I consider when choosing a predictive model technique?](./faq/choosing-technique.md) + +##### Logistic Regression + +- [What is the probabilistic interpretation of regularized logistic regression?](./faq/probablistic-logistic-regression.md) +- [Does regularization in logistic regression always results in better fit and better generalization?](./faq/regularized-logistic-regression-performance.md) +- [What is the major difference between naive Bayes and logistic regression?](./faq/naive-bayes-vs-logistic-regression.md) +- [What exactly is the "softmax and the multinomial logistic loss" in the context of machine learning?](./faq/softmax.md) + +##### Neural Networks + +- [What is the difference between deep learning and usual machine learning?](./faq/difference-deep-and-normal-learning.md) +- [Can you give a visual explanation for the back propagation algorithm for neural networks?](./faq/visual-backpropagation.md) +- [Why did it take so long for deep networks to be invented?](./faq/inventing-deeplearning.md) +- [What are some good books/papers for learning deep learning?](./faq/deep-learning-resources.md) +- [Why are there so many deep learning libraries?](./faq/many-deeplearning-libs.md) +- [Why do some people hate neural networks/deep learning?](./faq/deeplearning-criticism.md) + +##### Unsupervised Learning + +- [What are some of the issues with clustering?](./faq/issues-with-clustering.md) + +##### Preprocessing + +- [What are the different dimensionality reduction methods in machine learning?](./faq/dimensionality-reduction.md) +- [What is the difference between LDA and PCA for dimensionality reduction?](./faq/lda-vs-pca.md) +- [When should I apply data normalization/standardization?](./faq/when-to-standardize.md) +- [How do you attack a machine learning problem with a large number of features?](./faq/large-num-features.md) + +##### Naive Bayes + +- [What is the decision boundary for Naive Bayes?](./faq/naive-bayes-boundary.md) +- [Can I use Naive Bayes classifiers for mixed variable types?](./faq/naive-bayes-vartypes.md) +- [Is it possible to mix different variable types in Naive Bayes, for example, binary and continues features?](./naive-bayes-vartypes.md) + +##### Other + +- [What is Euclidean distance in terms of machine learning?](./faq/euclidean-distance.md) + + + + + +
+
+
+ + + + + +### Questions about the Book + - [Can I use paragraphs and images from the book in presentations or my blog?](./faq/copyright.md) - [How is this different from other machine learning books?](./faq/different.md) - [Which version of Python was used in the code examples?](./faq/py2py3.md) - [Which technologies and libraries are being used?](./faq/technologies.md) - [Which book version/format would you recommend?](./faq/version.md) -- [Why did you choose Python for machine learning?](./faq/why_python.md) -- [Why do you use so many leading and trailing underscores in the code examples?](./faq/underscore_convention.md) -- [Are There Any Prerequisites and Recommended Pre-Readings?](./faq/prerequisites.md) +- [Why did you choose Python for machine learning?](./faq/why-python.md) +- [Why do you use so many leading and trailing underscores in the code examples?](./faq/underscore-convention.md) +- [Are there any prerequisites and recommended pre-readings?](./faq/prerequisites.md) + ## Contact diff --git a/code/ch05/ch05.ipynb b/code/ch05/ch05.ipynb index 064162ad..8872e4ef 100644 --- a/code/ch05/ch05.ipynb +++ b/code/ch05/ch05.ipynb @@ -42,15 +42,15 @@ "output_type": "stream", "text": [ "Sebastian Raschka \n", - "Last updated: 08/20/2015 \n", + "Last updated: 11/07/2015 \n", "\n", - "CPython 3.4.3\n", - "IPython 3.2.1\n", + "CPython 3.5.0\n", + "IPython 4.0.0\n", "\n", - "numpy 1.9.2\n", - "scipy 0.15.1\n", + "numpy 1.10.1\n", + "scipy 0.16.0\n", "matplotlib 1.4.3\n", - "scikit-learn 0.16.1\n" + "scikit-learn 0.17\n" ] } ], @@ -61,7 +61,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 2, "metadata": { "collapsed": true }, @@ -119,7 +119,7 @@ }, { "cell_type": "code", - "execution_count": 1, + "execution_count": 3, "metadata": { "collapsed": true }, @@ -137,7 +137,7 @@ }, { "cell_type": "code", - "execution_count": 3, + "execution_count": 4, "metadata": { "collapsed": false }, @@ -149,7 +149,7 @@ "" ] }, - "execution_count": 3, + "execution_count": 4, "metadata": { "image/png": { "width": 400 @@ -164,7 +164,7 @@ }, { "cell_type": "code", - "execution_count": 1, + "execution_count": 5, "metadata": { "collapsed": false }, @@ -306,7 +306,7 @@ "4 4.32 1.04 2.93 735 " ] }, - "execution_count": 1, + "execution_count": 5, "metadata": {}, "output_type": "execute_result" } @@ -333,7 +333,7 @@ }, { "cell_type": "code", - "execution_count": 2, + "execution_count": 6, "metadata": { "collapsed": false }, @@ -356,7 +356,7 @@ }, { "cell_type": "code", - "execution_count": 3, + "execution_count": 13, "metadata": { "collapsed": false }, @@ -366,7 +366,56 @@ "\n", "sc = StandardScaler()\n", "X_train_std = sc.fit_transform(X_train)\n", - "X_test_std = sc.fit_transform(X_test)" + "X_test_std = sc.transform(X_test)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---\n", + "\n", + "**Note**\n", + "\n", + "Accidentally, I wrote `X_test_std = sc.transform(X_test)` instead of `X_test_std = sc.fit_transform(X_test)`. In this case, it wouldn't make a big difference since the mean and standard deviation of the test set should be (quite) similar to the training set. However, as remember from Chapter 3, the correct way is to re-use parameters from the training set if we are doing any kind of transformation -- the test set should basically stand for \"new, unseen\" data.\n", + "\n", + "My initial typo reflects a common mistake is that some people are *not* re-using these parameters from the model training/building and standardize the new data \"from scratch.\" Here's simple example to explain why this is a problem.\n", + "\n", + "Let's assume we have a simple training set consisting of 3 samples with 1 feature (let's call this feature \"length\"):\n", + "\n", + "- train_1: 10 cm -> class_2\n", + "- train_2: 20 cm -> class_2\n", + "- train_3: 30 cm -> class_1\n", + "\n", + "mean: 20, std.: 8.2\n", + "\n", + "After standardization, the transformed feature values are\n", + "\n", + "- train_std_1: -1.21 -> class_2\n", + "- train_std_2: 0 -> class_2\n", + "- train_std_3: 1.21 -> class_1\n", + "\n", + "Next, let's assume our model has learned to classify samples with a standardized length value < 0.6 as class_2 (class_1 otherwise). So far so good. Now, let's say we have 3 unlabeled data points that we want to classify:\n", + "\n", + "- new_4: 5 cm -> class ?\n", + "- new_5: 6 cm -> class ?\n", + "- new_6: 7 cm -> class ?\n", + "\n", + "If we look at the \"unstandardized \"length\" values in our training datast, it is intuitive to say that all of these samples are likely belonging to class_2. However, if we standardize these by re-computing standard deviation and and mean you would get similar values as before in the training set and your classifier would (probably incorrectly) classify samples 4 and 5 as class 2.\n", + "\n", + "- new_std_4: -1.21 -> class 2\n", + "- new_std_5: 0 -> class 2\n", + "- new_std_6: 1.21 -> class 1\n", + "\n", + "However, if we use the parameters from your \"training set standardization,\" we'd get the values:\n", + "\n", + "- sample5: -18.37 -> class 2\n", + "- sample6: -17.15 -> class 2\n", + "- sample7: -15.92 -> class 2\n", + "\n", + "The values 5 cm, 6 cm, and 7 cm are much lower than anything we have seen in the training set previously. Thus, it only makes sense that the standardized features of the \"new samples\" are much lower than every standardized feature in the training set.\n", + "\n", + "---" ] }, { @@ -378,7 +427,7 @@ }, { "cell_type": "code", - "execution_count": 4, + "execution_count": 10, "metadata": { "collapsed": false }, @@ -431,7 +480,7 @@ }, { "cell_type": "code", - "execution_count": 5, + "execution_count": 11, "metadata": { "collapsed": true }, @@ -444,16 +493,16 @@ }, { "cell_type": "code", - "execution_count": 6, + "execution_count": 12, "metadata": { "collapsed": false }, "outputs": [ { "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAakAAAEbCAYAAABgLnslAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8VNX9//HXh1WQLSGAyKqgthYVN8QFiYptRAUsBQFR\nUdza2oLWqmCVuHy1/BCr2EddkKWKglq1SFGhLlHrvgAuLAoKIqBFCbsSSD6/P+ZmOgmTZBJyMzPh\n/Xw85pG5d86987k3yXzmnHvuOebuiIiIpKI6yQ5ARESkLEpSIiKSspSkREQkZSlJiYhIylKSEhGR\nlKUkJSIiKSvUJGVmU83sWzP7uIzXzzOzRWb2kZm9YWaHhxmPiIikl7BrUtOAnHJe/wI42d0PB24F\nHgw5HhERSSOhJil3fx3IL+f1t9x9U7D4DtA+zHhERCS9pNI1qZHAc8kOQkREUke9ZAcAYGanABcD\nJyY7FhERSR1JT1JBZ4nJQI67x20aNDMNMCgiUku4uyVaNqnNfWbWEXgaGO7uy8sr6+616jFu3Lik\nx6Bj2juPqbYel44pPR6VFWpNysxmAr2BLDNbDYwD6gO4+wPATUAGcJ+ZAex09x5hxiQiIukj1CTl\n7kMreP0S4JIwYxARkfSVSr379irZ2dnJDqHa6ZjSR208Lh1T7WRVaSOsaWbm6RCniIiUz8zwSnSc\nSHrvvj0RXMcSkRShL5NS3dI6SYH+KURShb40Shh0TUpERFKWkpSIiKQsJSkREUlZSlK12PTp0+nV\nq1eVt+/bty+PPPJINUYUnpUrV1KnTh2KiooqLPv666/zk5/8JJQ4RowYwY033hjKvktLp9+PSFWl\nfccJqR65ubmsWLGixIfec8/VzkHpe/XqxdKlS0PZt5nVWAeC2vr7EYmlmpRINQu7x2lVx0ATSUdK\nUiFZvXo1v/zlL2ndujVZWVn87ne/AyI1lvPPPz9arnQzVXZ2NjfeeCMnnngiTZs2pV+/fnz33Xec\nd955NG/enB49erBq1aq42xZvP2XKlLgxjRo1io4dO9K8eXOOOeYY/vOf/wDwwgsvcMcdd/D444/T\ntGlTjjzyyBL72rFjBy1atODTTz+N7mv9+vU0btyY7777DoB//etfdO/enYyMDE488UQ+/vjjMs/N\n0qVLOf3002nZsiU/+clPePLJJwFYsWIFLVu2ZMGCBQCsXbuWVq1a8dprr0XjGTNmDMcddxzNmzdn\nwIAB5OfHn1Nz2rRpHHrooTRr1owuXbrw4IP/m/Q5Ly+PDh06RJc7d+7MxIkTOeKII2jRogVDhgxh\nx44d0dfLO7YFCxZw1FFH0axZM4YMGcKPP/4YN56KzmF+fj5nnXUWrVu3JjMzk7PPPps1a9ZEy2Zn\nZ/OnP/2JE088kSZNmvDFF1+U+F2vWLGCU089laysLFq1asXw4cPZtGlTdPuKjnH27Nl0796d5s2b\n07VrV+bNmwfApk2bGDlyJPvvvz/t27fnxhtvTKhJVaS6KEmFoLCwkLPOOosDDjiAVatWsWbNGoYO\njQxjmEhT0OOPP86MGTNYs2YNK1as4Pjjj2fkyJFs2LCBn/70p9x8881lbltec1OPHj1YtGgR+fn5\nDBs2jEGDBlFQUEBOTg5jx45lyJAhbNmyJZokivfVsGFDBg4cyMyZM6P7euKJJ8jOziYrK4sFCxYw\ncuRIJk+ezIYNG7j88svp168fBQUFu8Wwbds2Tj/9dIYPH8769euZNWsWv/nNb1iyZAldunRh/Pjx\nDB8+nB9++IGLLrqIiy66iJNPPjm6/SOPPMK0adNYt24d9erV4/e//33cY23Tpg1z585l8+bNTJs2\njauuuip6XPHO2ZNPPsm8efP48ssv+eijj5g+fTpAmce2c+dOCgoKGDBgABdeeCH5+fkMGjSIp556\nKu75r+gcujsjR47kq6++4quvvqJRo0ZceeWVJfYxY8YMHnroIbZs2UKnTp12+13fcMMNrFu3jiVL\nlrB69Wpyc3MTOsZ3332XCy+8kIkTJ7Jp0yZee+01OnfuDESusTVo0IAVK1awYMEC5s+fz0MPPRT3\nPIqEItnDtic4tLvHU9b6/71ePY/KevPNN71Vq1ZeWFi422vjxo3z4cOHR5e//PJLN7No2ezsbL/9\n9tujr//hD3/wvn37RpfnzJnj3bt3j7tt8fZTpkxxd/dp06b5SSedVGacGRkZ/tFHH8WNq/S+Xnzx\nRe/SpUv0tRNOOMEfeeQRd3e/4oor/MYbbyyx7SGHHOKvvvrqbu85a9Ys79WrV4l1l112md98883R\n5X79+nm3bt38iCOO8IKCghLxjBkzJrq8ePFib9CggRcVFcU9F7EGDBjg99xzj7u7v/LKK96+ffvo\na507d/ZHH300unzttdf6FVdcUeGxvfrqq77//vuXeO2EE07YrXyx8s5haQsWLPCMjIwSxz5u3LgS\nZWJ/P6U988wzfuSRRyZ0jJdddplfffXVu+3jm2++8YYNG/oPP/wQXffYY4/5KaecEvc9K/p/FHGP\n/p0k/PlfqztOJKvZfvXq1XTq1Ik6dapWUW3Tpk30+T777EPr1q1LLG/durVK+73zzjuZOnUqa9eu\nxczYvHlztLmuItnZ2Wzfvp13332X1q1bs2jRIs455xwAVq1axcMPP8y9994bLb9z507WrVu3235W\nrVrFO++8Q0ZGRnTdrl27uOCCC6LLl1xyCf3792fy5MnUr1+/xPaxzXQdO3Zk586dcY/h+eef5+ab\nb+bzzz+nqKiI7du3c/jhh5d5fPvtt1/0eaNGjVi7dm2Fx+butGvXrsR+OnXqVOb1ovLO4fbt27nq\nqquYN29etAlz69atuHu0thR77KV9++23jBo1iv/85z9s2bKFoqIiMjMzyz3G4t/P119/zZlnnrnb\nPletWsXOnTtp27ZtdF1RUREdO3YsMw6R6lark1SydOjQga+++orCwkLq1q1b4rUmTZqwffv26PI3\n33xT7r7Kax7cd999gcgHXJMmTcrd3+uvv86ECRN4+eWX+dnPfgZAZmZm9AO1ombIunXrMnjwYGbO\nnEnr1q05++yzo+/fsWNHbrjhBsaOHVvuPorL9u7dm/nz58d9fevWrYwePZpLLrmEcePG8ctf/rJE\nQvvqq69KPK9fvz5ZWVls27Ytun7Hjh0MHDiQGTNm0L9/f+rWrcs555xTqc4GxeejvGN79dVXS1w3\ngsgHe9euXePus7xzOHHiRD777LNoAlu4cCFHHXVUiSRV3u9o7Nix1K1bl08++YQWLVrwz3/+M3od\ntCIdOnRg+fLd5xzt0KEDDRs25Pvvv6/yFy6RPaW/vBAcd9xxtG3bluuvv57t27fz448/8uabbwLQ\nvXt3XnvtNVavXs2mTZu44447dts+9sO0vA/WVq1a0a5dOx555BEKCwuZOnUqK1asiFt2y5Yt1KtX\nj6ysLAoKCrjlllvYvHlz9PX99tuPlStX7vZ+scvDhg1j1qxZPPbYYwwbNiy6/tJLL+X+++/n3Xff\nxd3Ztm0bc+fOjVvjO+uss/jss8+YMWMGO3fuZOfOnbz33nvRLuGjRo2iR48ePPjgg5x55plcccUV\nJWKZMWMGS5YsYfv27dx0000MGjRotw/vgoICCgoKyMrKok6dOjz//PNlJsWyFB93ecd2wgknUK9e\nPSZNmsTOnTt5+umnee+998rdb1nncOvWrTRq1IjmzZuzYcOGuNcdy/tb2Lp1K/vuuy/NmjVjzZo1\nTJgwIeFjHDlyJNOmTePll1+mqKiINWvWsGzZMtq2bcvPf/5zrr766mjtbMWKFdGOLCI1QUkqBHXq\n1GHOnDksX76cjh070qFDB5544gkA+vTpw7nnnsvhhx/Osccey9lnn73bh2zscryOELHLkydPZsKE\nCWRlZbF48WJOPPHEuNvm5OSQk5PDwQcfTOfOnWnUqFGJZptBgwYB0LJlS4455pi479WjRw+aNGnC\nunXrOOOMM6Lrjz76aCZPnsyVV15JZmYmBx10EA8//HDcc9OkSRPmz5/PrFmzaNeuHW3btmXMmDHs\n2LGD2bNnM3/+fO677z4A7rrrLj788MNoZwMz4/zzz2fEiBG0bduWgoICJk2atFusTZs2ZdKkSQwe\nPJjMzExmzpxJ//79yzyHpcWet/KOrX79+jz99NNMnz6dli1b8sQTTzBw4MAy91veORw9ejQ//PAD\nWVlZnHDCCZxxxhnl/t5LGzduHB9++CHNmzfn7LPPZuDAgQkf47HHHhvtXNKiRQuys7OjNdaHH36Y\ngoICDj30UDIzMxk0aFCFtX+R6pTW80kF85IkISJJhlNOOYXzzz+fiy++ONmhSBz6f5REVHY+KdWk\nJK3oQ1Bk76IkJWlFcxaJ7F3U3Cci1UL/j5IINfeJiEitoSQlIiIpS0lKRERSlpKUiIikLCUpERFJ\nWUpSIenWrVuVh4+JnYK8MlOdlzedeGWmV69MfDWl9Dxc5fn1r3/NbbfdFkocderU4Ysvvghl37G+\n+uormjZtqt5ysterdQPMjh6dy8aN4e2/RQu4++7cCst98sknVX6P2CFrKjPVeU1NJ16TU6THvmei\niodVSmcdO3Zky5YtyQ5DJOlqXZLauBE6d84Nbf8rV4a371ip/g26puNL9fNRnXbt2kW9erXuX1Ok\nStTcF5LOnTvz8ssvA5GmqsGDB3PhhRfSrFkzunXrxgcffBAtW94U5LFTnY8fPz46EGyxUaNGMWrU\nKKDk1PGFhYVcc801tGrVii5dujB37tzd4nvppZeiy6Wb0wYNGkTbtm1p0aIFvXv3ZvHixQkf+9Sp\nU6MDkubk5EQHKx0/fjw9e/aksLAQiNR4unXrRkFBQbQ5cvLkybRr147999+fiRMnlvke5cUX2xyZ\nl5dH+/btueuuu2jTpg37779/dEZaiEzrcc0119CpUyf2228/fv3rX5c4/xMmTIhOnT516tQy43n8\n8cc59thjS6z7y1/+Eh3Ydu7cuRx55JE0b96cjh07lhjlvPjYp06dSqdOnejTpw+rVq0q0Tw7bdo0\nDj30UJo1a0aXLl148MEHo9tXdIw//PADf/jDH+jcuTMtWrSgV69e0WN8++23OeGEE8jIyKB79+68\n+uqrZR6jSDKEmqTMbKqZfWtmH5dTZpKZfW5mi8zsyDDjqUmlm6fmzJnD0KFD2bRpE/369YtODV6Z\nKciHDBnCc889F50Co7CwkCeffJLzzjsv+p7F202ePJm5c+eycOFC3n//ff7xj3+UO7p66fc788wz\nWb58OevXr+eoo46KvkdFZs+ezR133MEzzzzDd999R69evRg6dCgA1157LQ0bNuS2227j888/54Yb\nbuDRRx+lQYMG0e3z8vJYvnw58+fPZ/z48SUSaaLxlT62b7/9ls2bN7N27VqmTJnCb3/7WzZt2gTA\n9ddfz/Lly1m0aBHLly9nzZo13HLLLQC88MILTJw4kRdffJHPPvuMF198sczj7tevH8uWLSsxL9Nj\njz0WjatJkybMmDGDTZs2MXfuXO677z5mz55dYh+vvfYaS5cuZd68ebvVHNu0acPcuXPZvHlzdMTy\nBQsWJHSM11xzDQsWLOCtt95iw4YNTJgwgTp16rBmzRrOOussbrrpJvLz87nzzjsZOHBgwhNhitSE\nsGtS04Ccsl40s75AV3c/CLgMSP+LCWXo1asXOTk5mBnDhw9n0aJFQOSb7K5duxg1ahR169Zl4MCB\nu30jL9apUyeOOuoonnnmGQBefvllGjduTI8ePXYr+8QTT3DVVVfRrl07MjIyGDt2bKWazEaMGMG+\n++5L/fr1GTduHIsWLUroGsn999/PmDFjOOSQQ6hTpw5jxoxh4cKFrF69GjPj4YcfZtKkSfTv35/r\nrruOI444osT248aNo1GjRnTr1o2LLrooOk1HZeOLPdb69etz0003UbduXc444wyaNGnCsmXLcHcm\nT57MXXfdRYsWLWjSpAljxoxh1qxZ0XN48cUXc+ihh9K4ceO4czwVa9SoEf3794/G+/nnn7Ns2TL6\n9esHQO/evaOTTR522GEMGTJkt1pLbm4ujRo1omHDhrvtv2/fvhxwwAEAnHzyyfz85z/n9ddfr/AY\ni4qKmDZtGvfccw9t27alTp069OzZkwYNGjBjxgz69u1LTk7kX7RPnz4cc8wxNXZtUyQRoSYpd38d\nyC+nSD/g70HZd4AWZtamnPJpK3ZK+MaNG/Pjjz9SVFTE2rVr405BXpZhw4ZFPwhjv6mXtm7dut2m\nWk9UYWEh119/PV27dqV58+bRD8dEvmGvWrWKUaNGkZGRQUZGBi1btgSIzmDbqVMnsrOzWbVqFb/9\n7W932750zMXTuO9JfC1btiwxs2zjxo3ZunUr69evZ/v27Rx99NHReM8444zofip7Dkv/bs455xz2\n2WcfAN555x1OOeUUWrduTYsWLXjggQf4/vvvyzz20p5//nl69uxJy5YtycjI4LnnniuxfVnH+N13\n3/Hjjz/SpUuX3fa5atUqnnzyyeixZ2Rk8MYbb2i+KEkpyb4m1Q5YHbP8NdA+SbEkRdu2beNOQV6W\nX/3qV+Tl5bFmzRr++c9/lpjdtfR+S0+1HmvfffctMeX6unXrok1kjz32GM8++ywvvfQSmzZt4ssv\nvwQS67zQsWNHHnzwQfLz86OPbdu20bNnTyBybebtt9/mtNNO45prrtlt+9Ixl07gicaXSG/ArKws\nGjVqxOLFi6Oxbty4MTpjcUXnsLQ+ffqwfv16Fi1axKxZs0r8boYNG8aAAQP4+uuv2bhxI1dcccVu\ntwOUFfOOHTsYOHAg1157Lf/973/Jz8+nb9++Cf0+srKy2GeffeJOD9+xY0fOP//8Er+rLVu2cO21\n11a4X5GakuwkBVD6P3Pv6cYFHH/88ZWagrxVq1ZkZ2czYsQIDjzwQA455JC45QYPHsykSZNYs2YN\n+fn5/PnPfy7xevfu3Zk1axa7du3i/fff56mnnoq+tnXrVho2bEhmZibbtm1j7NixJbYt78Pxiiuu\n4Pbbb492ZNi0aRNPPvkkEKnpXHrppUyZMoXp06czZ84cnn/++RLb33bbbfzwww98+umnTJ8+nXPP\nPXe390gkvkQ+wOvUqcOll17K6NGjWb9+PRCp8RVPNT948GCmT58ena6+vOY+iDS5DRo0iGuuuYb8\n/HxOP/30EjFnZGTQoEED3n33XR577LGEu9UXFBRQUFBAVlYWderU4fnnn4/GmMgxXnzxxVx99dWs\nW7eOwsJC3nrrLQoKChg+fDhz5sxh/vz5FBYW8uOPP0a/AImkimQnqTVAbBtH+2DdbnJzc6OPvLy8\nmoit2pQ3BXyDBg0qnIK89LbDhg3jpZdeKrMWBXDppZfyi1/8giOOOIJjjjlmt+nEb731VlasWEFG\nRga5ubklmg0vuOACOnXqRLt27ejWrRvHH398hVPaFxswYADXXXcdQ4YMoXnz5hx22GHMmzcPgMsv\nv5wBAwaQk5NDZmYmU6ZM4ZJLLiE//38twr1796Zr16706dOHP/7xj/Tp02e396xsfOUlg/Hjx9O1\na1d69uxJ8+bNOf300/nss88AyMnJYfTo0Zx66qkcfPDBnHbaaRUmluLfzaBBg0o0v/3tb3/jpptu\nolmzZtx66627Jd94+y1e17RpUyZNmsTgwYPJzMxk5syZ0V6DiRzjnXfeyWGHHcaxxx5Ly5YtGTNm\nDEVFRbRv357Zs2dz++2307p1azp27MjEiROr5YZvSZ7MTDCr/CMseXl5JT6/Kyv0+aTMrDMwx90P\ni/NaX+BKd+9rZj2Bu929Z5xyCc8nlSo380rlrFy5kgMPPJBdu3aV+HCX9KH5pMqXmQn55V2hryYZ\nGbBhQ/jvU1WVnU8q1CRlZjOB3kAW8C0wDqgP4O4PBGX+SqQH4DbgInf/MM5+NOlhLacklf70/1g+\nM9DpqXySCvW2dncfmkCZK8OMQdKHpoYXkdI0fbyIVIt0/H+sqSY4SP1muJqSUs191UVJSiT1peP/\no5rgal5lk5Qa/0VEJGVpqGURSSk13QQnqU3NfSJSLarr/1FNcLVbSvXuqwnqESYiUnuldZJSLUpE\npHZL6yQlIjWnJkdMECmW1tekRKTm6FqRVAd1QRcRkVpDSUpERFKWrkmJpDHdUyS1na5JiaQxXSeS\ndKNrUiIiUmsoSYmISMqqMEmZWQcze8bM1gePp8ysfU0EJyIie7dEalLTgGeB/YPHnGCdiJQhMzNy\nvSjshzozSG1XYccJM1vk7kdUtC5M6jgh6UYdGkTiC6PjxPdmdr6Z1TWzemY2HPiu6iGKiIgkJpEk\ndTEwGPgGWAcMAi4KMygRERHQfVIioVBzn0h81TaflJld5+7jzezeOC+7u/++ShGKiIgkqLxhkRYH\nPz8AYr8TWqllERGRUJSZpNx9TvB0u7s/EfuamQ0ONSoRERES6zgxJsF1Iimtpu5d0v1LItWnvGtS\nZwB9gXZmNolIMx9AU2BnDcQmUq3y89WZQSTdlHdNai2R61H9g5/FSWozcFXIcYmIiCQ04kQDdy+o\noXjKikFd0GWPqVu4SPJVWxf0GJ3N7HbgUKBRsM7d/cCqBCgiIpKoRAeYvR/YBWQDfwceDTEmERER\nILEk1cjdXyTSNLjK3XOBM8MNS0REJLEk9aOZ1QWWm9mVZvZLYN9Edm5mOWa21Mw+N7Pr4ryeZWYv\nmNlCM/vEzEZULnwREanNEuk4cSywFGgB3Ao0A/6fu79dwXZ1gWVAH2AN8B4w1N2XxJTJBRq6+xgz\nywrKt3H3XaX2pY4TssfUcUIk+aq140SQaM5192uALcCISsTSA1ju7iuDfc0i0p19SUyZdcDhwfNm\nwPelE5TUfpmZkXuYwqYbbEXST7lJyt0Lzewkq1pVph2wOmb5a+C4UmUmAy+b2VoiNwlruKW9kG6y\nFZGyJNIFfSEw28yeBLYH69zdn65gu0Q+dsYCC90928y6AP82syPcfUvpgrm5udHn2dnZZGdnJ7B7\nERFJpry8PPLy8qq8fSLXpKYTJ+G4e7kTH5pZTyDX3XOC5TFAkbuPjynzHPB/7v5GsPwScJ27v19q\nX7omVYvpWpHI3qPab+Z19xFVjOV94CAz60xkiKVzgaGlyiwl0rHiDTNrAxwCfFHF9xMRkVomkea+\nKnH3XWZ2JTAPqAtMcfclZnZ58PoDwO3ANDNbRKQ7/LXuviGsmEREJL1o+nhJOjX3iew9Ktvcl8jN\nvCIiIklRYZIys/3MbIqZvRAsH2pmI8MPTURE9naJ1KSmA/OB/YPlz9F8UiIiUgMSSVJZ7v44UAjg\n7juJjIguIiISqkSS1FYza1m8ENz/tCm8kERERCIS6YL+B2AOcKCZvQm0An4ValSSdDU1nh5oTD0R\nKVtCXdDNrD6RG20NWBo0+dUYdUGveeoWLiJhqPYu6MENuU3c/RN3/xhoYma/2ZMgRUREEpHI2H2L\n3P2IUusWunv3UCMr+X6qSdUw1aREJAxh3Mxbx8yi5YI5pupXJTgREZHKSKTjxDxglpk9QOSa1OXA\nC6FGJSIiQmLNfXWBy4DTglX/Bh5y98KQY4uNQc19NUzNfSIShso292mAWYlLSUpEwlDt80mZ2UnA\nOKBzTHl39wOrFKGIiEiCEmnuWwaMBj4kGBoJwN2/Cze0EjGoJlXDVJMSkTBUe00K2Ojuz+9BTCIi\nIlWSSE3qz0Rm1n0a2FG83t0/DDe0EjGoJlXDVJMSkTBUe8cJM8sDdivk7qdUOroqUpKqeUpSIhIG\n9e6TaqEkJSJhCOOaFGZ2FnAosE/xOne/pfLhiYiIJC6RAWYfAAYDvycy4sRgoFPIcYmIiCR0Tepj\ndz/MzD5y98PNrAnwgrufVDMhqrmvWE3P8bRhQ828l4jsPcJo7vsh+LndzNoB3wP7VSU42TP5+bpO\nJCJ7l0SS1L/MLAOYAHwQrJscXkgiIiIRlerdZ2b7APu4+8bwQor7vmruQz3uRCT9VVtzn5md5u4v\nmdlASt0nFbzJ03sQp4iISIXKa+47GXgJOJs4N/MSGYFCREQkNOU29wUz8g5y98drLqS4cai5DzX3\niUj6q9bp4929CLh2j6MSERGpgkQHmP0OeBzYVrze3WvsLhrVpCJUkxKRdBfGALMriT/A7AEJBJMD\n3E1kFPWH3H18nDLZwF+A+sB37p4dp4ySFEpSIpL+UmaAWTOrCywD+gBrgPeAoe6+JKZMC+AN4Bfu\n/rWZZcWbTFFJKkJJSkTSXVgDzHZj9wFmH65gsx7AcndfGexjFtAfWBJTZhjwlLt/Heyzxmb7FRGR\n1JfIALO5wL3AX4FTgP8H9Etg3+2A1THLXwfrYh0EZJrZK2b2vpmdn0jQIiKyd0ikJvUr4AjgQ3e/\nyMzaAI8msF0iDVP1gaOA04DGwFtm9ra7f166YG5ubvR5dnY22dnZCexeRESSKS8vj7y8vCpvn0jH\niffc/Vgz+wA4FdgMLHX3QyrYrieQ6+45wfIYoCi284SZXQc0cvfcYPkhIiOs/6PUvnRNCl2TEpH0\nV633SQXeDwaYnQy8DywA3kxkO+AgM+tsZg2Ac4FnS5WZDZxkZnXNrDFwHLA40eBFRKR2q+wAswcA\nzdx9UYLlz+B/XdCnuPsdZnY5gLs/EJS5BrgIKAImu/ukOPtRTQrVpEQk/YVxn9QcYCYw2923lVs4\nJEpSEUpSIpLuwmjumwj0Ahab2VNm9qtgyg4REZFQJdzcZ2b1iHRBvxTIcfdmYQZW6r1Vk0I1KRFJ\nf2HdzNuIyL1Rg4l0Gf971cITERFJXIVJysyeINLr7gUiN/S+GoyOLoHMTMjPD/99MjLCfw8RkVSS\nSMeJHOBFd99VMyHFjSGlm/vUDCcikpiUGWC2OilJiYjUDmH07hMREUkKJSkREUlZZXacMLOjiQwS\na8Sf9PDDEOMSEREp+5qUmeURSU6NgKOBj4KXDgfed/fjayLAIBZdkxIRqQWq7ZqUu2e7+ynAWuAo\ndz/a3Y8GjgzWiYiIhCqRa1I/cfePixfc/RPgp+GFJCIiEpHIiBMfBfM8zSByfWoYkNAo6CIiInsi\nkZt5GwEM32ptAAAQUElEQVS/JjLILMBrwH3u/mPIscXGoGtSIiK1QCg38wYTEnZ096V7ElxVKUmJ\niNQO1X4zr5n1IzIb7wvB8pFmVnqGXRERkWqXSMeJXCIDzOYDuPsC4MAQYxIREQESS1I73X1jqXUa\nBV1EREKXSO++T83sPKCemR0E/B54M9ywREREEqtJ/Q74GbADmAlsBkaHGZSIiAhoqo5qod59IiKJ\nqfbp483sEOAaoHNMeXf3U6sUoYiISIISuZn3I+A+4EOgMFjt7v5ByLHFxqCalIhILVDtNSkivfvu\n24OYREREqiSRjhNzzOy3ZtbWzDKLH6FHJiIie71EmvtWEn/SwwNCiileDGruExGpBUIZuy/ZlKRE\nRGqHarsmZWanuftLZjaQ+DWpp6sYo4iISELK6zhxMvAScDZxkhSgJCUiIqFSc181UHOfiEhiwuiC\njpmdBRwK7FO8zt1vSWC7HOBuoC7wkLuPL6PcscBbwGA1I4qISLFE5pN6ABhMZGBZC553SmC7usBf\ngRwiCW6omf20jHLjicxXlXB2FRGR2i+R+6ROcPcLgA3ufjPQEzgkge16AMvdfaW77wRmAf3jlPsd\n8A9gfYIxi4jIXiKRJPVD8HO7mbUDdgH7JbBdO2B1zPLXwbqoYH/9iQy7BPE7aIiIyF4qkWtS/zKz\nDGACUDxe3+QEtksk4dwNXO/ubmaGmvtERCRGhUkqpoPEU2Y2F9gnzky98awBOsQsdyBSm4p1NDAr\nkp/IAs4ws53u/mzpneXm5kafZ2dnk52dnUAIIiKSTHl5eeTl5VV5+zK7oJe6idcoVTOqqBeemdUD\nlgGnAWuBd4Gh7r6kjPLTgDnx9qsu6CIitUN1dkEv6ybeYuUmKXffZWZXAvOIdEGf4u5LzOzy4PUH\nEg1SRET2TrqZtxqoJiUikpjK1qQSuU8qy8zuNbMFZvahmd1jZi33LEwREZGKJdK7bxbwKvBLItem\nhgGPA31CjKtajR6dy8ZEunoAf/97bqX3n5FR6U1ERCQBicwn9Ym7dyu17mN3PyzUyEq+3x41940Y\nkUvnzrnVFxCwcmUu06dX7z5FRGq7am/uA+ab2VAzqxM8zgXmVz1EERGRxCSSpC4DHgUKgsdM4DIz\n22Jmm8MMTkRE9m6J3MzbpCYCERERKS2R3n0jSy3XM7Nx4YUkIiISkUhzXx8ze87M9jezbkTmfWoW\nclwiIiIJNfcNNbMhwEfANuA8d/9P6JGJiMheL5HmvoOJTHj4NPAVMNzM9g07MBERkUSa+54FbnL3\ny4DewOfAe6FGJSIiQmIjThzn7psA3L0ImGhmc8INS0REpJyalJldC+Dum8xsUKmXR4QZlIiICJTf\n3Dc05vnYUq+dEUIsIiIiJSRyTUpERCQplKRERCRllddx4nAz2xI8bxTzHKBRiDGJiIgA5SQpd69b\nk4GIiIiUpuY+ERFJWUpSIiKSspSkREQkZSlJiYhIylKSEhGRlKUkJSIiKUtJSkREUpaSlIiIpCwl\nKRERSVlKUiIikrKUpEREJGUpSYmISMoKPUmZWY6ZLTWzz83sujivn2dmi8zsIzN7w8wODzsmERFJ\nD6EmKTOrC/wVyAEOBYaa2U9LFfsCONndDwduBR4MMyYREUkfYdekegDL3X2lu+8EZgH9Ywu4+1vu\nvilYfAdoH3JMIiKSJsJOUu2A1THLXwfryjISeC7UiEREJG2UNzNvdfBEC5rZKcDFwInhhSMiIukk\n7CS1BugQs9yBSG2qhKCzxGQgx93z4+0oNzc3+jw7O5vs7OzqjFNEREKQl5dHXl5elbc394QrO5Xf\nuVk9YBlwGrAWeBcY6u5LYsp0BF4Ghrv722Xsx/ckzhEjcuncObfK28ezcmUu06dX7z5FRGo7M8Pd\nLdHyodak3H2XmV0JzAPqAlPcfYmZXR68/gBwE5AB3GdmADvdvUeYcYVl9OhcNm6s/v22aAF3351b\n/TsWEUlxYTf34e7PA8+XWvdAzPNLgEvCjqMmbNxItdfYIFJrExHZG2nECRERSVlKUiIikrKUpERE\nJGUpSYmISMpSkhIRkZSlJCUiIilLSUpERFKWkpSIiKQsJSkREUlZSlIiIpKylKRERCRlKUmJiEjK\nUpISEZGUFfoo6BKOMKYF0ZQgIpJqlKTSVBjTgmhKEBFJNWruExGRlKUkJSIiKUtJSkREUpaSlIiI\npCwlKRERSVnq3ScVUnd3EUkWJSmpkLq7i0iyKElJylCNTURKU5KSlKEam4iUpo4TIiKSspSkREQk\nZam5T/ZKuv4lkh6UpGSvVFPXv8JIhqCEKHsPJSmREIWRDEEdQmTvoWtSIiKSskKtSZlZDnA3UBd4\nyN3HxykzCTgD2A6McPcFYcYkUlvV1HU2NWFKTQotSZlZXeCvQB9gDfCemT3r7ktiyvQFurr7QWZ2\nHHAf0DOsmFLJypV5dO6cnewwqpWOKbkq07SY6HHFa1asySbMyiTEb75ZyX77da6wXDolw7y8PLKz\ns5MdRlKFWZPqASx395UAZjYL6A8siSnTD/g7gLu/Y2YtzKyNu38bYlwpIZ0+/BKlY0of6XJclUu8\nuQmVLet6XirWRBcuzKN79+wqvU9tEWaSagesjln+GjgugTLtgVqfpEQktdRUj8+aTLy1QZhJyhMs\nZ1XcTkREylEb7gc093Bygpn1BHLdPSdYHgMUxXaeMLP7gTx3nxUsLwV6l27uMzMlLhGRWsLdS1dO\nyhRmTep94CAz6wysBc4FhpYq8yxwJTArSGob412PqswBiYhI7RFaknL3XWZ2JTCPSBf0Ke6+xMwu\nD15/wN2fM7O+ZrYc2AZcFFY8IiKSfkJr7hMREdlTKT3ihJnlmNlSM/vczK5LdjzVwcw6mNkrZvap\nmX1iZr9PdkzVxczqmtkCM5uT7FiqQ3BLxD/MbImZLQ6apNOamY0J/vY+NrPHzKxhsmOqLDObambf\nmtnHMesyzezfZvaZmc03sxbJjLGyyjimCcHf3iIze9rMmiczxsqKd0wxr/3BzIrMLLOi/aRskoq5\nGTgHOBQYamY/TW5U1WIncJW7/4zIjcu/rSXHBTAKWEzt6aF5D/Ccu/8UOJyS9/ilneD68KXAUe5+\nGJFm+CHJjKmKphH5XIh1PfBvdz8YeClYTifxjmk+8DN3PwL4DBhT41HtmXjHhJl1AE4HViWyk5RN\nUsTcDOzuO4Him4HTmrt/4+4Lg+dbiXzw7Z/cqPacmbUH+gIPsfttBWkn+Nbay92nQuQaq7tvSnJY\ne2ozkS9Jjc2sHtCYyGgwacXdXwfyS62ODgwQ/BxQo0HtoXjH5O7/dveiYPEdIveQpo0yfk8AdwHX\nJrqfVE5S8W70bZekWEIRfLM9ksgfYLr7C/BHoKiigmniAGC9mU0zsw/NbLKZNU52UHvC3TcAE4Gv\niPS43ejuLyY3qmoTO1LNt0CbZAYTgouB55IdxJ4ys/7A1+7+UaLbpHKSqi1NRnGZWRPgH8CooEaV\ntszsLOC/weDAaV+LCtQDjgL+5u5HEel9mm5NSCWYWRdgNNCZSO29iZmdl9SgQuCR3mC15vPDzG4A\nCtz9sWTHsieCL3ljgXGxqyvaLpWT1BqgQ8xyByK1qbRnZvWBp4AZ7v7PZMdTDU4A+pnZl8BM4FQz\nezjJMe2pr4l843svWP4HkaSVzo4B3nT37919F/A0kd9dbfCtme0HYGZtgf8mOZ5qYWYjiDSj14Yv\nE12IfEFaFHxWtAc+MLPW5W2UykkqejOwmTUgcjPws0mOaY+ZmQFTgMXufney46kO7j7W3Tu4+wFE\nLsS/7O4XJDuuPeHu3wCrzezgYFUf4NMkhlQdlgI9zaxR8HfYh0hHl9rgWeDC4PmFQNp/+QumOvoj\n0N/df0x2PHvK3T929zbufkDwWfE1kU485X6hSNkkFXzTK74ZeDHweOw0H2nsRGA4cErQXXtB8MdY\nm9SWppbfAY+a2SIivftuT3I8e8TdFwEPE/kCWHxN4MHkRVQ1ZjYTeBM4xMxWm9lFwJ+B083sM+DU\nYDltxDmmi4F7gSbAv4PPib8lNchKijmmg2N+T7ES+pzQzbwiIpKyUrYmJSIioiQlIiIpS0lKRERS\nlpKUiIikLCUpERFJWUpSIiKSspSkJG2ZWWFw/8jHZvaEmTUqo9wbVdz/0WZ2zx7El9bDXSXKzEaX\nde5F9pTuk5K0ZWZb3L1p8HwG8IG7/yXm9XrBTeFJj682C4a4Ocbdv092LFL7qCYltcXrQFcz621m\nr5vZbOAT+F+NxsyyzSzPzJ4MJpObUbyxmR1rZm+Y2UIze8fMmgTl5wSv55rZI2b2ZjCx3iXB+iZm\n9qKZfWBmH5lZv4oCNbMLgonsFhaPcRgM//VysP7FYM4dzGy6mf3NzN4ysxVBTH+3yCSM02L2udXM\n7rLIRJovmllWsL67mb1t/5s4r0WwPs/M/hwc6zIzOylYX9cik+29G2xzWXnnziKTdu4PvGJmL5lZ\nnSDmj4PzMXrPfq2y13N3PfRIywewJfhZD5gNXA70BrYCneKUywY2EvlQNSJDtpwANABWAEcH5ZoQ\nmRAwG5gTrMsFFgANgZZEprtoG5RrGpTJAj4v/b6lYv4ZsAzIDJZbBD/nAOcHzy8CngmeTwceC573\nIzIn1M+C+N8HDg9eKwKGBs9vBO4Nnn9EZF4sgJuBvwTPXwEmBM/PIDJhIMBlwA3B84bAe0QGBY17\n7oJyX8Ycz9HA/JjjbZ7svxM90vuhmpSks0ZmtoDIB+lKYCqRD9B33b2sWT/fdfe17u7AQiLzRh0C\nrHP3DyAyGaW7F5bazoHZ7r7DI81arxCZmNOAO4Lx/f4N7F/BqM6nAk94ZG4n3H1jsL4nUDwVwwzg\npJj3nRM8/wT4xt0/DeL/lEgCgUiSejx2ezNrRiRJvB6s/ztwckwsTwc/P4zZz8+BC4Lz+jaQCXQN\n4ih97jqzuxXAgWY2ycx+QSSpilRZvWQHILIHfnD3I2NXRAb3Zls52+yIeV5I5H+gqhdmnchgwVlE\nRnMuDK7P7FPBNmXNoVPW+oLgZxEl4y8i/v+wEf+YSu+/eF/F56HYle7+7xIbmmUT/9yV4O4bzexw\nItOGXwEMBkbGiUUkIapJyd7OiTS/tTWzYwDMrKmZ1S1VzoD+ZtbQzFoSaf56F2hGZMLHQjM7BehU\nwfu9DAwys8zgvTKC9W8SmeYEInMHvVbJ46gDDAqeDwNed/fNQH7x9SbgfCCvgv3MA35jkenlMbOD\nreIZibcQOQ8E56aeuz9NpNkx3efgkiRTTUrSWbzaQrxZWb2M55EV7jvN7Fzg3qAr9Xbg9FL7ciLX\nd14hUnO6xd2/MbNHgTlm9hGRa0Sx08nEe6/FZvZ/wKtmVkikqe1iItOCTDOzPxKZsO+iMvZTVq1v\nG9DDzP5EZPr0c4P1FwL3B4lmRan9lggt+PkQkWa8Dy1SLf0vcA7xz2uxB4EXzGwNcFVwHMVfgNN6\nNmNJPnVBF0mAmY0Dtrr7xGTHEs/e0t1d9j5q7hNJXCp/o0vl2ESqTDUpERFJWapJiYhIylKSEhGR\nlKUkJSIiKUtJSkREUpaSlIiIpCwlKRERSVn/H0Z1R7aS8QYmAAAAAElFTkSuQmCC\n", + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAakAAAEbCAYAAABgLnslAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8VNX9//HXh7AIsiQQQHYUW1tqFTekWCQq+kVUwFIQ\nEBXFrZUWXEoFq+Dy0/JDrOL354YsVVTUikWKCHWJ2rrggriAKFQQA1qUsCOB5PP7496Mk/2GZJJJ\neD8fj3lk7p1773zOHZjPnHPPPcfcHRERkWRUp7oDEBERKYmSlIiIJC0lKRERSVpKUiIikrSUpERE\nJGkpSYmISNJKaJIys5lm9o2ZfVTKNtPM7HMzW25mxyQyHhERqVkSXZOaBfQt6UUz6wcc7u4/Ai4H\n7k9wPCIiUoMkNEm5++tAdimb9Af+Gm77NpBqZq0TGZOIiNQc1X1Nqh2wPm75K6B9NcUiIiJJprqT\nFIAVWtY4TSIiAkDdan7/LKBD3HL7cF0BZqbEJSJSS7h74cpJiaq7JvUccCGAmfUAtrj7N8Vt6O61\n6jFx4sRqj0FlOjDLVFvLlexlSktzgoai8jwmlnuftLTqL2tpj/JKdBf0J4A3gCPMbL2ZXWJmV5jZ\nFQDu/jzwHzNbDTwI/DaR8YiIVJfsbHAv32PixPLvs3lzdZe0ciW0uc/dh0XYZnQiYxARkZqruq9J\nHbAyMjKqO4RKpzLVHLWxXPtTpubNgxpOVUhLK/8+tfFzKi/bnzbCqmZmXhPiFJGaxSxoIpOqY2Z4\nDeo4ISIiUiIlKRERSVq6JiUiSSXZrxNJ1dI1KRFJKrpOVLvpmpSIiNQaSlIiIpK0lKRERCRpqeOE\niERSVR0a1JlB4qnjhIhEog4NUhnUcUJERGoNJSkREUlaSlIiIpK01HFCpAbT6AxS26njhEgNps4M\nUtOo44SIiNQaSlIiIpK0lKRERCRpKUmJiEjSUu8+kQTQEEIilUO9+0QSQL3uRIqn3n0iIlJrKEmJ\niEjSUpISEZGkpSQlIiJJS0lKRESSlpKUiIgkLSUpERFJWrqZVw4YmtZCpObRzbxywNANtiLVTzfz\niohIraEkJSIiSUtJSkREkpaSlIiIJC0lKRERSVpKUiIikrSUpEREJGklNEmZWV8z+9TMPjezPxbz\nerqZvWBmH5jZx2Y2MpHxiIhIzZKwm3nNLAVYBfQBsoB3gGHuvjJum0lAA3cfb2bp4fat3X1foWPp\nZl6pMN3MK1L9kulm3u7Aandf6+57gbnAgELbbASahs+bAt8VTlBS+zVvHiSQRD80VJFIzZPIsfva\nAevjlr8CTiy0zXTgZTPbADQBhiQwHklS2dmq4YhI8RJZk4rytTMB+MDd2wLdgP9nZk0SGJOIiNQg\niaxJZQEd4pY7ENSm4vUE/g+Au68xsy+AI4B3Cx9s0qRJsecZGRlkZGRUbrQiIlLpMjMzyczM3O/9\nE9lxoi5BR4jTgA3AUop2nLgL2OruN5tZa+A94Ch331zoWOo4UYupQ4PIgaO8HScSVpNy931mNhpY\nDKQAM9x9pZldEb7+IHA7MMvMlhM0PY4rnKBEROTApfmkpNqpJiVy4EimLugiIiIVoiQlIiJJS0lK\nRESSlpKUiIgkLSUpERFJWkpSIiKStJSkREQkaZWZpMysg5k9a2abwsczZta+KoITEZEDW5Sa1Czg\nOaBt+FgQrpNarKqmz9AUGiJSmjJHnDCz5e5+dFnrEkkjTlQ9jQIhIomQiBEnvjOzC8wsxczqmtkI\n4Nv9D1FERCSaKEnqEoLJCL8mmEl3MHBxIoMSEREBDTArJVBzn4gkQqVN1WFmf3T3yWZ2bzEvu7v/\nfr8iFBERiai0+aRWhH/fo+BU8Ea0qeFFREQqpMQk5e4Lwqe73P2p+NfMbEhCoxIRESFaF/Rl7n5M\nWesSSdekqp6uSYlIIlTmNakzgX5AOzObRtDMB9AE2FuhKEVERCIo7ZrUBoLrUQPCv/lJahtwdYLj\nEhERidTcV9/dc6oonpJiUHNfFVNzn4gkQqU198XpbGa3A12BhuE6d/fD9idAERGRqKIOMPsAsA/I\nAP4KPJbAmERERIBoSaqhu79I0DS4zt0nAWclNiwREZFozX3fm1kKsNrMRhN0qDg4sWGJiIhE6zhx\nAvApkArcCjQF/q+7v5X48GIxqONEFVPHCRFJhPJ2nCg1SYU1qMnufl1lBLe/lKSqnpKUiCRCpfbu\nc/dcM/ulKUskhebNITu7at5Ls+WKSDKI0tz3AMG08U8Du8LV7u7zEhxbfAzKkah2IyI1XyLukzoI\n+A44tdD6KktSIiJyYNKkhzWIalIiUtOVtyYV5T4pERGRaqEkJSIiSUtJSkREklaZScrMDjGzGWb2\nQrjc1cxGJT40ERE50EWpSc0GlhB0Qwf4HM0nJSIiVSBKkkp39yeBXAB330swIrqIiEhCRUlSO8ys\nRf6CmfUAtiYuJBERkUCUm3mvBRYAh5nZG0BL4NcJjUpERISIN/OaWT3gCMCAT8Mmvyj79QXuBlKA\nh919cjHbZAB/AeoB37p7RjHb6GZedDOviNR8lToKenjA0cBj7p4dLqcBw9z9vjL2SwFWAX2ALOCd\ncL+VcdukAv8G/sfdvzKzdHf/tphjKUmhJCUiNV8iRpy4LD9BAYTPL4+wX3dgtbuvDWtec4EBhbYZ\nDjzj7l+Fxy6SoERE5MAVJUnVMbPYdmENqV6E/doB6+OWvwrXxfsR0NzMXjGzd83sggjHFRGRA0SU\njhOLgblm9iDBNakrgBci7BelYaoecCxwGtAIeNPM3nL3zyPsKyIitVyUJPVHgua934TL/wQejrBf\nFtAhbrkDQW0q3nqCzhK7gd1m9hpwNMENwwVMmjQp9jwjI4OMjIwIIYiISHXKzMwkMzNzv/dP2FQd\nZlaXoOPEacAGYClFO078BPhf4H+ABsDbwHnuvqLQsdRxAnWcEJGar9InPTSzXwITgc5x27u7H1ba\nfu6+L+wZuJigC/oMd19pZleErz/o7p+GYwJ+COQB0wsnKBEROXBF6YK+ChgLvE84NBJUbU881aQC\nqkmVzizyjzMRqQLFfW8nYvr4Le6+qDyBiVQX/ZgRSQ6V9aMxSk3qzwTNdfOAPfnr3f39SokgAtWk\nAqpJlS78hVbdYYgIJf9/TMSIE5kU053c3U+J+iYVpSQVUJIqnZKUSPKosiSVDJSkAkpSpVOSEkke\nlZWkIk0fb2Znm9k4M7sp/1GOWEUkCcyePZtevXrt9/79+vXj0UcfrcSIEmft2rXUqVOHvLy8Mrd9\n/fXX+clPfpKQOEaOHMmNN96YkGMXVpM+n/KI0gX9QaAhcCowHRhMcD+ThJo3h+zssrerqLS0xL+H\nCAQ3z69Zs6bAl97zzz9fjRElTq9evfj0008Tcmwzq7Jep7X184nSu6+nu//czD5095vNbCrRhkU6\nYGRnqxlORIqX6Cbo/OPX1lswojT37Q7/7jKzdgRTxx+SuJBEap/169fzq1/9ilatWpGens7vfvc7\nIKixXHDBD+MqF26mysjI4MYbb+Skk06iSZMm9O/fn2+//Zbzzz+fZs2a0b17d9atW1fsvvn7z5gx\no9iYxowZQ8eOHWnWrBnHH388//rXvwB44YUXuOOOO3jyySdp0qQJxxxzTIFj7dmzh9TUVD755JPY\nsTZt2kSjRo349tvg9sl//OMfdOvWjbS0NE466SQ++uijEs/Np59+yumnn06LFi34yU9+wtNPPw3A\nmjVraNGiBcuWLQNgw4YNtGzZktdeey0Wz/jx4znxxBNp1qwZAwcOJLuEJo1Zs2bRtWtXmjZtSpcu\nXXjooYdir2VmZtKhww8juHXu3JmpU6dy9NFHk5qaytChQ9mzJ9axudSyLVu2jGOPPZamTZsydOhQ\nvv/++2LjKescZmdnc/bZZ9OqVSuaN2/OOeecQ1ZWVmzbjIwM/vSnP3HSSSfRuHFj/vOf/xT4rNes\nWcOpp55Keno6LVu2ZMSIEWzd+sOE6mWVcf78+XTr1o1mzZpx+OGHs3jxYgC2bt3KqFGjaNu2Le3b\nt+fGG2+M1KRaEVGS1D/COaSmAO8Ba4EnEhmUSG2Sm5vL2WefzaGHHsq6devIyspi2LBhQLRfv08+\n+SRz5swhKyuLNWvW8Itf/IJRo0axefNmfvrTn3LzzTeXuG9pzU3du3dn+fLlZGdnM3z4cAYPHkxO\nTg59+/ZlwoQJDB06lO3bt8eSRP6xGjRowKBBg3jiiR++Bp566ikyMjJIT09n2bJljBo1iunTp7N5\n82auuOIK+vfvT05OTpEYdu7cyemnn86IESPYtGkTc+fO5be//S0rV66kS5cuTJ48mREjRrB7924u\nvvhiLr74Yk4++eTY/o8++iizZs1i48aN1K1bl9///vfFlrV169YsXLiQbdu2MWvWLK6++upYuYo7\nZ08//TSLFy/miy++4MMPP2T27NkAJZZt79695OTkMHDgQC666CKys7MZPHgwzzzzTLHnv6xz6O6M\nGjWKL7/8ki+//JKGDRsyevToAseYM2cODz/8MNu3b6dTp05FPusbbriBjRs3snLlStavX19g/NPS\nyrh06VIuuugipk6dytatW3nttdfo3LkzEFxjq1+/PmvWrGHZsmUsWbKEhx+OMpRrBbh75AdwEJBa\nnn0q4xGEmbySPLwDRln/ToJG2Yo/yuuNN97wli1bem5ubpHXJk6c6CNGjIgtf/HFF25msW0zMjL8\n9ttvj71+7bXXer9+/WLLCxYs8G7duhW7b/7+M2bMcHf3WbNm+S9/+csS40xLS/MPP/yw2LgKH+vF\nF1/0Ll26xF7r2bOnP/roo+7ufuWVV/qNN95YYN8jjjjCX3311SLvOXfuXO/Vq1eBdZdffrnffPPN\nseX+/fv7kUce6UcffbTn5OQUiGf8+PGx5RUrVnj9+vU9Ly+v2HMRb+DAgX7PPfe4u/srr7zi7du3\nj73WuXNnf+yxx2LL48aN8yuvvLLMsr366qvetm3bAq/17NmzyPb5SjuHhS1btszT0tIKlH3ixIkF\nton/fAp79tln/ZhjjolUxssvv9yvueaaIsf4+uuvvUGDBr579+7Yuscff9xPOeWUYt+zpP+P4frI\n3/8lXpMys9Pc/SUzG0Sh+6TCLoTzEpY5RRKguq4brl+/nk6dOlGnTqTOtEW0bt069vyggw6iVatW\nBZZ37NixX8e98847mTlzJhs2bMDM2LZtW6y5riwZGRns2rWLpUuX0qpVK5YvX865554LwLp163jk\nkUe49957Y9vv3buXjRs3FjnOunXrePvtt0mL6xW0b98+LrzwwtjypZdeyoABA5g+fTr16hWcyi6+\nma5jx47s3bu32DIsWrSIm2++mc8//5y8vDx27drFUUcdVWL5DjnkhysaDRs2ZMOGDWWWzd1p167g\nlHmdOnUq8ZpUaedw165dXH311SxevDjWhLljxw7cPVZbii97Yd988w1jxozhX//6F9u3bycvL4/m\nzZuXWsb8z+err77irLPOKnLMdevWsXfvXtq0aRNbl5eXR8eOHUuMozKU1nHiZOAl4ByKnxtKSUok\ngg4dOvDll1+Sm5tLSkpKgdcaN27Mrl27Ystff/11qccqrXnw4IMPBoIvuMaNG5d6vNdff50pU6bw\n8ssv87Of/QyA5s2bR74In5KSwpAhQ3jiiSdo1aoV55xzTuz9O3bsyA033MCECRNKPUb+tr1792bJ\nkiXFvr5jxw7Gjh3LpZdeysSJE/nVr35VIKF9+eWXBZ7Xq1eP9PR0du7cGVu/Z88eBg0axJw5cxgw\nYAApKSmce+655erQkH8+Sivbq6++WuC6EQRf7IcffnixxyztHE6dOpXPPvsslsA++OADjj322AJJ\nqrTPaMKECaSkpPDxxx+TmprK3//+99h10LJ06NCB1atXF7u+QYMGfPfdd/v9g2t/lPhO7j4xnJF3\nkbtfXPhRZRGK1HAnnngibdq04frrr2fXrl18//33vPHGGwB069aN1157jfXr17N161buuOOOIvvH\nf5mW9sXasmVL2rVrx6OPPkpubi4zZ85kzZo1xW67fft26tatS3p6Ojk5Odxyyy1s27Yt9vohhxzC\n2rVri7xf/PLw4cOZO3cujz/+OMOHD4+tv+yyy3jggQdYunQp7s7OnTtZuHBhsTW+s88+m88++4w5\nc+awd+9e9u7dyzvvvBPrEj5mzBi6d+/OQw89xFlnncWVV15ZIJY5c+awcuVKdu3axU033cTgwYOL\nfHnn5OSQk5NDeno6derUYdGiRSUmxZLkl7u0svXs2ZO6desybdo09u7dy7x583jnnXdKPW5J53DH\njh00bNiQZs2asXnz5mKvO5b2b2HHjh0cfPDBNG3alKysLKZMmRK5jKNGjWLWrFm8/PLL5OXlkZWV\nxapVq2jTpg1nnHEG11xzTax2tmbNmlhHlkQpNR26ex4wLqERiNRyderUYcGCBaxevZqOHTvSoUMH\nnnrqKQD69OnDeeedx1FHHcUJJ5zAOeecU+RLNn65uI4Q8cvTp09nypQppKens2LFCk466aRi9+3b\nty99+/blxz/+MZ07d6Zhw4YFmm0GDx4MQIsWLTj++OOLfa/u3bvTuHFjNm7cyJlnnhlbf9xxxzF9\n+nRGjx5N8+bN+dGPfsQjjzxS7Llp3LgxS5YsYe7cubRr1442bdowfvx49uzZw/z581myZAn3338/\nAHfddRfvv/9+rLOBmXHBBRcwcuRI2rRpQ05ODtOmTSsSa5MmTZg2bRpDhgyhefPmPPHEEwwYMKDE\nc1hY/HkrrWz16tVj3rx5zJ49mxYtWvDUU08xaNCgEo9b2jkcO3Ysu3fvJj09nZ49e3LmmWeW+rkX\nNnHiRN5//32aNWvGOeecw6BBgyKX8YQTToh1LklNTSUjIyNWY33kkUfIycmha9euNG/enMGDB5dZ\n+6+oqAPMfgs8CcTq0O6+OaGRFYzBy1M1r2oarig5aFikA8spp5zCBRdcwCWXXFLdoUgxKmtYpCg3\n8w4luCZ1VaH1h0Z9ExGRRNCPktqvzCTl7p2rIA4RkXKrraMsyA8ijYJuZkcCXQnukwLA3YtvZE4A\nNfdJFGruE0keVdbcZ2aTgN7Az4CFwJnAv4AqS1IiInJgitLZ/ddAH2Bj2PX8aCA1oVGJiIgQcYBZ\nd88F9plZM+C/QMm3OouIiFSSKL373g0HmJ0OvEvQDf2NhEYlIiJCOaePN7NDgabuvjxxIRX7vuo4\nIWVSxwmR5FFl08eb2QIzG25mB7v7F1WdoERqgyOPPHK/h4+Jn4K8PFOdlzadeHmmVy9PfFWl8Dxc\npfnNb37DbbfdlpA46tSpw3/+85+EHDvel19+SZMmTQ7IH2FRmvumAucBd5jZuwRzSf3D3YufzUsk\niYwdO4ktWxJ3/NRUuPvuSWVu9/HHH+/3e8QPWVOeqc6rajrxqpwiPf49o8ofVqkm69ixI9u3b6/u\nMKpFlJt5M4FMM6sLnAJcBswEmiY2NJGK27IFOneelLDjr12buGPHS/Zf0FUdX7Kfj8q0b98+6taN\nUp+onSKNt25mDYFBwJXACcBfExmUSG3TuXNnXn75ZSBoqhoyZAgXXXQRTZs25cgjj+S9996LbVva\nFOTxU51Pnjw5NhBsvjFjxjBmzBig4NTxubm5XHfddbRs2ZIuXbqwcOHCIvG99NJLseXCzWmDBw+m\nTZs2pKam0rt3b1asWBG57DNnzowNSNq3b9/YYKWTJ0+mR48e5ObmAkGN58gjjyQnJyfWHDl9+nTa\ntWtH27ZtmTp1aonvUVp88c2RmZmZtG/fnrvuuovWrVvTtm3b2Iy0EEzrcd1119GpUycOOeQQfvOb\n3xQ4/1OmTIlNnT5z5swS43nyySc54YQTCqz7y1/+EhvYduHChRxzzDE0a9aMjh07FhjlPL/sM2fO\npFOnTvTp04d169YVaJ6dNWsWXbt2pWnTpnTp0oWHHnootn9ZZdy9ezfXXnstnTt3JjU1lV69esXK\n+NZbb9GzZ0/S0tLo1q0br776aollrCpRrkk9BXwKnAr8L9DF3aNNTCIiQNHmqQULFjBs2DC2bt1K\n//79Y1ODl2cK8qFDh/L888/HpsDIzc3l6aef5vzzz4+9Z/5+06dPZ+HChXzwwQe8++67/O1vfyt1\ndPXC73fWWWexevVqNm3axLHHHht7j7LMnz+fO+64g2effZZvv/2WXr16MWzYMADGjRtHgwYNuO22\n2/j888+54YYbeOyxx6hfv35s/8zMTFavXs2SJUuYPHlygUQaNb7CZfvmm2/Ytm0bGzZsYMaMGVx1\n1VVs3boVgOuvv57Vq1ezfPlyVq9eTVZWFrfccgsAL7zwAlOnTuXFF1/ks88+48UXXyyx3P3792fV\nqlUF5mV6/PHHY3E1btyYOXPmsHXrVhYuXMj999/P/PnzCxzjtdde49NPP2Xx4sVFao6tW7dm4cKF\nbNu2LTZi+bJlyyKV8brrrmPZsmW8+eabbN68mSlTplCnTh2ysrI4++yzuemmm8jOzubOO+9k0KBB\nkSfCTJQoNamZBInpCnd/JZy+Q0QqoFevXvTt2xczY8SIESxfHvRHeuutt9i3bx9jxowhJSWFQYMG\nFflFnq9Tp04ce+yxPPvsswC8/PLLNGrUiO7duxfZ9qmnnuLqq6+mXbt2pKWlMWHChHI1mY0cOZKD\nDz6YevXqMXHiRJYvXx7pGskDDzzA+PHjOeKII6hTpw7jx4/ngw8+YP369ZgZjzzyCNOmTWPAgAH8\n8Y9/5Oijjy6w/8SJE2nYsCFHHnkkF198cWyajvLGF1/WevXqcdNNN5GSksKZZ55J48aNWbVqFe7O\n9OnTueuuu0hNTaVx48aMHz+euXPnxs7hJZdcQteuXWnUqFGxczzla9iwIQMGDIjF+/nnn7Nq1Sr6\n9+8PQO/evWOTTf785z9n6NChRWotkyZNomHDhjRo0KDI8fv168ehhwZjfJ988smcccYZvP7662WW\nMS8vj1mzZnHPPffQpk0b6tSpQ48ePahfvz5z5syhX79+9O3bFwimkTn++OOr7NpmScpMUu7+grvv\nq4pgRA4U8VPCN2rUiO+//568vDw2bNhQ7BTkJRk+fHjsizD+l3phGzduLDLVelS5ublcf/31HH74\n4TRr1iz25RjlF/a6desYM2YMaWlppKWl0aJFC4DYDLadOnUiIyODdevWcdVVhSdaKDo9fP407hWJ\nr0WLFgVmlm3UqBE7duxg06ZN7Nq1i+OOOy4W75lnnhk7TnnPYeHP5txzz+Wgg4LhT99++21OOeUU\nWrVqRWpqKg8++CDfffddiWUvbNGiRfTo0YMWLVqQlpbG888/X2D/ksr47bff8v3339OlS5cix1y3\nbh1PP/10rOxpaWn8+9//Tvh8UWWpujmARaRMbdq0KXYK8pL8+te/JjMzk6ysLP7+978XmN218HEL\nT7Ue7+CDDy4w5frGjRtjTWSPP/44zz33HC+99BJbt27liy++AKJ1XujYsSMPPfQQ2dnZscfOnTvp\n0aMHEFybeeuttzjttNO47rrriuxfOObCCTxqfFF6A6anp9OwYUNWrFgRi3XLli2xGYvLOoeF9enT\nh02bNrF8+XLmzp1b4LMZPnw4AwcO5KuvvmLLli1ceeWVRW4HKCnmPXv2MGjQIMaNG8d///tfsrOz\n6devX6TPIz09nYMOOqjY6eE7duzIBRdcUOCz2r59O+PGVe+8t0pSIknkF7/4RbmmIG/ZsiUZGRmM\nHDmSww47jCOOOKLY7YYMGcK0adPIysoiOzubP//5zwVe79atG3PnzmXfvn28++67PPPMM7HXduzY\nQYMGDWjevDk7d+5kwoQJBfYt7cvxyiuv5Pbbb491ZNi6dStPP/00ENR0LrvsMmbMmMHs2bNZsGAB\nixYtKrD/bbfdxu7du/nkk0+YPXs25513XpH3iBJflC/wOnXqcNlllzF27Fg2bdoEBDW+/KnmhwwZ\nwuzZs2PT1ZfW3AdBk9vgwYO57rrryM7O5vTTTy8Qc1paGvXr12fp0qU8/vjjkbvV5+TkkJOTQ3p6\nOnXq1GHRokWxGKOU8ZJLLuGaa65h48aN5Obm8uabb5KTk8OIESNYsGABS5YsITc3l++//z72A6g6\nldiv0cyOI5js0MK/Bbj7+wmMS6RSpKYmtpt46n4MtVzaFPD169dn3rx5XHbZZfzpT3+iX79+RaYg\nL7zv8OHDufDCC5kyZUqJ73nZZZfx2WefcfTRR9OsWTOuvfZaMjMzY6/feuutDBs2jLS0NHr37s35\n55/P5s3B5NsXXnghixcvpl27drRo0YJbbrmFBx98sNTy5Bs4cCA7duxg6NChrFu3jmbNmnHGGWcw\nePBgrrjiCgYOHBi7BjJjxgxGjRpV4J6y3r17c/jhh5OXl8cf/vAH+vTpU+Q9yxtfaclg8uTJ3HLL\nLfTo0YNvv/2Wdu3a8dvf/pYzzjiDvn37MnbsWE499VRSUlK49dZbS7xGlm/48OGcfPLJXHXVVQWa\n3+677z6uvfZaRo8eTe/evTnvvPPYEndDX3Ex5q9r0qQJ06ZNY8iQIezZs4dzzjkn1mswShnvvPNO\nxo8fzwknnMCOHTvo1q0bL7zwAu3bt2f+/PmMGzeOYcOGkZKSwoknnsh9991XahkTrcRhkcwskyA5\nNQSOAz4MXzoKeNfdf1EVAYaxaFgkKZOGRao91q5dy2GHHca+ffsKfLlLzZHwYZHcPcPdTwE2AMe6\n+3HufhxwTLhOREQkoaL8RPmJu3+Uv+DuHwM/TVxIIiKaGl4CZY6CbmZzgR3AHILrU8OBxu4+LPHh\nxWJQc5+USc19Ismjspr7oiSphsBvgF7hqteA+6tygFklKYlCSUokeVRZkgoP2gjo6O7Rhl/+Yb++\nwN1ACvCwu08uYbsTgDeBIe4+r5jXlaSkTEpSIsmjKueT6g8sA14Il48xs+ci7JdCMNZfX6ArMMzM\nilzLCrebHB5fjdAiIhITZfz3ScCJwCsA7r7MzA6LsF93YLW7r4XYta0BwMpC2/0O+BvB6OoiFaKL\n7SK1S5TJ1EemAAAPvklEQVQktdfdtxT6zx9lkNl2wPq45a8Ikl2MmbUjSFynEiQptdXIflNTn0jt\nE6UL+idmdj5Q18x+ZGb3Am9E2C/KN8bdwPXhBSdDzX0iIhInSk3qd8ANwB6CqeMXA7dG2C8LiB/G\ntwNBbSreccDcsJaWDpxpZnvdvcg1r0mTJsWeZ2RkkJGRESEEERGpTpmZmQWG4CqvSL379uvAwXTz\nq4DTCEaoWAoMc/fC16Tyt58FLFDvPhGR2qu8vfvKrEmZ2RHAdUDnuO3d3U8tbT9332dmowlqXinA\nDHdfaWZXhK8/WNr+IiIiUW7m/RC4H3gfyA1Xu7u/l+DY4mNQTUpEpBao9JoUQe+++ysQk4iIyH6J\n0rtvgZldZWZtzKx5/iPhkYmIyAEvSnPfWoqf9PDQBMVUXAwVau4bO3YScfOJleqvf51U7uOnpUE4\nP5yIiJSi0pv73L1zhSJKAlu2QOfOkyJtO3FitGOuXTuJ2bOjHVNERPZPadPHn+buL5nZIIqvSRXp\nKi4iIlKZSqtJnQy8BJxD8aNHKEmJiEhClZik3H1i+HdklUUjIiISJ0oXdMzsbILpNg7KX+futyQq\nKBEREYg2n9SDwBDg9wQDwA4BOiU4LhERkUj3SfV09wuBze5+M9ADOCKxYYmIiERLUrvDv7vC+Z/2\nAYckLiQREZFAlGtS/zCzNGAKkD9e3/TEhSQiIhKIcjNvfgeJZ8xsIXCQu0ccv0FERGT/lXYzb/xN\nvBb3PH9YC90nJSIiCVVaTaqkm3jzKUmJiEhClXYz78gqjENERKSIKPdJpZvZvWa2zMzeN7N7zKxF\nVQQnIiIHtihd0OcC/wV+Bfwa2AQ8mcigREREIFoX9EPc/da45dvM7LxEBSQiIpIvSk1qiZkNM7M6\n4eM8YEmiAxMREYmSpC4HHgNywscTwOVmtt3MtiUyOBERObBFuZm3cVUEIiIiUliU3n2jCi3XNbOI\nk6yLiIjsvyjNfX3M7Hkza2tmRwJvAk0THJeIiEik5r5hZjYU+BDYCZzv7v9KeGQiInLAi9Lc92OC\nCQ/nAV8CI8zs4EQHJiIiEqW57zngJne/HOgNfA68k9CoREREiHYz74nuvhXA3fOAqWa2ILFhiYiI\nlFKTMrNxAO6+1cwGF3p5ZCKDEhERgdKb+4bFPZ9Q6LUzExCLiIhIAVGuSYmIiFQLJSkREUlapXWc\nOMrMtofPG8Y9B2iYwJhERESA0mfmTanKQERERApTc5+IiCQtJSkREUlaSlIiIpK0lKRERCRpJTxJ\nmVlfM/vUzD43sz8W8/r5ZrbczD40s3+b2VGJjklERGqGhCYpM0sB/hfoC3QFhpnZTwtt9h/gZHc/\nCrgVeCiRMYmISM2R6JpUd2C1u691973AXGBA/Abu/mb+ALbA20D7BMckIiI1RKKTVDtgfdzyV+G6\nkowCnk9oRCIiUmNEmaqjIjzqhmZ2CnAJcFLiwhERkZok0UkqC+gQt9yBoDZVQNhZYjrQ192zizvQ\npEmTYs8zMjLIyMiozDhFRCQBMjMzyczM3O/9zT1yZaf8BzerC6wCTgM2AEuBYe6+Mm6bjsDLwAh3\nf6uE43hF4hw5chKdO0/a7/2Ls3btJGbPrtxjiojUdmaGu1vU7RNak3L3fWY2GlgMpAAz3H2lmV0R\nvv4gcBOQBtxvZgB73b17IuMSEZGaIdHNfbj7ImBRoXUPxj2/FLg00XFUhbFjJ7FlS+UfNzUV7r57\nUuUfWEQkySU8SR1Itmyh0psVIWhaFBE5EGlYJBERSVpKUiIikrSUpEREJGkpSYmISNJSkhIRkaSl\nJCUiIklLSUpERJKWkpSIiCQtJSkREUlaSlIiIpK0lKRERCRpKUmJiEjSUpISEZGkpSQlIiJJS0lK\nRESSlpKUiIgkLU16WEMlYhZgzQAsIslGSaqGSsQswJoBWESSjZr7REQkaSlJiYhI0lKSEhGRpKUk\nJSIiSUtJSkREkpaSlIiIJC11QZcy6Z4sEakuSlJSJt2TJSLVRc19IiKStFSTkqShZkURKUxJSpKG\nmhVFpDA194mISNJSkhIRkaSl5j45IOn6l0jNoCQlB6Squv6ViGQISohy4FCSEkmgRCRDUIcQOXDo\nmpSIiCQtJSkREUlaau4TqSWqqjOIrrNJVUpokjKzvsDdQArwsLtPLmabacCZwC5gpLsvS2RMIrVV\nVXUGqcrrbOqFKQlLUmaWAvwv0AfIAt4xs+fcfWXcNv2Aw939R2Z2InA/0CNRMSWTtWsz6dw5o7rD\nqFQqU81RU8pVnoQYtUwldTpJxoSYmZlJRkZGpcVTEyWyJtUdWO3uawHMbC4wAFgZt01/4K8A7v62\nmaWaWWt3/yaBcSWFmvIlUR4qU81RG8tV0TIl420JH3yQSbduGWVuV5trh4lMUu2A9XHLXwEnRtim\nPVDrk5SIHJjKVzucFGnbmlQ7LK9EJimPuJ3t534iIlKK2jBos7knJieYWQ9gkrv3DZfHA3nxnSfM\n7AEg093nhsufAr0LN/eZmRKXiEgt4e6FKyclSmRN6l3gR2bWGdgAnAcMK7TNc8BoYG6Y1LYUdz2q\nPAUSEZHaI2FJyt33mdloYDFBF/QZ7r7SzK4IX3/Q3Z83s35mthrYCVycqHhERKTmSVhzn4iISEUl\n9bBIZtbXzD41s8/N7I/VHU9lMLMOZvaKmX1iZh+b2e+rO6bKYmYpZrbMzBZUdyyVIbwl4m9mttLM\nVoRN0jWamY0P/+19ZGaPm1mD6o6pvMxsppl9Y2Yfxa1rbmb/NLPPzGyJmaVWZ4zlVUKZpoT/9pab\n2Twza1adMZZXcWWKe+1aM8szs+ZlHSdpk1TczcB9ga7AMDP7afVGVSn2Ale7+88Ibly+qpaUC2AM\nsILa00PzHuB5d/8pcBQF7/GrccLrw5cBx7r7zwma4YdWZ0z7aRbB90K864F/uvuPgZfC5ZqkuDIt\nAX7m7kcDnwHjqzyqiimuTJhZB+B0YF2UgyRtkiLuZmB33wvk3wxco7n71+7+Qfh8B8EXX9vqjari\nzKw90A94mKK3FdQ44a/WXu4+E4JrrO6+tZrDqqhtBD+SGplZXaARwWgwNYq7vw5kF1odGxgg/Duw\nSoOqoOLK5O7/dPe8cPFtgntIa4wSPieAu4BxUY+TzEmquBt921VTLAkR/rI9huAfYE33F+APQF5Z\nG9YQhwKbzGyWmb1vZtPNrFF1B1UR7r4ZmAp8SdDjdou7v1i9UVWa+JFqvgFaV2cwCXAJ8Hx1B1FR\nZjYA+MrdP4y6TzInqdrSZFQsM2sM/A0YE9aoaiwzOxv4bzg4cI2vRYXqAscC97n7sQS9T2taE1IB\nZtYFGAt0Jqi9Nzaz86s1qATwoDdYrfn+MLMbgBx3f7y6Y6mI8EfeBGBi/Oqy9kvmJJUFdIhb7kBQ\nm6rxzKwe8Awwx93/Xt3xVIKeQH8z+wJ4AjjVzB6p5pgq6iuCX3zvhMt/I0haNdnxwBvu/p277wPm\nEXx2tcE3ZnYIgJm1Af5bzfFUCjMbSdCMXht+THQh+IG0PPyuaA+8Z2atStspmZNU7GZgM6tPcDPw\nc9UcU4WZmQEzgBXufnd1x1MZ3H2Cu3dw90MJLsS/7O4XVndcFeHuXwPrzezH4ao+wCfVGFJl+BTo\nYWYNw3+HfQg6utQGzwEXhc8vAmr8j79wqqM/AAPc/fvqjqei3P0jd2/t7oeG3xVfEXTiKfUHRdIm\nqfCXXv7NwCuAJ+On+ajBTgJGAKeE3bWXhf8Ya5Pa0tTyO+AxM1tO0Lvv9mqOp0LcfTnwCMEPwPxr\nAg9VX0T7x8yeAN4AjjCz9WZ2MfBn4HQz+ww4NVyuMYop0yXAvUBj4J/h98R91RpkOcWV6cdxn1O8\nSN8TuplXRESSVtLWpERERJSkREQkaSlJiYhI0lKSEhGRpKUkJSIiSUtJSkREkpaSlNRYZpYb3j/y\nkZk9ZWYNS9ju3/t5/OPM7J4KxFejh7uKyszGlnTuRSpK90lJjWVm2929Sfh8DvCeu/8l7vW64U3h\n1R5fbRYOcXO8u39X3bFI7aOalNQWrwOHm1lvM3vdzOYDH8MPNRozyzCzTDN7OpxMbk7+zmZ2gpn9\n28w+MLO3zaxxuP2C8PVJZvaomb0RTqx3abi+sZm9aGbvmdmHZta/rEDN7MJwIrsP8sc4DIf/ejlc\n/2I45w5mNtvM7jOzN81sTRjTXy2YhHFW3DF3mNldFkyk+aKZpYfru5nZW/bDxHmp4fpMM/tzWNZV\nZvbLcH2KBZPtLQ33uby0c2fBpJ1tgVfM7CUzqxPG/FF4PsZW7GOVA56766FHjXwA28O/dYH5wBVA\nb2AH0KmY7TKALQRfqkYwZEtPoD6wBjgu3K4xwYSAGcCCcN0kYBnQAGhBMN1Fm3C7JuE26cDnhd+3\nUMw/A1YBzcPl1PDvAuCC8PnFwLPh89nA4+Hz/gRzQv0sjP9d4KjwtTxgWPj8RuDe8PmHBPNiAdwM\n/CV8/gowJXx+JsGEgQCXAzeEzxsA7xAMClrsuQu3+yKuPMcBS+LK26y6/53oUbMfqklJTdbQzJYR\nfJGuBWYSfIEudfeSZv1c6u4b3N2BDwjmjToC2Oju70EwGaW75xbaz4H57r7Hg2atVwgm5jTgjnB8\nv38CbcsY1flU4CkP5nbC3beE63sA+VMxzAF+Gfe+C8LnHwNfu/snYfyfECQQCJLUk/H7m1lTgiTx\nerj+r8DJcbHMC/++H3ecM4ALw/P6FtAcODyMo/C560xRa4DDzGyamf0PQVIV2W91qzsAkQrY7e7H\nxK8IBvdmZyn77Il7nkvwf2B/L8w6wWDB6QSjOeeG12cOKmOfkubQKWl9Tvg3j4Lx51H8/2Gj+DIV\nPn7+sfLPQ77R7v7PAjuaZVD8uSvA3beY2VEE04ZfCQwBRhUTi0gkqknJgc4Jmt/amNnxAGbWxMxS\nCm1nwAAza2BmLQiav5YCTQkmfMw1s1OATmW838vAYDNrHr5XWrj+DYJpTiCYO+i1cpajDjA4fD4c\neN3dtwHZ+debgAuAzDKOsxj4rQXTy2NmP7ayZyTeTnAeCM9NXXefR9DsWNPn4JJqppqU1GTF1RaK\nm5XVS3gerHDfa2bnAfeGXal3AacXOpYTXN95haDmdIu7f21mjwELzOxDgmtE8dPJFPdeK8zs/wCv\nmlkuQVPbJQTTgswysz8QTNh3cQnHKanWtxPobmZ/Ipg+/bxw/UXAA2GiWVPouAVCC/8+TNCM974F\n1dL/AudS/HnN9xDwgpllAVeH5cj/AVyjZzOW6qcu6CIRmNlEYIe7T63uWIpzoHR3lwOPmvtEokvm\nX3TJHJvIflNNSkREkpZqUiIikrSUpEREJGkpSYmISNJSkhIRkaSlJCUiIklLSUpERJLW/we7TbGc\n0Bk8DQAAAABJRU5ErkJggg==\n", "text/plain": [ - "" + "" ] }, "metadata": {}, @@ -493,7 +542,7 @@ }, { "cell_type": "code", - "execution_count": 7, + "execution_count": 13, "metadata": { "collapsed": true }, @@ -508,7 +557,7 @@ }, { "cell_type": "code", - "execution_count": 8, + "execution_count": 14, "metadata": { "collapsed": false }, @@ -542,7 +591,7 @@ }, { "cell_type": "code", - "execution_count": 9, + "execution_count": 15, "metadata": { "collapsed": false }, @@ -551,7 +600,7 @@ "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAakAAAEbCAYAAABgLnslAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XuYVNWZ7/Hvy00QuxFEgxc8oNEcg4Ioo+aMaJkMl1y8\n4GiiieaYMZNEEzQ+ZDKJqLQG9SQZg6hPEj1eEpN4CQomGMcWlZI244kaERU1Gp9BMRcULwEjSgPv\n+WNXdVc31dV12VX7Ur/P89Rj1a5dVe8GrLfWWu9ay9wdERGROBoQdQAiIiJ9UZISEZHYUpISEZHY\nUpISEZHYUpISEZHYUpISEZHYijxJmdlAM1tpZkujjkVEROIl8iQFnAs8C2jCloiI9BBpkjKzvYBP\nANcDFmUsIiISP1G3pBYA/wZsizgOERGJociSlJl9CnjN3VeiVpSIiBRhUa3dZ2aXAacDW4ChQCtw\np7t/vuAcjVOJiKSYu5dspETWknL38919rLuPB04BHixMUAXnNeVt3rx5kcega9e167p17fW8lSPq\nMalCajWJiEgPg6IOAMDdHwIeijoOERGJlzi1pKRAJpOJOoTI6NqbT7NeNzT3tZcjssKJcpiZxzk+\nERGpnpnhcS2cEBER6Y+SlIiIxJaSlIiIxJaSlIiIxJaSlIiIxJaSlIiIxJaSlIiIxJaSlIiIxJaS\nlIiIxJaSlIiIxJaSlIiIxJaSlIiIxJaSlIiIxJaSlIiIxJaSlIiIxFZkScrMhprZ78zsSTN71swu\njyoWERGJp8i2j3f398zsGHd/18wGAQ+b2ZHu/nBUMYmISLxE2t3n7u/m7g4BBgJvRhiOiIjETKRJ\nyswGmNmTwDpgubs/G2U8IiISL1G3pLa5+8HAXsBRZpaJMh4REYmXyMakCrn738zsN8AUIFv4XFtb\nW9f9TCZDJpNpZGgiIhKSbDZLNput6DXm7vWJpr8PNhsNbHH3t81sGNAOXOzuDxSc41HFJyISllGt\nrby1cWPX45EtLby5YUOEEcWDmeHuVuqcKFtSuwM/NbMBBN2OPytMUCIiafHWxo0U/ty2goQlpUXW\nkiqHWlIikgZm1jNJAfpuK68lpRUnREQktpSkmlR7ezvTPzWd6Z+aTnt7e9ThiKTayJYWDLpuI1ta\nIo4oOZSkmlB7ezuzTpnFssHLWDZ4GbNOmVVVolKiEynPmxs24O5dNxVNlE9jUk1o+qems2zwMjg4\nd+BJmNY5jfvuvq/s98gnuk2ZTQAMyw5jyW1LmDFjRh0iFpE00piU1M0VV18RJKiDgYNhU2YTV1x9\nRdRhiUjKKEk1oTmz5zAsOwyeBJ4MWkFzZs+JOiwRke0oSTWhGTNmsOS2JUzrnMa0zmlVddMp0YlI\nI2hMSqrW3t7e1cU3Z/acHomu1HMiIlDemJSSlISuq6jigE2wFga8OYBLzr+EuXPnRh2aiMSICick\nEldcfUWQoJ4CpsC26du46NKLQi9TVwm8SPopSUl9rAWm0VX9t23GtlCr/8Ka6yUi8aYkJaGbM3sO\nA96s7z8tlcBLsxnV2oqZdd1GtbZGHVJDKElJ6GbMmMEl51/CgPYBqv4TCUl+JfX87a0mWUldSapJ\nNHr8ZsqUKUyaNIlRj45i8p8nh74ahUrgRZpDLHbmlfrqvYTRw6c8XNcljHp83hjYlN0U+mfk53p1\nlbnfpjJ3kTRSCXoTCGOtvrxy5j8V+7yW37ZwxGFHaM6USJXSuLuvStAlVLVU1G3cYaOq8ERq0Kwr\nqUeapMxsrJktN7PVZvaMmZ0TZTxpFdb4TbkVdXNmz+lRNMF9QKb0a0REiom6JdUJnOfuE4AjgK+a\n2QERx5Q6YazVV+nnTTpoEjwOPARMAj5Yt48TkRSL1ZiUmd0FXO3uD+Qea0wqBvLjUOtfX8/q51ez\n+aObgdJ7SPVYGmkVMJ1+XyMizSVRa/eZ2TiC390T3P2d3DElqYj1rgwcsmwIEw6awOhdRvdbBFGY\n3BhAWa8RkeaRmCRlZjsBWWC+u99VcFxJKmKHHH4IK/+8ElqAQ4F3qq8MFBEpVE6SinyelJkNBu4E\nfl6YoPLa2tq67mcyGTKZTMNia3bt7e2senoV5Bs+S4CJwKgIgxKRxMpms2Sz2YpeE2lLyswM+Cnw\nhrufV+R5taQiVGy+04D7BnDP4nvUZScNkca5QdItCfOk/hE4DTjGzFbmbjMjjimxGrH00aSJk5Sg\npGHKWa+u1MKr1S7K2qyLucZRLMak+qKWVPl6FziEUUVXj/cUqYSZUfgNYEDv74RS55Tz+mo/V2qX\nmMKJvihJlS/MpY8KaRt4iVJSkpS6JauTiMIJibcZM2YoMUlkRra0YL2+/OMo3y2ZZ02yjUYjRD0m\nJSHR1hWSRoXr1Y1saeGtjRu3Gyca2dKCQdetMJGVeq6Ual8n4VN3X4qoa07SLM7jRHGOLc40JiWh\nUyKUMFQzhhPnRKAxqeooSUmoVO0nYakm4cQ5SUl1kjBPShKk3K06ROqhmnEizXdKPlX3iUgiVNN9\npqq75FNLSsqmCkIJi6rnpFwak5KKqHBCkiSMcaxSRRG9nxsEtKhoomwqnBCRplYsiWyhsuq7Sle0\nABV0lEuFE1KzRixam4QYJJ76K4zITwaGYIHaTvpeqFbiSUkqZcL8Qs+XnC8bvIxlg5cx65RZDU8S\ncYhB6q+vZNNfEipnlfQw4pDoqLsvRcKex1SvRWuTFoPUX19dav2NKZU75lTLefnlmPI0JhUedfc1\nGc1jEimulmrCwvUD3b1HAur9XGev56V2SlLSpziUnMchBomvcpNPqUQj8aYklSLFvtCP/sjRVY9R\nzZgxgyW3LWFa5zSmdU6LZAmkOMQg9ddXsukvCYWdfDR/K340JpUyhfOYjv7I0Vz6H5dqrT0RiaXY\nz5MysxuBTwKvuftBRZ5XkqqBig5EJM6SUDhxEzAz4hhEJMFUNp5ukSYpd+8A3ooyhjRT0YE0g7Dn\nSkm8RN2SkjpS0YFIN7W4kin2W3W0tbV13c9kMmQymchiiavei74CPR5rDEpE23bEQTabJZvNVvSa\nyKv7zGwcsFSFE9XpvcrEkGVDYCBs/uhmQBV9kn7lbt2unX3jJwmFE1Kj3qtMbB65OUhQWnVCEq7c\n7jlN1E23SJOUmd0K/Bewv5mtNbMvRBmPiMRH2AUR+Ym6g+neUqN38tO4VfxE3t1Xirr7+qfuPkmr\nenXPVbo/lL6D6if2k3n7oyRVnv4KJ5SgJC7KHT8CJalmoCQlDaNt5aUclSSBIWZ0FjweDGxWkkoV\nFU5IQ2hjwuSpdeyl9+sH12H8Jr+Lbv7WWfr0spVaRFYLzMaPWlJSM60RmDy1thiKvZ4y3qOSz1Wr\nJv3Ukkq4MLeCj1qarkWqp5aKVEotqZgKeyv4empvb+e4Tx/XVVE45MEh/PqXv+6KNUnX0iyiaklV\nopIiC0kmtaQSLHFbwW8FHs/dtvZ8KnHX0gRqbdH0fv2gKt6jP5qkK6AkJWXor6vuiquvYPO0zfBF\n4IuwedpmJaGYqzUB9H59p5KI1EnsF5htVnNmz+HhUx5mE91dZHNua8w2G6V29334lIcr7qqL8lpE\nJNk0JhVjUcw96j1+NKB9ANsO3gb5jy5SuVfOmJPmUdWfxnDqQ3+u9aPJvFKxYuXkPE7QlZd7PPnP\nkxm962igO+HUkoSUwMKR9pLtqJJF2v9co6QkJRUrlqQG3DeAbdO3AeGvDajKv/D092Wa9BZBVMlC\nSap+lKSkYsWSxtxvzOWhRx4CYP0b61k5ZmVoE3c1ETg8/X2ZJv3LVkkqfVSCLhUrtuX83Llzue/u\n+5gzew4vv/Jy0P33x/LeT5N4GyeMibLaqmJ7moAcLbWkpKT8eNH619ez+vnVXd183AdMgmHP9d09\nV25Bhbr7GqOcFkEjWw2Vdj8mvbtStqfuvpSIqrCgRwJ5HJhCj265UY+O4pYbb+kznnK78lQ40Rjl\nfMk3MkmpG03KSVKaJxVzvVsa1cxTqlaPlSL+sP3zh04+NJQ4ZsyYocTUAI1sdajVI2GJevv4mWb2\nvJm9aGb/HmUscRWbJYUOJejiezK4DcsO69pgsS9zZs9hWHZYRa+pF42NlSes8Zewt36X5hVZkjKz\ngcA1wEzgw8CpZnZAVPHI9nokmXdgiA1h8l8ndxVU9Nf6KVaEEUWLSftdla+/5ZLCLKxQQYKUI7Ix\nKTP7CDDP3WfmHn8LwN3/T8E5TT8mFXVhQRrGi1TmHp5yx5GiGG9SF2PyxH1Mak9gbcHjV4HDI4ol\ntvKtka5EcVtjE0Ul40Vr18KaNTB1avC4owPGjYOxY+sWnsTUyJYWrFfCqJf8v7t8F2MHRzKONey9\n8dW6faY0TpRJqqyfVW1tbV33M5kMmUymTuHEV1IKC9asgRNPhEWLgscnnwyLF0efpLTAbeM1sgWT\n/3cHR5MFTmYRizmR4HevxEk2myWbzVb0mii7+44A2gq6+74NbHP37xac0/TdfUmTzcIxxwT3ly+H\nuPymSEO3ZRzEpUutd6v9qqvg3HOD+8vJkOEhlbQnQNy7+x4H9jOzccCfgc8Ap0YYj6RYUlqjcRfV\nGE/v5LjTsOkMHd7e1Wq/8MLuc3O/kVSIkRIlq/vM7AAz+5iZ7dTr+MxaP9jdtwBfA9qBZ4Hb3f25\nWt9XotPREXTxLV8e3E4+OTiWBmvX9ryWjo7gmDRG75L2dzbdx6JFQav9mGPArPvf3ejRWVas0CaM\nadFnS8rMzgG+CjwH3Ghm57r7XbmnLwfurfXD3f0/gf+s9X0kHsaNC8ag8l0wixcHx9IgruNtErjk\nku6u5TT9u5MSY1Jm9gxwhLu/k+uSuwP4ubtfaWYr3X1y3YPTmFRdaZymMnEdb2sG25e0H8no0R3b\n/WjI/0CSZKh1TMrc/R0Ad19jZhngTjP7HwTTHiTBolxuSaRSvUvaRwxfn9pWu/RUqiW1HDjP3Z8s\nODYYuAE4zd3rvlqFWlL1owmulenoKN7dp1/uItWrdT+pzwN/LTzg7p3A/waOqj08aSZJXzsvP96W\nyQQ3/XIXaQxt1dGkGrncUtRLO4lIPGk/KSmpUYUT6lpsTnGZ+CvxFffJvBIxTXCVesrPbcozbdch\nVehzTMrM9jOzI4scP9LM9q1vWJImcdpXSkSSpVThxJVAsbb5htxzImWJy75SIpI8pUrQH3f3KX08\n94y7H1jXyNCYlEiSaUxK+lNrCfrOJZ4bWl1IIumgtfyKK9y5962NGxnZ0tLnLr8i5SiVpB43sy/1\nPmhm/wr8vn4hicRffi2/bDa4nXhicKzZ9V4I9i0VS0iNSnX3jQGWAJvpTkqHAjsAs9z9L3UPTt19\nEmNJXsuvXrsoR7FtvCRXTSXo7v5X4CNmdgyQH3+6290fDDFGEYmAVnWXpCjVkhoGfAX4IPAUcGNu\nWaSGUUtKwhZWCyINa/nVoyWoYgmpRK2TeX9K0NXXAXwc+DBwbnjhiTReWC2INO+dVQslJAlbqZbU\n0+5+UO7+IOCxRuwh1SsGtaQkdEkeSwpLuS3Balue9RrzknSptQR9S/5Obqv30JjZyWa22sy2mtkh\nYb63iPSv3FXdq61iVPWjhKVUS2or8G7BoWHAptx9d/fWqj/U7H8C24BrgTnu/kQf56klJf2q5Fd7\nGsaSGq3alqdarNKfWqv7BoYfUtd7Pw9BgCK1qmScSWNJIskS6VYdud1/1ZKSmjXyV3scxlsaFUO1\nLU+1WKUckW7VYWbLgDFFnjrf3ZeW+z5tbW1d9zOZDBn1GSROHL7UwxSHOUaNiqHalqdarFJMNpsl\nm81W9Bq1pKTu6v2rOopf7fVuuZWT2DXmI0mXlE0PNTCVclOnBgmk8As1zASSxl/tcWiticRBJEnK\nzGYBVwGjgd+Y2Up3/3gUsUjyjR3b88u73uMeHR1B0li+PHhcacutnFZSf4m91hhEkiKSJOXuSwgW\nr5UmkLYv1FpbbmG0ktLYehQpJtIxqf5oTCod0lY4EYb+xpNUHSfNICljUpJyje6OS4OoWkr6QSFx\nU2pZJJHYS+IOuYXdn8uXB/cLrwGCpFCYzKdOrV+iKPwzXLMGjj0WfvlLLWck8aAkJYm1di386lfd\na8RddRUcf3z8v1TLXTcvTKWSeeE6e1u3gjt85jNBd+SiRX23fJP4A0GSR919klhr1sDFF8Pcud3j\nOwsXxr87MYruz1LFGr0rCRcsgPPOq+09RcKiwglJtMICBIDbb4fdd9eYSjGlijUKnxsxAu66K7jf\nX8GGJhRLLWrdqkMk9p56qvv+ggXwpS8FYyraImJ7r73W83G+e65wjOz224PnBg6svCuy8O9CXX8S\nFnX3SWJ1dATdfQsXwsSJwRft/Pmw2271W90iqTo6ggTe2hr8mZ1wQnB86dLtKwl33707MZX6sytM\nbk89BV//enA8/3ehrj8Jg5KUJNa4cUG3VO8y7ZdeijKqeBo3LkhIW7d2J/Dbb+/+s6tmjKwwueW7\n+c49N/ivfhxIWNTdJ4lVrEx7zZr+y7ubUe8/KwhanGG+58SJ5b9WlYFSLiUpSZUoyruTopz5WY16\nb20vL+VSdZ9Ik6jnahLVvLcqA0XLIolIl3rOz9LSV1Iv6u6TRNAYRrrUs+tR0kVJShKhXmMYSn7R\n0NihlEtjUpIY9RjD0JYYItHRmJRIP+q9tb2I1Cay7j4z+76ZPWdmq8xssZmNiCoWib++xjDi1F0X\np1hE0iLKMan7gAnuPgl4Afh2hLFIzPU1hlHrWFW1A/jFEtLvfhcsN3TVVd2x/OpX9U1USoySeu4e\n+Q2YBfy8yHEX6c/y5e7BLkjB/Uq88or7ihXdj1esCI71Z8UK99Gjg89bvjy4v2KF+8KF3bEsWNB9\nvF76iqOYaq9VpF5y3/Gl80N/JzTiBiwFPlvkeD3+XCRlaklSYX9u4bFS8RQmjFdecb/66u6EUWny\nKPf6K0lo/VHCkzCUk6Tq2t1nZsvM7Okit2MLzpkLbHb3W+oZi6RTnObb5GNZsKD7WOH2FYUKuymX\nLIFzzgn+W88lggqLRPrbdbc/WtZIGqWu1X3uPq3U82Z2BvAJ4GN9ndPW1tZ1P5PJkNHaKVKg9zYT\njZpvU5gcIbj/ox/BvHnBVhj57Sva2mDy5O2TQbHdcKtZQbxYHI0ooVdVpFQjm82SzWYrek1k86TM\nbCZwBXC0u6/v4xyPKj6RUnqvVXfHHcF/Dz+8u0WRL+zoaw27wnlfhVu2VzIHrJI188KeE6a196RW\n5cyTijJJvQgMAd7MHXrE3c/udY6SlCRCpQmg8Pz8hoFXXtlzw8CwWyZhLjCrSdAShlgnqXIoSUmS\nVNKyKEwYa9cGperHHx8kjI4OGDIENm+uz4rlvT+/mvev54rq0jzKSVJau08kAoUbBo4dC1/7WvcX\n/NSpQYIqLEw4/vggkeXVOh+q1sKHYhtOKkFJPaglJRKCenR/FbbMFi6E73ynfu+vMSWJgtbuE2mQ\nelcZTpyoajppTuruEymh3GWHxo4NklL+3KlTg+6zarvkis3/6mvOVVjvH7e1EEVASUqkpErGbsKc\n4Np7rcLC+VdhTFqu11qIImHTmJRIPyoZu6nXOE8jq+k0ViWNojEpkTrrnTzC7JIrNHZsz4Sk8Shp\nFolMUmYlE29iqJUYf/0tO5TvHiuclLtwYX0n5dZTVMssifQlkd19uSZiBBGFJw3X0Az662ZbuzZY\nGDa/7t4558A3vhG8Zt26YJmkJM0f0iRdaaTUrjiRhi/4NFyDBF/ixx4Lf/tb8Hj4cLjssmBOk1og\nIqVpTEqkAQp/a/z970GrSvOYRMKhEnSRGowbF7SaRKQ+lKREarBmTZCkFi6E1lYYMSK4H+XmiyJp\noiQVsmuuuYYpU6YwdOhQvvCFL0QdjoSkr5UY8pNiZ82Cu++GpUuD+30ti6QVHUQqozGpkO25555c\neOGFtLe3s2nTpqjDkZAUlppDz9LsfOVbYQVcX9Vwfb2PqudEilN1X51ceOGFvPrqq9x0001Fn0/C\nNUhPYa3EoBUdRALaT6qIUa2tmFnXbVRra10+RwlIRKR2kSQpM/uOma0ysyfN7AEzC6Wzo3cCKpaE\n3tq4EYeu21sbN1b8HuVIy6oYEuhr1fCo3kekWUQ1JvU9d78QwMxmA/OAL9b6pvkEVMh6JaFGvAeo\nJZU2Ye0XVe99p0TSJpIk5e6F3/o7AeujiKOe1JJKl7AWeNVCsSKViay6z8wuBU4H3gWOaNTnjmxp\n6dEyGtnSEur7b926lc7OTrZs2cLWrVt5//33GTRoEAMHDgz1c0REmkHdqvvMbBkwpshT57v70oLz\nvgV8yN23m1RUaXXfqNbW7caYRra08OaGDZXEvX13H+V337W1tXHJJZdsd+yiiy7a/nPUJSgiTSwR\nC8ya2d7APe5+YJHnfN68eV2PM5kMmUymrl/wYSS6cihJiUizyWazZLPZrscXX3xxPJOUme3n7i/m\n7s8GDnP304ucl9h5Uv1JwzWIiNQizqugX25mHwK2Ai8BZ0UUh4iIxFjk3X2lqCUlIpJeWnFCREQS\nTUlKRERiS0lKRERiS0lKRERiS0lKRERiS0lKRERiS0kqZJs3b+bMM89k3LhxtLa2MnnyZO69996o\nwxIRSSQlqZBt2bKFvffemxUrVrBhwwbmz5/Ppz/9aV5++eWoQxMRSZymSlKPPALXXdf9+LrrgmNh\n2nHHHZk3bx577703AJ/85CcZP348TzzxRLgfJCLSBFKVpBYsgD/9Kbi/aRN85zvw/vvdz3/gA3DZ\nZfDDHwa3yy4LjhV6553Sjyu1bt06XnjhBSZMmFDbG4mINKHI9pOqh85OyGTg3nvh7LNhl12gcBun\nffaBBx+EffcNHr/0UnAs79134aCDghbWtGlw5ZWwdCk88EC18XTyuc99jjPOOIP999+/6usSEWlW\nqUpS3/wmvPcefPCDcNRRcPPNMKjXFRbWMOSTWd6OO8LPfgYnngiHHQbPPQfLl1cXy7Zt2zj99NMZ\nOnQo11xzTXVvIiLS5FKVpDZtgt/+Nri/di2sWwd77tn9/F13wfe+F7SgAD76UdhjDzjhhO5zjjwy\nSHB33gk33AC5oaWKuDtnnnkmr7/+Ovfcc4925RURqVKqktRnPxt08XV2wg9+AMccA08+GbSQAKZP\nhxUruhPPihUwenTP97jySli5En7xC/j612Hs2KDrrxJnnXUWzz//PPfffz877LBD7RcmItKkUrVV\nx6pVMGFCdxffE0/AIYeU/3nvvgv//M9w7bVBInv44WB86uaby3+Pl19+mfHjxzN06NAeLajrrruO\nU089td9rEBFpFonYPr4U7Scl0r+1a2HNGpg6NXjc0QHjxgW9ACJxpv2kRJrAmjVBsU82G9xOPDE4\nJpIGkbakzGwO8H1gtLu/WeR5taREypDNBmOwEFSkZjJRRiNSnli3pMxsLDAN0HpBIiJSVJTdfT8A\nvhnh54ukQkcHnHxy0IJavjy439ERdVQi4YikBN3MjgdedfenzEq29ESkH+PGweLF3YUTixcHx0TS\noG5jUma2DBhT5Km5wPnAdHffYGb/DUxx9zeKvIfGpEREUqqcMam6taTcvegUWDM7EBgPrMq1ovYC\nfm9mh7n7a73Pb2tr67qfyWTIaERYRCSRstks2Wy2otdEPk8q15I6VNV9IiLNJdbVfQX0TS0iIkVF\nnqTcfZ9iragkO+2009h9991pbW1ln3324dJLL406JBGRRIq8u6+UpHb3rV69mn333ZehQ4fyhz/8\ngaOPPpqf/OQnzJw5s+ucuF+DiEi9JaW7r2HcnfmXz2fXPXZl1z12Zf7l8+uSKCZMmMDQoUO7Hg8a\nNIjddtst9M8REUm7VCWp9evXc+yJxzJm7zEcPvVwnn322R7P//jaH3P5Dy9n/QnrWX/Cei7/4eX8\n+Nofb/c+jz76KIsWLeKFF16oOpazzz6b4cOHM2HCBC644AIOqWQ5dhERAVLU3efuHHL4IawetJrO\nQzqxNcbIx0by4nMvMmrUKACOnnE0K3ZeAR/OvehZOOrto3io/aGu9zl3zrlc/7PrGbTHIDrXdHLt\n1ddy+umnVxW/u/PQQw9x0kkncc8993DYYYeVvAYRkWbSVN1969at47nnn6NzWifsCv4PzpZdtvDI\nI490nTNq51HY291/Hva2scvOu3Q9fuKJJ7j+5ut591/eZcOsDWz63Ca+dNaXeO+996qKyczIZDKc\nfPLJ3HrrrdVfnIhIk0pNkho2bBjbOrfB+7kD22DbO9sYPnx41znzL5rP8MeGM6h9EIPaBzH8seHM\nnze/6/m1a9cyaPdBMCx3YDewwcYbb2y3GEZFOjs7e8QhIiLlSc328SNGjODLX/4yN916E3//0N8Z\n9qdhHDj+QI488siucyZMmMCqx1d1tWo++4vPMn78+K7nJ02axJZXtsCfgT2Ap2CnHXdizJhiqzsV\n9/rrr/PAAw9w7LHHMnToUO6//34WLVrE/fffH9aliog0jdSMSUEwBnTLLbfwyKOPsN8++/GVr3yF\nHXbYoaLPXLx4MaefcTpbfSsjRoyg/e52Dj744LJfv379ek466SRWrVqFu7P//vtzwQUXcNxxx5V1\nDSIizULbx1dpy5YtvP3224waNYoBA+rTI6okJSLNTkkqxtJwDSIitWiq6j4REUkfJSkREYktJSkR\nEYktJSkREYktJSkREYktJSkREYmtxK44YVayalFERFIgkiRlZm3AF4HXc4e+7e73lvt6zS8SEWkO\nUXX3OfADd5+cu5WdoJpFNpuNOoTI6NqbT7NeNzT3tZcjyjEp9deV0Mz/cHXtzadZrxua+9rLEWWS\nmm1mq8zsBjPbOcI4REQkpuqWpMxsmZk9XeR2HPAjYDxwMPAX4Ip6xSEiIskV+QKzZjYOWOruBxV5\nThUSIiIp1t8Cs1FV9+3u7n/JPZwFPF3svP6CFxGRdItqntR3zexggiq//wa+HFEcIiISY5F394mI\niPQlEcsimdlsM3vOzJ4xs+9GHU8jmdkcM9tmZqOijqVRzOz7ub/vVWa22MxGRB1TvZnZTDN73sxe\nNLN/jzonbWeAAAAERUlEQVSeRjGzsWa23MxW5/7/PifqmBrJzAaa2UozWxp1LI1kZjub2R25/8+f\nNbMj+jo39knKzI4BjgMmuvuBwH9EHFLDmNlYYBrwctSxNNh9wAR3nwS8AHw74njqyswGAtcAM4EP\nA6ea2QHRRtUwncB57j4BOAL4ahNdO8C5wLMEQx/NZCFwj7sfAEwEnuvrxNgnKeAs4HJ37wRw99f7\nOT9NfgB8M+ogGs3dl7n7ttzD3wF7RRlPAxwG/NHd1+T+nd8GHB9xTA3h7n919ydz998h+LLaI9qo\nGsPM9gI+AVxPEy1ukOsZmeruNwK4+xZ3/1tf5ychSe0HHGVm/8/MsmY2JeqAGsHMjgdedfenoo4l\nYv8C3BN1EHW2J7C24PGruWNNJTcdZTLBD5NmsAD4N2BbfyemzHjgdTO7ycyeMLP/a2Y79nVyLFZB\nN7NlwJgiT80liHGkux9hZv8A/BLYp5Hx1Us/1/1tYHrh6Q0JqkFKXPv57r40d85cYLO739LQ4Bqv\n2bp6tmNmOwF3AOfmWlSpZmafAl5z95Vmlok6ngYbBBwCfM3dHzOzK4FvARf1dXLk3H1aX8+Z2VnA\n4tx5j+WKCHZx9zcaFmCd9HXdZnYgwa+NVbktSfYCfm9mh7n7aw0MsW5K/Z0DmNkZBF0hH2tIQNH6\nEzC24PFYgtZUUzCzwcCdwM/d/a6o42mQ/wUcZ2afAIYCrWZ2s7t/PuK4GuFVgl6ix3KP7yBIUkUl\nobvvLuCjAGa2PzAkDQmqFHd/xt0/4O7j3X08wV/qIWlJUP0xs5kE3SDHu/t7UcfTAI8D+5nZODMb\nAnwG+HXEMTWEBb/CbgCedfcro46nUdz9fHcfm/v/+xTgwSZJULj7X4G1ue9zgH8CVvd1fixaUv24\nEbjRzJ4GNgNN8RfZS7N1B10NDAGW5VqSj7j72dGGVD/uvsXMvga0AwOBG9y9z2qnlPlH4DTgKTNb\nmTtW0f5yKdFs/4/PBn6R+1H2EvCFvk7UZF4REYmtJHT3iYhIk1KSEhGR2FKSEhGR2FKSEhGR2FKS\nEhGR2FKSEhGR2FKSEmkAM9ua25LhaTP7pZkNyx0fY2a3mdkfzexxM/uNme1X5PU3mtm63HxBkaah\nJCXSGO+6+2R3P4hgUvpXcseXEKw28EF3n0KwZuMHirz+JoKtPESaShJWnBBJmw5gYm6vtM3ufl3+\nib5WvXf3jtwq4SJNRS0pkQYys0HAx4GngAOB30cbkUi8KUmJNMaw3Np0jxHstHxjxPGIJIK6+0Qa\nY5O7Ty48YGargZMiikckEdSSEomIuz8I7GBm/5o/ZmYTzezICMMSiRUlKZHG6Gu7gVnAP+VK0J8B\nLgX+0vskM7sV+C9gfzNba2Z9bm0gkibaqkNERGJLLSkREYktJSkREYktJSkREYktJSkREYktJSkR\nEYktJSkREYktJSkREYktJSkREYmt/w/CVImiEe/4GwAAAABJRU5ErkJggg==\n", "text/plain": [ - "" + "" ] }, "metadata": {}, @@ -578,7 +627,7 @@ }, { "cell_type": "code", - "execution_count": 10, + "execution_count": 16, "metadata": { "collapsed": false }, @@ -589,7 +638,7 @@ "array([ 2.59891628, 0.00484089])" ] }, - "execution_count": 10, + "execution_count": 16, "metadata": {}, "output_type": "execute_result" } @@ -615,7 +664,7 @@ }, { "cell_type": "code", - "execution_count": 11, + "execution_count": 17, "metadata": { "collapsed": false }, @@ -628,7 +677,7 @@ " 0.01635336, 0.01284271, 0.00642076])" ] }, - "execution_count": 11, + "execution_count": 17, "metadata": {}, "output_type": "execute_result" } @@ -643,7 +692,7 @@ }, { "cell_type": "code", - "execution_count": 12, + "execution_count": 18, "metadata": { "collapsed": false }, @@ -652,7 +701,7 @@ "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEPCAYAAABCyrPIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGqpJREFUeJzt3X20XFV5x/Hvj0QgGDE3pgUJ0YACCm0QUKCWyvDSGqiS\ntoo0oghooVVUXL4g2JqLVqt1qSguMQIGLGqwEitRCgRkWgoKBPIGSTBR0yZBQCFR3tS8PP3j7Bsm\nk7n3npnMmTNz7++z1qw558w+5z53cjPP7L3P3lsRgZmZ2S5lB2BmZt3BCcHMzAAnBDMzS5wQzMwM\ncEIwM7PECcHMzICCE4Kkr0l6RNKyIcp8UdIqSUskHVZkPGZmNriiawhzgOmDvSjpZOClEXEAcA5w\nWcHxmJnZIApNCBFxO7BhiCKnAFensncBEyTtVWRMZmbWWNl9CJOBtTX764B9S4rFzGxUKzshAKhu\n33NpmJmVYGzJP389MKVmf990bDuSnCTMzFoQEfVfugdVdg3heuAMAElHAxsj4pFGBSOiZx+zZs0q\nPYbRGn8vx+742//o6wuyRoi8j1lNle/rK/93rH00q+jbTr8F3AkcJGmtpLMlnSvpXICIuAH4maTV\nwGzgnUXGY2aj24YNEJH/MWtWc+Uff7zs33DnFNpkFBEzc5Q5r8gYzMwsn7L7EEaFSqVSdgg7pZfj\n7+XYYfTFP3Fi9i2+KH19zZXv9fe/WWqlnanTJEUvxGlmO0fKml6sPSQRPdSpbGZmXcIJwczMAPch\nmFkTuq2N39rLfQhmlpvb+HuL+xDMzKwlTghmZga4D8FsRHEbv+0M9yGYjSBu47da7kMwM7OWOCGY\nmRnghGBmZok7lc06yJ2+1s3cqWzWQe70tU5yp7KZmbXECcHMzAAnBDMzS9ypbFbDnb42mrlT2ayG\nO31tJHGnspmZtcQJwczMACcEMzNLnBDMzAxwQjAzs8QJwczMACcEMzNLPDDNeooHjpkVxwPTrKd4\n4JhZfh6YZmZmLXFCMDMzwAnBzMwSJwQzMwOcEMzMLHFCMDMzwAnBzMySQhOCpOmSVkpaJemCBq9P\nknSjpMWS7pd0ZpHxmJnZ4AobmCZpDPAgcCKwHrgHmBkRK2rK9AO7RcSFkial8ntFxOa6a3lgWg8p\ncjRxXx88/ngx1zYbaZodmFbk1BVHAqsjYg2ApLnADGBFTZlfANPS9p7AY/XJwHrPhg0eTWzWi4pM\nCJOBtTX764Cj6spcDvxQ0kPA84A3FRiPmZkNociEkOc74kXA4oioSHoJsEDSoRHxRH3B/v7+bduV\nSoVKpdKuOM3MRoRqtUq1Wm35/CL7EI4G+iNietq/ENgaEZ+uKXMD8ImIuCPt3wpcEBEL667lPoQe\n4gnozLpDN01utxA4QNJUSbsCpwHX15VZSdbpjKS9gIOAnxUYk5mZDaKwJqOI2CzpPOAmYAxwZUSs\nkHRuen028ElgjqQlZMnpQxHhe0jMzErg9RCs7dxkZNYduqnJyMzMeogTgpmZAU4IZmaWDJsQJE2R\n9F1Jv0yP6yTt24ngzMysc/LUEOaQ3S66T3rMT8fMzGwEGfYuI0lLIuLQ4Y4VyXcZ9RbfZWTWHYq4\ny+gxSW+VNEbSWElvAX7VeohmZtaN8iSEs8kmnXuYbHbSU4GzigzKijVxYvYtvqhHX1/Zv6GZtcID\n00YhN+mYjQ5tWw9B0gUR8WlJlzZ4OSLiPS1FaGZmXWmouYyWp+d72X4qa5FvamszM+shgyaEiJif\nNp+OiG/XvibJC9mYmY0weW47XRQRhw13rEjuQ2gv9yGYjQ7t7EM4CTgZmCzpi2RNRZAtdblpp6I0\nM7OuM1QfwkNk/Qcz0vNAQvgN8L6C4zIzsw7L02S0a0T8vkPxDBaDm4zayE1GZqND25qMakyV9Eng\nYGBcOhYRsX8rAZqZWXfKO7ndV4DNQAW4GvhGgTGZmVkJ8iSEcRFxC1nz0v9GRD/wl8WGZWZmnZan\nyei3ksYAqyWdR9bZ/NxiwzIzs07L06n8KmAlMAH4OLAn8K8R8ePiw9sWgzuV28idymajQ7OdykMm\nhFQz+HREfKAdwbXKCaG9nBDMRoe2rocQEVuAYyTlvqCZmfWmPH0Ii4HvSfp34Ol0LCJiXnFhmZlZ\np+VJCLsDjwHH1x13QijIxImwYUNx1/cCNmbWiBfI6UJu4zezdihiTWUzMxsFnBDMzAxwQjAzs2TY\nhCBpb0lXSrox7R8s6e3Fh2ZmZp2Up4ZwFXAzsE/aX4XXQzAzG3HyJIRJEXEtsAUgIjaRzXxqZmYj\nSJ6E8KSkFwzsSDoa+HVxIZmZWRnyDEx7PzAf2F/SncAfAG8sNCozM+u4XAPTJD0HOIhsXeWVqdmo\nYzwwzcyseW0fmJbWQBgfEfdHxDJgvKR35gxmuqSVklZJumCQMhVJiyTdL6maN3AzM2uvPOshLImI\nQ+uOLY6IVwxz3hjgQeBEYD1wDzAzIlbUlJkA3AG8NiLWSZoUEb9qcC3XEMzMmlTE1BW7SNpWLn3Q\nPyfHeUcCqyNiTWpimgvMqCvzZuC6iFgH0CgZmJlZZ+RJCDcBcyWdIOlEsg/2G3OcNxlYW7O/Lh2r\ndQAwUdJtkhZKemueoM3MrP3y3GV0AXAO8A9pfwFwRY7z8jR6PAc4HDgB2AP4kaQfR8SqHOeamVkb\nDZsQ0qppl6VHM9YDU2r2p5DVEmqtBX4VEc8Az0j6b+BQstHQ2+nv79+2XalUqFQqTYZjZjayVatV\nqtVqy+fn6VQ+BpgFTOXZBBIRsf8w540l61Q+AXgIuJsdO5VfBnwJeC2wG3AXcFpELK+7ljuVzcya\n1Gyncp4moyuB84H7SNNX5BERm9MtqzcBY4ArI2KFpHPT67MjYmWaNG8psBW4vD4ZmJlZZ+SpIdwV\nEUd1KJ7BYnANwcysSc3WEPIkhE+RfcOfB/xu4HhE3NdqkM1yQjAza14RCaFKgzuGIuK4pqNrkROC\nmVnz2p4QuoETgplZ84roVEbS64CDgd0HjkXEx5oPz8zMutWwCUHSbGAccDxwOXAq2e2ho9bEibBh\nQ3HX7+sr7tpmZoPJ04ewLCL+WNLSiJgmaTxwY0Qc05kQu6/JyE06ZtYLipjc7pn0/LSkyWTLZ+7d\nSnBmZta98vQhfF9SH/AZ4N507PLiQjIzszI0dZeRpN2B3SNiY3EhNfy5bjIyM2tS2+4yknRCRNwq\n6Q3UjUNIP2TeTsRpZmZdZqgmo9cAtwKvp/FU1k4IZmYjyJBNRmmltFMj4trOhdQwDjcZmZk1qa13\nGUXEVuBDOx2VmZl1vbyT2/0KuBZ4auB4RDxebGjbxeAagplZk4qY3G4NjSe326/p6FrkhGBm1jxP\nbtcBTghm1guKmtzuj9hxcruvNx+emZl1qzyT2/UDxwKHAD8ATgL+B3BCMDMbQfLMZfRG4ETgFxFx\nFnAoMKHQqMzMrONyTW4XEVuAzZKeDzwKTCk2LDMz67Q8fQgL0+R2lwMLyW49vbPQqMzMrOOandxu\nP2DPiFhSXEgNf67vMjIza1Lb10OQNF/SmyU9NyJ+3ulkYGZmnZGnD+GzwJ8ByyVdJ+mNaRpsMzMb\nQXI3GUkaCxwH/B0wPSL2LDKwup/tJiMzsyYVNTBtHHAK8CbgcODq1sIzM7NulWdg2reBo4AbgS8B\n/5VmQTUzsxEkz+R204FbImJzZ0JqGIObjMzMmuTJ7TrACcHMekHbbzs1M7PRwQnBzMyAITqVJR1B\ntjCOaLxAzn0FxmVmZh02aB+CpCpZIhgHHAEsTS9NAxZGxJ90IsAUi/sQzMya1LY+hIioRMRxwEPA\n4RFxREQcARyWjpmZ2QiSpw/hZRGxbGAnIu4HXl5cSGZmVoY8I5WXSroCuIasP+HNgCe4MzMbYfLU\nEM4ClgPvBd6Tts/Kc3FJ0yWtlLRK0gVDlHuVpM2S/ibPdc3MrP1yDUyTtAfwoohYmfvC0hjgQbLl\nN9cD9wAzI2JFg3ILgKeBORFxXYNruVPZzKxJRayHcAqwiGwuIyQdJun6HNc+ElgdEWsiYhMwF5jR\noNy7ge8Av8wbtJmZtV+eJqN+ssntNgBExCJg/xznTQbW1uyvS8e2kTSZLElclg75e7eZWUnyJIRN\nEbGx7lie2U7zfLhfAnw4tQcpPczMrAR57jJ6QNLpwFhJB5B1LN+Z47z1wJSa/SlktYRaRwBzJQFM\nAk6StCkidmiS6u/v37ZdqVSoVCo5QjAzGz2q1SrVarXl8/NMf/1c4CPAX6RDNwEfj4jfDnPeWLJO\n5RPIBrLdTYNO5Zryc4D5ETGvwWvuVDYza1LbV0yLiKeAi9Ijt4jYLOk8sgQyBrgyIlZIOje9PruZ\n65mZWbHy1BAOAj4ATOXZBBIRcXyxoW0Xg2sIZmZNavsCOZKWkt0FdB+wJR2OiLi35Sib5IRgZta8\ntjcZkd1ldNnwxbrP+ef3s7H+/qgGrr66v6nr9vW1Fo+ZWTfLkxDmS3oXMA/43cDBiHi8sKjaZONG\nmDq1f9hys2YNf601a/q56qrhr2Vm1qvyJIQzycYUfKDu+H5tj8bMzEqT5y6jqR2Iw8zMSjbUEpon\nRMStkt5A4yU0dxgvYGZmvWuoGsJrgFuB19N4GgonBDOzEWTQhBARs9LzmR2LxszMSpOnUxlJrwMO\nBnYfOBYRHysqKDMz67w86yHMBt5ENqmd0vaLC47LzMw6LM/016+OiDOAxyPiYuBo4KBiwzIzs07L\nkxCeSc9PpwVtNgN7FxeSmZmVIU8fwvcl9QGfAQbmL7q8uJDMzKwMeQamDXQeXyfpB8DuDVZQMzOz\nHjfUwLTaAWmq2R6YQc/jEMzMRpChagiDDUgb4IRgZjaCDDUw7cwOxmFmZiXLMw5hkqRLJS2SdJ+k\nL0h6QSeCMzOzzslz2+lc4FHgb4A3Ar8Eri0yKDMz67w8t53uHREfr9n/Z0mnFRWQmZmVI08N4WZJ\nMyXtkh6nATcXHZiZmXVWnoRwDvAN4Pfp8S3gHElPSPpNkcGZmVnn5BmYNr4TgZiZWbny3GX09rr9\nsZJyLEtvZma9JE+T0YmSbpC0j6Q/An4E7FlwXGZm1mF5moxmSvpbYCnwFHB6RPxP4ZGZmVlH5Wky\nOpBscZx5wP8Bb5H03KIDMzOzzsrTZHQ98NGIOAc4FlgF3FNoVGZm1nF5BqYdFRG/BoiIrcBnJc0v\nNiwzM+u0QWsIkj4EEBG/lnRq3ctnFhmUmZl13lBNRjNrti+qe+2kAmIxM7MS5elDMDOzUcAJwczM\ngKE7ladJeiJtj6vZBhhXYExmZlaCoVZMG9PJQMzMrFxuMjIzM6ADCUHSdEkrJa2SdEGD10+XtETS\nUkl3SJpWdExmZrajQhOCpDHAl4DpwMHATEkvryv2M+A1ETEN+Djw1SJjMjOzxoquIRwJrI6INRGx\niWx95hm1BSLiRwMjoYG7gH0LjsnMzBooOiFMBtbW7K9LxwbzduCGQiMyM7OG8sxltDMib0FJxwFn\nA3/a6PX+/v5t25VKhUqlspOhmZmNLNVqlWq12vL5RSeE9cCUmv0pZLWE7aSO5MuB6RGxodGFahOC\nmZntqP7L8sUXX9zU+UU3GS0EDpA0VdKuwGlk02lvI+lFZGstvCUiVhccj5mZDaLQGkJEbJZ0HnAT\nMAa4MiJWSDo3vT4b+CjQB1wmCWBTRBxZZFztcv75/Wzc2J5rTZgAl1zS356LmZm1oOgmIyLiP4H/\nrDs2u2b7HcA7io6jCBs3wtSp/W251po17bmOmVmrPFLZzMwAJwQzM0ucEMzMDHBCMDOzxAnBzMwA\nJwQzM0ucEMzMDHBCMDOzxAnBzMwAJwQzM0ucEMzMDHBCMDOzxAnBzMwAJwQzM0ucEMzMDOjAegjW\nOi/AY2ad5ITQxbwAj5l1kpuMzMwMcEIwM7PECcHMzAAnBDMzS5wQzMwMcEIwM7PEt52OYh7nYGa1\nnBBGMY9zMLNaTghWGNdAzHqLE4IVxjUQs97iTmUzMwOcEMzMLHFCMDMzwH0I1sPcaW3WXk4I1rOK\n7rR2wrHRxgnBbBC+S8pGG/chmJkZ4BqCWWncJGXdptCEIGk6cAkwBrgiIj7doMwXgZOAp4EzI2JR\nkTGZdQv3gVi3KSwhSBoDfAk4EVgP3CPp+ohYUVPmZOClEXGApKOAy4Cji4qpLGvWVJk6tVJ2GC3r\n5fh7OXbYufi7IeE8/PAa9t576rDXr0847Uxmja6fV7VapVKptC+QLldkDeFIYHVErAGQNBeYAayo\nKXMKcDVARNwlaYKkvSLikQLj6rjR/KFUtl6OHbo7/jwJZ82a/lxJqT7htDOZNbp+3oSzeHGVV7yi\nMmSZkVR7KjIhTAbW1uyvA47KUWZfYEQlBDPrLnkTTp6ENpKa64pMCJGznFo8z8ysK/XqLcuKKObz\nV9LRQH9ETE/7FwJbazuWJX0FqEbE3LS/Eji2vslIkpOEmVkLIqL+S/egiqwhLAQOkDQVeAg4DZhZ\nV+Z64DxgbkogGxv1HzTzC5mZWWsKSwgRsVnSecBNZLedXhkRKySdm16fHRE3SDpZ0mrgKeCsouIx\nM7OhFdZkZGZmvaWrp66QNF3SSkmrJF1QdjzNkDRF0m2SHpB0v6T3lB1TKySNkbRI0vyyY2lWuo35\nO5JWSFqemiV7hqQL09/PMknflLRb2TENRdLXJD0iaVnNsYmSFkj6iaSbJU0oM8ahDBL/Z9LfzxJJ\n8yQ9v8wYB9Mo9prX3i9pq6SJw12naxNCzcC26cDBwExJLy83qqZsAt4XEYeQDbZ7V4/FP+C9wHJ6\n8+6vLwA3RMTLgWlsPwamq6W+t78DDo+IPyZrdv3bMmPKYQ7Z/9daHwYWRMSBwK1pv1s1iv9m4JCI\nOBT4CXBhx6PKp1HsSJoC/Dnwv3ku0rUJgZqBbRGxCRgY2NYTIuLhiFictp8k+zDap9yomiNpX+Bk\n4Ap2vD24q6Vvcn8WEV+DrE8rIn5dcljN+A3Zl4o9JI0F9iAb8d+1IuJ2YEPd4W2DT9PzX3U0qCY0\nij8iFkTE1rR7F9k4qa4zyHsP8DngQ3mv080JodGgtcklxbJT0re9w8j+oHrJ54EPAluHK9iF9gN+\nKWmOpPskXS5pj7KDyisiHgc+C/wf2V16GyPilnKjakntzAOPAHuVGcxOOhu4oewg8pI0A1gXEUvz\nntPNCaEXmyh2IGk88B3gvamm0BMkvQ54NE022FO1g2QscDjw5Yg4nOwutm5urtiOpJcA5wNTyWqW\n4yWdXmpQOymyO1h68v+1pI8Av4+Ib5YdSx7py89FwKzaw8Od180JYT0wpWZ/ClktoWdIeg5wHXBN\nRPxH2fE06dXAKZJ+DnwLOF7S10uOqRnryL4d3ZP2v0OWIHrFK4E7I+KxiNgMzCP7N+k1j0jaG0DS\nC4FHS46naZLOJGs67aWE/BKyLxNL0v/hfYF7Jf3hUCd1c0LYNrBN0q5kA9uuLzmm3CQJuBJYHhGX\nlB1PsyLiooiYEhH7kXVm/jAizig7rrwi4mFgraQD06ETgQdKDKlZK4GjJY1Lf0snknXu95rrgbel\n7bcBPfXFKE3h/0FgRkT8tux48oqIZRGxV0Tsl/4PryO7QWHIhNy1CSF9KxoY2LYcuLZ26uwe8KfA\nW4Dj0m2bi9IfV6/qxar+u4FvSFpCdpfRJ0uOJ7eIWAJ8neyL0UAb8FfLi2h4kr4F3AkcJGmtpLOA\nTwF/LuknwPFpvys1iP9s4FJgPLAg/R/+cqlBDqIm9gNr3vtauf7/emCamZkBXVxDMDOzznJCMDMz\nwAnBzMwSJwQzMwOcEMzMLHFCMDMzwAnBOkzSlnQ/9zJJ35Y0bpByd7R4/SMkfWEn4uuZ6UV2hqTz\nB3vvbfTyOATrKElPRMTz0vY1wL0R8fma18emQYmlxzeSpekMXhkRj5Udi3UP1xCsTLcDL5V0rKTb\nJX0PuB+e/aYuqSKpKunf00Il1wycLOlVku6QtFjSXZLGp/Lz0+v9kv5N0p1pgZZ3pOPjJd0i6V5J\nSyWdMlygks5Ii6QsHpjTKU2r8sN0/JY09zySrpL0ZUk/kvTTFNPVyhbpmVNzzSclfU7ZAkq3SJqU\njr9C0o/17KIsE9LxqqRPpd/1QUnHpONjlC3kcnc655yh3jtlizXtA9wm6VZJu6SYl6X34/yd+2e1\nnhURfvjRsQfwRHoeC3wPOBc4FngSeHGDchVgI9kHmMiG578a2BX4KXBEKjeebBGZCjA/HesHFgG7\nAS8gm0r6hanc81KZScCq+p9bF/MhwIPAxLQ/IT3PB96ats8Cvpu2rwK+mbZPIVvb4JAU/0JgWnpt\nKzAzbf8TcGnaXkq2lgPAxcDn0/ZtwGfS9klkC88AnAN8JG3vBtxDNrFZw/culft5ze9zBHBzze/7\n/LL/Tvwo5+EagnXaOEmLyD601gBfI/uwujsiBlvV6e6IeCgiAlhMttbBQcAvIuJeyBYhiogtdecF\n8L2I+F1kTSO3kS28JOBf0hxHC4B9hpkF8njg25GtUUBEbEzHjwYGpkO+Bjim5ucOLDl6P/BwRDyQ\n4n+A7MMasoRwbe35kvYk+0C+PR2/GnhNTSzz0vN9Ndf5C+CM9L7+GJgIvDTFUf/eTWVHPwX2l/RF\nSa8lS2A2Co0tOwAbdZ6JiMNqD2STefLUEOf8rmZ7C9nfbaudX0E26eAkstkft6T29N2HOWewueQH\nO/779LyV7ePfSuP/d6Lx71R//YFrDbwPA86LiAXbnShVaPzebSciNkqaRrYE498DbwLe3iAWG+Fc\nQ7BeFGRNOC+U9EoASc9Ttg53LQEzJO0m6QVkTSh3A3uSLf6zRdJxwIuH+Xk/BE5VWqRcUl86fifP\nrnN8OvDfTf4euwCnpu03A7dHxG+ADQP9A8Bbgeow17kJeKeypTaRdKCGXx3uCbL3gfTejI2IeWRN\nV720boS1kWsI1mmNvgU3WkkrBtnODkRsknQacGm6ffJpssXEa68VZO3xt5HVCD4WEQ9L+gYwX9JS\nsjb92mnVG/2s5ZI+AfyXpC1kzTVnk02vPUfSB8kWfjlrkOsMVpt5CjhS0j+SLS95Wjr+NuAr6UP9\np3XX3S609HwFWVPQfcqqW48Cf03j93XAV4EbJa0H3pd+j4EviD2zspy1l287tRFL0izgyYj4bNmx\nNDJabnG13uEmIxvpuvkbTzfHZqOQawhmZga4hmBmZokTgpmZAU4IZmaWOCGYmRnghGBmZokTgpmZ\nAfD/rkPCdeFHCccAAAAASUVORK5CYII=\n", "text/plain": [ - "" + "" ] }, "metadata": {}, @@ -669,7 +718,7 @@ }, { "cell_type": "code", - "execution_count": 13, + "execution_count": 19, "metadata": { "collapsed": false }, @@ -682,7 +731,7 @@ }, { "cell_type": "code", - "execution_count": 14, + "execution_count": 20, "metadata": { "collapsed": false }, @@ -691,7 +740,7 @@ "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAEPCAYAAACqZsSmAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHrFJREFUeJzt3X+Q3PV93/Hne4EL17ExSKJSKDJQglsbjHygepTiVusm\nd4dpChFKp/aM4iOeQuw2xNEcRIibiHM4SmgsyzGdNiNisBIcezzGUNEwt2hirxq5bsaAEMKY1vYQ\nDZhfwcw0YZBH4Hv3j/3eaW9vf3x39/vdz/e739djZmdud7/73c/u7b73831/3t/Px9wdEREpllLo\nBoiIyOAp+IuIFJCCv4hIASn4i4gUkIK/iEgBKfiLiBRQ8OBvZqeY2WEzezh0W0REiiJ48Ac+DTwD\n6IQDEZEBCRr8zexc4CrgTwAL2RYRkSIJ3fPfA9wMLARuh4hIoQQL/mb2K8Cr7n4Y9fpFRAbKQs3t\nY2b/Cfh14G3gdOAM4AF3/3jdNhoHEBHpgbu37VQH6/m7+63uvt7dLwA+CnyzPvDXbTe0l9tuuy14\nG/T69PqK9tqK8PriCJ3zr6devojIgJwaugEA7n4QOBi6HSIiRZGlnn/hlMvl0E1IlV5ffg3za4Ph\nf31xBBvwjcPMPMvtExHJIjPDszrgKyIi4Sj4i4gUkIK/iEgBKfiLiBSQgr+ISAEp+IuIFJCCv4hI\nASn4i4gUkIK/iEgBKfiLiBSQgr+ISAEp+IuIFJCCv4hIASn4i4gUUMgF3E83s782syfN7BkzuzNU\nW0REiibYSl7u/lMz+7C7v2lmpwKHzOxD7n4oVJtERIoiaNrH3d+M/hwBTgFeD9gcEZHCCBr8zaxk\nZk8CrwDfcvdnQrZHRKQogi7g7u4LwAfM7F1AxczK7l6t32Z2dnbp73K5PLRrb1YqFXbv3gvA9PQN\nTE5OBm6RiORFtVqlWq129ZjMrOFrZr8HHHf3z9bdVog1fCuVClu2THH8+F0AjI7u4MEH9+kHQER6\nkuk1fM1sjZmdGf09CowDh0O1J6Tdu/dGgX8KqP0ILB4FiIikIWTa5+eBfWZWovYj9Gfu/pcB2yMi\nUhghSz2PApeFev4smZ6+gUOHpjh+vHZ9dHQH09P7wjZKRIZaZnL+zRQl5w8a8BVZpO9C/+Lk/BX8\nRSRxvQZwFT8kQ8FfRAaunwA+MbGVAweuplb8ALCP8fH9PProA+k1eAjFCf5B6/xFZPgsr16D48dr\nt6n3ni0K/iKSGSp+GBxN6SwiPatUKkxMbGViYiuVSgWoBfDR0R3APmBfFMBviLW/yclJHnywluoZ\nH9+vfH+KlPMXkZ7ccccd7Nq1h4WFi4ArGB29fylYq2InLA34ikgqgbhSqXDVVR9jYWFPdMsOYBvj\n489pcDYDNOArUnCNlTeHDk0lkkrZvXtvFPin6m79Y+CcvvYrg6Ocf4Ka5T9FQhrkvFGl0g9i5/Yl\nPPX8E5JWD0skixqrckql7fz+70/r854j6vknRDNzShb1U3nTzmJVztjYPaxadTsbNlzKxo0b+95v\nIx1Np0c9f5EhthikTw74Jns0+uyzP+T48bt4/XXYsiXZo10dTafM3TN7qTUvH+bn5310dK3Dlxy+\n5KOja31+fj50s0QSNz8/7+Pj1/o73/nu6PPu0eVLPj5+bcvtx8ev7eo7MT5+baz9y0pR7GwbX9Xz\nT0jaPSyRLFjeG3+xy+3Ve8+UTr8OIS/kqOcvUgTLe+PzDmuWjnZLpbNW9Oz76b3raLp3ZL3nb2br\ngT8F/iHgwF53/0LINolIXJPUChxuB85mw4b3Jdqj19F0uoKe4Wtm64B17v6kmb0DeBz4VXf/fnS/\nh2yfiCzXmMaBm4CpZVM7tNu+VNrOhg3v4847f0+BPEW5m97BzB4C7vZoLV8Ff5HsWZwu4rXXfgK8\nzZo1a9tOG1GpVNi5806OHHmahYXrgPdrkZaU5Sr4m9n5wEHgYnd/I7pNwV9kCJxcpGUdsBd4kbGx\nU3jiiUOBWzaccjO3T5Ty+Trw6cXAv2h2dnbp73K5TLlcHmjbpDnN2ijdO0ptArhaCujIke1UKhV9\ndhJQrVapVqvdPajTiHDaF+A0oAL8TpP7Ehj3Lo5u66nbbd/pPlVhSDfm5+e9VFqtuv0BIUa1T+jA\nb9Sqffa0uD+VN2YYdRuQ223faV86+UZ6MTa2OdHPTa8njxVBHoL/h4AF4EngcHS5su7+tN6bXInz\nIe82ILfbvtO+FPylF0keMeros704wT9ozt/dD6HJ5drK4hmSWmdVepFk3b4Wie9fJgZ8pbW4H/Ju\nA3K77VvdVz/IOzNzIwcP7o+2V8mexDM5OanPSkYo+A+JbntV7bZvdh/QcASiOm0JR0ef/ctMnX8z\nqvNfmfYJdXLMyTrtxWX79jE+vr/teq0qB5U06fPVWm7q/KW1tOc3SesLlMWxCgkjrc+YUkh96jQi\nHPKCqn1S1U3FRLfVFaoIEndV5YRC1qt9JKxuKiY0w6L0QlU52aXgnwPtDpsHmffs5jBbA3IiGdfp\n0CDkBaV96g6bpx02eam02ufm5hru6+2QOu1Dcp2BKUr7hEGMtI+qfTKuVmVzAXA/ixNilUrbeeSR\nr7B7996uK3AWnZyW9xXgVNasWa2KCUmFqnIGL061j86uzYVvUwv8U8AUCwt7lr5MzVQqFSYmtjIx\nsZVKpdL0/i1bpjhw4GoOH76eZ599tumXstN+un1eyZZB/b8mJyd59NEHljol+oxkRKdDg5AXlPZp\nOxtis0Pqubm5jofZcSpxOh2uN6Z0dHifLyH+X0nP7aOUYmtkfWK3jo1T8Hd397m5OS+Vzmo5A2f9\nlyBOYO93m2Zf4rGxK1TamSMhSnGTek51NDqLE/xV7ZMDMzMzbNy4seVUDPXpmnbpoEX9VuI0K987\nduz22I8X6YfKR5Oh4J8Tccss4wT2ODX73f5AnHfeuRw/vkOlnTkRohS3n+esHzSurR0sfet0aBDy\ngtI+PUkqH9pqP60Ou5WHzZcQ/69enrPx8zYycqaPjJyttE8bqNRT0qLyPRmUZpMKjo3dw5o1awF9\n/prJ/MRuZnYv8K+BV939/SHbIst1Cu6aVEtCWrNmbazzWaS10Dn/+4C7qa3jKxmhGTklSzRVSDqC\np33M7Hzg4WY9f6V90tOuZ9/L3P0iaVKasTuZT/tIGOrZS94ozZi8zAf/2dnZpb/L5TLlcjlYW4ZF\npzppHWZLljUeBQCFPyqoVqtUq9XuHtSpHCjtC3A+cLTFfcnVPslSmd2qVRdGs4S2PtMyTkmeSjtl\n0FaWfZ7tIyNnquyzAXmY3kHBP3nNgnLjlwbOiH4AevvC6BR7CaHZFBGwSdOKNIgT/EOXen4F2Ays\nNrPngV3ufl/INuVdq3x+Y6oHYNWq27n88ud6WpWrm1PsNVgnkj1Bg7+7fyzk8w+jVkG5mcsv35B6\nBY8GlyVJjeNRIyM3A29x4kRtTErjU/FlfsBXkpH0IO709A0cPPhRTpz4YwBGRp5levqrK7brdhIu\nLTIj7aycl+rPANrOUyUtdMoLhbygnH/X2uXikxygnZ+fXza/ysjI2U332c00vsuXrFzT9XiCBqDD\n0vufHeRhwLdt4xT8ezKIL2HcoN7NwPDJfXY/77sGoMPS+58tcYK/0j5DKEsnxMSZPjoJmuN9MFoN\n3mf5/VfBQXMK/tKTbsYQul+LYBtw09LtGsTLhjwO3uexzQPT6dAg5AWlfTItjfTS4j7Hxq7wsbHN\nsfettEP62qX65ubmonNHTp5HMjc3F7jFYZarzAKU9pE0pZFe6nWfg0ovSXMHDz4BXA/sj265noMH\nn2BmZvl2SsFkSKdfh5AX1PPvSBUW3dN71pt2R1dxetjNVuTq5ugu6TYPM1TtM9wG/cEehqBZ1GCQ\nlG6X9qy3/Adivqdy3iTbPMwU/IfcIPOZeQuarb7wRc0Bd6vXtXbrH9N4ffl7r/9DmuIEf+X8JZYs\nl/I1UoVHf3p9/+rHa5rtY2bmRg4d2hFViL2Y5kuQODr9OoS8oJ5/W4PsjXfbY077ULvd/tu1NW9H\nMCEkcXTUah/11Vz1Z4jr/5AslPYZfoPKZ3YTNNMOsJ323yl4FTEH3I24g7ft3sOV+f1NvmrVhSvG\nCfR/SIeCvyQq7pc17bx6nOCu3n3vOr1/cd7fVvM0jYyc7WNjVyjgp0zBX4JIoucYcv/Se1qtcR+1\nVeOaLb6i6rQ0ZT74A1cCzwI/AHY0uT+dd0ZSlUTPsZ/9S7q6ObJrvvLWtapOS1mmgz9wCvBDass4\nngY8Cby3YZu03htJWb89x069tiL36uqFeB/6Gf+ppYDmUw/+RS/pzXrw/0Vgvu76LcAtDduk885I\nS1mYDrrovba4Qr5P3XxOTlb4bB7YYusK/tkO/r8G3FN3fRtwd8M26bwz0tTyQbpNXiqtTmVyrn6r\ndaQmj+9TFqvThlGc4B/yJC8P+NzSRO1Erm3A/cBdLCzArl3b2bhxY6InSGkStuHSzWRtg1prQp+x\nzkIG/x8D6+uurwdeaNxodnZ26e9yuUy5XE67XQX3beDkmbwLC8vP5E1qVsZ2QSDp9YaHVRbepyyf\nTZ2lRY3SVq1WqVar3T2o06FBWhdqPzw/ojbgO4IGfIObn5/3Umn1ilTCqlUX+vj4tT43NzewQ2kN\n6MYT+n3KY+qpCOg35w+8F/gl4B0Nt1/ZacdxLsBHgP9DrepnZ5P703t3pKm5uTkvlc6qq844IxoD\n+FLTHwZ90fsXOoD3Q8E/m/oK/sBvR4H5IeAY8Kt19x3utOMkLgr+YSwGo9oJOtN1X+xN+qInLO8D\nk3lv/7DqN/g/vdjjj1IzjwG/4wr+hbGyVze97KhAX/T+DUPPOckjlzwfBWVJnODfbsDX3P2NKAL/\njZmVgQfM7DzAuhtZkDxaOaB4PzMz0xw8uD+6PxsDexJWUgOrWR48HkZW+5FocofZt4Dt7v5k3W2n\nAV8Etrl7KfXGmXmr9slgaM3VdDUGvNHRHYUNeBMTWzlw4GoWK81gH+Pj+3n00QdCNiuXzAx3b9tJ\nb9fz/zjwVv0N7v6WmU0BexNon+RAkcrlQshbPbo6A8OjZc8/C9TzH04KIPmU9lGKjoKSE6fnr+Av\nA6UveH4NIi2jjkEy+k37iCQuT2sBy+ApzTg4LYO/mV0ErHX3Qw23fwh4yd1/lHbjRCQ7sjCdhCSn\nXcXO54G/a3L730X3iXRtevoGRkd3APuAmyiVpnnttZ9QqVRCN006WBycHh/fz/j4fqXrcq5dqedj\n7r6xxX1Pu/slqbYM5fyHVaVSYefO2zly5BkWFvYAyv2LJClOzr9dz//MNved3luTRGo9yDVr1kaB\nfwqoDQAvDvQNu0qlwsTEViYmtuqIR4JpF/wfM7MbGm80s+uBx9NrksjwWqx2OnDgag4cuJotW6b0\nAyBBtEv7rAMeBE5wMthfDvwcsMXdX0q9cUr7DK00Sj7zUCaos1hlEPoq9XT3l4FfNLMPA4v5/f/h\n7t9MsI1SUEmf2ap5YUS6067nPwp8EvgF4CngXnd/q+nGKVHPX+LKS486iSOePBzhSFj9Dvjuo5bm\neYraoiufTbBtIl0ZlkHSfsslNWYgiWk11zNwtO7vU0lwDn/g3wLfA34GXNZmu16mspaM63bO9jgL\nhhRlUZFhmP9f0keM+fzb9fzfrvuBeLvNdr04CmwB/mfC+5WM66XnunxKiOZloToBaaVhOVqSdLSb\n2+dSM/v7uuujddfd3c/o9Und/Vmo5aWkWNKc26cI88LEnWJBA+DSScuev7uf4u7vrLucWvd3z4Ff\npFvLp4TYFwW8Faeg9CxED7nX54x7hBPnaEkKrlNeqNcLcIBaeqfx8m/qtvkWyvkXSq+5+bTWdg0x\nVjCI50xrbEBr7OYDMXL+Qefzj5aKnHb3J1rc77fddtvS9XK5TLlcHlDrJC1ZKlXsVCKaRlsHNS9+\nGifRaS2GbKpWq1Sr1aXrn/nMZzqWeqbW849zodbzv7zN/Yn/IorUa9dDTquHPqiKnaR76ao0yg9i\n9PyDLOZiZluALwBrgL8ws8Pu/pEQbZFiazeAGmdwupcjg0HNi1+EAXDpXZDg7+4PUps3SCSofqaZ\n6LWiJm+Lti/SYi7DRWv4irTQKcedlyklkpSl8RppTWv4ylAaVADKaw89TUolDQ/1/CVXGnvjIyM3\nc/HF72HNmrUD74nmofpFPfViitPzD1rt0+mCqn0KJU51SrOKE9i0ohpnUPXoWa57P1mtNO2wyUul\n1T43Nxe6WTIAZLXaR6RRf9MRnEPtLFaWermDmtoga2mQ+p7+a6+9wvHj24D7gbtYWIBdu7azcePG\nTLVZAun06xDygnr+Q2+x57xq1YWxasgba+9hjcP8sseMjW0uZD1643tTKp3l8E8L+V4UHX3O6imS\nqvoZPl9//exYj6mf22Zs7D5GRt4GXmZxzp/Nmy/jyJGnU213VjXO57OwsAd4NXCrJKuU9pFglger\ndcC2pfva1ZDXp1qWD2jWKnMWFq4DdixtXyptZ3r6K+m8iIy78MJ389xz21lYqF1Xbb4sUrWPpCJO\nlcnKOvmbWLXqIS6/fEPPlSkn97kO2Au8yNjYKTzxxKFeX0putKo+AlTxUzCq9pEg4s6Jk8bcOUVZ\n0auVLFcfyeCQ9Vk9O1HPP5+6OfM1jTp01bZL0ekMX8m8NEols1Z+KZJFqvaRxKW98pb0R2v7CmjA\nV1Ki1Euykno/8zAlhfQvTtpHwV8k45IM2EWcibSIlPMXGQJxFpUR6VawnL+Z/aGZfd/MjpjZN8zs\nXaHaIsU0iNx36Px64/NrPEaWdKoFTesCjAOl6O8/AP6gyTbJFb6K1BnE+QDz8/M+MnL20nOMjJzt\n8/PzXdfi99rWVo/TuQDDjxh1/sFP5Kq1ky3A/U1uT+N9kSHX69TQ3Ux4Fuc5mk0wd+GFH+g5kHcb\nsLXgenHFCf5Zyfl/Aijm5CuyTL9VLf1NDR1v/zt33s6RI89EE6e1fo5jx15Y8fhjx17k7bf/M93m\n73XugiQt1eBvZgeoTbLS6FZ3fzjaZgY44e5/3mwfs7OzS3+Xy2XK5XLyDZVMSCJwxx0c7WUx8pPt\nuwDY0/E5zjtvHa+/flPdLTdx+umn8cYbsV9OX7TgenFUq1Wq1Wp3D+p0aJDmBbgO+DZweov7Uzgg\nkqxKIk3RzT66TaWc3He856jl/M+MVhrb5Gbv8KmpqYHOPaT8fjGR5bSPmV0J3AxsdvefhmqHDJdu\neru9p1Ju4GSdfOvnmJycZNeum9i1aw8LCxfh/pt87Wv3MzNzIwcP7o/am+4JVkoXSUudfh3SugA/\nAI4Bh6PLf22yTSq/ipJNSVXgpNXbXd6+aS+VVvvY2Oa2z6FBVwmBLPf83f2iUM8t2bS4Slf94izd\n9FoHMUPo8vZ9Wb1qyS1N7yBDIakpEOqD/ebNl3HHHXf3tU/NpSMhaG4fKYwk5qxpDNSl0nYWFj4B\nfLbnfS7uV5PcySDFCf6a0llyJ60pE5ovgP7txPbfTuhpIKSAOg0KhLygAV9p0G7Kgn4Hi5sNzpZK\nq/vaZ5x2FX3pSUkeeZneoWXjFPylQbvqmX6rfJoF4bm5ub72GafaRxVBkrQ4wT8r0zuI9K3fmvZW\n1UYzM0m1EOAojz9+hImJrcr/S1Aa8JVcyVv1zPL2HgXuAb4AnGw7kKvXJNmnah8ZSnmrnlls73e+\n8x3eeOM84BxqZwm/vFQ9lLfXJNmmlbxkKOVxyoLXXnuFN954E/hkdMsUsG3p/jy+Jsk3BX+RFC2f\nCfSPqJ8TqFSaZnr6y8HaJsWmOn+RFJ08d+CcFfdt2HCJevsSjHr+IgOxcibQO+/U3PoSjnr+In1q\nd3buyQXTXwa2USpNMzZ237JqHp3dKyGo2kekD3FKT9tV8uStdFXyQaWeIinrd0K5JCakE2mkid1E\nRKSpIAO+ZnY7cDXgwE+A69z9+RBtEelHv4uka5F1CSVI2sfM3unufx/9fSOwwd3/fZPtlPaRzOv3\n7Fyd3StJy0XO38x2Au9y91ua3KfgLyLSpUxP72BmdwC/DrwJbArVDhGRIkot+JvZAWBdk7tudfeH\n3X0GmDGzW4A9wG8028/s7OzS3+VymXK5nHxjRURyrFqtUq1Wu3pMFtI+7wYecfdLmtyntI+ISJcy\nW+ppZhfVXb0GOByiHSIiRRUq53+nmf0T4GfAj4BPBWqHiEghBU/7tKO0j4hI9zKb9hERkbAU/EVE\nCkjBX0SkgBT8RUQKSMFfRKSAFPxFRApIwV9EpIAU/EVECkjBX0SkgBT8RUQKSMFfRKSAFPxFRApI\nwV9EpIAU/EVECkjBXwqpUqkwMbGViYmtVCqV0M0RGTjN5y+FU6lU2LJliuPH7wJgdHQHDz64j8nJ\nycAtE0lG5ufzN7NpM1sws1Uh2yHFsnv33ijwTwG1H4Hdu/eGbpbIQAUL/ma2HhgHjoVqg4hIUYVa\nwxfgc8DvAv89YBukgKanb+DQoSmOH69dHx3dwfT0vrCNEhmwIDl/M7sGKLv7djN7Drjc3V9vsp1y\n/pKKSqWylOqZnr5B+X4ZKnFy/qn1/M3sALCuyV0zwE5gon7zVvuZnZ1d+rtcLlMul5NpoBTa5OSk\nAr4MjWq1SrVa7eoxA+/5m9klwF8Cb0Y3nQv8GPigu7/asK16/iIiXYrT8w9e6qm0j4hIsjJf6hlR\ndBcRGbDgPf921PMXEeleXnr+IiIyYAr+IiIFpOAvIlJACv4iIgWk4C8iUkAK/iIiBaTgLyJSQAr+\nIiIFpOAvIlJACv4iIgWk4C8iUkAK/iIiBaTgLyJSQAr+IiIFpOAvIlJAQYK/mc2a2Qtmdji6XBmi\nHSIiRRWq5+/A59x9LLrMB2pHUN0uuJw3en35NcyvDYb/9cURMu3TdpWZIhj2D6BeX34N82uD4X99\ncYQM/jea2REz+6KZnRmwHSIihZNa8DezA2Z2tMnlauC/ARcAHwBeAnan1Q4REVkp+ALuZnY+8LC7\nv7/JfVq9XUSkB50WcD91UA2pZ2Y/7+4vRVe3AEebbdep8SIi0psgwR+4y8w+QK3q5zngNwO1Q0Sk\nkIKnfUREZPBycYavmd1oZt83s6fN7K7Q7UmDmU2b2YKZrQrdlqSY2R9G/7cjZvYNM3tX6DYlwcyu\nNLNnzewHZrYjdHuSZGbrzexbZva96Pv226HblAYzOyU6wfTh0G1JmpmdaWZfj757z5jZpmbbZT74\nm9mHgauBS939EuCzgZuUODNbD4wDx0K3JWGPAhe7+wbg/wI7A7enb2Z2CvBfgCuB9wEfM7P3hm1V\not4Ctrv7xcAm4D8O2etb9GngGWqp52HzR8Aj7v5e4FLg+802ynzwBz4F3OnubwG4+98Gbk8aPgf8\nbuhGJM3dD7j7QnT1r4FzQ7YnIR8EfujufxN9Jr8KXBO4TYlx95fd/cno7zeoBY5zwrYqWWZ2LnAV\n8CcM2cmm0dH1v3D3ewHc/W13/3/Nts1D8L8I+Jdm9r/NrGpmG0M3KElmdg3wgrs/FbotKfsE8Ejo\nRiTgHwHP111/Ibpt6ERl2GPUfriHyR7gZmCh04Y5dAHwt2Z2n5k9YWb3mNk/aLZhqGqfZczsALCu\nyV0z1Np4lrtvMrN/BnwN+MeDbF+/Ory+ncBE/eYDaVRC2ry2W9394WibGeCEu//5QBuXjmFME6xg\nZu8Avg58OjoCGApm9ivAq+5+2MzKoduTglOBy4DfcvfvmtnngVuAXc02DM7dx1vdZ2afAr4Rbffd\naFB0tbv/ZGAN7FOr12dml1D7pT5iZlBLizxuZh9091cH2MSetfvfAZjZddQOsX9pIA1K34+B9XXX\n11Pr/Q8NMzsNeAC4390fCt2ehP1z4Gozuwo4HTjDzP7U3T8euF1JeYFaJuG70fWvUwv+K+Qh7fMQ\n8K8AzOw9wEieAn877v60u6919wvc/QJq/7jL8hL4O4mm6r4ZuMbdfxq6PQl5DLjIzM43sxHg3wH7\nA7cpMVbrhXwReMbdPx+6PUlz91vdfX30ffso8M0hCvy4+8vA81GsBPhl4HvNts1Ez7+De4F7zewo\ncAIYmn9UE8OWUrgbGAEOREc233H3/xC2Sf1x97fN7LeACnAK8EV3b1pNkVNXANuAp8zscHTbziGe\ndn3YvnMANwJfjjonPwJ+o9lGOslLRKSA8pD2ERGRhCn4i4gUkIK/iEgBKfiLiBSQgr+ISAEp+IuI\nFJCCv0gDM/tZNN3vUTP7mpmNRrevM7OvmtkPzewxM/sLM7uoyePvNbNXonNTRDJJwV9kpTfdfSxa\nV/oE8Mno9gepnRH6C+6+kdq8TGubPP4+alM+i2RWHs7wFQnpr4BLo3UlTrj73sU7Ws3E6u5/Fc2I\nKZJZ6vmLtGBmpwIfAZ4CLgEeD9sikeQo+IusNBrNa/Ndaqur3Ru4PSKJU9pHZKXj7j5Wf4OZfQ/4\ntUDtEUmcev4iMbj7N4GfM7PrF28zs0vN7EMBmyXSMwV/kZVaTXW7BfjlqNTzaeAO4KXGjczsK8D/\nAt5jZs+bWdMpdUVC0pTOIiIFpJ6/iEgBKfiLiBSQgr+ISAEp+IuIFJCCv4hIASn4i4gUkIK/iEgB\nKfiLiBTQ/wcJy8IUIL4AXgAAAABJRU5ErkJggg==\n", "text/plain": [ - "" + "" ] }, "metadata": {}, @@ -707,7 +756,7 @@ }, { "cell_type": "code", - "execution_count": 15, + "execution_count": 21, "metadata": { "collapsed": true }, @@ -749,7 +798,7 @@ }, { "cell_type": "code", - "execution_count": 16, + "execution_count": 22, "metadata": { "collapsed": false }, @@ -763,7 +812,7 @@ }, { "cell_type": "code", - "execution_count": 17, + "execution_count": 23, "metadata": { "collapsed": false }, @@ -772,7 +821,7 @@ "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAakAAAEaCAYAAACrcqiAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xt4VNW5P/DvC6kETCKChksEFEkiDYIXEH4iELwUrCIq\nar2gx8LjtbWearWHiy0KyrHtwZ7C4wEteAoUPa3QcqIVKmIMakWwGAED6AHDRW6K5IKQkLB+f+zM\nZGaYmcxk9uy19t7fz/PkcWbPzsyboZ133rXevZYopUBERGSiNroDICIiioVJioiIjMUkRURExmKS\nIiIiYzFJERGRsTJ0BxCPiLD1kIjIB5RSEu240UkKANS8ebpDsMW0khJMGzNGdxjG4/sUx/z5mPHc\nAxh78cV4fvrzePCJB3VHZDy+T4nR/T71z+wf8zEO9xG5RX4+cOyY7iiIHGV8JUVETYYPx9dLs/F2\nj826IyFyDCsphxQXFOgOwRX4PsV38H86AgAGDR+kORJ34PuUGJPfJyYphxQXFuoOwRX4PiVm0Ahz\nP1RMwvcpMSa/T0xSRERkLCYpIiIyFpMUEREZi0mKiIiMxSRFRETGYpIiIiJjMUkREZGxmKSIiMhY\nTFJERGQsJikiIjIWkxSRy1TvOqw7BCLHMEkRucjiZ3bpDoHIUUxSRERkLCYpIiIylrYkJSKZIrJW\nRD4WkU9FZKauWIiIyEzaduZVSh0TkZFKqW9FJAPAuyJymVLqXV0xERGRWbQO9ymlvm26eQqAtgAO\naQyHiIgMozVJiUgbEfkYwH4AbyulPtUZDxERmUXbcB8AKKVOALhARE4DsFJEipVSpaHnTCspCd4u\nLijg9uJERC637p11WFe2LqFzRSmV5nASIyJPADiqlPpNyDGl5s3TGFX6XP7MM6iuqQnez8nOxurJ\nkzVGRG4xo88JjB06VHcYRLbpn9kfSimJ9pi2SkpEzgDQoJQ6LCLtAVwF4Eld8TituqYG67OygvcH\nhiQsIiKy6Bzu6wbgDyLSBtbc2CKl1Fsa4yEiIsPobEHfCOAiXa9PRETm09o44Wc52dlhQ3w52dka\noyEiXco/KMfK5SsBAKPGjsKAIQM0R2QWYxonovFy4wRRa83ocwJo2xZjhwzRHYov2ZlUyj8ox6zp\nszDsnmEAgDUvrsEjTzziu0QVr3GCa/cRuczUz9sAjY26w/ClQFLpNLgTOg3uhFnTZ6H8g/JWP9/K\n5Ssx7J5hGDB6AAaMHoBh9wwLJkCycLiPiChBoUkl9JjfKh8nMUkREWkyauwozJo+K3g/MNxHzZik\niIgSZHdSGTBkAB554pHgEJ8f56NawsYJIhfiqhP6sBvPfkauOEFEZJpEEtCAIQOYmBzE7j4iIgCv\nzn8Vk+6fhC1fbEHd6XUpd+6RPVhJEZGRnBxWK/+gHAtfWIgrf3YlTjn1FKx5cQ36Ffdj554BWEkR\nkXHsvh6pJYHW8r5X9EW/Uf0w7J5h2LFhR9pejxLHSipB3FqDyDk6rkfqkNUBNV9Z/x+vP1KPfRX7\n8PDDD6ft9SgxrKQSFNhaI/BTza01iDxj1NhR+HDRhzi4+SA2vb4JpbNLMeGhCRzqMwArKSIyjtMX\nuYZer3Twy4PoW9QX2z/fjvIPyh1PVGxxD8dKioiME0gah9YewqG1hxy5yHXAkAEYNXYUDnx1APnX\n5TsyFxbJ6bk4N2AllSBurUGmWf7RRxh78cW6w0gbp65HCq1ctm/Zjn439EOXfl2Q0zEHQPy5MLur\nHq4NeDImqQSxSYJMMnXSS5jx3AO6w3C90K0y6o7VYcurW9D7ut7IODUD+/fuR92xuoR+FwBmTZ/F\nZY3SgEmKiHwrtHLZt3sfhowfgnfmvoMOHTug/kg9SmeXYvp/Tm/xd0OPpZKkuODsybTOSYlIDxF5\nW0Q2i8gmEfmJzniIyN9yz81F9/zuqHy/Epte34TC8wodrYx0zMWZTncldRzAT5VSH4tIFoCPRORN\npVSF5riIyAdCK5e6Y3VY+ZuVuOz2y9CtoBvWvLgG458Yn9DvAvZVPVwbMJxRq6CLyF8BzFZKvdV0\nn6ugE0Uzfz5mPPeApxsnnBLa/NC7T29s/3w7gMQaISIbJwCwfbwVXLEKuoicDeBCAGv1RkJEfpJK\n5RL6u4FGij4j+mDHhh0ovd+6IPimiTfZGa7vGJGkmob6XgXwsFKqVnc8RETJWrl8JfqM6INNpZsw\n7J5hqD9Sj4WzFyK/KJ8VVQq0JykR+Q6ApQAWK6X+Gvn4tJKS4O3iggIUFxY6GB2limsekp/s2LAD\nw+4Zhn6j+uFo1VE0Hmv0/XVO0ax7Zx3Wla1L6FytSUpEBMB8AJ8qpX4b7ZxpY8Y4GxTZmlgCax4G\nDOSah+RRo8aOQun9pag/Uo+jVUdR81UNOmR1wBEc0R2acQaNGIRBIwYF7899em7Mc3VXUkMBjAfw\niYhsaDo2SSm1QmNMrpdqkkkksbBCIgo3YMgATHhoAhbOXojGY43okNUBHy76MC3XOflpfT+tSUop\n9S64fqDtnKheWCERneymiTchvygfK5evxBEcSct1Tn5b6UJ3JUVxeKFa4ZqH5Dfpvs7Jb+v7MUkZ\nTFe1YmdicVtSJSKzMEl5UKpJJpHEEu81vFABGm/iRODYMWyu24yidkW6oyEH+W19PyYpD3IiIcR7\nDc5XOePrpaehz0DdUZDTQjdoBODp+SiAScpofpzPYRVGbqOj085P6/sxSRnMjx/OrMLITfzWaacD\nkxRFlUpF48cKkPzJb512OjBJUVSpVDR+rACJKD2YpMgorMLITfzWaacDk5ThLn/mGXy6ezfaNu37\n1UYE+WedFVateKnZwK1xkz/5rdNOByYpw1XX1OAsAOszrH+qTxsbcVfE0Fs6mg1Y0RAlxk+ddjow\nSVEYL1Vl5F0mL7BqcmxuxCTlYa1JOMlUZXYkNCZFSpbJbd9OxOa3JMgkZbic7Gx8WlWFvIYGAE1z\nUhFDb7GG5tJ9zVGizx8vEfG6KEqWyW3f6Y7N5ASdLkxShkukqjC98mAiIrKHyQk6XZikKAwbJtzl\n8/WHUTRUdxTOMrnt2+TY3IpJysNak3CSqcrsSGhMiq333IAazNAdhAYmt33bHVvk/JMfk6Coputv\nTCQiSs2bpzsMShGbI9JnRp8TGDvUZ6WUT0TOP4UmJK81TvTP7A+llER7jJUUpR0TElHyYs0/PT7z\ncU8kpkRpTVIisgDANQAOKKXO1xkLJYfVERE5QXcl9RKA2QAWao6DksSOPaL08uP8UzRtdL64UmoN\ngG90xkBEZKJAE8ahtYdwaO0hoxpEnKS7kiKDcAiPyCxcF9AFSWpaSUnwdnFBAYoLCzVG423JDOGx\ndZzIWbGWQ3LjMknr3lmHdWXrEjrX/CQ1ZozuECgKVlhEzom1HFLgttuWSRo0YhAGjRgUvD/36bkx\nzzU+SZE3cCiRqPVitaMD8PwySbpb0F8GMAJAZxHZBeAXSqmXdMbkNS0lh9DH91dVoWtVFc467bTg\nuXZhNyARtYbWJKWUuk3n6/tBS8kh7PGsLAysrcX6mTOdDJGIWhCvHd3rbeoc7iNjcYiQyBJvTUBT\n1zG0C5MUOWJvbS3yDh9uPpDR8v/0OESYmOXvvcf1+3wgVju619vUmaQ8rqVWcadaybtlZYUnnNra\ntLxOQKAK211VhbZKoY0Iupx2WqurMVOruqmft8GMPid0h+Fpbmzx9hImKY9r6YPUhA/adAhUYQOr\nqrA+IwOfNjbiu1lZra7GWNX5kx93wjUNkxQZixcMU7q1VCX5cSdc0zBJkSPSvQEjUbISrZLqjtVh\n3+59wdtu5sahSyYpcoTTCSeQFHcDyGtosOakamtbXY2xqvOeRKqk3n1648V/fxGjfjbKevw3K3HP\nQ/c4Hqsd3Dp0ySRFnmR3UmRV50/bP9+Oy26/DHvL9wIALrv9Mmz/fLvmqFrHrUOXTFJE5EuJ7tfU\nraAbRv9kNACgfEU5Dq09FPM53TicZjomKY8wtUXa7/jvYq54F8gGJLPxYLThtDE3jglWXrqTlls3\nUWSS8gi2SJuJ/y5ma+lC2HiJLLJqihxO27ttLxa+sBDXTboOgP45oESSsomYpMh2fqge/PA3up1d\nQ2/RElm0qin3jFx0QqfgOTs27DBuDsiNq1MwSZHt/FA9+OFvdLN0d7JFa0L47H8/w5oX1wTv76vY\nh4vHXmzL6/kZk5RHsEXaTPx30UNHJ1vnLp1x+723B6u3CQ9NQMmiErTLbAfAPXNApmGS8ggvDDXp\nHkJLx+t74d/FD5IdGgxtQti7bS/+seQfOP+i8wEAj898PHheflG+6+aATMMkRbZrbfWgewgtmdc3\nrUJavnkzxhYVaY3BJKl25bWUUAJNCIv/azG2btmKa/7tGrTLbBe1oy80aVHymKTIdn6oHkz6G6dO\negkznntAdxhGSaaTrbVDgwOGDMDK5StReENh8HfrjtVhwb8vwI3TbgSgv6MPcP+1W0xS5Eq6hwaN\nUVaGFw5cD2RmYk/DHuRl5OmOyBg6Otm+rf0WXft2Naajz61LIYXSmqREZDSA3wJoC+D3SqlndcZD\neiUzhJaOoUHThvBOUlYGAHjhvZBhvdyRQC5Qd7Aj1uMw1sPaWHJgYUcAMCZpmfxtPpWLXKP97qBx\ng2yPsbXcuhRSKG1JSkTaApgD4EoAewCsE5H/VUpV6IqJ9EpXJZRo1WVqJfbCzK+bbhUBublALoCJ\nE8PO6VEJoNK6XdGwEe8d3IN2ZzYnrZxsYGRPPXNWpn+bT+Ui18jfveveu1CyrATlBeUA2NFnB52V\n1CUAPldKfQEAIvIKgLEAmKQM5sZhNt0NGclqTkpNcnNPSkrx9M04H6g8P5i0AODAwFewfPNmAFbC\nApxLWm74Np/K0GDk75rU0efWpZBC6UxSeQB2hdzfDWCwplgoQdU1NXi9/dn4srELLjxlMwbW1GDD\nzs7o3vFbdMk56lgcxg/NJeGkpAQAkybZ+hq5628FEBwxxHmPhCet3C5AUTt2B9rBpFUd3LoUUiid\nSUolctK0kpLg7eKCAhQXFqYtIErMl41d8Ng3k/Hs6TNxpOEoHls6GL8et9bRJGV69RZTWRnw2WdW\ns0NAkpVSKoYPb7oRJWl9Ditp9eljHbMrabnx2/zX+zNwYG8G+l5wDABQ8XEmcrs1oHOXBs2RJc+k\npBmw7p11WFe2LqFzRamEcoXtRGQIgGlKqdFN9ycBOBHaPCEiSs2bpyU+im7gpElYn5WFj+r74b5D\nM1HReAJlP9+Oi3t9pTu0mLQPUUYmpqFDrf8GM4Y5Kho2AgA6D9kcPJaTDRR075hSE4bJjRPRVHyc\niVmTuuGnT1v7SD03pRsembk3mLTIXv0z+0MpJdEe05mkMgBsBXAFgC8BfAjgttDGCSYp8wQ+8I80\nDEDlkd9B2rRD2WMbcXGvr8KG/bQnBs1SnVcyya5eK9DuzMPB+365aPjTf2biqR9bifkXc/bguxcx\nQaVLvCSlbbhPKdUgIj8GsBJWC/p8dvaZb/XkydiwszMeWzoYUy/div9c3Q8PLrkMP7l8I156vzA4\n7Oe2ZoVUeSkpAUB1NXD4MNCzJ9CjcjR2rgE6dgSqzl8RnMsK6NOH81mUPlqvk1JKvQHgDZ0xUPK6\nd/wWvx63Fhf2/Brnda3CnQtG4lcrB2DBv5Thwp5RmgA8yGtJKdLhw8DSpcCN1sIJWLYMGDcO6Fk5\nOqxrsKwMQMh8ltOdg+lS8XEmnpvSDb+YswcAh/t0ipukRKQvgO4A1iqlakOOj1ZKrUh3cGSmLjlH\nw5okOpzivsnkpM2fDwDhDQ82d+CZpGdPK0EtXmzdHz/eOhZp+HBEbcIIVFt9+gAd26Y2n6VDbreG\nsKT0yMy9yO1m///O3TZXp0PMJCUiPwHwI1jXLS0QkYeVUn9tengmACYpn9uwszN+vuwSzBtv7aHz\n82WXBCss17eIz58fnpBwvdXwMNG8ZgdTRHYOVjRsxNcfWE0YgYuK7e4cTJfOXcI7+dJRQZl+kbMp\nYjZOiMgmAEOUUrUicjaAVwEsVkr9VkQ2KKUuTHtwbJww2v7q9vjycIfgEJ+O66VsU1YWvtyQwR14\nTti5M8ZwX5RqKlHROgfdWmnZ4VeTfoVOgzsFL3IuX1GOQ2sP+XLV9NY2TkhgiE8p9YWIFANYKiK9\nAER9MvKXyGE/t81Hhc8rFXluXikVHTuGJ6Vx46xjqeibYe23hPXWf8vKgLo7V4Qt3+R05yCH28wX\nL0kdEJELlFIfA0BTRXUtgPkA+jsSHZGNnFjZwStycqyfgFQqqFiGDwcQ0oixq9fJnYMDC9NXZeke\nbnPjRc46xBvu6wHguFJqX8RxATBUKfVu2oPjcB+1VuTwXQCTkmtUNGw86aJiwL7OQROG21jJWVo1\n3KeU2hXjuAKQ9gRFFEvMC4XDmh2KrHkln84peUHfjPPDhgYBb3QOhjJxySLTcNNDcp3ghcK1tThY\n2x7DqqqahvKuZ2LyqGhrDn596cawzsFk99DicJs7aFsWKREc7qOgkA3/Zu36Fda0PRXIyAAyMzEM\nR/HIzMoWnoC8rKJhI7LO3RO2fFMiSYvDbWZo1XCfiOQD6BI59yQilwHYq5T6P3vDJIoQNq8UsuFf\ndQ6QdWbzebUubHknW0XuoZXoxo8cbjNfvOG+3wKINstc3fTYmLRERL52Ult4lOG77zwzF8NqDjTf\nz851KDpyi3gbPwYaMFJd2Z2cEa+7b71SamCMxzYppfqlNTJwuM8PUm0Ln/3MIByPSFgPTU5snxoK\nX0gWsC7i7dgxvP3ca8rKgDMvPblzkBs/6tPai3njXbqXmVpI5EvR1r9L8QLa4zUHsCZk6C+0wqKW\nxVpI1stJyirMo3cOpmvjR2q9eElqvYjcq5R6IfSgiNwD4KP0hkWeEbbhX1P3XT5c1YHn5Woj0YVk\nvSzemoOBpJVs5yDZJ16S+lcAfxGRO9CclC4G0A7ADekOjFzqpItoi4DckcAk9y435Mdqw88il2/a\n1WsF3juIsCYMJi3nxLuYdx+A/yciIwEE5p9eU0qtdiQyco2weaXckVYHnkNr4H0nOzftTRRerjZ2\n7rSS7vjx1n07FpIFvFV99qgcbd2I0znIjR/TJ14LensA9wPoA+ATAAuUUsedCozMZdKGf2ySSE06\nFpIFvF19RnYOhm786JVNH00Sr7vvTwDqAawBcDWASqXUww7Gxu4+Uxi04V+q39Bb8/vp2LbCDyor\nw6vPXr30xuOEsjLgnKaV3QPYhNGy1nb39VVKnQ8AIjIfAL+y+kFYo0PA9cYszJrqN/TW/H66qg3y\nnsiV3UOXb2LnYOvEq6TCNja0c6NDEbkZwDQA5wEYpJT6Z4zzWEmlW8hyQ0GGb/iX6jd0P37Ddxqr\nz+iibfzIJozWV1L9RaQm5H77kPtKKZXK6PJGWB2CzECaNM8rhSw3xA3/yCbJVp9earSIJ5HOQac3\nfjRdvO6+tul6UaXUFgCwtqYiJ3hlw79Uu9HS1c2WCJM/iO2OLdlNE73caBFPZOdgtI0f/d45qHUV\ndBF5G8CjHO6zmYc3/NPROGEXk4fATIjN6WFYk780BJSVWSthBERbJNcLWjvclxIReRNA1ygPTVZK\nlST6PNNKmk8tLihAcWGhDdF5kE82/Et1W3MntkWPJZHrrXR9cHr5WrBY3FC9DR+OsD20Qjd9BNy7\n8eO6d9ZhXVlivXispNwqLCk18Whi8pKWqgWdFY3OhhJdf7ebm2iiNWG4NWlpqaSSwImplkTrwMu9\nFxiaz6TkIonMhyVT0dhZdemcqwPY5t8akU0YoWsOemklDC2VlIjcAOB3AM4AUAVgg1Lq6ijn+beS\nipxXym1a7ocdeK6VaFJJ9Nu9ndWHG+Zn7GbCPFw6Rbuw2NTOwXiVFLePN8hJHXgcvvOdZD843Txc\npZvfEvOuXuEJKyfbnI0fTR/u8y2vtIWTfTjs5RydTTQ69AhZCQOwhgerQ4YGTV13kJWUU9Kw4R/Z\nw63fqL0+XEXOCd34EXB+p2IO9+kSuQ6e4csN+VXoh/2RI8Dy5cAdd1gf9iYnLLcm13i8+De5UUXD\nRmSduydseDCdyzcxSTkhagcemx2c1toPucDcTn09kJkJ3H67dTxadcIP0vQJfGH43veA2lrg/fet\n9x/ge6xTrKRlV8LinFQacQ08s6R6geYppwAjR8ZvAXfDRaBuFWjBX7AA+OYb68uCUnyPdYvcQ6t5\n48fma7TS1TnISipJJm34R9El2/EWObezZAlw7JiVsGL9PrvqUherIv3mm+aqFoj/70BmODAwfOkm\nILkGDFZSKWAHntmifdAdOZLcc4R21O3caR275RYgKyt8uC/yterrgQz+P6jVolWkQ4cC771nJaW9\ne4GXX2Z3oxvkNi3dBDRfn7W8prnKSqURg5VUqGgb/rFSMlq0KgiIP6cUT7z5psBrXXopsHq19fjl\nlzfPm7CrLnmRFenpp1vvP9D8XmdnAytX8j12q8jOwWibPrJxIhYXbvhHJwv9oLv+eutDLV1NDZWV\nwB/+ADQ2AhMmWENQbJxovVjDpmxO8a5oTRhPXHwbh/uCwpYbamp24Bp42tn1oZSVdfJaeHZr29b6\nSedr+EG89QL9dqGtn0Q2YezqtSLu+b5IUmHzSrkj2YFnoNZ2zDm5MKruRVi9hqtrEBCy8WMM3hvu\ni7bhH+eVXKE1HXNODgtxCIooPe67Tzw+3OeTDf/oZE4OC3EIish57ktS0TrwcD3bwl2OQ2nUElay\n/mT+cN8ddwBgB57X8QOIWsIFdb0r3nCf8Ulq3tnPcA08IsM59SWDK314k7vnpDiMR2Q8rmdI6WJ+\nkiIi4wUWho23MG8iWlrxg/OW/tNG1wuLyK9FpEJEykVkmYicpisWonSprm5eDxCwbldX+zeOlgQq\nsspK62fp0uZlkgLXVfXqZf3wuip/0JakAPwdQJFSagCAbQA4rkfGS/bDPt6HrpP27AH++MfmOP74\nR+uYXckrtMoZP966Hfq8iQqtyBYvtm4HKqWcnJNXE+FwovdpS1JKqTeVUiea7q4FcJauWIgSlWzS\nifehm4zQZFJdDXz4YXMySSSxnHqq9d8FC6yfwDG7kiirHEoXnZVUqAkA/qY7CPKGdA5t2ZV0khWa\nTCoqrC0sKioSTyw9e1qbOR46ZP2MHGkds+vvsavKsasiI+9Ia+OEiLwJoGuUhyYrpUqazpkCoF4p\ntSTac5SUTAveLigoRmFhsf2BkqeY1Glm12R/ZGPCD34ArFpl3U6kSWHnTuDtt4FOnaz7b78NdO9u\nXtMB1/Pzh61bS7FtW2lC56Y1SSmlror3uIjcDeD7AK6Idc6YMdPsDYo8z65Os2iSTTqmfOjW1lr/\nnTDB+u+SJdYx0zrmuPSUPxQWhhccr732ZMxztbWgi8hoAI8BGKGUOqYrDqJkJJt07PrQDU0m+/YB\nr7wC3Hor0LVrYonlrLOsvbZErPPuuMNKUm3bmpFEiWLRtuKEiHwG4BQAh5oO/UMp9WDEOWrePHNX\nxCAzeXH5nNDrh6qrgS1bgPPOa75+KJHVHbz4vpA3uHtZJCYpSpKdS/ToWFMwna/JZYXIRPGSlCnd\nfUS2sfN6mnRe5xSrC9GUa6uITMBlkYjiSGcTRqwuxHS9pmlNEkSJYJIi0iSdCTAaUzoNiZLB4T7y\nvXgX/+q4uNSUC1rdst4feRuTFPmerkVNYyWjdL1msnNdnBsjE7C7jwh6ut50dA4m+3eyG5CcwO4+\nIgPpXNW7sdG6KDiAQ3lkKlc2Ttx3X9SE6zqsEs3gl663aKtWAOGrVoQmSb+8L2Q2VyYpADB5mDIR\nIt5ItF7gl6630L/z9NOBq64C/v53oE0b4MorT/6b/fK+kNlcm6SI7OKXRU0Df2OgY6+8HDh61Fq/\nb/VqoFu38PfBL+8LmY1zUkQ+EujY27sXqKsDamqs+SkiUzFJEflI4ALilSuB+nprf6lTTwUuv5xD\neWQmDvcR+VBjo7VVx+23W8N8y5adPNxHZAJWUkQ+EujYu+UWa0+pf/zD2mOKTRFkKiYpm82ZMwcD\nBw5EZmYmfvjDH+oOh3wgmeWLAh17RUXA4MHNySn0Gi0uh0QmYZKyWV5eHp544glMCOzTTZRmySxf\nlMgFxFwOiUziuzmpywcNQvWBA8H7Obm5WL1unW3Pf8MNNwAA1q9fj927d9v2vESx2L2autOrsxPF\n46kkFZmAgJOTUPWBA1h/5pnB+wMjzk/kORLh9ouNiYhMoCVJich0ANcBUAC+BnC3UmpXqs8bmYCA\nk5OQE88BcEUJco7dyxdxOSQyia45qV8ppQYopS4A8FcAv9QUR9qwkiKn2L21Rzq3JyFKlpZKSilV\nE3I3C8BXTr12Tm5uWGWUk5ubltdhJUVOsXv5Ii6HRCbRNiclIk8DuBPAtwCG2PGckQkocCyUnU0S\n0TQ2NuL48eNoaGhAY2Mj6urqkJGRgbZt26b1dYmIvChtSUpE3gTQNcpDk5VSJUqpKQCmiMi/AXgO\nQMoXFdmRgBJJdPFMnz4dTz31VPD+4sWLMW3aNPziF79IOTYiIr/RvjOviPQE8DelVL8oj6lrr22e\nriooKEZhYXFgF0cnw7SdiHA/KSLypa1bS7FtW2nw/muvPRlzZ15d3X35SqnPmu6OBbAh1rljxkxz\nJCYiInJGYaFVcAS89tqTMc/VNSc1U0QKATQC+D8AD2iKg4iIDKaru+8mHa9LRETuwrX7iIjIWExS\nRERkLCYpIiIyFpMUEREZi0mKiIiMxSRFRETGYpKyWX19PSZOnIizzz4bOTk5uPDCC7FixQrdYRER\nuZKvktQnn1h74wQsW2Yds1NDQwN69uyJsrIyVFdXY8aMGbjllltQWVlp7wsREfmAp5LUkiVAYG3Y\nujrg978H6uubH+/cGXjpJeDPf7Z+XnrJOhbq22/j329Jhw4d8Mtf/hI9m/Y3uOaaa3DOOefgn//8\nZ5J/DREReSpJNTQA990H7N4NPPoosGMHELpDRl4eMHcu8Oyz1s/cudaxgGPHgFtvBdaute4vWWI9\nTyr279/06vWQAAAJo0lEQVSPbdu2oaioKLUnIiLyIW37SaXDXXdZFdT11wMXXQQ89VR4kgKA998P\nv33zzc33MzOt33nsMaCoCPjiCyuRtdbx48dxxx134O6770ZBQUHrn4iIyKc8VUnV1QHl5dbt/fuB\nr78Of7y0FFi4EFi+3PpZuNA6FuqCC6wE9+67wIQJQNdoO2Il4MSJE7jzzjuRmZmJOXPmtO5JiIh8\nzlNJasoU4LTTrOG6ceOA+++3hvAChgwBXnzRGuLLy7NuD4nYE3jJEmDrVmDGDGD27Oahv2QopTBx\n4kQcPHgQS5cu5a68RESt5KnhvnvvBc491xriu+su4JJLrCG8gMzM8Moosko6dgz44ANriK9rV+vn\nL38BBg9OLo4HHngAW7ZswapVq9CuXbvW/0FERD6nfWfeeERERdu91uSdeSsrK3HOOecgMzMzrIJ6\n4YUXcNtttwXvc2deIiJL02e6OTvzelmvXr1w4sQJ3WEQEXmCp+akiExSXQ3s3Nl8f+dO6xgRJY5J\niihNDh8Gli4FKiutn6VLrWNElDgO9xGlSc+ewI03AosXW/fHj7eOEVHitFZSIvKoiJwQkU464yAi\nIjNpq6REpAeAqwBw5VXypJ07rUWMx4+37i9bZl2/x2qKKHE6K6lZAB7X+PpEadWxo5WUevWyfsaN\ns44RUeK0VFIiMhbAbqXUJyJRW+OJXC8nx/oJYAVFlLy0JSkReRNAtJXvpgCYBOB7oafHep6SkmnB\n2wUFxSgsLLYnQCIi0mLr1lJs21aa0LmOrzghIv0AvAUgsFPTWQD2ALhEKXUg4lzXrTiRKK44QURk\nibfihONzUkqpTUqpLkqpc5RS5wDYDeCiyATlZuPHj0e3bt2Qk5OD3r174+mnn9YdEhGRK5lwMa9j\n5YRSCgv+ewGuuOYKXHHNFVjw3wvSUpFNmjQJO3bsQHV1Nd544w3Mnj0bK1assP11iIi8TvvFvEqp\n3nY91+HDhzHj2Rn4ZPMn6JHXA1Mem4LevZuffumypVj0+iKMfXosAGDRjEXIyc7BTeNuCnuezZs3\nY+/evcjPz0evXr2SjiNyF96MjAzk5ua24i8iIvI3EyopWyil8NCjD+FA1gFc+/S1yB6SjYk/moiq\nqqrgOavXrMbQu4aiS+8u6NK7C4b+y1CsXrM67Hme+91zuO+x+zC3ZC5+MOEHeP1vr7cqngcffBCn\nnnoqioqKMHXqVFx00UUp/X1ERH7kmSR16NAhbNm+BaN+NApn9joTg28YjOwe2di4cWPwnOysbHyz\n75vg/W/2fYOc7OYe4S1btuDVN17FhBcnYNz0cbj1P27Fk88+ibq6uqTjef7551FbW4tVq1Zh6tSp\n+PDDD1P7A4mIfEj7cJ9d2rVrh8b6RhyrPYYOOR1wovEEar+pRfv27YPn3PvDe3HXvXehap9VXe0o\n3YFFLy4KPr5//36c2ftMtM+2fqdL7y5oc0obVFVVtWq4TkRQXFyMm2++GS+//DIuueSSFP9KIiJ/\n8UySysrKwu033Y4lP1uCgpEF2PPJHvQ+ozcuuOCC4Dnnnnsu/rTwT8EmhmcXPou8vLzg4/n5+dhX\nsQ97tu5BXmEeyleVI/uUbHTu3Dml2I4fP57ycxAR+ZFnkhQAPPqvj6JoRRE2V2zG8MuGY9y4cWG7\n4wJAXl4eJk6cGPX3u3fvjmemPoPJj01Gg2pAp+xOmDNrzknPEc/Bgwfx1ltvYcyYMcjMzMSqVavw\n5z//GatWrUrpbyMi8iNPJSkRwdVXX42rr7661c9x+eWX470R76GmpgY5OTlo0ya5aTsRwdy5c/HA\nAw9AKYWCggIsWrQIgwYNanVMRER+5akkZZe2bduiYytXAj3jjDNQWlpqb0BERD7lme4+IiLyHiYp\nIiIyFpMUEREZi0mKiIiMxSRFRETGYpIiIiJjubYFndvOExF5nyuTFHe0JSLyBw73OWTr1lLdIbgC\n36fE8H1KDN+nxJj8PjFJOWTbtlLdIbgC36fE8H1KDN+nxJj8PjFJERGRsZikiIjIWKKUuU0IImJu\ncEREZBulVNSWbaOTFBER+RuH+4iIyFhMUkREZCwmKSIiMhaTlMNE5FEROSEinXTHYioR+bWIVIhI\nuYgsE5HTdMdkChEZLSJbROQzEfm57nhMJCI9RORtEdksIptE5Ce6YzKZiLQVkQ0iUqI7lmiYpBwk\nIj0AXAWgUncshvs7gCKl1AAA2wBM0hyPEUSkLYA5AEYD+C6A20Skr96ojHQcwE+VUkUAhgD4Ed+n\nuB4G8CkAI7vomKScNQvA47qDMJ1S6k2l1Immu2sBnKUzHoNcAuBzpdQXSqnjAF4BMFZzTMZRSu1T\nSn3cdLsWQAWA7nqjMpOInAXg+wB+D8DIVbuZpBwiImMB7FZKfaI7FpeZAOBvuoMwRB6AXSH3dzcd\noxhE5GwAF8L6skMnew7AYwBOtHSiLq5cBd1UIvImgK5RHpoCa8jqe6GnOxKUoeK8V5OVUiVN50wB\nUK+UWuJocOYycjjGVCKSBeBVAA83VVQUQkSuBXBAKbVBRIp1xxMLk5SNlFJXRTsuIv0AnAOgvGkf\nrLMAfCQilyilDjgYojFivVcBInI3rGGIKxwJyB32AOgRcr8HrGqKIojIdwAsBbBYKfVX3fEY6lIA\n14nI9wFkAsgRkYVKqbs0xxWGK05oICI7AFyslDqkOxYTichoAP8BYIRS6ivd8ZhCRDIAbIWVuL8E\n8CGA25RSFVoDM4xY3wT/AOBrpdRPdcfjBiIyAsDPlFJjdMcSiXNSevCbQXyzAWQBeLOpNfZ53QGZ\nQCnVAODHAFbC6sb6HyaoqIYCGA9gZNP/fjY0ffGh+Iz8XGIlRURExmIlRURExmKSIiIiYzFJERGR\nsZikiIjIWExSRERkLCYpIiIyFpMUkUNEpLHpmp2NIvInEWnfdLyriLwiIp+LyHoReV1E8qP8/gIR\n2S8iG52PnkgPJiki53yrlLpQKXU+gHoA9zcd/wuA1UqpPkqpgbDWeewS5fdfgrVNB5FvcO0+Ij3W\nAOgvIiNhLaL7QuCBWCvlK6XWNK3qTeQbrKSIHNa0Bt/VAD4B0A/AR3ojIjIXkxSRc9qLyAYA62Dt\nzrxAczxExuNwH5FzjiqlLgw9ICKbAdykKR4i47GSItJIKbUaQDsRuSdwTET6i8hlGsMiMgaTFJFz\nYm05cAOAK5ta0DcBeBrA3siTRORlAO8DKBCRXSLyw/SFSmQGbtVBRETGYiVFRETGYpIiIiJjMUkR\nEZGxmKSIiMhYTFJERGQsJikiIjIWkxQRERnr/wNcfvn73uyhYwAAAABJRU5ErkJggg==\n", "text/plain": [ - "" + "" ] }, "metadata": {}, diff --git a/docs/errata.md b/docs/errata.md index 50e94f44..6da8f2f6 100644 --- a/docs/errata.md +++ b/docs/errata.md @@ -11,23 +11,24 @@ I would be happy if you just write me a short [mail](mailto:mail@sebastianraschk ## Donations -- Current amount for the next donation: $37.00 +- Current amount for the next donation: $46.00 - Amount donated to charity: $0.00 ## Leaderboard 1. Ryan S. ($16.00) -2. S.R. ($4.00) -3. Joseph Gordon ($3.00) -4. T.S. Jayram ($2.00) -5. Andrei R. ($2.00) -9. Ilmo S. ($2.00) -10. Edgar C. ($3.00) -5. Elias R. ($1.00) -6. Haitham H. Saleh ($1.00) -7. Muqueet M. ($1.00) -10. Renato R. ($1.00) -11. Michael L. ($1.00) +2. Christopher Galpin ($8.00) +3. S.R. ($4.00) +4. Joseph Gordon ($3.00) +5. Edgar C. ($3.00) +6. T.S. Jayram ($2.00) +7. Andrei R. ($2.00) +8. Ilmo S. ($3.00) +9. Elias R. ($1.00) +10. Haitham H. Saleh ($1.00) +11. Muqueet M. ($1.00) +12. Renato R. ($1.00) +13. Michael L. ($1.00) ... @@ -36,6 +37,15 @@ I would be happy if you just write me a short [mail](mailto:mail@sebastianraschk **I am really sorry about you seeing many typos up in the equations so far. Unfortunately, the layout team needed to retype the equations for compatibility reasons. There were a lot of typos introduced during this process, and I tried my very best to eliminate all of these by comparing the pre-finals against my draft. Cross-comparing 450 pages was a tedious task, and it appears that several typos slipped through, so if you see something that does not make quite sense to you, please let me know.** +### Current Errata: + +####[View PDF on GitHub](./errata_pdf/errata_2nd.pdf) + +#### [External View Link](http://sebastianraschka.com/pdf/books/pymle/errata_2nd.pdf) + +####[Download PDF](https://github.com/rasbt/python-machine-learning-book/raw/master/docs/errata_pdf/errata_2nd.pdf) + + **E-book Update (2015-10-20):** Good news! I just heard back from the publisher; all the typos and errors which are listed below will be fixed by next week. If you bought the book via Packt, you'd just need to re-download the book. I hope that the updates will also be reflected in the Amazon Kindle version. If not, please contact me, and I'll be happy to make an arrangement with the publisher so that you'll get the updated e-book. @@ -55,14 +65,14 @@ The easiest way may be to go to page viii, the second page of the **Preface**. I
-### Previous Errata: Sep 23 - Oct 25, 2015 +### Old Errata: Sep 23 - Oct 25, 2015 I converted the errata into a PDF document where I highlighted the errors and inserted the corrections as they appear in the book as many readers suggested. Please find the PDF version of the errata below. Thanks so much for your feedback so far, I truly appreciate it and will do my best to get this fixed as soon as possible! -####[View PDF on GitHub](./errata_pdf/errata_2015-10-23.pdf) +####[View PDF on GitHub](./errata_pdf/errata_1st.pdf) -#### [External View Link](http://sebastianraschka.com/pdf/books/pymle/errata_2015-10-23.pdf) +#### [External View Link](http://sebastianraschka.com/pdf/books/pymle/errata_1st.pdf) -####[Download PDF](https://github.com/rasbt/python-machine-learning-book/raw/master/docs/errata_pdf/errata_2015-10-23.pdf) +####[Download PDF](https://github.com/rasbt/python-machine-learning-book/raw/master/docs/errata_pdf/errata_1st.pdf) diff --git a/docs/errata_pdf/errata_1st.pdf b/docs/errata_pdf/errata_1st.pdf new file mode 100644 index 00000000..5a0c8bb6 Binary files /dev/null and b/docs/errata_pdf/errata_1st.pdf differ diff --git a/docs/errata_pdf/errata_2015-10-23.pdf b/docs/errata_pdf/errata_2015-10-23.pdf deleted file mode 100644 index e16f1bcc..00000000 Binary files a/docs/errata_pdf/errata_2015-10-23.pdf and /dev/null differ diff --git a/docs/errata_pdf/errata_2nd.pdf b/docs/errata_pdf/errata_2nd.pdf new file mode 100644 index 00000000..71d18f30 Binary files /dev/null and b/docs/errata_pdf/errata_2nd.pdf differ diff --git a/docs/feedback.md b/docs/feedback.md index 742a4924..bc42876e 100644 --- a/docs/feedback.md +++ b/docs/feedback.md @@ -81,8 +81,7 @@ your book is the best for me to use as a textbook. > An amazing book, really can't praise it enough. Presentation of technical subjects are explained very clearly. The author is very thorough in his writing making sure to fill in the details so you don't get left behind. It was a wonderful experience. Clearly written examples showing the theory and practice of machine learning. An invaluable tutorial. As a bonus, chapter on "Parallelizing Neural Network Training with Theano" was a great addition! A very exciting topic, author really went all out. Hands down the best Python Machine learning book available. - -NazHuz via [Amazon](http://www.amazon.com/gp/customer-reviews/RURYHN1G3SRMZ/ref=cm_cr_pr_rvw_ttl?ie=UTF8&ASIN=1783555130) +-- NazHuz via [Amazon](http://www.amazon.com/gp/customer-reviews/RURYHN1G3SRMZ/ref=cm_cr_pr_rvw_ttl?ie=UTF8&ASIN=1783555130)
@@ -91,3 +90,9 @@ I am reading your recent book on ML and I think this is on the best book I ever -- Michaël Lambé [![](./images/armand_g_tweet.png)](https://twitter.com/arm_gilles/status/658927560932401154) + + +
+ +This is the first Python Machine Learning book that actually made sense. Sebastian managed to combine theory (math behind the models, how to implement an algorithm from scratch) with practice (how to actually implement it using scikit-learn) in a way no book has done thus far. I was really impressed (and surprised) to see how 'easy' it is to implement the more simple algorithms from scratch [...] +-- C. Herther via [Amazon](http://www.amazon.com/gp/customer-reviews/R2I0D8HNIQODVA/ref=cm_cr_pr_rvw_ttl?ie=UTF8&ASIN=B00YSILNL0) diff --git a/faq/README.md b/faq/README.md index 7315235a..52bf4a01 100644 --- a/faq/README.md +++ b/faq/README.md @@ -17,11 +17,95 @@ Sebastian ## FAQ + + +### General Questions about Machine Learning and 'Data Science' + +- [Why do you and other people sometimes implement machine learning algorithms from scratch?](./implementing-from-scratch.md) +- [What learning path/discipline in data science I should focus on?](./data-science-career.md) +- [At what point should one start contributing to open source?](./open-source.md) +- [How important do you think having a mentor is to the learning process?](./mentor.md) +- [Where are the best online communities centered around data science/machine learning or python?](./ml-python-communities.md) + +### Questions about Machine Learning Concepts + +##### Tree models + +- [How does the random forest model work? How is it different from bagging and boosting in ensemble models?](./bagging-boosting-rf.md) +- [What are the disadvantages of using classic decision tree algorithm for large dataset?](./decision-tree-disadvantages.md) +- [Why are implementations of decision tree algorithms usually binary, and what are the advantages of the different impurity metrics?](./decision-tree-binary.md) + +##### About the Machine Learning Field + +- [What are the different fields of study in data mining?](./datamining-overview.md) +- [What are differences in research nature between the two fields: machine learning & data mining?](./datamining-vs-ml.md) +- [How do I know if the problem is solvable through machine learning?](./ml-solvable.md) +- [What are the origins of machine learning?](./ml-origins.md) +- [How was classification, as a learning machine, developed?](./classifier-history.md) +- [Which machine learning algorithms can be considered as among the best?](./best-ml-algo.md) +- [What are the broad categories of classifiers?](./classifier-categories.md) + +##### Model evaluation + +- [Is it always better to have the largest possible number of folds when performing cross validation?](./number-of-kfolds.md) +- [When training an SVM classifier, is it better to have a large or small number of support vectors?](./num-support-vectors.md) +- [How do I evaluate a model?](./evaluate-a-model.md) +- [What is the best validation metric for multi-class classification?](./multiclass-metric.md) +- [What factors should I consider when choosing a predictive model technique?](./choosing-technique.md) + +##### Logistic Regression + +- [What is the probabilistic interpretation of regularized logistic regression?](./probablistic-logistic-regression.md) +- [Does regularization in logistic regression always results in better fit and better generalization?](./regularized-logistic-regression-performance.md) +- [What is the major difference between naive Bayes and logistic regression?](./naive-bayes-vs-logistic-regression.md) +- [What exactly is the "softmax and the multinomial logistic loss" in the context of machine learning?](./softmax.md) + +##### Neural Networks + +- [What is the difference between deep learning and usual machine learning?](./difference-deep-and-normal-learning.md) +- [Can you give a visual explanation for the back propagation algorithm for neural networks?](./visual-backpropagation.md) +- [Why did it take so long for deep networks to be invented?](./inventing-deeplearning.md) +- [What are some good books/papers for learning deep learning?](./deep-learning-resources.md) +- [Why are there so many deep learning libraries?](./many-deeplearning-libs.md) +- [Why do some people hate neural networks/deep learning?](./deeplearning-criticism.md) + +##### Unsupervised Learning + +- [What are some of the issues with clustering?](./issues-with-clustering.md) + +##### Preprocessing + +- [What are the different dimensionality reduction methods in machine learning?](./dimensionality-reduction.md) +- [What is the difference between LDA and PCA for dimensionality reduction?](./lda-vs-pca.md) +- [When should I apply data normalization/standardization?](./when-to-standardize.md) +- [How do you attack a machine learning problem with a large number of features?](./large-num-features.md) + +##### Naive Bayes + +- [What is the decision boundary for Naive Bayes?](./naive-bayes-boundary.md) +- [Is it possible to mix different variable types in Naive Bayes, for example, binary and continues features?](./naive-bayes-vartypes.md) + +##### Other + +- [What is Euclidean distance in terms of machine learning?](./euclidean-distance.md) + + + + +
+
+
+ + + + +### Questions about the Book + - [Can I use paragraphs and images from the book in presentations or my blog?](./copyright.md) - [How is this different from other machine learning books?](./different.md) - [Which version of Python was used in the code examples?](./py2py3.md) - [Which technologies and libraries are being used?](./technologies.md) - [Which book version/format would you recommend?](./version.md) -- [Why did you choose Python for machine learning?](./why_python.md) -- [Why do you use so many leading and trailing underscores in the code examples?](./underscore_convention.md) -- [Are There Any Prerequisites and Recommended Pre-Readings?](./prerequisites.md) +- [Why did you choose Python for machine learning?](./why-python.md) +- [Why do you use so many leading and trailing underscores in the code examples?](./underscore-convention.md) +- [Are there any prerequisites and recommended pre-readings?](./prerequisites.md) diff --git a/faq/bagging-boosting-rf.md b/faq/bagging-boosting-rf.md new file mode 100644 index 00000000..ba429455 --- /dev/null +++ b/faq/bagging-boosting-rf.md @@ -0,0 +1,46 @@ +# How does the random forest model work? How is it different from bagging and boosting in ensemble models? + +Let's assume we use a decision tree algorithms as base classifier for all three: boosting, bagging, and (obviously :)) the random forest. + + +Why and when do we want to use any of these? Given a fixed-size number of training samples, our model will increasingly suffers from the "curse of dimensionality" if we increase the number of features. The challenge of individual, unpruned decision trees is that the hypothesis often ends up being too complex for the underlying training data -- decision trees are prone to overfitting. + + +**tl;dr: Bagging and random forests are "bagging" algorithms that aim to reduce the complexity of models that overfit the training data. In contrast, boosting is an approach to increase the complexity of models that suffer from high bias, that is, models that underfit the training data.** + + +## Bagging + + +Now, let's take a look at the probably "simplest" case, bagging. Here, we train a number (ensemble) of decision trees from bootstrap samples of your training set. Bootstrap sampling means drawing random samples from our training set with replacement. E.g., if our training set consists of 7 training samples, our bootstrap samples (here: n=7) can look as follows, where C1, C2, ... Cm shall symbolize the decision tree classifiers: + + +![](./bagging-boosting-rf/bagging.png) + + +After we trained your (m) decision trees, we can use them to classify new data via majority rule. For instance, we'd let each decision tree make a decision and predict the class label that received more votes. Typically, this would result in a less complex decision boundary, and the bagging classifier would have a lower variance (less overfitting) than an individual decision tree. Below is a plot comparing a single decision tree (left) to a bagging classifier (right) for 2 variables from the Wine dataset (Alcohol and Hue). + + +![](./bagging-boosting-rf/bagging-regions.png) + + +## Boosting + + +In contrast to bagging, you use very simple classifiers as base classifiers, so-called "weak learners." Picture these weak learners as "decision tree stumps" -- decision trees with only 1 splitting rule. Below, we will refer to the probably most popular example of boosting, AdaBoost. Here, we start with one decision tree stump (1) and "focus" on the samples it got wrong. In the next round, we train another decision tree stump that attempts to get these samples right (2); we achieve this by putting a larger weight on these training samples. Again, this 2nd classifier will likely get some other samples wrong, so you'd re-adjust the weights ... + + +![](./bagging-boosting-rf/boosting.png) + + +In a nutshell, we can summarize "Adaboost" as "adaptive" or "incremental" +learning from mistakes. Eventually, we will come up with a model that has a lower bias than an individual decision tree (thus, it is less likely to underfit the training data). + + +![](./bagging-boosting-rf/boosting-regions.png) + + +## Random forests + + +The random forest algorithm is actually a bagging algorithm: also here, we draw random bootstrap samples from your training set. However, in addition to the bootstrap samples, we also draw random subsets of features for training the individual trees; in bagging, we provide each tree with the full set of features. Due to the random feature selection, the trees are more independent of each other compared to regular bagging, which often results in better predictive performance (due to better variance-bias trade-offs), and I'd say that it's also faster than bagging, because each tree learns only from a subset of features. diff --git a/faq/bagging-boosting-rf/bagging-regions.png b/faq/bagging-boosting-rf/bagging-regions.png new file mode 100644 index 00000000..fe1acf31 Binary files /dev/null and b/faq/bagging-boosting-rf/bagging-regions.png differ diff --git a/faq/bagging-boosting-rf/bagging.png b/faq/bagging-boosting-rf/bagging.png new file mode 100644 index 00000000..49a1ea57 Binary files /dev/null and b/faq/bagging-boosting-rf/bagging.png differ diff --git a/faq/bagging-boosting-rf/boosting-regions.png b/faq/bagging-boosting-rf/boosting-regions.png new file mode 100644 index 00000000..93c825a5 Binary files /dev/null and b/faq/bagging-boosting-rf/boosting-regions.png differ diff --git a/faq/bagging-boosting-rf/boosting.png b/faq/bagging-boosting-rf/boosting.png new file mode 100644 index 00000000..7a4fbae5 Binary files /dev/null and b/faq/bagging-boosting-rf/boosting.png differ diff --git a/faq/best-ml-algo.md b/faq/best-ml-algo.md new file mode 100644 index 00000000..b1eb717c --- /dev/null +++ b/faq/best-ml-algo.md @@ -0,0 +1,17 @@ +# Which machine learning algorithms can be considered as among the best? + +I recommend taking a look at + +Wolpert, D.H., Macready, W.G. (1997), "[No Free Lunch Theorems for Optimization](http://ti.arc.nasa.gov/m/profile/dhw/papers/78.pdf)", IEEE Transactions on Evolutionary Computation 1, 67. + +Unfortunately, there's no real answer to this question: different datasets, questions, and assumptions require different algorithms -- or in other words: we haven't found the Master Algorithm, yet. + +But let me write down thoughts about different classifiers at least: + +- both logistic regression and SVMs work great for linear problems, logistic regression may be preferable for very noisy data +- naive Bayes may work better than logistic regression for small training set sizes; the former is also pretty fast, e.g., if you have a large multi-class problem, you'd only have to train one classifier whereas you'd have to use One-vs-Rest or One-vs-One with in SVMs or logistic regression (alternatively, you could implement multinomial/softmax regression though); another point is that you don't have to worry so much about hyperparameter optimization -- if you are estimating the class priors from the training set, there are actually no hyperparameters +- kernel SVM/logistic regression is preferable for nonlinear data vs. the linear models +- k-nearest neighbor can also work quite well in practice for datasets with large number of samples and relatively low dimensionality +- Random Forests & Extremely Randomized trees are very robust and work well across a whole range of problems -- linear and/or nonlinear problems + +Personally, I tend to prefer a multi-layer neural network in most cases, given that my dataset is sufficiently large. In my experience, the generalization performance was almost always superior to one of the other approaches I listed above. But again, it really depends on the given dataset. diff --git a/faq/choosing-technique.md b/faq/choosing-technique.md new file mode 100644 index 00000000..9deaf53b --- /dev/null +++ b/faq/choosing-technique.md @@ -0,0 +1,47 @@ +# What factors should I consider when choosing a predictive model technique? + +This is a very broad question, and the answer would basically fill an entire book. In a nutshell, I would come up with the + +### 1. How does your target variable look like? + +- continuous target variable? -> regression +- categorical (nominal) target variable? -> classification +- ordinal target variable? -> ranked classification +- no target variable and want to find structure in data? -> cluster analysis, projection + +### 2. Is computational performance an issue? + +- use "cheaper" models/algorithms +- dimensionality reduction +- feature selection +- lazy learner (e.g,. k-nearest neighbors) + +### 3. Does my dataset fit into memory? If no: + +- out of core learning +- distributed systems + +### 4. Is my data linearly separable? + +- hard to know the answer upfront +- always a good idea to compare different models + +### 5. Finding a good bias variance threshold. Does my model overfit? + +- increase regularization strength if supported by the model +- dimensionality reduction or feature selection otherwise +- collect more training data if possible (check via learning curves first) + +### 6. Are you planning to update your model with new data on the fly? + +- one option are lazy learners (e.g., K-nearest neighbors); needs to keep training data around; no learning necessary but more expensive predictions +- it's generally relatively cheap to update generative models +- another option is stochastic gradient descent for online learning + +... + +The list goes on and on :). I think Andreas Mueller's scikit-learn algorithm "cheat-sheet" is an excellent resource. (Click on the image to view the original, interactive version on scikit-learn) + +[![](./choosing-technique/scikit-cheatsheet.png)](http://scikit-learn.org/dev/tutorial/machine_learning_map/index.html) + +[Source: http://scikit-learn.org/dev/tutorial/machine_learning_map/index.html] diff --git a/faq/choosing-technique/scikit-cheatsheet.png b/faq/choosing-technique/scikit-cheatsheet.png new file mode 100644 index 00000000..b30a0c27 Binary files /dev/null and b/faq/choosing-technique/scikit-cheatsheet.png differ diff --git a/faq/classifier-categories.md b/faq/classifier-categories.md new file mode 100644 index 00000000..5a6efeed --- /dev/null +++ b/faq/classifier-categories.md @@ -0,0 +1,60 @@ +# What are the broad categories of classifiers? + + +### A (broad) categorization could be "discriminative" vs. "generative" classifiers: + + +Discriminative algorithms: +- a direct mapping of x -> y +- intuition: "Distinguishing between people who are speaking different languages without actually learning the language" +- e.g., Logistic regression, SVMs, Neural networks, ... + +Generative algorithms: +- model how the data was generated (joint probability distributions p(x, y)) +- e.g., naive Bayes, Bayesian belief networks, Restricted Boltzmann machines + + +### Or, we could categorize classifiers as "lazy" vs. "eager" learners: + +Lazy learners: +- don't "learn" a decision rule (or function) +- no learning step involved but require to keep training data around +- e.g., K-nearest neighbor classifiers + +### A third possibility could be "parametric" vs. "non-parametric" + +(in context of machine learning; the field of statistics interprets use terms a little bit differently.) + +non-parametric: +- representations grow with the training data size +- e.g., Decision trees, K-nearest neighbors + +parametric: +- representations are "fixed" +- e.g., most linear classifiers like logistic regression etc. + +### Pedro Domingo's 5 Tribes of Machine Learning + +In his new book ([The Master Algorithm](http://www.amazon.com/Master-Algorithm-Ultimate-Learning-Machine/dp/0465065708/ref=sr_1_1?ie=UTF8&qid=1447045562&sr=8-1&keywords=pedro+domingos)), Pedro Domingo's mentioned the 5 tribes of machine learning, which is another nice categorization. Summarizing from the book (pp. 51-53) + +**Symbolists** +- manipulating symbols (like mathematicians replace expressions by expressions), or in other words, using pre-existing knowledge to fill in the missing pieces +- "master algorithm:" inverse deduction + + +**Connectionists** +- reverse-engineering a biological brain, i.e., strengthening the connections between neurons +- "master algorithm:" backpropagation + + +**Evolutionaries** +- whereas connectionism is about fine-tuning the brain, evolution is about creating the brain +- "master algorithm:" genetic programming + +**Bayesians** +- based on probabilistic inference, i.e., incorporating a priori knowledge: certain outcomes are more likely +- "master algorithm:" Bayes' theorem and its derivatives + +**Analogizers** +- generalizing from similarity, i.e., recognizing similarities or in other words: remember experiences (training data) and how to combine them to make new predictions +- "master algorithm:" support vector machine diff --git a/faq/classifier-history.md b/faq/classifier-history.md new file mode 100644 index 00000000..f426a957 --- /dev/null +++ b/faq/classifier-history.md @@ -0,0 +1,46 @@ +# How was classification, as a learning machine, developed? + + +There are two fundamental milestones I'd say. +The first one is Fisher's Linear Discriminant [1], later generalized by Rao [2] to what we know as Linear Discriminant Analysis (LDA). Essentially, LDA is a linear transformation (or projection) technique, which is mainly used for dimensionality reduction (i.e., the objective is to find the k-dimensional feature subspace that -- linearly -- separates the samples from different classes best. +Given the objective to maximize class separability, projecting the 2D dataset below onto "x-axis component," would be a better choice than the "y-axis component." + + + +![](./classifier-history/lda.png) + +Keep in mind though that LDA is a projection technique; the feature axes of your new feature subspace are (almost certainly) different from your original axes. +In other words, LDA aims to find a new feature subspace that retains most of the class-discriminatory information. Where do I want to go with this? Intuitively, we can come up with a criterion function to minimize the ratio of the distance between the (class) sample means, and the within class scatter. And maximizing our criterion function plus plugging in a threshold function yields us a linear classifier. +Anyway, subjectively, I would group LDA with its closed-form solution into the more classical statistics field (or probabilistic learning if you will due to its relation to ANOVA and Bayes' Theorem). + + +Another timely take on classification would be the perceptron algorithm, which is in turn based on the concept of the McCulloch-Pitt (MCP) Neuron -- an early (maybe first?) model of how a neuron in a mammal's brain could work [3]. In contrast to the LDA classifier, Rosenblatt's perceptron [4] is an incremental learner. For each training sample, it compares the predicted class labes to the actual class label and modify the model weights accordingly. +Rosenblatt's initial perceptron rule is fairly simple and can be summarized by the following steps: + +1. Initialize the weights to 0 or small random numbers. +2. For each training sample x(i): +- Calculate the output value. +- Update the weights. +The value for updating the weights at each increment is calculated by the learning rule + +![](./classifier-history/perceptron-rule.png) + +![](./classifier-history/perceptron-figure.png) + +There were few problems with this approach, i.e., the algorithm will never converge if the data is not perfectly separable by a line or hyperplane. +An improvement over the perceptron was the adaptive linear neuron (Adaline) [5]. It is closely related to linear regression (if you use an optimization algorithm like gradient descent rather than the closed-form solution); we update the weights by comparing the real-value output to the actual class label. (Remember, the perceptron algorithms compares the binary-values class labels, i.e., the predicted and actual class labels.) After we trained the model, we use a threshold function to turn the real-valued output into a class label: + +![](./classifier-history/adaline.png) + +A very similar concept is Logistic Regression, however, instead of a linear function, we minimize a logistic function [6]. Again, it wouldn't say the early beginning of logistic regression would be necessarily the "machine learning" approach until incremental learning (gradient descent, stochastic gradient descent, and other optimization algorithms were being used to learn the model weights). In any case, scientists came up with many other cost functions until then, I'd say they are likely inspired by the perceptron, Adaline, and logistic regression. +![](./classifier-history/activation-functions.png) + +For example, the hinge loss for SVM. Another direction of the analogizer's SVM approach would be the "connectionism," i.e., combining neuron units to multi-layer neural networks. Although scientists combined Adaline units to multiple Adalines (Madaline), the problem with Adaline was that a combination of linear units is ... well, it's still linear. In any case, there is so much to write about, but I hope that satisfies your curiosity towards the early beginning to some extend. + + +[1] Fisher, R. A. (1936). "The Use of Multiple Measurements in Taxonomic Problems". Annals of Eugenics 7 (2): 179–188. doi:10.1111/j.1469-1809.1936.tb02137.x +[2] Rao, R. C. (1948). "The utilization of multiple measurements in problems of biological classification". Journal of the Royal Statistical Society, Series B 10 (2): 159–203. +[3] McCulloch, W. and Pitts, W. (1943). A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biophysics, 5:115–133. +[4] F. Rosenblatt. The perceptron, a perceiving and recognizing automaton Project Para. Cornell Aeronautical Laboratory, 1957. +[5] B. Widrow et al. Adaptive ”Adaline” neuron using chemical ”memistors”. Number Technical Report 1553-2. Stanford Electron. Labs., Stanford, CA, October 1960. +[6] Berkson, Joseph. "Application of the logistic function to bio-assay." Journal of the American Statistical Association 39.227 (1944): 357-365. diff --git a/faq/classifier-history/activation-functions.png b/faq/classifier-history/activation-functions.png new file mode 100644 index 00000000..d15bc32f Binary files /dev/null and b/faq/classifier-history/activation-functions.png differ diff --git a/faq/classifier-history/adaline.png b/faq/classifier-history/adaline.png new file mode 100644 index 00000000..9c4d18de Binary files /dev/null and b/faq/classifier-history/adaline.png differ diff --git a/faq/classifier-history/lda.png b/faq/classifier-history/lda.png new file mode 100644 index 00000000..437e0645 Binary files /dev/null and b/faq/classifier-history/lda.png differ diff --git a/faq/classifier-history/perceptron-figure.png b/faq/classifier-history/perceptron-figure.png new file mode 100644 index 00000000..db586f6b Binary files /dev/null and b/faq/classifier-history/perceptron-figure.png differ diff --git a/faq/classifier-history/perceptron-rule.png b/faq/classifier-history/perceptron-rule.png new file mode 100644 index 00000000..fc59dde5 Binary files /dev/null and b/faq/classifier-history/perceptron-rule.png differ diff --git a/faq/data-science-career.md b/faq/data-science-career.md new file mode 100644 index 00000000..609ae201 --- /dev/null +++ b/faq/data-science-career.md @@ -0,0 +1,7 @@ +# What learning path/discipline in data science I should focus on? + +The tl;dr: "Data science" is a broad field including many different specializations; which particular sub-role do you find most appealing? Statistics, machine learning, software engineering, data visualization...? + +Hm, how can I say this … it’s a pity that our day only has 24 hours, and there is only so much that we can learn as an individual person. So, I think it is somewhat important to reflect on your goals and your interest when you are picking your study topics. Although it has a “bad ring” to it, the phrase “being a jack of all trades” is certainly very tempting and useful if you work as an individual. As a “data scientist” there are endless topics you can get lost in, from programming to statistics, databases, differential calculus, … Of course, it’s useful to know a bit of everything, but I think it is impossible to become a master in everything in a given amount of time. For example, data scientists rarely work on their own but are often part of bigger teams with different areas of responsibilities. Some people are really good at stats; some people are responsible for building the framework to collect, clean, and extract data. Some people develop new algorithms, and some are “data scientist-programmers” who focus on implementing them most efficiently. For example, I’d say that I am a pretty good Python programmer, but my Java skills are really rudimentary. I think a better knowledge of Java would help me here and there, but my focus is more on the Machine Learning part; Python is already sufficient for me to implement all my ideas. So rather than investing in learning a brand new programming language, I try to focus more on my strength, e.g., picking up fresh ML concepts and staying on top with the current developments in the field. Of course, I’d like to learn a new programming language since it could be useful, but I also know that I only have so much time to learn all these things … ;) + +What I am trying to say is that it is probably not a bad idea to think about where you want to be in xx years, and what does it take to get there. Which are the things and skills that are necessary to get you there? I would focus on these first! diff --git a/faq/datamining-overview.md b/faq/datamining-overview.md new file mode 100644 index 00000000..b85e4954 --- /dev/null +++ b/faq/datamining-overview.md @@ -0,0 +1,21 @@ +# What are the different fields of study in data mining? + + +I would roughly define the different application areas as + +1) Clustering (unsupervised learning) +e.g., to find groups of customers based on some similarity + +2) Predictive modeling (supervised learning) +2.1) Classification + e.g., medical diagnosis (sick/healthy), image classification etc. +2.2) Regression + e.g., stock trade change prediction +2.3) Ranking + e.g., search engine results + +3) Association rule mining +e.g., which products do customers frequently buy together + +4) Anomaly detection +e.g., credit fraud detection diff --git a/faq/datamining-vs-ml.md b/faq/datamining-vs-ml.md new file mode 100644 index 00000000..927983be --- /dev/null +++ b/faq/datamining-vs-ml.md @@ -0,0 +1,4 @@ +# What are differences in research nature between the two fields: Machine Learning & Data Mining? + +In a nutshell, Data Mining is about the discovery of patterns in datasets or "gaining knowledge and insights" from data. Machine Learning is closely related though. We can think of Machine Learning algorithms as one of he work horses of Data Mining; most Data Mining approaches are based on Machine Learning algorithms. Maybe it helps to think of Data Mining as a pipeline of steps and approaches, and the use of a Machine Learning algorithm is one part of this pipeline. +Or in other words, Data Mining is not "just" Machine Learning. E.g., data visualization or summarization is also part of Data Mining. What I was trying to say is that Machine Learning is one part, one set of techniques, that is/are being used in Data Mining. diff --git a/faq/decision-tree-binary.md b/faq/decision-tree-binary.md new file mode 100644 index 00000000..88b083a4 --- /dev/null +++ b/faq/decision-tree-binary.md @@ -0,0 +1,60 @@ +# Why are implementations of decision tree algorithms usually binary and what are the advantages of the different impurity metrics? + +For practical reasons (combinatorial explosion) most libraries implement decision trees with binary splits. The nice thing is that they are NP-complete (Hyafil, Laurent, and Ronald L. Rivest. "Constructing optimal binary decision trees is NP-complete." Information Processing Letters 5.1 (1976): 15-17.) + +Our objective function (e.g., in CART) is to maximize the information gain (IG) at each split: + + +![](./decision-tree-binary/information-gain.png) + + +where *f* is the feature to perform the split, and *D_p* and *D_j* are the datasets of the parent and *j*th child node, respectively. *I* is the impurity measure. *N* is the total number of samples, and *N_j* is the number of samples at the *j*th child node. +Now, let's take a look at the most commonly used splitting criteria for classification (as described in CART). For simplicity, I will write the equations for the binary split, but of course it can be generalized for multiway splits. So, for a binary split we can compute *IG* as + +![](./decision-tree-binary/information-gain-2.png) + +Now, the two impurity measures or splitting criteria that are commonly used in binary decision trees are Gini Impurity(*I_G*) and Entropy (*I_H*) and the Classification Error (*I_E*). Let us start with the definition of Entropy, which is defined as + +![](./decision-tree-binary/entropy.png) + +for all “non-empty” classes + +![](./decision-tree-binary/empty-classes.png) + +and *p(i|t)* is the proportion of the samples that belong to class *c* for a particular node *t*. The entropy is therefore 0 if all samples at a node belong to the same class, and the Entropy is maximal if we have an uniform class distribution +Intuitively, the Gini Impurity can be understood as a criterion to minimize the probability of misclassification + +![](./decision-tree-binary/gini-impurity.png) + +Similar to the Entropy, the Gini Impurity is maximal if the classes are perfectly mixed. +However, in practice both Gini Impurity and Entropy typically yield very similar results and it is often not worth spending much time on evaluating trees using different impurity criteria rather than experimenting with different pruning cut-offs. +Another impurity measure is the Classification Error + +![](./decision-tree-binary/error.png) + +which is a useful criterion for pruning but not recommend for growing a decision tree since it is less sensitive to changes in the class probabilities of the nodes. + +![](./decision-tree-binary/overview-plot.png) + +So let me illustrate what I mean by "the classification error is less sensitive to changes in the class probabilities" by looking at the two possible splitting scenarios shown in the figure below. + +![](./decision-tree-binary/split.png) + +We start with a data set *D_p* at the parent node that consists 40 samples from class 1 and 40 samples from class 2 that we split into two datasets D_left and D_right, respectively. The information gain using the Classification Error as splitting criterion would be the same (*IG_E* = 0.25) in both scenario *A* and *B*: + +![](./decision-tree-binary/calc_1.png) + +![](./decision-tree-binary/calc_2.png) + +However, the Gini Impurity would favor the split in scenario B (0.1666) over scenario A (0.125), which is indeed more “pure”: + +![](./decision-tree-binary/calc_3.png) + +Similarly, the entropy criterion would favor scenario B(IGH = 0.31) over scenario A(IGH = 0.19): + + +![](./decision-tree-binary/calc_5.png) + +![](./decision-tree-binary/calc_6.png) + +Maybe some more words about Gini vs. Entropy. As mentioned before, the resulting trees are typically very similar in practice. Maybe an advantage of Gini would be that you don't need to compute the log, which can make it a bit faster in your implementation. diff --git a/faq/decision-tree-binary/calc_1.png b/faq/decision-tree-binary/calc_1.png new file mode 100644 index 00000000..d0a44062 Binary files /dev/null and b/faq/decision-tree-binary/calc_1.png differ diff --git a/faq/decision-tree-binary/calc_2.png b/faq/decision-tree-binary/calc_2.png new file mode 100644 index 00000000..d0a44062 Binary files /dev/null and b/faq/decision-tree-binary/calc_2.png differ diff --git a/faq/decision-tree-binary/calc_3.png b/faq/decision-tree-binary/calc_3.png new file mode 100644 index 00000000..f123fe81 Binary files /dev/null and b/faq/decision-tree-binary/calc_3.png differ diff --git a/faq/decision-tree-binary/calc_5.png b/faq/decision-tree-binary/calc_5.png new file mode 100644 index 00000000..e3c07cec Binary files /dev/null and b/faq/decision-tree-binary/calc_5.png differ diff --git a/faq/decision-tree-binary/calc_6.png b/faq/decision-tree-binary/calc_6.png new file mode 100644 index 00000000..b1780de2 Binary files /dev/null and b/faq/decision-tree-binary/calc_6.png differ diff --git a/faq/decision-tree-binary/empty-classes.png b/faq/decision-tree-binary/empty-classes.png new file mode 100644 index 00000000..72d814c1 Binary files /dev/null and b/faq/decision-tree-binary/empty-classes.png differ diff --git a/faq/decision-tree-binary/entropy.png b/faq/decision-tree-binary/entropy.png new file mode 100644 index 00000000..81c9adea Binary files /dev/null and b/faq/decision-tree-binary/entropy.png differ diff --git a/faq/decision-tree-binary/error.png b/faq/decision-tree-binary/error.png new file mode 100644 index 00000000..49f2fce2 Binary files /dev/null and b/faq/decision-tree-binary/error.png differ diff --git a/faq/decision-tree-binary/gini-impurity.png b/faq/decision-tree-binary/gini-impurity.png new file mode 100644 index 00000000..806e1487 Binary files /dev/null and b/faq/decision-tree-binary/gini-impurity.png differ diff --git a/faq/decision-tree-binary/information-gain-2.png b/faq/decision-tree-binary/information-gain-2.png new file mode 100644 index 00000000..e89625de Binary files /dev/null and b/faq/decision-tree-binary/information-gain-2.png differ diff --git a/faq/decision-tree-binary/information-gain.png b/faq/decision-tree-binary/information-gain.png new file mode 100644 index 00000000..8d1a7615 Binary files /dev/null and b/faq/decision-tree-binary/information-gain.png differ diff --git a/faq/decision-tree-binary/overview-plot.png b/faq/decision-tree-binary/overview-plot.png new file mode 100644 index 00000000..57558d3a Binary files /dev/null and b/faq/decision-tree-binary/overview-plot.png differ diff --git a/faq/decision-tree-binary/split.png b/faq/decision-tree-binary/split.png new file mode 100644 index 00000000..c5360374 Binary files /dev/null and b/faq/decision-tree-binary/split.png differ diff --git a/faq/decision-tree-disadvantages.md b/faq/decision-tree-disadvantages.md new file mode 100644 index 00000000..713eaf0c --- /dev/null +++ b/faq/decision-tree-disadvantages.md @@ -0,0 +1,19 @@ +# What are the disadvantages of using classic decision tree algorithm for large dataset? + + + +### The computational efficiency perspective + +It's a combinatorial search problem: at each split, we want to find the features that give us "the best bang for the buck" (maximizing information gain). If we choose a"brute" force approach, our computational complexity is O(m^2), where m is the number of features in our training set, and O(n^2) for the number of n training cases (I think it can be O(n log(n) if you are lucky). + +Let's take a look at a simple dataset, Iris (150 flowers, 3 classes, 4 continuous features). At each split, we have to re-evaluate all 4 features, and for each feature we have to find the optimal value to split on, e.g,. sepal length <3.4 cm (this is for a binary split). Computational complexity is one of the reasons why people implement *binary* decision trees most of the time. + +### The predictive performance perspective + +An unpruned model is much more likely to overfit as a consequence of the curse of dimensionality. However, instead of pruning a single decision tree, it often a better idea to use ensemble methods. We could + +- combine decision tree stumps that learn from each other by focusing on samples that are hard to classify (AdaBoost) +- create an ensemble of unpruned decision trees; draw bootstrap samples, and do random feature selection (random forests) +- forget about bagging and use all training samples as input for your unpruned trees; choose both the splitting feature and splitting value at random (= Extremely randomized trees) + +(Related topic: [How does the random forest model work? How is it different from bagging and boosting in ensemble models?](../bagging-boosting-rf.md)) diff --git a/faq/deep-learning-resources.md b/faq/deep-learning-resources.md new file mode 100644 index 00000000..4069d1f8 --- /dev/null +++ b/faq/deep-learning-resources.md @@ -0,0 +1,11 @@ +# What are some good books/papers for learning deep learning? + + +A good overview and introduction is the recent deep learning review published in Nature (http://www.nature.com/nature/journal/v521/n7553/full/nature14539.html); it references a lot of useful literature to follow up on + +- LeCun, Yann, Yoshua Bengio, and Geoffrey Hinton. "Deep learning." Nature521.7553 (2015): 436-444. + + +As a good textbook resource, I would like to recommend Yoshua Bengio's upcoming "Deep Learning" book. The book is freely accessible at: http://goodfeli.github.io/dlbook/ + +- Bengio, Yoshua, Ian Goodfellow, and Aaron Courville. "Deep learning." An MIT Press book in preparation. Draft chapters available at http://www. iro. umontreal. ca/∼ bengioy/dlbook (2014). diff --git a/faq/deeplearning-criticism.md b/faq/deeplearning-criticism.md new file mode 100644 index 00000000..a9a14bf5 --- /dev/null +++ b/faq/deeplearning-criticism.md @@ -0,0 +1,10 @@ +# Why do some people hate neural networks/deep learning? + +I also know many people who make disrespectful remarks about neural networks in genaral. Personally, I find recurrent and convolutional neural networks truly beautiful. However, there is this popular saying: "that if al that you have is a hammer everything starts to look like a nail" + +The math behind neural nets is probably a bit harder to understand, but I don't think they are really black boxes. I think a neural net is not more of a black box than standard techniques like kernel SVMs of random forests. Actually, I think it is easier to explain backpropagation than kernel methods. + +However, I think people in the biosciences prefer "interpretable" results, e.g., decision trees where they can follow the "reasoning" step by step. Unarguably, random forests are better at solving the prediction task since you don't have to worry so much about overfitting or pruning your tree; at the same time, we lose some of this "interpretability." Although, I think this is not really true. Feature importance computed from e.g., extremely randomized trees might be even more useful than looking at a single decision tree. + +Please note that I don't blame the bio-research field for this kind of thinking, they really try to solve different problems. +Assuming biologists want to know which functional groups of a ligand are "interacting" with residues in the protein binding site. Of course, the primary goal is often to a good agonist or antagonist (inhibitor or drug) in a million-compound database to solve a particular problem; in addition, they are also trying to "understand" and "explain" the results. diff --git a/faq/difference-deep-and-normal-learning.md b/faq/difference-deep-and-normal-learning.md new file mode 100644 index 00000000..4bed42f1 --- /dev/null +++ b/faq/difference-deep-and-normal-learning.md @@ -0,0 +1,38 @@ +# What is the difference between deep learning and usual machine learning? + +That's an interesting question, and I try to answer this is a very general way. +The tl;dr version of this is: Deep learning is essentially a set of techniques that help we to parameterize deep neural network structures, neural networks with many, many layers and parameters. + + +And if we are interested, a more concrete example: +Let's start with multi-layer perceptrons (MLPs) ... + + +On a tangent: The term "perceptron" in MLPs may be a bit confusing since we don't really want only linear neurons in our network. Using MLPs, we want to learn complex functions to solve non-linear problems. Thus, our network is conventionally composed of one or multiple "hidden" layers that connect the input and output layer. Those hidden layers normally have some sort of sigmoid activation function (log-sigmoid or the hyperbolic tangent etc.). For example, think of a log-sigmoid unit in our network as a logistic regression unit that returns continuous values outputs in the range 0-1. A simple MLP could look like this + +![](./difference-deep-and-normal-learning/mlp.png) + + +where y_hat is the final class label that we return as the prediction based on the inputs (x) if this are classification tasks. The "a"s are our activated neurons and the "w"s are the weight coefficients. +Now, if we add multiple hidden layers to this MLP, we'd also call the network "deep." The problem with such "deep" networks is that it becomes tougher and tougher to learn "good" weights for this network. When we start training our network, we typically assign random values as initial weights, which can be terribly off from the "optimal" solution we want to find. During training, we then use the popular backpropagation algorithm (think of it as reverse-mode auto-differentiation) to propagate the "errors" from right to left and calculate the partial derivatives with respect to each weight to take a step into the opposite direction of the cost (or "error") gradient. **Now, the problem with deep neural networks is the so-called "vanishing gradient" -- the more layers we add, the harder it becomes to "update" our weights because the signal becomes weaker and weaker. Since our network's weights can be terribly off in the beginning (random initialization) it can become almost impossible to parameterize a "deep" neural network with backpropagation.** + +**Deep Learning** + +Now, this is where "deep learning" comes into play. Roughly speaking, we can think of deep learning as "clever" tricks or algorithms that can help we with the training of such "deep" neural network structures. There are many, many different neural network architectures, but to continue with the example of the MLP, let me introduce the idea of convolutional neural networks (ConvNets). We can think of those as an "add-on" to our MLP that helps we to detect features as "good" inputs for our MLP. + +In applications of "usual" machine learning, there is typically a strong focus on the feature engineering part; the model learned by an algorithm can only be so good as its input data. Of course, there must be sufficient discriminatory information in our dataset, however, the performance of machine learning algorithms can suffer substantially when the information is buried in meaningless features. The goal behind deep learning is to automatically learn the features from (somewhat) noisy data; it's about algorithms that do the feature engineering for us to provide deep neural network structures with meaningful information so that it can learn more effectively. **We can think of deep learning as algorithms for automatic "feature engineering," or we could simply call them "feature detectors," which help us to overcome the vanishing gradient challenge and facilitate the learning in neural networks with many layers.** + + + +Let's consider a ConvNet in context of image classification. +Here, we use so-called "receptive fields" (think of them as "windows") that slide over our image. We then connect those "receptive fields" (for example of the size of 5x5 pixel) with 1 unit in the next layer, this is the so-called "feature map." After this mapping, we have constructed a so-called convolutional layer. Note that our feature detectors are basically replicates of one another -- they share the same weights. The idea is that if a feature detector is useful in one part of the imagine it is likely that it is useful somewhere else, but at the same time it allows each patch of image to be represented in several ways. + +![](./difference-deep-and-normal-learning/convolution.png) + + +Next, we have a "pooling" layer, where we reduce neighboring features from our feature map into single units (by taking the max feature or by averaging them, for example). We do this over many rounds and eventually arrive at an almost scale invariant representation of our image (the exact term is "equivariant"). This is very powerful since we can detect objects in an image no matter where they are located. + + +![](./difference-deep-and-normal-learning/convnet.png) + +In essence, the "convolutional" add-on that acts as a feature extractor or filter to our MLP. Via the convolutional layers we aim to extract the useful features from the images, and via the pooling layers, we aim to make the features somewhat equivariant to scale and translation. diff --git a/faq/difference-deep-and-normal-learning/convnet.png b/faq/difference-deep-and-normal-learning/convnet.png new file mode 100644 index 00000000..45cfe16f Binary files /dev/null and b/faq/difference-deep-and-normal-learning/convnet.png differ diff --git a/faq/difference-deep-and-normal-learning/convolution.png b/faq/difference-deep-and-normal-learning/convolution.png new file mode 100644 index 00000000..f357abb4 Binary files /dev/null and b/faq/difference-deep-and-normal-learning/convolution.png differ diff --git a/faq/difference-deep-and-normal-learning/mlp.png b/faq/difference-deep-and-normal-learning/mlp.png new file mode 100644 index 00000000..97b79c84 Binary files /dev/null and b/faq/difference-deep-and-normal-learning/mlp.png differ diff --git a/faq/dimensionality-reduction.md b/faq/dimensionality-reduction.md new file mode 100644 index 00000000..763e9abc --- /dev/null +++ b/faq/dimensionality-reduction.md @@ -0,0 +1,31 @@ +# What are the different dimensionality reduction methods in machine learning? + +Since there are so many different approaches, let's break it down to "feature selection" and "feature extraction." + +Some examples of feature selection: + +- L1 regularization (e.g., Logistic regression) and sparsity +- variance thresholds +- recursive feature elimination based on the weights of linear models +- random forests / extra trees and feature importance (calculated as average information gain) +- sequential forward/backward selection +- genetic algorithms +- exhaustive search + +Some examples of feature extraction: + +- Principal Component Analysis (PCA), unsupervised, returns axes of maximal variance given the constraint that those axes are orthogonal to each other +- Linear Discriminant Analysis (LDA; not to be confused with Latent Dirichlett Allocation), supervised, returns axes that maximizes class separability (same constraint that axes are also orthogonal); and another article: Linear Discriminant Analysis bit by bit +- kernel PCA: uses kernel trick to transform non-linear data to a feature space were samples may be linearly separable (in contrast, LDA and PCA are linear transformation techniques +- supervised PCA +- and many more non-linear transformation techniques, which you can find nicely summarized here: [Nonlinear dimensionality reduction](https://en.wikipedia.org/wiki/Nonlinear_dimensionality_reduction) + +** So, which technique should we use? ** + +This also follows the "No Lunch Theorem" principle in some sense: there is no method that is always superior; it depends on your dataset. Intuitively, LDA would make more sense than PCA if you have a linear classification task, but empirical studies showed that it is not always the case. Although kernel PCA can separate concentric circles, it fails to unfold the Swiss Rroll, for example; here, locally linear embedding (LLE) would be more appropriate. + +![](./dimensionality-reduction/swiss-roll.png) + +![](./dimensionality-reduction/rbf-kpca.png) + +![](./dimensionality-reduction/lle.png) diff --git a/faq/dimensionality-reduction/lle.png b/faq/dimensionality-reduction/lle.png new file mode 100644 index 00000000..390d4c5b Binary files /dev/null and b/faq/dimensionality-reduction/lle.png differ diff --git a/faq/dimensionality-reduction/rbf-kpca.png b/faq/dimensionality-reduction/rbf-kpca.png new file mode 100644 index 00000000..8c998571 Binary files /dev/null and b/faq/dimensionality-reduction/rbf-kpca.png differ diff --git a/faq/dimensionality-reduction/swiss-roll.png b/faq/dimensionality-reduction/swiss-roll.png new file mode 100644 index 00000000..17e6ca65 Binary files /dev/null and b/faq/dimensionality-reduction/swiss-roll.png differ diff --git a/faq/euclidean-distance.md b/faq/euclidean-distance.md new file mode 100644 index 00000000..1631fc47 --- /dev/null +++ b/faq/euclidean-distance.md @@ -0,0 +1,12 @@ +# What is Euclidean distance in terms of machine learning? + + +It is just a distance measure between a pair of samples *p* and *q* in an *n*-dimensional feature space: + +![](./euclidean-distance/eucl-1.png) + +For example, picture it as a "straight, connecting" line in a 2D feature space: + +![](./euclidean-distance/eucl-2.png) + +The Euclidean is often the "default" distance used in e.g., K-nearest neighbors (classification) or K-means (clustering) to find the "k closest points" of a particular sample point. Another prominent example is hierarchical clustering, agglomerative clustering (complete and single linkage) where you want to find the distance between clusters. diff --git a/faq/euclidean-distance/eucl-1.png b/faq/euclidean-distance/eucl-1.png new file mode 100644 index 00000000..5cfab3d0 Binary files /dev/null and b/faq/euclidean-distance/eucl-1.png differ diff --git a/faq/euclidean-distance/eucl-2.png b/faq/euclidean-distance/eucl-2.png new file mode 100644 index 00000000..50177da5 Binary files /dev/null and b/faq/euclidean-distance/eucl-2.png differ diff --git a/faq/evaluate-a-model.md b/faq/evaluate-a-model.md new file mode 100644 index 00000000..1d7d634b --- /dev/null +++ b/faq/evaluate-a-model.md @@ -0,0 +1,50 @@ +# How do I evaluate a model? + +The short answer is to keep an independent test set for your final model -- this has to be data that your model hasn't seen before. + + +However, it all depends on your goal & approach. + + +### Scenario 1: + + +- Just train a simple model. + + +Split the dataset into a separate test and training set. Train the model on the former, evaluate the model on the latter (by "evaluate" I mean calculating performance metrics such as the error, precision, recall, ROC auc, etc.) + + +### Scenario 2: + + +- Train a model and tune (optimize) its hyperparameters. + + +Split the dataset into a separate test and training set. Use techniques such as k-fold cross-validation on the training set to find the "optimal" set of hyperparameters for your model. If you are done with hyperparameter tuning, use the independent test set to get an unbiased estimate of its performance. +Below I inserted a figure to illustrate the difference: + +![](./evaluate-a-model/evaluate_overview.png) + + +The first row refers to "Scenario 1", and the 3rd row describes a more "classic" approach where you further split your training data into a training subset and a validation set. Then, you train your model on the training subset and evaluate in on the validation set to optimize its hyperparameters, for example. Eventually, you test it on the independent test set. The fourth row describes the "superior" (more unbiased) approach using k-fold cross-validation as described in "Scenario 2." + + +Also, let me attach an overview of k-fold cross validation in case you are not familiar with it, yet: + + +![](./evaluate-a-model/k-fold.png) + + +(Here: E = prediction error, but you can also substitute it by precision, recall, f1-score, ROC auc or whatever metric you prefer for the given task.) + +### Scenario 3: + + +- Build different models and compare different algorithms (e.g., SVM vs. logistic regression vs. Random Forests, etc.). + + +Here, we'd want to use nested cross-validation. In nested cross-validation, we have an outer k-fold cross-validation loop to split the data into training and test folds, and an inner loop is used to select the model via k-fold cross-validation on the training fold. After model selection, the test fold is then used to evaluate the model performance. After we have identified our "favorite" algorithm, we can follow-up with a "regular" k-fold cross-validation approach (on the complete training set) to find its "optimal" hyperparameters and evaluate it on the independent test set. Let's consider a logistic regression model to make this clearer: +Using nested cross-validation you will train *m* different logistic regression models, 1 for each of the *m* outer folds, and the inner folds are used to optimize the hyperparameters of each model (e.g., using gridsearch in combination with k-fold cross-validation. If your model is stable, these *m* models should all have the same hyperparameter values, and you report the average performance of this model based on the outer test folds. Then, you proceed with the next algorithm, e.g., an SVM etc. + +![](./evaluate-a-model/nested-k-fold.png) diff --git a/faq/evaluate-a-model/evaluate_overview.png b/faq/evaluate-a-model/evaluate_overview.png new file mode 100644 index 00000000..72eb03c8 Binary files /dev/null and b/faq/evaluate-a-model/evaluate_overview.png differ diff --git a/faq/evaluate-a-model/k-fold.png b/faq/evaluate-a-model/k-fold.png new file mode 100644 index 00000000..6db9f929 Binary files /dev/null and b/faq/evaluate-a-model/k-fold.png differ diff --git a/faq/evaluate-a-model/nested-k-fold.png b/faq/evaluate-a-model/nested-k-fold.png new file mode 100644 index 00000000..13067369 Binary files /dev/null and b/faq/evaluate-a-model/nested-k-fold.png differ diff --git a/faq/implementing-from-scratch.md b/faq/implementing-from-scratch.md new file mode 100644 index 00000000..a146d00b --- /dev/null +++ b/faq/implementing-from-scratch.md @@ -0,0 +1,18 @@ +# Why do you and other people sometimes implement machine learning algorithms from scratch? +There are several different reasons why implementing algorithms from scratch can be useful: + +1. it can help us to understand the inner works of an algorithm +2. we could try to implement an algorithm more efficiently +3. we can add new features to an algorithm or experiment with different variations of the core idea +4. we circumvent licensing issues (e.g., Linux vs. Unix) or platform restrictions +5. we want to invent new algorithms or implement algorithms no one has implemented/shared yet +6. we are not satisfied with the API and/or we want to integrate it more "naturally" into an existing software library + +Let us narrow down the phrase "implementing from scratch" a bit further in context of the 6 points I mentioned above. When we talk about "implementing from scratch," we need to narrow down the scope to make this question really tangible. Let's talk about a particular algorithm, simple logistic regression, to address the different points using concrete examples. I'd claim that logistic regression has been implemented more than thousand times. + +One reason why we'd still want to implement logistic regression from scratch could be that we don't have the impression that we fully understand how it works; we read a bunch of papers, and kind of understood the core concept though. Using a programming language for prototyping (e.g., Pyhon, MATLAB, R, and so forth), we could take the ideas from paper and try to express them in code -- step by step. An established library, such as scikit-learn, can help us than double-check the results and to see if our implementation -- our idea of how the algorithm is supposed to work -- is correct. Here, we don't really care about efficiency; although we spend so much time to implement the algorithm, we probably want to use an established library if we want to perform some serious analysis in our research lab and/or company. Established libraries are typically more trustworthy -- they have been battle-tested by many people, people who may have already encountered certain edge cases and made sure that there are no weird surprises. Furthermore, it is also more likely that this code was highly optimized for computational efficiency over time. Here, implementing from scratch simply serves the purpose of self-assessment. Reading about a concept is one thing, but putting it to action is a +whole other level of understanding -- and being able to explain it to others is the icing on the cake. + +Another reason why we want to re-implement logistic regression from scratch may be that we are not satisfied with the "features" of other implementations. Let's us naively assume that other implementations don't have regularization parameters, or it doesn't support multi-class settings (i.e., via One-vs-All, One-vs-One, or softmax). Or if computational (or predictive) efficiency is an issue, maybe we want to implement it with another solver (e.g., Newton vs. Gradient Descent vs. Stochastic Gradient Descent, etc.). But improvements concerning computational efficiency does not necessarily need to be in terms of modifications of the algorithms, but we could use lower-level programming languages, for example, Scala instead of Python, or Fortran instead of Scala, ... this can go all down to assembly or machine code, or designing a chip that is optimized for running such kind of analysis. However, if you are a machine learning (or "data science") practitioner or researcher, this is probably something you should delegate to the software engineering team. + +To come back to the main question: Different people implement algorithms from scratch for various reasons. Personally, when I implement algorithms from scratch, I do it because of the learning experience. diff --git a/faq/inventing-deeplearning.md b/faq/inventing-deeplearning.md new file mode 100644 index 00000000..ac16452a --- /dev/null +++ b/faq/inventing-deeplearning.md @@ -0,0 +1,14 @@ +# Why did it take so long for deep networks to be invented? + +It's not that "deep networks" haven't been around in the 1960s, but the problem was how to train them. In the 1970s, backpropagation was "invented" or re-discovered -- I don't want to quote a single resource here not to offend any of the parties involved since this is a sensitive topic those days ... In any case, the problem was the "vanishing gradient," when gradient-based methods were used for learning the weights. It was observed that there was no gain going beyond 1-3 hidden layers. + + +So back to the question: Deep network structures existed, but it was hard/impossible to train them appropriately. I'd say the 2 main reasons why this field experienced such a leap in the recent years +are + + +1. availability of computing resources +2. clever ideas to pre-train a neural network + + +The second point is what deep learning is all about; in a nutshell, we pre-train our deep neural networks using unsupervised learning, but this goes beyond the scope of the question ... diff --git a/faq/issues-with-clustering.md b/faq/issues-with-clustering.md new file mode 100644 index 00000000..f0b6038e --- /dev/null +++ b/faq/issues-with-clustering.md @@ -0,0 +1,29 @@ +# What are some of the issues with Clustering? + +I wouldn't necessarily call most of them "issues" but rather "challenges". For example, *k*-means: + + +- The different results via *k*-means with distinct random initializations are definitely a problem. However, we could use *k*-means++ as an alternative, and if it's computationally feasible, we want to run your algorithm multiple times with different seeds and pick the one with e.g., lowest within cluster SSE (sum of squared errors) + + +- The number of clusters is (typically) not known a priori (that's basically the characteristic of unsupervised learning problems), but there are a few "performance" or "evaluation metrics one can use to infer a "satisfying" grouping against the value of K; this is also called the elbow method: +![](./issues-with-clustering/elbow.png) + + +Here, it seems that k=3 would be a good pick. Let's have a look at the accompanying 2D dataset that I used to train the *k*-means algorithm and see if our intuition agrees: + +![](./issues-with-clustering/clusters_unlabeled.png) + +![](./issues-with-clustering/clusters_kmeans.png) + +I'd say k=3 is definitely a reasonable pick. However, note that the "elbow" is typically not as clear as shown above. Moreover, note that in practice we normally work with higher-dimensional datasets so that we can't simply plot our data and double-check visually. (We could use unsupervised dimensionality reduction techniques though such as PCA). In fact, if we already knew that the 3 clusters belong to three different groups, this would be a classification task. + + +Anyway, there are other useful evaluation metrics such as the silhouette coefficient, which gives us some idea of the cluster sizes and shapes. Using the same dataset, let me give you a "good" silhouette plot (with k=3) and a not so decent one (k=2) + + +![](./issues-with-clustering/silhouette_good.png) + +![](./issues-with-clustering/silhouette_bad.png) + +I would say that the biggest "shortcoming" in *k*-means may be that we assume that the groups come in spherical or globular shapes, which is rarely the case with "real-world" data. In contrast, I could think of choosing the "optimal" *k* as just another hyperparameter optimization procedure, which is also necessary for almost every supervised learning algorithm. diff --git a/faq/issues-with-clustering/clusters_kmeans.png b/faq/issues-with-clustering/clusters_kmeans.png new file mode 100644 index 00000000..d38253d4 Binary files /dev/null and b/faq/issues-with-clustering/clusters_kmeans.png differ diff --git a/faq/issues-with-clustering/clusters_unlabeled.png b/faq/issues-with-clustering/clusters_unlabeled.png new file mode 100644 index 00000000..fb5c446e Binary files /dev/null and b/faq/issues-with-clustering/clusters_unlabeled.png differ diff --git a/faq/issues-with-clustering/elbow.png b/faq/issues-with-clustering/elbow.png new file mode 100644 index 00000000..6666a35f Binary files /dev/null and b/faq/issues-with-clustering/elbow.png differ diff --git a/faq/issues-with-clustering/silhouette_bad.png b/faq/issues-with-clustering/silhouette_bad.png new file mode 100644 index 00000000..f575c967 Binary files /dev/null and b/faq/issues-with-clustering/silhouette_bad.png differ diff --git a/faq/issues-with-clustering/silhouette_good.png b/faq/issues-with-clustering/silhouette_good.png new file mode 100644 index 00000000..681a11b7 Binary files /dev/null and b/faq/issues-with-clustering/silhouette_good.png differ diff --git a/faq/large-num-features.md b/faq/large-num-features.md new file mode 100644 index 00000000..de44ce00 --- /dev/null +++ b/faq/large-num-features.md @@ -0,0 +1,46 @@ +# How do you attack a machine learning problem with a large number of features? + + +There are 3 main strategies to reduce the number of features if necessary to avoid overfitting (due to the curse of dimensionality) and/or reduce the computational complexity (i.e., increase the computational efficiency). + + +**1) Regularization and Sparsity** + +- If supported by the model, I would recommend L1 or ElasticNet regularization to zero-out some features. + + +**2) Feature Selection** + +- We could try various different feature selection algorithms (e.g., selecting by variance or by greedy search: sequential backward/forward selection, genetic algorithms, etc.) + + +**3) Feature Extraction** + +- We could compress your feature space via transformation onto a lower-dimensional subspace. One popular example would be Principal Component Analysis. But we have to keep in mind that PCA is a linear transformation technique, which may be problematic in non-linear problems. For example, let's consider a simple "concentric circles" dataset: + + +![](./large-num-features/concentric-circles.png) + +Let's assume the blue samples belong to one class, and the red circles belong to a second class. Our goal is to train a model for classification. Furthermore, we assume that this dataset has too many dimensions (okay, we only have 2 features here, but we need to keep it "simple" for visualization purposes). Now, we want to compress the data onto a lower-dimensional subspace, here: 1 dimension. +Let's start with "standard" PCA. Can you spot the problem? + + +![](./large-num-features/pca-pc1.png) + + +The two classes are not separable anymore... +Let's use kernel PCA: + +![](./large-num-features/kpca-pc1.png) + +This is much better; we can now train a linear classifier to separate those two classes. However, the problem is that we introduce an additional hyperparameter (gamma) that needs to be tuned. Also, this "kernel trick" does not work for any dataset, and there are also many more manifold learning techniques that are "more powerful"/appropriate than kernel PCA. +For example, locally linear embedding (LLE) to unfold the famous Swiss Roll: + + +![](./large-num-features/swiss-roll.png) + + +![](./large-num-features/lle1.png) + + +![](./large-num-features/lle2.png) diff --git a/faq/large-num-features/concentric-circles.png b/faq/large-num-features/concentric-circles.png new file mode 100644 index 00000000..46a52e44 Binary files /dev/null and b/faq/large-num-features/concentric-circles.png differ diff --git a/faq/large-num-features/kpca-pc1.png b/faq/large-num-features/kpca-pc1.png new file mode 100644 index 00000000..fcc02c94 Binary files /dev/null and b/faq/large-num-features/kpca-pc1.png differ diff --git a/faq/large-num-features/lle1.png b/faq/large-num-features/lle1.png new file mode 100644 index 00000000..390d4c5b Binary files /dev/null and b/faq/large-num-features/lle1.png differ diff --git a/faq/large-num-features/lle2.png b/faq/large-num-features/lle2.png new file mode 100644 index 00000000..f9aacfc9 Binary files /dev/null and b/faq/large-num-features/lle2.png differ diff --git a/faq/large-num-features/pca-pc1.png b/faq/large-num-features/pca-pc1.png new file mode 100644 index 00000000..1bbc97cd Binary files /dev/null and b/faq/large-num-features/pca-pc1.png differ diff --git a/faq/large-num-features/swiss-roll.png b/faq/large-num-features/swiss-roll.png new file mode 100644 index 00000000..17e6ca65 Binary files /dev/null and b/faq/large-num-features/swiss-roll.png differ diff --git a/faq/lda-vs-pca.md b/faq/lda-vs-pca.md new file mode 100644 index 00000000..e9578cc0 --- /dev/null +++ b/faq/lda-vs-pca.md @@ -0,0 +1,15 @@ +# What is the difference between LDA and PCA for dimensionality reduction? + +Both LDA and PCA are linear transformation techniques: LDA is a supervised whereas PCA is unsupervised -- PCA ignores class labels. + +We can picture PCA as a technique that finds the directions of maximal variance: + +![](./lda-vs-pca/pca.png) + +In contrast to PCA, LDA attempts to find a feature subspace that maximizes class separability (note that LD 2 would be a very bad linear discriminant in the figure above). + +![](./lda-vs-pca/lda.png) + + +Remember that LDA makes assumptions about normally distributed classes and equal class covariances. +If you are interested in an empirical comparison: A. M. Martinez and A. C. Kak. PCA versus LDA. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 23(2):228–233, 2001). (PCA tends to result in better classification results in an image recognition task if the number of samples for a given class was relatively small.) diff --git a/faq/lda-vs-pca/lda.png b/faq/lda-vs-pca/lda.png new file mode 100644 index 00000000..c877d131 Binary files /dev/null and b/faq/lda-vs-pca/lda.png differ diff --git a/faq/lda-vs-pca/pca.png b/faq/lda-vs-pca/pca.png new file mode 100644 index 00000000..ee15d38d Binary files /dev/null and b/faq/lda-vs-pca/pca.png differ diff --git a/faq/many-deeplearning-libs.md b/faq/many-deeplearning-libs.md new file mode 100644 index 00000000..21100cf7 --- /dev/null +++ b/faq/many-deeplearning-libs.md @@ -0,0 +1,14 @@ +# Why are there so many deep learning libraries? + +In my opinion, the main reason is that everything in the Deep Learning field is still highly experimental. The libraries are usually a side-product of someone's own research. In addition, deep learning algorithms are not so general (in terms of writing a general code that can be applied to many different problems) in comparison to other algorithms, e.g., Random forests, logistic regression, SVMs, etc. Thus, everyone has his or her particular idea how a good interface may look like; hence, they may end up developing a new library. Also, they probably want to incorporate their personal research in the respective library and get credit for it -- it's much easier to have your personal library that you can tweak and change as you wish. + +However, I also think that it may only seem that there are so many libraries because it is a truly trendy topic at the moment, and we are living in the time and age where open-source and code sharing is (fortunately) very popular. I guess there are 100x more libraries that implement SVMs, logistic regression etc. than deep learning libraries. + + +Anyway, if you are interested in implementing neural networks yourself, have a look at [Theano](http://deeplearning.net/software/theano/) -- NumPy on steroids as how it is commonly called. Not only does it allow you to use numerical expressions more efficiently, but they also implement tensors and let you utilize GPUs. Theano is actually what most of the "many deep learning libraries" in Python are using, e.g,. + +- [Lasagne](https://github.com/Lasagne/Lasagne) +- [Keras](http://keras.io) +- [PyLearn 2](https://github.com/lisa-lab/pylearn2) + +... diff --git a/faq/mentor.md b/faq/mentor.md new file mode 100644 index 00000000..cb0e2dd9 --- /dev/null +++ b/faq/mentor.md @@ -0,0 +1,3 @@ +# How important do you think having a mentor is to the learning process? + +I guess it really depends on what type of learner you are. From what I understand, a mentor is like a "personal trainer," someone who gives you homework, tells you what books to read, etc. (in contrast to a teacher or professor who teaches a whole class at once). I think it is important to have friends who share similar interests to bounce of ideas, and I think it is important to know someone more knowledgeable than you who you could ask questions for your personal growth (e.g., a Prof who supervises your Ph.D. thesis would be such a person). However, I am not sure how important a "mentor" is for technical questions -- I think that's the 1-person equivalent of an answer poll via cross-validated or Quora. Anyways, I think a mentor may be a nice thing to have for certain people, but I wouldn't say everyone needs a dedicated "mentor." diff --git a/faq/ml-origins.md b/faq/ml-origins.md new file mode 100644 index 00000000..74c79248 --- /dev/null +++ b/faq/ml-origins.md @@ -0,0 +1,11 @@ +# What are the origins of machine learning? + +I think that it all started with the McCulloch-Pitt (MCP) Neuron, a first model of how a neuron in a mammal's brain could work: +W. S. McCulloch and W. Pitts. [A logical calculus of the ideas immanent in nervous activity](http://link.springer.com/article/10.1007/BF02459570). The bulletin of mathematical biophysics, 5(4):115–133, 1943. + +Note that other methods like linear regression were already invented (F. Galton. [Regression towards mediocrity in hereditary stature](http://www.jstor.org/stable/2841583). Journal of the Anthropological Institute of Great Britain and Ireland, pages 246–263, 1886.). Here, I would like to make the distinction between ML and statistics in terms of how ML evolved. I see ML as a field that emerged from artificial intelligence research, hence, the MCP neuron. +However, ML is deeply intertwined with statistics. For example, I would describe a linear regression analysis based on the closed-form solution (normal equation) primarily as a technique that came from the statistics field, and I would associate linear regression with stochastic gradient descent learning as an ML technique. I think the early goal in ML was how the algorithm can "learn" a function by itself rather than solving an equation mathematically. + +So, I would say that the first ML algorithm really was the perceptron (F. Rosenblatt. The perceptron, a perceiving and recognizing automaton Project Para. Cornell Aeronautical Laboratory, 1957.) What followed was the gradient descent algorithm used in adaptive linear neurons (B. Widrow. Adaptive ”Adaline” neuron using chemical ”memistors”. Number Technical Report 1553-2. Stanford Electron. Labs., Stanford, CA, October 1960.). Those single learning units were then connected to multi-layer architectures, and what followed was the multi-layer perceptron also around the first half of the 20th century. + +Note that I would put, for example, Fisher's Linear Discriminant Analysis (R. A. Fisher. [The use of multiple measurements in taxonomic problems](http://onlinelibrary.wiley.com/store/10.1111/j.1469-1809.1936.tb02137.x/asset/j.1469-1809.1936.tb02137.x.pdf;jsessionid=F0E7EFF219B2C2D94FF2B2981D3533E8.f04t03?v=1&t=igrkwkvk&s=631d3f737becda820356e6862bffc239e9b1f2d6). Annals of eugenics, 7(2):179–188, 1936) into developments in the statistics department -- note that this was before the term ML was coined. diff --git a/faq/ml-python-communities.md b/faq/ml-python-communities.md new file mode 100644 index 00000000..0994504d --- /dev/null +++ b/faq/ml-python-communities.md @@ -0,0 +1,3 @@ +# Where are the best online communities centered around data science/machine learning or python? + +I think [Twitter](https://twitter.com/rasbt) is great to stay in touch and see what other people are up to, and to share news of course. [Cross-validated](http://stats.stackexchange.com) & [Quora](https://www.quora.com/topic/Machine-Learning) are good websites to ask specific questions and collect multiple opinions about a topic. There's also a [machine learning group on Google+](https://plus.google.com/communities/107785538899595981479) that has a lot of members, but is not that active anymore I fear. The [MachineLearning subreddit](https://www.reddit.com/r/MachineLearning/) is another nice place to discuss certain news and articles. diff --git a/faq/ml-solvable.md b/faq/ml-solvable.md new file mode 100644 index 00000000..7f9f59fc --- /dev/null +++ b/faq/ml-solvable.md @@ -0,0 +1,32 @@ +# How do I know if the problem is solvable through machine learning? + + +In general, most ML algorithm assume that our training samples are i.i.d. Since we want your training set to be representative of the population, we need to divide it into training and test sets randomly. + +Also, we want to use our test set only once; we don't want retrain your model and evaluate it on the random test set over and over again, or our estimate will be hugely overoptimistic otherwise. We should use k-fold cross-validation or nested cross-validation instead. + +There are many different reasons why our performance may be not satisfactory: + +- our data is skewed +- there is a lot of noise +- there are many outliers +- our features are not informative enough +- we don't have enough training samples + +In brief: our algorithm suffers from high variance (overfitting) or high bias (underfitting). + + +![](./ml-solvable/bias-variance.png) + + +It may help to get a better grasp of our problem by plotting "learning curves" + +For example, here I plotted the average accuracies of a model (using 10-fold cross validation). The blue line (training accuracy) shows the average accuracy on the training folds and the green line shows the average accuracy on the test fold for different sizes of the initial training set. + + +![](./ml-solvable/learning_curve.png) + + +Similarly, we can evaluate a the model performance for a particular tuning parameter (here, I plotted different values for C, the inverse regularization parameter of an SVM). + +![](./ml-solvable/param_curve.png) diff --git a/faq/ml-solvable/bias-variance.png b/faq/ml-solvable/bias-variance.png new file mode 100644 index 00000000..f6db4e89 Binary files /dev/null and b/faq/ml-solvable/bias-variance.png differ diff --git a/faq/ml-solvable/learning_curve.png b/faq/ml-solvable/learning_curve.png new file mode 100644 index 00000000..8a2726d0 Binary files /dev/null and b/faq/ml-solvable/learning_curve.png differ diff --git a/faq/ml-solvable/param_curve.png b/faq/ml-solvable/param_curve.png new file mode 100644 index 00000000..c91bb729 Binary files /dev/null and b/faq/ml-solvable/param_curve.png differ diff --git a/faq/multiclass-metric.md b/faq/multiclass-metric.md new file mode 100644 index 00000000..0d8568f9 --- /dev/null +++ b/faq/multiclass-metric.md @@ -0,0 +1,22 @@ +# What is the best validation metric for multi-class classification? + +It really depends on our "goal" and our dataset. Classification Accuracy (or misclassification error) makes sense if our class labels are uniformly distributed. Even better, we can compute the ROC area under the curve (even for multi-class sytems), e.g., have a look at the nice [ICML'04 tutorial](http://www.cs.bris.ac.uk/~flach/ICML04tutorial/) on ROC analysis. +Similarly, we can generalize all the binary performance metrics such as precision, recall, and F1-score etc. to multi-class settings. In the binary case, we have + + +![](./multiclass-metric/conf_mat.png) + + +![](./multiclass-metric/pre-rec.png) + + +![](./multiclass-metric/mcc.png) + +(PRE=precision, REC=recall, F1=F1-Score, MCC=Matthew's Correlation Coefficient) +And to generalize this to multi-class, assuming we have a One-vs-All (OvA) classifier, we can either go with the "micro" average or the "macro" average. In "micro averaging," we'd calculate the performance, e.g., precision, from the individual true positives, true negatives, false positives, and false negatives of the the k-class model: + +![](./multiclass-metric/micro.png) + +And in macro-averaging, we average the performances of each individual class: + +![](./multiclass-metric/macro.png) diff --git a/faq/multiclass-metric/conf_mat.png b/faq/multiclass-metric/conf_mat.png new file mode 100644 index 00000000..fe3fdb12 Binary files /dev/null and b/faq/multiclass-metric/conf_mat.png differ diff --git a/faq/multiclass-metric/macro.png b/faq/multiclass-metric/macro.png new file mode 100644 index 00000000..4b457058 Binary files /dev/null and b/faq/multiclass-metric/macro.png differ diff --git a/faq/multiclass-metric/mcc.png b/faq/multiclass-metric/mcc.png new file mode 100644 index 00000000..a5f9cc13 Binary files /dev/null and b/faq/multiclass-metric/mcc.png differ diff --git a/faq/multiclass-metric/micro.png b/faq/multiclass-metric/micro.png new file mode 100644 index 00000000..e0cb446f Binary files /dev/null and b/faq/multiclass-metric/micro.png differ diff --git a/faq/multiclass-metric/pre-rec.png b/faq/multiclass-metric/pre-rec.png new file mode 100644 index 00000000..b65beda6 Binary files /dev/null and b/faq/multiclass-metric/pre-rec.png differ diff --git a/faq/naive-bayes-boundary.md b/faq/naive-bayes-boundary.md new file mode 100644 index 00000000..71767752 --- /dev/null +++ b/faq/naive-bayes-boundary.md @@ -0,0 +1,13 @@ +# What is the decision boundary for Naive Bayes? + + +It's a (piecewise) quadratic decision boundary for the Gaussian model. The multinomial model has a linear boundary. +Below, I plotted some examples: + +1) UCI Wine Dataset + +![](./naive-bayes-boundary/gaussian_1.png) + +2) An XOR toy dataset + +![](./naive-bayes-boundary/gaussian_2.png) diff --git a/faq/naive-bayes-boundary/gaussian_1.png b/faq/naive-bayes-boundary/gaussian_1.png new file mode 100644 index 00000000..cf731187 Binary files /dev/null and b/faq/naive-bayes-boundary/gaussian_1.png differ diff --git a/faq/naive-bayes-boundary/gaussian_2.png b/faq/naive-bayes-boundary/gaussian_2.png new file mode 100644 index 00000000..ceaaae68 Binary files /dev/null and b/faq/naive-bayes-boundary/gaussian_2.png differ diff --git a/faq/naive-bayes-vartypes.md b/faq/naive-bayes-vartypes.md new file mode 100644 index 00000000..c64faab5 --- /dev/null +++ b/faq/naive-bayes-vartypes.md @@ -0,0 +1,66 @@ +# Is it possible to mix different variable types in Naive Bayes, for example, binary and continues features? + + +Yes, this is definitely possible. + + +Let's briefly recapitulate the concept behind Naive Bayes: Our objective function is to maximize the posterior probability given the training data: + + +![](./naive-bayes-vartypes/bayes-theorem-in-words.png) + + +![](./naive-bayes-vartypes/bayes-theorem.png) + +Let + +![](./naive-bayes-vartypes/bayes-theorem-notation.png) + + +And based on the objective function, we can formulate the decision rule as: + + +![](./naive-bayes-vartypes/naive-bayes-decision.png) + + +In the context of this question, let's not worry about the priors for now; these are typically computed via Maximum Likelihood Estimation (MLE), for instance, the class frequency Nωj/N (number of samples in class ωj divided by the number of all samples in the training set). + +A short note about the evidence term: I wrote it for completeness, but we can simple cancel it from the decision function, because it is a constant term for all classes. +Now, the class-conditional probabilities, let's call them likelihoods, are computed as the product of the likelihoods of the individual features *d*: + + +![](./naive-bayes-vartypes/naive-bayes-likelihood.png) + + +Here, we make the "naive" conditional independence assumption, which states that features are independent of each other -- that's how Naive Bayes got its name. What we are basically saying is "The probability of observing this combination of features is equal to the product of observing each feature separately." + +Another assumption that we make is that *p(xi = b | ωj )* is drawn from a particular distribution -- that's why Naive Bayes is also called a "generative model." +To come back to the original question, let us consider the multi-variate Bernoulli model for binary features and the Gaussian Naive Bayes model for continuous features. + + +#### The (Multi-variate) Bernoulli Model + + +We use the Bernoulli distribution to compute the likelihood of a binary variable. +For example, we could estimate P(xk=1 | ωj) via MLE as the frequency of occurences in the training set: +θ = P̂(xk=1 | ωj) = Nxk, ωj / N ωj +which reads "number of training samples in class ωj that have the property xk=1 (Nxk, ωj) divided by by all training samples in ωj (N ωj)." In context of text classification, this is basically the set of documents in class ωj that contain a particular word divided by all documents in ωj. +Now, we can compute the likelihood of the binary feature vector **x** given class ωj as + +![](./naive-bayes-vartypes/likelihood-bernoulli.png) + +#### The Gaussian Model + +Typically, we use the Gaussian Naive Bayes model for variables on a continuous scale -- assuming that our variables are normal distributed. + +![](./naive-bayes-vartypes/gaussian-likelihood.png) + +In the equation above, we have 2 parameters we need to estimate, the mean μ of the samples associated with class ωj and the variance σ2 associated with class ωj, respectively. This should be straight-forward, so let's skip the details here. After we plugged the estimated parameters into the equation, we can compute the likelihood of a continuous feature vector as (similar to the Bernoulli model above). + +![](./naive-bayes-vartypes/naive-bayes-likelihood_shorter.png) + +Since we have the conditional independence assumption in Naive Bayes, we see that mixing variables is not a problem. We can compute the likelihoods of binary variables via Bernoulli Bayes and compute the likelihoods of the continuous variables via the Gaussian model. To compute the class-conditional probability of a sample, we can simply form the product of the likelihoods from the different feature subsets: + +![](./naive-bayes-vartypes/combined.png) + +(The same concept applies to Multinomial Naive Bayes and other models.) diff --git a/faq/naive-bayes-vartypes/bayes-theorem-in-words.png b/faq/naive-bayes-vartypes/bayes-theorem-in-words.png new file mode 100644 index 00000000..d66ac1f7 Binary files /dev/null and b/faq/naive-bayes-vartypes/bayes-theorem-in-words.png differ diff --git a/faq/naive-bayes-vartypes/bayes-theorem-notation.png b/faq/naive-bayes-vartypes/bayes-theorem-notation.png new file mode 100644 index 00000000..d3d45351 Binary files /dev/null and b/faq/naive-bayes-vartypes/bayes-theorem-notation.png differ diff --git a/faq/naive-bayes-vartypes/bayes-theorem.png b/faq/naive-bayes-vartypes/bayes-theorem.png new file mode 100644 index 00000000..8e7dad8d Binary files /dev/null and b/faq/naive-bayes-vartypes/bayes-theorem.png differ diff --git a/faq/naive-bayes-vartypes/combined.png b/faq/naive-bayes-vartypes/combined.png new file mode 100644 index 00000000..c0d19550 Binary files /dev/null and b/faq/naive-bayes-vartypes/combined.png differ diff --git a/faq/naive-bayes-vartypes/gaussian-likelihood.png b/faq/naive-bayes-vartypes/gaussian-likelihood.png new file mode 100644 index 00000000..9ba7244b Binary files /dev/null and b/faq/naive-bayes-vartypes/gaussian-likelihood.png differ diff --git a/faq/naive-bayes-vartypes/likelihood-bernoulli.png b/faq/naive-bayes-vartypes/likelihood-bernoulli.png new file mode 100644 index 00000000..b7c58fa9 Binary files /dev/null and b/faq/naive-bayes-vartypes/likelihood-bernoulli.png differ diff --git a/faq/naive-bayes-vartypes/multi-variate-bernoulli.png b/faq/naive-bayes-vartypes/multi-variate-bernoulli.png new file mode 100644 index 00000000..cf219265 Binary files /dev/null and b/faq/naive-bayes-vartypes/multi-variate-bernoulli.png differ diff --git a/faq/naive-bayes-vartypes/naive-bayes-decision.png b/faq/naive-bayes-vartypes/naive-bayes-decision.png new file mode 100644 index 00000000..6040e3b5 Binary files /dev/null and b/faq/naive-bayes-vartypes/naive-bayes-decision.png differ diff --git a/faq/naive-bayes-vartypes/naive-bayes-likelihood.png b/faq/naive-bayes-vartypes/naive-bayes-likelihood.png new file mode 100644 index 00000000..6e533e05 Binary files /dev/null and b/faq/naive-bayes-vartypes/naive-bayes-likelihood.png differ diff --git a/faq/naive-bayes-vartypes/naive-bayes-likelihood_shorter.png b/faq/naive-bayes-vartypes/naive-bayes-likelihood_shorter.png new file mode 100644 index 00000000..690aeac4 Binary files /dev/null and b/faq/naive-bayes-vartypes/naive-bayes-likelihood_shorter.png differ diff --git a/faq/naive-bayes-vs-logistic-regression.md b/faq/naive-bayes-vs-logistic-regression.md new file mode 100644 index 00000000..28e64d0f --- /dev/null +++ b/faq/naive-bayes-vs-logistic-regression.md @@ -0,0 +1,13 @@ +# What is the major difference between naive Bayes and logistic regression? + +On a high-level, I would describe it as "generative vs. discriminative" models. + +- Generative classifiers learn a model of joint probabilities p(x, y) and use Bayes rule to calculate p(x|y) to make a prediction +- Discriminative models learn the posterior probability p(x|y) "directly" + +You can think of discriminative models as "distinguishing between people that speak different languages without actually learning the language". + +In discriminative models, you have "less assumptions", e.g,. in naive Bayes and classification, you assume that your p(x|y) follows (typically) a Gaussian, Bernoulli, or Multinomial distribution, and you even violate the assumption of conditional independence of the features. In favor of discriminative models, Vapnik wrote once "one should solve the classification problem directly and never solve a more general problem as an intermediate step". +(Vapnik, Vladimir Naumovich, and Vlamimir Vapnik. Statistical learning theory. Vol. 1. New York: Wiley, 1998.) + +I think it really depends on your problem though which method to prefer. I can't find a reference now, but e.g. in classification, naive Bayes converges quicker but has typically a higher error than logistic regression. On small datasets you'd might want to try out naive Bayes, but as your training set size grows, you likely get better results with logistic regression. diff --git a/faq/num-support-vectors.md b/faq/num-support-vectors.md new file mode 100644 index 00000000..7808ef96 --- /dev/null +++ b/faq/num-support-vectors.md @@ -0,0 +1,9 @@ +# When training an SVM classifier, is it better to have a large or small number of support vectors? + + +Unfortunately, like so often in machine learning applications, it really depends on the dataset. If we train an RBF kernel SVM we will typically end up with more support vectors than in a linear model. If our data is linearly separable, the latter may be better, and if that's not the case, the former may be better. + +Also, we have to differentiate between computational efficiency and generalization performance. If we increase the number of support vectors, our classification becomes more "expensive", especially in kernel SVM where we have to recalculate the distances between every new sample and the entire training set. + + +I'd say the best way to tackle the predictive performance problem is simply to evaluate the model, plot learning curves, do k-fold and/or cross validation, and see what works best on our given dataset. diff --git a/faq/number-of-kfolds.md b/faq/number-of-kfolds.md new file mode 100644 index 00000000..2c4824a9 --- /dev/null +++ b/faq/number-of-kfolds.md @@ -0,0 +1,8 @@ +# Is it always better to have the largest possible number of folds when performing cross validation? + +Let's assume we mean k-fold cross-validation used for hyperparameter tuning of algorithms for classification, and with "better," we mean better at estimating the generalization performance. +In this case, my answer would be no, otherwise we would always use LOOCV (Leave one out cross validation) instead of k-fold CV. (A useful reference: Shao, Jun. [Linear model selection by cross-validation.](http://www.sciencedirect.com/science/article/pii/S0378375803003719) Journal of the American statistical Association 88.422 (1993): 486-494.). + +In practice, I would say most commonly used (default) value is k=10 in k-fold CV, which is often an appropriate a good choice. But if we are working with small(er) training sets, I would increase the number of folds to use more training data in each iteration; this will lower the bias towards estimating the generalization error. On the other hand, it will also increase the run-time and variance of your estimate. The reason for the increasing variance of the estimate is that the overlap between training sets increases with an increasing size of *k* -- note that the test sets never overlap though. + +And far as computational efficiency is concerned -- for example, think of training deep neural nets on large(r) datasets including hyperparameter tuning -- I would think carefully about the size of *k*. If our dataset is large, I'd therefore recommend choosing smaller values for *k*, but it is all a balancing act between bias and variance and computational efficiency, and for our final estimate, we still have our independent test set anyway. diff --git a/faq/open-source.md b/faq/open-source.md new file mode 100644 index 00000000..9234fea6 --- /dev/null +++ b/faq/open-source.md @@ -0,0 +1,9 @@ +# At what point should one start contributing to open source? + +Please don't feel pressured to contribute to open source if this is not your thing -- I see it more as an opportunity to learn and "give back" but it is certainly not a "duty" :). + +> ... small tasks like fixing typos [...] time could be spent learning instead + +I'd say it depends. I think contributing to open-source projects shouldn't be all about personal benefits, and expanding/fixing documentation where it is useful is also valuable for a project. Eventually, it is up to you where you want to be on the spectrum between "how does this benefit me personally" and "would this help the community and the people who put so much effort into the things that I am using daily for free." -- it's certainly not a black or white decision, and there is no right or wrong :) + +Btw. depending on your experience level, "fixing typos" can be a valuable learning experience as well. For instance, doing "data science" is not only about analyzing data, but communications skills and team skills are important as well. If you are not familiar with version control, see fixing typos as an opportunity to learn about it or practice working with certain frameworks/workflows. diff --git a/faq/probablistic-logistic-regression.md b/faq/probablistic-logistic-regression.md new file mode 100644 index 00000000..0041e479 --- /dev/null +++ b/faq/probablistic-logistic-regression.md @@ -0,0 +1,37 @@ +# What is the probabilistic interpretation of regularized logistic regression? + +Let's start directly with the maximum likelihood function: + +![](./probablistic-logistic-regression/1.png) + +where phi is your conditional probability, i.e., sigmoid (logistic) function: + +![](./probablistic-logistic-regression/2.png) + +![](./probablistic-logistic-regression/3.png) + +and z is simply the *net input* (a scalar): + +![](./probablistic-logistic-regression/4.png) + + +So, by maximizing the likelihood we maximize the probability. Since we are talking about "cost", lets reverse the likelihood function so that we can minimize a cost function J. First, let's take the log so that we arrive at the equation that most people are familiar with (it's particularly handy to use the "addition trick" in the partial derivative e.g,. if you are using gradient or stochastic gradient descent): + +![](./probablistic-logistic-regression/5.png) + +![](./probablistic-logistic-regression/6.png) + +Now, imagine we plot the cost as follows, as a function of the 2 weights in a 2D dataset. For the unregularized cost, we would find the global cost minimum (the dot at the center) for a particular w1 and w2 combination. The key idea is that we increase the weights as much as necessary to reach the global cost minimum. + +![](./probablistic-logistic-regression/7.png) + + +Now, let's add a regularization term, e.g., L2 + +![](./probablistic-logistic-regression/8.png) + +This basically means that we will increase the cost by the squared Euclidean norm of your weight vector. Or in other words, we are constraint now, and we can't reach the global minimum anymore due to this increasingly large penalty. Basically, we have to find the sweet spot now: the point that minimizes the cost under the constraint that ywer can't go to far on the w1 and w2 axes, respectively. (In the image below, the size of the sphere depends on an additional hyperparameter, lambda.) + +![](./probablistic-logistic-regression/9.png) + +It is as if we would add a prior to the weights. Instead of maximizing the likelihood (minimizing the cost function) given the training data, we maximize the likelihood given the additional information bias. diff --git a/faq/probablistic-logistic-regression/1.png b/faq/probablistic-logistic-regression/1.png new file mode 100644 index 00000000..558e8b51 Binary files /dev/null and b/faq/probablistic-logistic-regression/1.png differ diff --git a/faq/probablistic-logistic-regression/2.png b/faq/probablistic-logistic-regression/2.png new file mode 100644 index 00000000..f665c0aa Binary files /dev/null and b/faq/probablistic-logistic-regression/2.png differ diff --git a/faq/probablistic-logistic-regression/3.png b/faq/probablistic-logistic-regression/3.png new file mode 100644 index 00000000..632a8edf Binary files /dev/null and b/faq/probablistic-logistic-regression/3.png differ diff --git a/faq/probablistic-logistic-regression/4.png b/faq/probablistic-logistic-regression/4.png new file mode 100644 index 00000000..e8c32c2e Binary files /dev/null and b/faq/probablistic-logistic-regression/4.png differ diff --git a/faq/probablistic-logistic-regression/5.png b/faq/probablistic-logistic-regression/5.png new file mode 100644 index 00000000..b16f567d Binary files /dev/null and b/faq/probablistic-logistic-regression/5.png differ diff --git a/faq/probablistic-logistic-regression/6.png b/faq/probablistic-logistic-regression/6.png new file mode 100644 index 00000000..2e8e46e4 Binary files /dev/null and b/faq/probablistic-logistic-regression/6.png differ diff --git a/faq/probablistic-logistic-regression/7.png b/faq/probablistic-logistic-regression/7.png new file mode 100644 index 00000000..7a2c2600 Binary files /dev/null and b/faq/probablistic-logistic-regression/7.png differ diff --git a/faq/probablistic-logistic-regression/8.png b/faq/probablistic-logistic-regression/8.png new file mode 100644 index 00000000..b2789c1d Binary files /dev/null and b/faq/probablistic-logistic-regression/8.png differ diff --git a/faq/probablistic-logistic-regression/9.png b/faq/probablistic-logistic-regression/9.png new file mode 100644 index 00000000..ae36ee62 Binary files /dev/null and b/faq/probablistic-logistic-regression/9.png differ diff --git a/faq/regularized-logistic-regression-performance.md b/faq/regularized-logistic-regression-performance.md new file mode 100644 index 00000000..cac3de61 --- /dev/null +++ b/faq/regularized-logistic-regression-performance.md @@ -0,0 +1,23 @@ +# Does regularization in logistic regression always results in better fit and better generalization? + + +Regularization does NOT improve the performance on the data set that the algorithm used to learn the model parameters (feature weights). However, it **can improve the generalization performance**, i.e., the performance on new, unseen data, which is exactly what we want. + +In intuitive terms, we can think of regularization as a penalty against complexity. Increasing the regularization strength penalizes "large" weight coefficients -- our goal is to prevent that our model picks up "peculiarities," "noise," or "imagines a pattern where there is none." + +**Again, we don't want the model to memorize the training dataset, we want a model that generalizes well to new, unseen data.** + +In more specific terms, we can think of regularization as adding (or increasing the) bias if our model suffers from (high) variance (i.e., it overfits the training data). On the other hand, too much bias will result in underfitting (a characteristic indicator of high bias is that the model shows a "bad" performance for both the training and test dataset). +We know that our goal in an unregularized model is to minimize the cost function, i.e., we want to find the feature weights that correspond to the global cost minimum (remember that the logistic cost function is convex). + +![](./regularized-logistic-regression-performance/unregularized.png) + +Now, if we regularize the cost function (e.g., via L2 regularization), we add an additional to our cost function (J) that increases as the value of your parameter weights (w) increase; keep in mind that the regularization we add a new hyperparameter, lambda, to control the regularization strength. + +![](./regularized-logistic-regression-performance/l2-term.png) + +Therefore, our new problem is to minimize the cost function given this added constraint. + +![](./regularized-logistic-regression-performance/regularized.png) + +Intuitively, we can think of the "sphere" at the coordinate center in the figure above as our "budget." Now, our objective is still the same: we want to minimize the cost function. However, we are now constraint by the regularization term; we want to get as close as possible to the global minimum while staying within our "budget" (i.e., the sphere). diff --git a/faq/regularized-logistic-regression-performance/l2-term.png b/faq/regularized-logistic-regression-performance/l2-term.png new file mode 100644 index 00000000..45cb7bd9 Binary files /dev/null and b/faq/regularized-logistic-regression-performance/l2-term.png differ diff --git a/faq/regularized-logistic-regression-performance/regularized.png b/faq/regularized-logistic-regression-performance/regularized.png new file mode 100644 index 00000000..aa55ccf1 Binary files /dev/null and b/faq/regularized-logistic-regression-performance/regularized.png differ diff --git a/faq/regularized-logistic-regression-performance/unregularized.png b/faq/regularized-logistic-regression-performance/unregularized.png new file mode 100644 index 00000000..420678a0 Binary files /dev/null and b/faq/regularized-logistic-regression-performance/unregularized.png differ diff --git a/faq/softmax.md b/faq/softmax.md new file mode 100644 index 00000000..df701842 --- /dev/null +++ b/faq/softmax.md @@ -0,0 +1,15 @@ +# What exactly is the "softmax and the multinomial logistic loss" in the context of machine learning? + +The softmax function is simply a generalization of the logistic function that allows us to compute meaningful class-probabilities in multi-class settings (multinomial logistic regression). In softmax, we compute the probability that a particular sample (with net input z) belongs to the *i*th class using a normalization term in the denominator that is the sum of all *M* linear functions: + +![](./softmax/softmax_1.png) + +In contrast, the logistic function: + +![](./softmax/logistic.png) + +And for completeness, we define the net input as + +![](./softmax/net_input.png) + +where the weight coefficients of your model are stored as vector "w" and "x" is the feature vector of your sample. diff --git a/faq/softmax/logistic.png b/faq/softmax/logistic.png new file mode 100644 index 00000000..de9b14cc Binary files /dev/null and b/faq/softmax/logistic.png differ diff --git a/faq/softmax/net_input.png b/faq/softmax/net_input.png new file mode 100644 index 00000000..d9af13d1 Binary files /dev/null and b/faq/softmax/net_input.png differ diff --git a/faq/softmax/softmax_1.png b/faq/softmax/softmax_1.png new file mode 100644 index 00000000..edff52e6 Binary files /dev/null and b/faq/softmax/softmax_1.png differ diff --git a/faq/standardize-param-reuse.md b/faq/standardize-param-reuse.md new file mode 100644 index 00000000..9ccfe9ac --- /dev/null +++ b/faq/standardize-param-reuse.md @@ -0,0 +1,37 @@ +# Why do we re-use parameters from the training set to standardize the test set and new data? + +Let me give you an example to this very common question to show why we don't want to standardize new (or test) data "from scratch." + +Let's assume we have a simple training set consisting of 3 samples with 1 feature (let's call this feature "length"): + +- train_1: 10 cm -> class_2 +- train_2: 20 cm -> class_2 +- train_3: 30 cm -> class_1 + +mean: 20, std.: 8.2 + +After standardization, the transformed feature values are + +- train_std_1: -1.21 -> class_2 +- train_std_2: 0 -> class_2 +- train_std_3: 1.21 -> class_1 + +Next, let's assume our model has learned to classify samples with a standardized length value < 0.6 as class_2 (class_1 otherwise). So far so good. Now, let's say we have 3 unlabeled data points that we want to classify: + +- new_4: 5 cm -> class ? +- new_5: 6 cm -> class ? +- new_6: 7 cm -> class ? + +If we look at the "unstandardized "length" values in our training dataset, it is intuitive to say that all of these samples are likely belonging to class_2. However, if we standardize these by re-computing standard deviation and and mean you would get similar values as before in the training set and your classifier would (probably incorrectly) classify samples 4 and 5 as class 2. + +- new_std_4: -1.21 -> class 2 +- new_std_5: 0 -> class 2 +- new_std_6: 1.21 -> class 1 + +However, if we use the parameters from your "training set standardization," we'd get the values: + +- sample5: -18.37 -> class 2 +- sample6: -17.15 -> class 2 +- sample7: -15.92 -> class 2 + +The values 5 cm, 6 cm, and 7 cm are much lower than anything we have seen in the training set previously. Thus, it only makes sense that the standardized features of the "new samples" are much lower than every standardized feature in the training set. diff --git a/faq/underscore_convention.md b/faq/underscore-convention.md similarity index 100% rename from faq/underscore_convention.md rename to faq/underscore-convention.md diff --git a/faq/visual-backpropagation.md b/faq/visual-backpropagation.md new file mode 100644 index 00000000..6cc168a1 --- /dev/null +++ b/faq/visual-backpropagation.md @@ -0,0 +1,27 @@ +# Can you give a visual explanation for the back propagation algorithm for neural networks? + +Let's assume we are really into mountain climbing, and to add a little extra challenge, we cover eyes this time so that we can't see where we are and when we accomplished our "objective," that is, reaching the top of the mountain. + +Since we can't see the path upfront, we let our intuition guide us: assuming that the mountain top is the "highest" point of the mountain, we think that the steepest path leads us to the top most efficiently. +We approach this challenge by iteratively "feeling" around you and taking a step into the direction of the steepest ascent -- let's call it "gradient ascent." But what do we do if we reach a point where we can't ascent any further? I.e., each direction leads downwards? At this point, we may have already reached the mountain's top, but we could just have reached a smaller plateau ... we don't know. +Essentially, this is just an analogy of gradient ascent optimization (basically the counterpart of minimizing a cost function via gradient descent). However, this is not specific to backpropagation but just one way to minimize a convex cost function (if there is only a global minima) or non-convex cost function (which has local minima like the "plateaus" that let us think we reached the mountain's top). Using a little visual aid, we could picture a non-convex cost function with only one parameter (where the blue ball is our current location) as follows: + + ![](./visual-backpropagation/nonconvex-cost.png) + +Now, backpropagation is just back-propagating the cost over multiple "levels" (or layers). E.g., if we have a multi-layer perceptron, we can picture forward propagation (passing the input signal through a network while multiplying it by the respective weights to compute an output) as follows: + +![](./visual-backpropagation/forward-propagation.png) + +And in backpropagation, we "simply" backpropagate the error (the "cost" that we compute by comparing the calculated output and the known, correct target output, which we then use to update the model parameters): + +![](./visual-backpropagation/backpropagation.png) + + +It may be some time ago since pre-calc, but it's essentially all based on the simple chain-rule that we use for nested functions + +![](./visual-backpropagation/chain_rule_1.png) + +![](./visual-backpropagation/chain_rule_2.png) + + +Instead of doing this "manually" we can use computational tools (called "automatic differentiation"), and backpropagation is basically the "reverse" mode of this auto-differentiation. Why reverse and not forward? Because it is computationally cheaper! If we'd do it forward-wise, we'd successively multiply large matrices for each layer until we multiply a large matrix by a vector in the output layer. However, if we start backwards, that is, we start by multiplying a matrix by a vector, we get another vector, and so forth. So, I'd say the beauty in backpropagation is that we are doing more efficient matrix-vector multiplications instead of matrix-matrix multiplications. diff --git a/faq/visual-backpropagation/backpropagation.png b/faq/visual-backpropagation/backpropagation.png new file mode 100644 index 00000000..7fa8674b Binary files /dev/null and b/faq/visual-backpropagation/backpropagation.png differ diff --git a/faq/visual-backpropagation/chain_rule_1.png b/faq/visual-backpropagation/chain_rule_1.png new file mode 100644 index 00000000..e7f62c67 Binary files /dev/null and b/faq/visual-backpropagation/chain_rule_1.png differ diff --git a/faq/visual-backpropagation/chain_rule_2.png b/faq/visual-backpropagation/chain_rule_2.png new file mode 100644 index 00000000..ed10e184 Binary files /dev/null and b/faq/visual-backpropagation/chain_rule_2.png differ diff --git a/faq/visual-backpropagation/forward-propagation.png b/faq/visual-backpropagation/forward-propagation.png new file mode 100644 index 00000000..75d01225 Binary files /dev/null and b/faq/visual-backpropagation/forward-propagation.png differ diff --git a/faq/visual-backpropagation/nonconvex-cost.png b/faq/visual-backpropagation/nonconvex-cost.png new file mode 100644 index 00000000..896db0e3 Binary files /dev/null and b/faq/visual-backpropagation/nonconvex-cost.png differ diff --git a/faq/when-to-standardize.md b/faq/when-to-standardize.md new file mode 100644 index 00000000..deac8d65 --- /dev/null +++ b/faq/when-to-standardize.md @@ -0,0 +1,23 @@ +# When should I apply data normalization/standardization? + + +The only family of algorithms that I could think of being scale-invariant are tree-based methods. Let's take the general CART decision tree algorithm. Without going into much depth regarding information gain and impurity measures, we can think of the decision as "is feature x_i >= some_val?" Intuitively, we can see that it really doesn't matter on which scale this feature is (centimeters, Fahrenheit, a standardized scale -- it really doesn't matter). + + +Some examples of algorithms where feature scaling matters are: + + +- k-nearest neighbors with an Euclidean distance measure if want all features to contribute equally +- k-means (see k-nearest neighbors) +- logistic regression, SVMs, perceptrons, neural networks etc. if you are using gradient descent/ascent-based optimization, otherwise some weights will update much faster than others +- linear discriminant analysis, principal component analysis, kernel principal component analysis since you want to find directions of maximizing the variance (under the constraints that those directions/eigenvectors/principal components are orthogonal); you want to have features on the same scale since you'd emphasize variables on "larger measurement scales" more. + + +There are many more cases than I can possibly list here ... I always recommend you to think about the algorithm and what it's doing, and then it typically becomes obvious whether we want to scale your features or not. + + +In addition, we'd also want to think about whether we want to "standardize" or "normalize" (here: scaling to [0, 1] range) our data. Some algorithms assume that our data is centered at 0. For example, if we initialize the weights of a small multi-layer perceptron with tanh activation units to 0 or small random values centered around zero, we want to update the model weights "equally." +As a rule of thumb I'd say: When in doubt, just standardize the data, it shouldn't hurt. + + + diff --git a/faq/why_python.md b/faq/why-python.md similarity index 98% rename from faq/why_python.md rename to faq/why-python.md index fad7895e..71fd6351 100644 --- a/faq/why_python.md +++ b/faq/why-python.md @@ -32,7 +32,7 @@ By now, you may have already started wondering about this blog. I haven't writte Maybe I should start with the short answer. You are welcome to stop reading this article below this paragraph because it really nails it. I am a scientist, I like to get my stuff done. I like to have an environment where I can quickly prototype and jot down my models and ideas. I need to solve very particular problems. I analyze given datasets to draw my conclusions. This is what matters most to me: How can I get the job done most productively? What do I mean by "productively"? Well, I typically run an analysis only once (the testing of different ideas and debugging aside); I don't need to repeatedly run a particular piece of code 24/7, I am not developing software applications or web apps for end users. When I *quantify* "productivity," I literally estimate the sum of (1) the time that it takes to get the idea written down in code, (2) debug it, and (3) execute it. To me, "most productively" means "how long does it take to get the results?" Now, over the years, I figured that Python is for me. Not always, but very often. Like everything else in life, Python is not a "silver bullet," it's not the "best" solution to every problem. However, it comes pretty close if you compare programming languages across the spectrum of common and not-so common problem tasks; Python is probably the most versatile and capable all-rounder. -![](./images/the_general_problem.png) +![](./why-python/the_general_problem.png) (Source: [https://xkcd.com/974/](https://xkcd.com/974/)) Remember: "Premature optimization is the root of all evil" (Donald Knuth). If you are part of the software engineering team that wants to optimize the next game-changing high-frequency trading model from your machine learning and data science division, Python is probably not for you (but maybe it was the language of choice by the data science team, so it may still be useful to learn how to read it). So, my little piece of advice is to evaluate your daily problem tasks and needs when you choose a language. "If all that you have is a hammer, everything starts to look like a nail" -- you are too smart to fall for this trap! However, keep in mind that there is a balance. There are occasions where the hammer may be the best choice even if a screwdriver would probably be the "nicer" solution. Again, it comes down to productivity. @@ -44,12 +44,12 @@ I needed to develop a bunch of novel algorithms to "screen" 15 million small, ch Trust me, time was really "limited:" We just got our grant application accepted and research funded a few weeks before the results had to be collected (our collaborators were doing experiments on larvae of a certain fish species that only spawns in Spring). Therefore, I started thinking "How could I get those results to them as quickly as possible?" Well, I know C++ and FORTRAN, and if I implement those algorithms in the respective languages executing the "screening" run may be faster compared to a Python implementation. This was more of an educated guess, I don't really know if it would have been substantially faster. But there was one thing I knew for sure: If I started developing the code in Python, I could be able to get it to run in a few days -- maybe it would take a week to get the respective C++ versions coded up. I would worry about a more efficient implementation later. At that moment, it was just important to get those results to my collaborators -- "Premature optimization is the root of all evil." On a side node: The same train of thought applies to data storage solutions. Here, I just went with SQLite. CSV didn't make quite sense since I had to annotate and retrieve certain molecules repeatedly. I surely didn't want to scan or rewrite a CSV from start to end every time I wanted to look up a molecule or manipulate its entry -- issues in dealing with memory capacities aside. Maybe MySQL would have been even better but for the reasons mentioned above, I wanted to get the job done quickly, and setting up an additional SQL server ... there was no time for that, SQLite was just fine to get the job done. -![](./images/automation.png) +![](./why-python/automation.png) (Source: [https://xkcd.com/1319/](https://xkcd.com/1319/)) The verdict: **Choose the language that satisfies *your* needs!** -However, there is once little caveat here! How can a beginning programmer possibly know about the advantages and disadvantages of a language before learning it, and how should the programmer know if this language will be useful to her at all? This is what I would do: Just search for particular applications and solutions related to your most common problem tasks on Google and [GitHub](https://github.com). You don't need to read and understand the code. Just look at the end product. +However, there is once little caveat here! How can a beginning programmer possibly know about the advantages and disadvantages of a language before learning it, and how should the programmer know if this language will be useful to her at all? This is what I would do: Just search for particular applications and solutions related to your most common problem tasks on Google and [GitHub](https://github.com). You don't need to read and understand the code. Just look at the end product. > In the one and only true way. The object-oriented version of 'Spaghetti code' is, of course, 'Lasagna code'. (Too many layers). — Roberto Waltman. @@ -68,7 +68,7 @@ If you are interested, those are my favorite and most frequently used Python "to - [scikit-learn](http://scikit-learn.org/stable/): The most convenient API for the daily, more basic machine learning tasks. - [matplotlib](http://matplotlib.org): My library of choice when it comes to plotting. Sometimes I also use [seaborn](http://stanford.edu/~mwaskom/software/seaborn/index.html) for particular plots, for example, the heat maps are particularly great! -[](./images/heatmap.png) +[](./why-python/heatmap.png) (Source: [http://stanford.edu/~mwaskom/software/seaborn/examples/structured_heatmap.html](http://stanford.edu/~mwaskom/software/seaborn/examples/structured_heatmap.html)) @@ -77,7 +77,7 @@ If you are interested, those are my favorite and most frequently used Python "to - [pandas](http://pandas.pydata.org): Working with relatively small datasets, mostly from CSV files. - [sqlite3](https://docs.python.org/2/library/sqlite3.html): Annotating and querying "medium-sized" datasets. - [IPython notebooks](http://ipython.org): What can I say, 90% of my research takes place in IPython notebooks. It's just a great environment to have everything in one place: Ideas, code, comments, LaTeX equations, illustrations, plots, outputs, ... -![](./images/ipython_notebook.png) +![](./why-python/ipython_notebook.png) Note that the IPython Project recently evolved into [Project Jupyter](https://jupyter.org). Now, you can use Jupyter notebook environment not only for Python but R, Julia, and many more. @@ -92,7 +92,7 @@ prototyping after all! Since it was built with linear algebra in mind (MATLAB fo However, keep in mind that MATLAB comes with a big price tag, and I think it is slowly fading from academia as well as industry. Plus, I am a big fan open-source enthusiast after all ;). In addition, its performance is also not that compelling compared to other "productive" languages looking at the benchmarks below: -![](./images/julia_benchmark.png) +![](./why-python/julia_benchmark.png) (Benchmark times relative to C -- smaller is better, C performance = 1.0; Source: [http://julialang.org/benchmarks/](http://julialang.org/benchmarks/)) @@ -132,7 +132,7 @@ To be honest, I have to admit that I am not necessarily a big fan of the "@" sym [[back to top](#table-of-contents)] -I think Julia is a great language, and I would like to recommend it to someone who's getting started with programming and machine learning. I am not sure if I really should though. Why? There is this sad, somewhat paradoxical thing about committing to programming languages. With Julia, we cannot tell if it will become "popular" enough in the next few years. +I think Julia is a great language, and I would like to recommend it to someone who's getting started with programming and machine learning. I am not sure if I really should though. Why? There is this sad, somewhat paradoxical thing about committing to programming languages. With Julia, we cannot tell if it will become "popular" enough in the next few years. > There are only two kinds of languages: the ones people complain about and the ones nobody uses — Bjarne Stroustrup @@ -163,13 +163,13 @@ I just wanted to bring up Theano and computing on GPUs as a big plus for Python, To take one of my favorite Python quotes out of its original context: "We are all adults here" -- let's not waste our time with language wars. Choose the tool that "clicks" for you. When it comes to perspectives on the job market: There is no right or wrong here either. I don't think a company that wants to hire you as a "data scientist" really bothers about your favorite toolbox -- programming languages are just "tools" after all. The most important skill is to think like a "data scientist," to ask the right questions, to solve a problems. The hard part is the math and machine learning theory, a new programming language can easily be learned. Just think about, you learned how to swing a hammer to drive the nail in, how hard can it possibly be to pick up a hammer from a different manufacturer? But if you are still interested, look at the Tiobe Index for example, *one* measure of popularity of programming languages: -![](./images/tiobe.png) +![](./why-python/tiobe.png) (Source: [http://www.tiobe.com/index.php/content/paperinfo/tpci/index.html](http://www.tiobe.com/index.php/content/paperinfo/tpci/index.html)) However, if we look at the [The 2015 Top Ten Programming Languages](http://spectrum.ieee.org/computing/software/the-2015-top-ten-programming-languages) by Spectrum IEEE, the R language is climbing fast (left column: 2015, right column: 2014). -![](./images/spectrum.jpg) +![](./why-python/spectrum.jpg) (Source: [http://spectrum.ieee.org/computing/software/the-2015-top-ten-programming-languages](http://spectrum.ieee.org/computing/software/the–2015-top-ten-programming-languages)) @@ -205,8 +205,6 @@ Speaking of hammers and nails again, Python is extremely versatile, the largest Well, this is a pretty long answer to a seemingly very simple question. Trust me, I can go on for hours and days. But why complicate things? Let's bring the talks to a conclusion: -![](./images/python.png) +![](./why-python/python.png) (Source: [https://xkcd.com/353/](https://xkcd.com/353/)) - - diff --git a/faq/images/automation.png b/faq/why-python/automation.png similarity index 100% rename from faq/images/automation.png rename to faq/why-python/automation.png diff --git a/faq/images/heatmap.png b/faq/why-python/heatmap.png similarity index 100% rename from faq/images/heatmap.png rename to faq/why-python/heatmap.png diff --git a/faq/images/ipython_notebook.png b/faq/why-python/ipython_notebook.png similarity index 100% rename from faq/images/ipython_notebook.png rename to faq/why-python/ipython_notebook.png diff --git a/faq/images/julia_benchmark.png b/faq/why-python/julia_benchmark.png similarity index 100% rename from faq/images/julia_benchmark.png rename to faq/why-python/julia_benchmark.png diff --git a/faq/images/python.png b/faq/why-python/python.png similarity index 100% rename from faq/images/python.png rename to faq/why-python/python.png diff --git a/faq/images/revor.png b/faq/why-python/revor.png similarity index 100% rename from faq/images/revor.png rename to faq/why-python/revor.png diff --git a/faq/images/spectrum.jpg b/faq/why-python/spectrum.jpg similarity index 100% rename from faq/images/spectrum.jpg rename to faq/why-python/spectrum.jpg diff --git a/faq/images/the_general_problem.png b/faq/why-python/the_general_problem.png similarity index 100% rename from faq/images/the_general_problem.png rename to faq/why-python/the_general_problem.png diff --git a/faq/images/tiobe.png b/faq/why-python/tiobe.png similarity index 100% rename from faq/images/tiobe.png rename to faq/why-python/tiobe.png