{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# $$CatBoost\\ Tutorial$$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/catboost/tutorials/blob/master/python_tutorial.ipynb)\n", "\n", "In this tutorial we would explore some base cases of using catboost, such as model training, cross-validation and predicting, as well as some useful features like early stopping, snapshot support, feature importances and parameters tuning.\n", " \n", "You could run this tutorial in Google Colaboratory environment with free CPU or GPU. Just click on this link." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## $$Contents$$\n", "* [1. Data Preparation](#$$1.\\-Data\\-Preparation$$)\n", " * [1.1 Data Loading](#1.1-Data-Loading)\n", " * [1.2 Feature Preparation](#1.2-Feature-Preparation)\n", " * [1.3 Data Splitting](#1.3-Data-Splitting)\n", "* [2. CatBoost Basics](#$$2.\\-CatBoost\\-Basics$$)\n", " * [2.1 Model Training](#2.1-Model-Training)\n", " * [2.2 Model Cross-Validation](#2.2-Model-Cross-Validation)\n", " * [2.3 Model Applying](#2.3-Model-Applying)\n", "* [3. CatBoost Features](#$$3.\\-CatBoost\\-Features$$)\n", " * [3.1 Using the best model](#3.1-Using-the-best-model)\n", " * [3.2 Early Stopping](#3.2-Early-Stopping)\n", " * [3.3 Using Baseline](#3.3-Using-Baseline)\n", " * [3.4 Snapshot Support](#3.4-Snapshot-Support)\n", " * [3.5 User Defined Objective Function](#3.5-User-Defined-Objective-Function)\n", " * [3.6 User Defined Metric Function](#3.6-User-Defined-Metric-Function)\n", " * [3.7 Staged Predict](#3.7-Staged-Predict)\n", " * [3.8 Feature Importances](#3.8-Feature-Importances)\n", " * [3.9 Eval Metrics](#3.9-Eval-Metrics)\n", " * [3.10 Learning Processes Comparison](#3.10-Learning-Processes-Comparison)\n", " * [3.11 Model Saving](#3.11-Model-Saving)\n", "* [4. Parameters Tuning](#$$4.\\-Parameters\\-Tuning$$)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## $$1.\\ Data\\ Preparation$$\n", "### 1.1 CatBoost installation\n", "If you have not already installed CatBoost, you can do so by running '!pip install catboost' command. \n", " \n", "Also you should install ipywidgets package and run special command before launching jupyter notebook to draw plots." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!pip install catboost\n", "!pip install scikit-learn\n", "!pip install ipywidgets\n", "!jupyter nbextension enable --py widgetsnbextension" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1.2 Data Loading\n", "The data for this tutorial can be obtained from [this page](https://www.kaggle.com/c/titanic/data) (you would have to register a kaggle account or just login with facebook or google+) or you could use catboost.datasets as in code below." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
PassengerIdSurvivedPclassNameSexAgeSibSpParchTicketFareCabinEmbarked
0103Braund, Mr. Owen Harrismale22.010A/5 211717.2500NaNS
1211Cumings, Mrs. John Bradley (Florence Briggs Th...female38.010PC 1759971.2833C85C
2313Heikkinen, Miss. Lainafemale26.000STON/O2. 31012827.9250NaNS
3411Futrelle, Mrs. Jacques Heath (Lily May Peel)female35.01011380353.1000C123S
4503Allen, Mr. William Henrymale35.0003734508.0500NaNS
\n", "
" ], "text/plain": [ " PassengerId Survived Pclass \\\n", "0 1 0 3 \n", "1 2 1 1 \n", "2 3 1 3 \n", "3 4 1 1 \n", "4 5 0 3 \n", "\n", " Name Sex Age SibSp \\\n", "0 Braund, Mr. Owen Harris male 22.0 1 \n", "1 Cumings, Mrs. John Bradley (Florence Briggs Th... female 38.0 1 \n", "2 Heikkinen, Miss. Laina female 26.0 0 \n", "3 Futrelle, Mrs. Jacques Heath (Lily May Peel) female 35.0 1 \n", "4 Allen, Mr. William Henry male 35.0 0 \n", "\n", " Parch Ticket Fare Cabin Embarked \n", "0 0 A/5 21171 7.2500 NaN S \n", "1 0 PC 17599 71.2833 C85 C \n", "2 0 STON/O2. 3101282 7.9250 NaN S \n", "3 0 113803 53.1000 C123 S \n", "4 0 373450 8.0500 NaN S " ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from catboost.datasets import titanic\n", "import numpy as np\n", "\n", "train_df, test_df = titanic()\n", "\n", "train_df.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1.3 Feature Preparation\n", "First of all let's check how many absent values do we have:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Age 177\n", "Cabin 687\n", "Embarked 2\n", "dtype: int64" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "null_value_stats = train_df.isnull().sum(axis=0)\n", "null_value_stats[null_value_stats != 0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As we can see, **`Age`**, **`Cabin`** and **`Embarked`** indeed have some missing values, so let's fill them with some number way out of their distributions - so the model would be able to easily distinguish between them and take it into account:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "train_df.fillna(-999, inplace=True)\n", "test_df.fillna(-999, inplace=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now let's separate features and label variable:" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "X = train_df.drop('Survived', axis=1)\n", "y = train_df.Survived" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Pay attention that our features are of different types - some of them are numeric, some are categorical, and some are even just strings, which normally should be handled in some specific way (for example encoded with bag-of-words representation). But in our case we could treat these string features just as categorical one - all the heavy lifting is done inside CatBoost. How cool is that? :)" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "PassengerId int64\n", "Pclass int64\n", "Name object\n", "Sex object\n", "Age float64\n", "SibSp int64\n", "Parch int64\n", "Ticket object\n", "Fare float64\n", "Cabin object\n", "Embarked object\n", "dtype: object\n" ] } ], "source": [ "print(X.dtypes)\n", "\n", "categorical_features_indices = np.where(X.dtypes != float)[0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1.4 Data Splitting\n", "Let's split the train data into training and validation sets." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "from sklearn.model_selection import train_test_split\n", "\n", "X_train, X_validation, y_train, y_validation = train_test_split(X, y, train_size=0.75, random_state=42)\n", "\n", "X_test = test_df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## $$2.\\ CatBoost\\ Basics$$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's make necessary imports." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "from catboost import CatBoostClassifier, Pool, metrics, cv\n", "from sklearn.metrics import accuracy_score" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.1 Model Training\n", "Now let's create the model itself. We will go here with default parameters, as they provide a _really_ good baseline almost all the time. The only thing we would like to specify here is `custom_loss` parameter, as this would give us an ability to see what's going on in terms of this competition metric - accuracy, as well as to be able to watch for logloss, as it would be more smooth on dataset of such size." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "model = CatBoostClassifier(\n", " custom_loss=[metrics.Accuracy()],\n", " random_seed=42,\n", " logging_level='Silent'\n", ")" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "0c292d3acabb4c818f76d73fb0ac5f03", "version_major": 2, "version_minor": 0 }, "text/plain": [ "MetricVisualizer(layout=Layout(align_self='stretch', height='500px'))" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "model.fit(\n", " X_train, y_train,\n", " cat_features=categorical_features_indices,\n", " eval_set=(X_validation, y_validation),\n", "# logging_level='Verbose', # you can uncomment this for text output\n", " plot=True\n", ");" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As you can see, it is possible to watch our model learn through verbose output or with nice plots (personally I would definately go with the second option - just check out those plots: you can, for example, zoom in areas of interest!)\n", "\n", "With this we can see that the best accuracy value of **0.8296** (on validation set) was acheived on **150** boosting step." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.2 Model Cross-Validation\n", "\n", "It is good to validate your model, but to cross-validate it - even better. And also with plots! So with no more words:" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "5d740a481999493ea723de678e714428", "version_major": 2, "version_minor": 0 }, "text/plain": [ "MetricVisualizer(layout=Layout(align_self='stretch', height='500px'))" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "cv_params = model.get_params()\n", "cv_params.update({\n", " 'loss_function': metrics.Logloss()\n", "})\n", "cv_data = cv(\n", " Pool(X, y, cat_features=categorical_features_indices),\n", " cv_params,\n", " plot=True\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we have values of our loss functions at each boosting step averaged by 3 folds, which should provide us with a more accurate estimation of our model performance:" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Best validation accuracy score: 0.83±0.02 on step 355\n" ] } ], "source": [ "print('Best validation accuracy score: {:.2f}±{:.2f} on step {}'.format(\n", " np.max(cv_data['test-Accuracy-mean']),\n", " cv_data['test-Accuracy-std'][np.argmax(cv_data['test-Accuracy-mean'])],\n", " np.argmax(cv_data['test-Accuracy-mean'])\n", "))" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Precise validation accuracy score: 0.8294051627384961\n" ] } ], "source": [ "print('Precise validation accuracy score: {}'.format(np.max(cv_data['test-Accuracy-mean'])))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As we can see, our initial estimation of performance on single validation fold was too optimistic - that is why cross-validation is so important!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.3 Model Applying\n", "All you have to do to get predictions is" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[0 0 0 0 1 0 1 0 1 0]\n", "[[0.85473931 0.14526069]\n", " [0.76313031 0.23686969]\n", " [0.88972889 0.11027111]\n", " [0.87876173 0.12123827]\n", " [0.3611047 0.6388953 ]\n", " [0.90513381 0.09486619]\n", " [0.33434185 0.66565815]\n", " [0.78468564 0.21531436]\n", " [0.39429048 0.60570952]\n", " [0.94047549 0.05952451]]\n" ] } ], "source": [ "predictions = model.predict(X_test)\n", "predictions_probs = model.predict_proba(X_test)\n", "print(predictions[:10])\n", "print(predictions_probs[:10])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "But let's try to get a better predictions and Catboost features help us in it." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## $$3.\\ CatBoost\\ Features$$\n", "You may have noticed that on model creation step I've specified not only `custom_loss` but also `random_seed` parameter. That was done in order to make this notebook reproducible - by default catboost chooses some random value for seed:" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Random seed assigned for this model: 0\n" ] } ], "source": [ "model_without_seed = CatBoostClassifier(iterations=10, logging_level='Silent')\n", "model_without_seed.fit(X, y, cat_features=categorical_features_indices)\n", "\n", "print('Random seed assigned for this model: {}'.format(model_without_seed.random_seed_))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's define some params and create `Pool` for more convenience. It stores all information about dataset (features, labeles, categorical features indices, weights and and much more)." ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [], "source": [ "params = {\n", " 'iterations': 500,\n", " 'learning_rate': 0.1,\n", " 'eval_metric': metrics.Accuracy(),\n", " 'random_seed': 42,\n", " 'logging_level': 'Silent',\n", " 'use_best_model': False\n", "}\n", "train_pool = Pool(X_train, y_train, cat_features=categorical_features_indices)\n", "validate_pool = Pool(X_validation, y_validation, cat_features=categorical_features_indices)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3.1 Using the best model\n", "If you essentially have a validation set, it's always better to use the `use_best_model` parameter during training. By default, this parameter is enabled. If it is enabled, the resulting trees ensemble is shrinking to the best iteration." ] }, { "cell_type": "code", "execution_count": 33, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Simple model validation accuracy: 0.7982\n", "\n", "Best model validation accuracy: 0.8251\n" ] } ], "source": [ "model = CatBoostClassifier(**params)\n", "model.fit(train_pool, eval_set=validate_pool)\n", "\n", "best_model_params = params.copy()\n", "best_model_params.update({\n", " 'use_best_model': True\n", "})\n", "best_model = CatBoostClassifier(**best_model_params)\n", "best_model.fit(train_pool, eval_set=validate_pool);\n", "\n", "print('Simple model validation accuracy: {:.4}'.format(\n", " accuracy_score(y_validation, model.predict(X_validation))\n", "))\n", "print('')\n", "\n", "print('Best model validation accuracy: {:.4}'.format(\n", " accuracy_score(y_validation, best_model.predict(X_validation))\n", "))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3.2 Early Stopping\n", "If you essentially have a validation set, it's always easier and better to use early stopping. This feature is similar to the previous one, but only in addition to improving the quality it still saves time." ] }, { "cell_type": "code", "execution_count": 42, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "CPU times: user 9.42 s, sys: 368 ms, total: 9.79 s\n", "Wall time: 1.1 s\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 42, "metadata": {}, "output_type": "execute_result" } ], "source": [ "%%time\n", "model = CatBoostClassifier(**params)\n", "model.fit(train_pool, eval_set=validate_pool)" ] }, { "cell_type": "code", "execution_count": 43, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "CPU times: user 1.52 s, sys: 108 ms, total: 1.62 s\n", "Wall time: 195 ms\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 43, "metadata": {}, "output_type": "execute_result" } ], "source": [ "%%time\n", "earlystop_params = params.copy()\n", "earlystop_params.update({\n", " 'od_type': 'Iter',\n", " 'od_wait': 40\n", "})\n", "earlystop_model = CatBoostClassifier(**earlystop_params)\n", "earlystop_model.fit(train_pool, eval_set=validate_pool);" ] }, { "cell_type": "code", "execution_count": 44, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Simple model tree count: 500\n", "Simple model validation accuracy: 0.7982\n", "\n", "Early-stopped model tree count: 82\n", "Early-stopped model validation accuracy: 0.8072\n" ] } ], "source": [ "print('Simple model tree count: {}'.format(model.tree_count_))\n", "print('Simple model validation accuracy: {:.4}'.format(\n", " accuracy_score(y_validation, model.predict(X_validation))\n", "))\n", "print('')\n", "\n", "print('Early-stopped model tree count: {}'.format(earlystop_model.tree_count_))\n", "print('Early-stopped model validation accuracy: {:.4}'.format(\n", " accuracy_score(y_validation, earlystop_model.predict(X_validation))\n", "))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "So we get better quality in a shorter time.\n", "\n", "Though as was shown earlier simple validation scheme does not precisely describes model out-of-train score (may be biased because of dataset split) it is still nice to track model improvement dynamics - and thereby as we can see from this example it is really good to stop boosting process earlier (before the overfitting kicks in)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3.3 Using Baseline\n", "It is posible to use pre-training results (baseline) for training." ] }, { "cell_type": "code", "execution_count": 45, "metadata": {}, "outputs": [], "source": [ "current_params = params.copy()\n", "current_params.update({\n", " 'iterations': 10\n", "})\n", "model = CatBoostClassifier(**current_params).fit(X_train, y_train, categorical_features_indices)\n", "# Get baseline (only with prediction_type='RawFormulaVal')\n", "baseline = model.predict(X_train, prediction_type='RawFormulaVal')\n", "# Fit new model\n", "model.fit(X_train, y_train, categorical_features_indices, baseline=baseline);" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3.4 Snapshot Support\n", "Catboost supports snapshots. You can use it for recovering training after an interruption or for starting training with previous results. " ] }, { "cell_type": "code", "execution_count": 46, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0:\tlearn: 0.8053892\ttest: 0.7937220\tbest: 0.7937220 (0)\ttotal: 1.39ms\tremaining: 5.54ms\n", "1:\tlearn: 0.8008982\ttest: 0.7982063\tbest: 0.7982063 (1)\ttotal: 3ms\tremaining: 4.5ms\n", "2:\tlearn: 0.8008982\ttest: 0.7937220\tbest: 0.7982063 (1)\ttotal: 4.13ms\tremaining: 2.75ms\n", "3:\tlearn: 0.8113772\ttest: 0.7892377\tbest: 0.7982063 (1)\ttotal: 5.52ms\tremaining: 1.38ms\n", "4:\tlearn: 0.8173653\ttest: 0.8026906\tbest: 0.8026906 (4)\ttotal: 6.67ms\tremaining: 0us\n", "\n", "bestTest = 0.802690583\n", "bestIteration = 4\n", "\n", "5:\tlearn: 0.8173653\ttest: 0.8026906\tbest: 0.8026906 (4)\ttotal: 8.14ms\tremaining: 5.89ms\n", "6:\tlearn: 0.8248503\ttest: 0.8026906\tbest: 0.8026906 (4)\ttotal: 9.64ms\tremaining: 4.46ms\n", "7:\tlearn: 0.8233533\ttest: 0.8026906\tbest: 0.8026906 (4)\ttotal: 10.8ms\tremaining: 2.76ms\n", "8:\tlearn: 0.8233533\ttest: 0.8026906\tbest: 0.8026906 (4)\ttotal: 11.4ms\tremaining: 1.19ms\n", "9:\tlearn: 0.8233533\ttest: 0.8026906\tbest: 0.8026906 (4)\ttotal: 12.7ms\tremaining: 0us\n", "\n", "bestTest = 0.802690583\n", "bestIteration = 4\n", "\n" ] } ], "source": [ "params_with_snapshot = params.copy()\n", "params_with_snapshot.update({\n", " 'iterations': 5,\n", " 'learning_rate': 0.5,\n", " 'logging_level': 'Verbose'\n", "})\n", "model = CatBoostClassifier(**params_with_snapshot).fit(train_pool, eval_set=validate_pool, save_snapshot=True)\n", "params_with_snapshot.update({\n", " 'iterations': 10,\n", " 'learning_rate': 0.1,\n", "})\n", "model = CatBoostClassifier(**params_with_snapshot).fit(train_pool, eval_set=validate_pool, save_snapshot=True)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3.5 User Defined Objective Function\n", "It is possible to create your own objective function. Let's create logloss objective function." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# for performance reasons it is better to install `numba` package for working with user defined functions\n", "!pip install numba" ] }, { "cell_type": "code", "execution_count": 49, "metadata": {}, "outputs": [], "source": [ "class LoglossObjective(object):\n", " def calc_ders_range(self, approxes, targets, weights):\n", " # approxes, targets, weights are indexed containers of floats\n", " # (containers which have only __len__ and __getitem__ defined).\n", " # weights parameter can be None.\n", " #\n", " # To understand what these parameters mean, assume that there is\n", " # a subset of your dataset that is currently being processed.\n", " # approxes contains current predictions for this subset,\n", " # targets contains target values you provided with the dataset.\n", " #\n", " # This function should return a list of pairs (der1, der2), where\n", " # der1 is the first derivative of the loss function with respect\n", " # to the predicted value, and der2 is the second derivative.\n", " #\n", " # In our case, logloss is defined by the following formula:\n", " # target * log(sigmoid(approx)) + (1 - target) * (1 - sigmoid(approx))\n", " # where sigmoid(x) = 1 / (1 + e^(-x)).\n", " \n", " assert len(approxes) == len(targets)\n", " if weights is not None:\n", " assert len(weights) == len(approxes)\n", " \n", " result = []\n", " for index in range(len(targets)):\n", " e = np.exp(approxes[index])\n", " p = e / (1 + e)\n", " der1 = (1 - p) if targets[index] > 0.0 else -p\n", " der2 = -p * (1 - p)\n", "\n", " if weights is not None:\n", " der1 *= weights[index]\n", " der2 *= weights[index]\n", "\n", " result.append((der1, der2))\n", " return result" ] }, { "cell_type": "code", "execution_count": 52, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0:\tlearn: 0.6827074\ttotal: 16.7ms\tremaining: 150ms\n", "1:\tlearn: 0.6723302\ttotal: 30.8ms\tremaining: 123ms\n", "2:\tlearn: 0.6619449\ttotal: 43.3ms\tremaining: 101ms\n", "3:\tlearn: 0.6521466\ttotal: 60.3ms\tremaining: 90.5ms\n", "4:\tlearn: 0.6435227\ttotal: 73.4ms\tremaining: 73.4ms\n", "5:\tlearn: 0.6353848\ttotal: 87.8ms\tremaining: 58.5ms\n", "6:\tlearn: 0.6277210\ttotal: 99.6ms\tremaining: 42.7ms\n", "7:\tlearn: 0.6210282\ttotal: 111ms\tremaining: 27.8ms\n", "8:\tlearn: 0.6141958\ttotal: 123ms\tremaining: 13.6ms\n", "9:\tlearn: 0.6073236\ttotal: 135ms\tremaining: 0us\n" ] } ], "source": [ "model = CatBoostClassifier(\n", " iterations=10,\n", " random_seed=42, \n", " loss_function=LoglossObjective(), \n", " eval_metric=metrics.Logloss()\n", ")\n", "# Fit model\n", "model.fit(train_pool)\n", "# Only prediction_type='RawFormulaVal' is allowed with custom `loss_function`\n", "preds_raw = model.predict(X_test, prediction_type='RawFormulaVal')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3.6 User Defined Metric Function\n", "Also it is possible to create your own metric function. Let's create logloss metric function." ] }, { "cell_type": "code", "execution_count": 53, "metadata": {}, "outputs": [], "source": [ "class LoglossMetric(object):\n", " def get_final_error(self, error, weight):\n", " return error / (weight + 1e-38)\n", "\n", " def is_max_optimal(self):\n", " return False\n", "\n", " def evaluate(self, approxes, target, weight):\n", " # approxes is a list of indexed containers\n", " # (containers with only __len__ and __getitem__ defined),\n", " # one container per approx dimension.\n", " # Each container contains floats.\n", " # weight is a one dimensional indexed container.\n", " # target is float.\n", " \n", " # weight parameter can be None.\n", " # Returns pair (error, weights sum)\n", " \n", " assert len(approxes) == 1\n", " assert len(target) == len(approxes[0])\n", "\n", " approx = approxes[0]\n", "\n", " error_sum = 0.0\n", " weight_sum = 0.0\n", "\n", " for i in range(len(approx)):\n", " w = 1.0 if weight is None else weight[i]\n", " weight_sum += w\n", " error_sum += -w * (target[i] * approx[i] - np.log(1 + np.exp(approx[i])))\n", "\n", " return error_sum, weight_sum" ] }, { "cell_type": "code", "execution_count": 54, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Learning rate set to 0.5\n", "0:\tlearn: 0.5521578\ttotal: 6.34ms\tremaining: 57.1ms\n", "1:\tlearn: 0.4885686\ttotal: 11.9ms\tremaining: 47.5ms\n", "2:\tlearn: 0.4607664\ttotal: 17.4ms\tremaining: 40.5ms\n", "3:\tlearn: 0.4418819\ttotal: 22.6ms\tremaining: 33.9ms\n", "4:\tlearn: 0.4278162\ttotal: 28.4ms\tremaining: 28.4ms\n", "5:\tlearn: 0.4151036\ttotal: 35.1ms\tremaining: 23.4ms\n", "6:\tlearn: 0.4099336\ttotal: 40.6ms\tremaining: 17.4ms\n", "7:\tlearn: 0.4095363\ttotal: 45.8ms\tremaining: 11.5ms\n", "8:\tlearn: 0.4032867\ttotal: 51.3ms\tremaining: 5.7ms\n", "9:\tlearn: 0.3929586\ttotal: 56.6ms\tremaining: 0us\n" ] } ], "source": [ "model = CatBoostClassifier(\n", " iterations=10,\n", " random_seed=42, \n", " loss_function=metrics.Logloss(),\n", " eval_metric=LoglossMetric()\n", ")\n", "# Fit model\n", "model.fit(train_pool)\n", "# Only prediction_type='RawFormulaVal' is allowed with custom `loss_function`\n", "preds_raw = model.predict(X_test, prediction_type='RawFormulaVal')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3.7 Staged Predict\n", "CatBoost model has `staged_predict` method. It allows you to iteratively get predictions for a given range of trees." ] }, { "cell_type": "code", "execution_count": 55, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "First class probabilities using the first 3 trees: [0.53597869 0.41039128 0.42057479 0.64281031 0.46576685]\n", "First class probabilities using the first 5 trees: [0.63722688 0.42492029 0.46209302 0.70926021 0.44280772]\n", "First class probabilities using the first 7 trees: [0.66964764 0.42409144 0.46124982 0.76101033 0.47205986]\n" ] } ], "source": [ "model = CatBoostClassifier(iterations=10, random_seed=42, logging_level='Silent').fit(train_pool)\n", "ntree_start, ntree_end, eval_period = 3, 9, 2\n", "predictions_iterator = model.staged_predict(validate_pool, 'Probability', ntree_start, ntree_end, eval_period)\n", "for preds, tree_count in zip(predictions_iterator, range(ntree_start, ntree_end, eval_period)):\n", " print('First class probabilities using the first {} trees: {}'.format(tree_count, preds[:5, 1]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3.8 Feature Importances\n", "Sometimes it is very important to understand which feature made the greatest contribution to the final result. To do this, the CatBoost model has a `get_feature_importance` method." ] }, { "cell_type": "code", "execution_count": 56, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Sex: 59.0040920142686\n", "Pclass: 16.340887169747038\n", "Ticket: 6.028107169932206\n", "Cabin: 3.8347242202560192\n", "Fare: 3.712969667934385\n", "Age: 3.4844512041824824\n", "Parch: 3.378089740355865\n", "Embarked: 2.313999407289956\n", "SibSp: 1.902679406033451\n", "PassengerId: 0.0\n", "Name: 0.0\n" ] } ], "source": [ "model = CatBoostClassifier(iterations=50, random_seed=42, logging_level='Silent').fit(train_pool)\n", "feature_importances = model.get_feature_importance(train_pool)\n", "feature_names = X_train.columns\n", "for score, name in sorted(zip(feature_importances, feature_names), reverse=True):\n", " print('{}: {}'.format(name, score))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This shows that features **`Sex`** and **`Pclass`** had the biggest influence on the result." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3.9 Eval Metrics\n", "The CatBoost has a `eval_metrics` method that allows to calculate a given metrics on a given dataset. And to draw them of course:)" ] }, { "cell_type": "code", "execution_count": 57, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "f1e9bb188e6c48b6bc168ae5d6ebed2f", "version_major": 2, "version_minor": 0 }, "text/plain": [ "MetricVisualizer(layout=Layout(align_self='stretch', height='500px'))" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "model = CatBoostClassifier(iterations=50, random_seed=42, logging_level='Silent').fit(train_pool)\n", "eval_metrics = model.eval_metrics(validate_pool, [metrics.AUC()], plot=True)" ] }, { "cell_type": "code", "execution_count": 58, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[0.8627368774106994, 0.8623176253563642, 0.8602213650846889, 0.8514170719436525, 0.8495723629045783, 0.8569092738554419]\n" ] } ], "source": [ "print(eval_metrics['AUC'][:6])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3.10 Learning Processes Comparison\n", "You can also compare different models learning process on a single plot." ] }, { "cell_type": "code", "execution_count": 61, "metadata": {}, "outputs": [], "source": [ "model1 = CatBoostClassifier(iterations=100, depth=1, train_dir='model_depth_1/', logging_level='Silent')\n", "model1.fit(train_pool, eval_set=validate_pool)\n", "model2 = CatBoostClassifier(iterations=100, depth=5, train_dir='model_depth_5/', logging_level='Silent')\n", "model2.fit(train_pool, eval_set=validate_pool);" ] }, { "cell_type": "code", "execution_count": 62, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "36ab5a837f5d429cbd054a35d762da00", "version_major": 2, "version_minor": 0 }, "text/plain": [ "MetricVisualizer(layout=Layout(align_self='stretch', height='500px'))" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from catboost import MetricVisualizer\n", "widget = MetricVisualizer(['model_depth_1', 'model_depth_5'])\n", "widget.start()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3.11 Model Saving\n", "It is always really handy to be able to dump your model to disk (especially if training took some time)." ] }, { "cell_type": "code", "execution_count": 63, "metadata": {}, "outputs": [], "source": [ "model = CatBoostClassifier(iterations=10, random_seed=42, logging_level='Silent').fit(train_pool)\n", "model.save_model('catboost_model.dump')\n", "model = CatBoostClassifier()\n", "model.load_model('catboost_model.dump');" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# $$4.\\ Parameters\\ Tuning$$\n", "While you could always select optimal number of iterations (boosting steps) by cross-validation and learning curve plots, it is also important to play with some of model parameters, and we would like to pay some special attention to `l2_leaf_reg` and `learning_rate`.\n", "\n", "In this section, we'll select these parameters using the **`hyperopt`** package." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!pip install hyperopt" ] }, { "cell_type": "code", "execution_count": 76, "metadata": {}, "outputs": [], "source": [ "import hyperopt\n", "\n", "def hyperopt_objective(params):\n", " model = CatBoostClassifier(\n", " l2_leaf_reg=int(params['l2_leaf_reg']),\n", " learning_rate=params['learning_rate'],\n", " iterations=500,\n", " eval_metric=metrics.Accuracy(),\n", " random_seed=42,\n", " verbose=False,\n", " loss_function=metrics.Logloss(),\n", " )\n", " \n", " cv_data = cv(\n", " Pool(X, y, cat_features=categorical_features_indices),\n", " model.get_params(),\n", " logging_level='Silent',\n", " )\n", " best_accuracy = np.max(cv_data['test-Accuracy-mean'])\n", " \n", " return 1 - best_accuracy # as hyperopt minimises" ] }, { "cell_type": "code", "execution_count": 78, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [03:59<00:00, 4.79s/trial, best loss: 0.16386083052749711]\n", "{'l2_leaf_reg': 1.0, 'learning_rate': 0.0450866712211308}\n" ] } ], "source": [ "from numpy.random import default_rng\n", "\n", "params_space = {\n", " 'l2_leaf_reg': hyperopt.hp.qloguniform('l2_leaf_reg', 0, 2, 1),\n", " 'learning_rate': hyperopt.hp.uniform('learning_rate', 1e-3, 5e-1),\n", "}\n", "\n", "trials = hyperopt.Trials()\n", "\n", "best = hyperopt.fmin(\n", " hyperopt_objective,\n", " space=params_space,\n", " algo=hyperopt.tpe.suggest,\n", " max_evals=50,\n", " trials=trials,\n", " rstate=default_rng(123)\n", ")\n", "\n", "print(best)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now let's get all cv data with best parameters:" ] }, { "cell_type": "code", "execution_count": 79, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Training on fold [0/3]\n", "\n", "bestTest = 0.8417508418\n", "bestIteration = 262\n", "\n", "Training on fold [1/3]\n", "\n", "bestTest = 0.8451178451\n", "bestIteration = 269\n", "\n", "Training on fold [2/3]\n", "\n", "bestTest = 0.8215488215\n", "bestIteration = 284\n", "\n" ] } ], "source": [ "model = CatBoostClassifier(\n", " l2_leaf_reg=int(best['l2_leaf_reg']),\n", " learning_rate=best['learning_rate'],\n", " iterations=500,\n", " eval_metric=metrics.Accuracy(),\n", " random_seed=42,\n", " verbose=False,\n", " loss_function=metrics.Logloss(),\n", ")\n", "cv_data = cv(Pool(X, y, cat_features=categorical_features_indices), model.get_params())" ] }, { "cell_type": "code", "execution_count": 80, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Precise validation accuracy score: 0.8361391694725029\n" ] } ], "source": [ "print('Precise validation accuracy score: {}'.format(np.max(cv_data['test-Accuracy-mean'])))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Recall that with default parameters out cv score was 0.8283, and thereby we have (probably not statistically significant) some improvement." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Make submission\n", "Now we would re-train our tuned model on all train data that we have" ] }, { "cell_type": "code", "execution_count": 81, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "" ] }, "execution_count": 81, "metadata": {}, "output_type": "execute_result" } ], "source": [ "model.fit(X, y, cat_features=categorical_features_indices)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And finally let's prepare the submission file:" ] }, { "cell_type": "code", "execution_count": 82, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "submisstion = pd.DataFrame()\n", "submisstion['PassengerId'] = X_test['PassengerId']\n", "submisstion['Survived'] = model.predict(X_test)" ] }, { "cell_type": "code", "execution_count": 83, "metadata": {}, "outputs": [], "source": [ "submisstion.to_csv('submission.csv', index=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Finally you can make submission at [Titanic Kaggle competition](https://www.kaggle.com/c/titanic).\n", "\n", "That's it! Now you can play around with CatBoost and win some competitions! :)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.7" }, "widgets": { "state": { "c26d03b66add4e078d26695cab837033": { "views": [ { "cell_index": 21 } ] } }, "version": "1.2.0" } }, "nbformat": 4, "nbformat_minor": 4 }