使用 PyCaret 进行异常检测

Anomaly detection is one of the main tasks in unsupervised machine learning. Its goal is to identify dataset instances that differ significantly from the majority. Those instances are known as outliers. There are various incentives to detect them depending on the context and domain of each application. There are also semi-supervised and fully supervised methods for anomaly detection, but we’ll focus on the unsupervised approach. Local outlier factor is one of the main anomaly detection models defined in the following equation.

$$\text{LOF}_{k}(A)=\frac{\sum_{B\in N_k(A)} \frac{\text{lrd}_{k}(B)}{\text{lrd}_{k}(A)}}{|N_{k}(A)|}$$

• $\text{LOF}_{k}(A)$ is the local outlier factor of the dataset instance $A$.
• $\text{lrd}_{k}(A)$ is the local reachability density of $A$.
• $k$ is the number of nearest neighbors to $A$.
• $N_{k}(A)$ is the set of $k$ nearest neighbors.

The local outlier factor of an instance is the average local reachability density of its neighbors divided by the local reachability of the instance itself. Values that are significantly larger than 1 indicate that the instance is an outlier, while smaller values suggest an inlier.

Anomaly detection using PyCaret

As we mentioned earlier, the local outlier factor is a popular anomaly detection model, but numerous others are available as well: Isolation forest, k-nearest neighbors detector, subspace outlier detection, and clustering-based local outlier. In the rest of this chapter, we’ll see how we can train and plot an anomaly detection model using the PyCaret library.

Importing the necessary libraries

We’ll import the libraries necessary for this project: pandas, Matplotlib, Seaborn, and the PyCaret Anomaly Detection module. We’ll also import the get_data()function to load the dataset of our preference. Finally, we’ll set the Matplotlib figure DPI to 300 to get high-quality images for this course.

1# Importing necessary libraries
2
3import pandas as pd
4import matplotlib.pyplot as plt 
5import matplotlib as mpl
6import seaborn as sns
7from pycaret.datasets import get_data 
8from pycaret.anomaly import * 
9mpl.rcParams['figure.dpi'] = 300

Loading the dataset

This chapter will use the Wholesale Customers dataset provided freely by the UCI Machine Learning Repository. This dataset includes the annual spending of a wholesale distributor’s customers on several product categories. The dataset features 'Fresh', 'Milk', 'Grocery', 'Frozen', 'Detergents_Paper', and 'Delicatessen', constitute the product categories and are all numeric variables. Channel and Region are categorical variables because they refer to the attributes of each customer. The Channel categories include Horeca (Hotel/Restaurant/Cafe) and Retail, while the Region categories are Lisbon, Oporto and Other. We use the get_data() function to load the Wholesale Customers dataset to a pandas dataframe.

1# Loading the dataset
2
3numeric = ['Fresh','Milk','Grocery','Frozen','Detergents_Paper','Delicassen']
4categorical = ['Channel','Region']
5replace_dict = { "Channel": {1: "Horeca", 2: "Retail"},
6                 "Region":  {1: "Lisbon", 2: "Oporto", 3: "Other"} }
7data = get_data('wholesale', verbose = False)
8data.replace(replace_dict, inplace = True) 
9data.head(10)

The categorical variables are encoded as integer numbers, so we’ll restore the original categories with the replace() function. This may seem redundant because the categorical variables will be encoded again during the PyCaret environment initialization to train the anomaly detection model. Regardless, having the original categories available during the “Exploratory Data Analysis” lesson of the chapter will be more informative and helpful. After executing the replace() function, we’ll use the head() pandas function to display the first rows of the dataset.