Python K-nearest neighbors

Last modified on: October 20, 2024

<div class="flex-shrink-0 flex flex-col relative items-end">
<div>
<div class="pt-0">
<div class="gizmo-bot-avatar flex h-8 w-8 items-center justify-center overflow-hidden rounded-full">
<div class="relative p-1 rounded-sm flex items-center justify-center bg-token-main-surface-primary text-token-text-primary h-8 w-8">&nbsp;</div>
</div>
</div>
</div>
</div>
<div class="group/conversation-turn relative flex w-full min-w-0 flex-col agent-turn">
<div class="flex-col gap-1 md:gap-3">
<div class="flex max-w-full flex-col flex-grow">
<div class="min-h-[20px] text-message flex w-full flex-col items-end gap-2 whitespace-normal break-words [.text-message+&amp;]:mt-5" dir="auto" data-message-author-role="assistant" data-message-id="e3ccd99e-b1fb-4272-adaa-2a5257785ebb">
<div class="flex w-full flex-col gap-1 empty:hidden first:pt-[3px]">
<div class="markdown prose w-full break-words dark:prose-invert light">
<h3><strong>K-Nearest Neighbors (KNN) in Python</strong></h3>
<p><strong>K-Nearest Neighbors (KNN)</strong> is a simple, intuitive, and effective algorithm used for both classification and regression tasks. It works by finding the <span class="katex"><span class="katex-mathml">KK</span><span class="katex-html" aria-hidden="true"><span class="base"><span class="mord mathnormal">K</span></span></span></span> closest training samples in the feature space to a given input point and making predictions based on those neighbors.</p>
<hr />
<h3><strong>Key Concepts of KNN</strong></h3>
<ul>
<li><strong>Distance Metric</strong>: The most common distance metric used is Euclidean distance, but other metrics (like Manhattan or Minkowski) can also be employed.</li>
<li><strong>Voting</strong>: For classification tasks, KNN predicts the class label based on a majority vote from the <span class="katex"><span class="katex-mathml">KK</span><span class="katex-html" aria-hidden="true"><span class="base"><span class="mord mathnormal">K</span></span></span></span> nearest neighbors. For regression, it averages the values of the nearest neighbors.</li>
</ul>
<h3><strong>Advantages of KNN</strong></h3>
<ul>
<li>Simple and easy to understand.</li>
<li>No training phase; the algorithm is lazy, meaning it only computes distances when making predictions.</li>
</ul>
<h3><strong>Disadvantages of KNN</strong></h3>
<ul>
<li>Computationally intensive during prediction, especially with large datasets.</li>
<li>Sensitive to the scale of data and irrelevant features.</li>
</ul>
<hr />
<h3><strong>Tutorial: Implementing K-Nearest Neighbors in Python</strong></h3>
<p>We will use Python&rsquo;s <code>scikit-learn</code> library to demonstrate KNN for classification on a sample dataset.</p>
<h3><strong>1. Import Required Libraries</strong></h3>
<div class="dark bg-gray-950 contain-inline-size rounded-md border-[0.5px] border-token-border-medium relative">
<div class="flex items-center text-token-text-secondary bg-token-main-surface-secondary px-4 py-2 text-xs font-sans justify-between rounded-t-md h-9">
<pre class="language-python"><code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import accuracy_score, classification_report, confusion_matrix</code></pre>
</div>
</div>
<h3><strong>2. Preparing the Data</strong></h3>
<p>For this example, we will use the <strong>Iris dataset</strong>.</p>
<div class="dark bg-gray-950 contain-inline-size rounded-md border-[0.5px] border-token-border-medium relative">
<div class="flex items-center text-token-text-secondary bg-token-main-surface-secondary px-4 py-2 text-xs font-sans justify-between rounded-t-md h-9">
<pre class="language-python"><code># Load the Iris dataset
iris = load_iris()
X = iris.data  # Features
y = iris.target  # Target variable

# Split the dataset into training and test sets (80% train, 20% test)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)</code></pre>
</div>
</div>
<h3><strong>3. Choosing the Number of Neighbors (K)</strong></h3>
<p>It's important to select the right <span class="katex"><span class="katex-mathml">KK</span><span class="katex-html" aria-hidden="true"><span class="base"><span class="mord mathnormal">K</span></span></span></span>. A common method is to test several values and choose the one with the best performance on the validation set.</p>
<div class="dark bg-gray-950 contain-inline-size rounded-md border-[0.5px] border-token-border-medium relative">
<div class="flex items-center text-token-text-secondary bg-token-main-surface-secondary px-4 py-2 text-xs font-sans justify-between rounded-t-md h-9">
<pre class="language-python"><code># Testing different values of K
k_values = range(1, 11)
scores = []

for k in k_values:
    knn = KNeighborsClassifier(n_neighbors=k)
    knn.fit(X_train, y_train)
    y_pred = knn.predict(X_test)
    scores.append(accuracy_score(y_test, y_pred))

# Plotting the accuracy scores for different K values
plt.figure(figsize=(10, 6))
plt.plot(k_values, scores, marker='o')
plt.title('K-Nearest Neighbors Classifier Accuracy')
plt.xlabel('Number of Neighbors (K)')
plt.ylabel('Accuracy')
plt.xticks(k_values)
plt.grid()
plt.show()</code></pre>
</div>
</div>
<h3><strong>4. Training the KNN Model</strong></h3>
<p>After selecting the optimal <span class="katex"><span class="katex-mathml">KK</span><span class="katex-html" aria-hidden="true"><span class="base"><span class="mord mathnormal">K</span></span></span></span>, we will train the KNN model.</p>
<div class="dark bg-gray-950 contain-inline-size rounded-md border-[0.5px] border-token-border-medium relative">
<div class="flex items-center text-token-text-secondary bg-token-main-surface-secondary px-4 py-2 text-xs font-sans justify-between rounded-t-md h-9">
<pre class="language-python"><code># Choose the best K value (e.g., K = 3 from the plot)
best_k = 3
knn = KNeighborsClassifier(n_neighbors=best_k)

# Fit the model on the training data
knn.fit(X_train, y_train)</code></pre>
</div>
</div>
<h3><strong>5. Making Predictions</strong></h3>
<p>Now, we can make predictions on the test set.</p>
<div class="dark bg-gray-950 contain-inline-size rounded-md border-[0.5px] border-token-border-medium relative">
<div class="flex items-center text-token-text-secondary bg-token-main-surface-secondary px-4 py-2 text-xs font-sans justify-between rounded-t-md h-9">
<pre class="language-python"><code># Predict on the test set
y_pred = knn.predict(X_test)</code></pre>
</div>
</div>
<h3><strong>6. Evaluating the Model</strong></h3>
<p>We can evaluate the model's performance using accuracy, confusion matrix, and classification report.</p>
<div class="dark bg-gray-950 contain-inline-size rounded-md border-[0.5px] border-token-border-medium relative">
<div class="flex items-center text-token-text-secondary bg-token-main-surface-secondary px-4 py-2 text-xs font-sans justify-between rounded-t-md h-9">
<pre class="language-python"><code># Calculate accuracy
accuracy = accuracy_score(y_test, y_pred)
print(f'Accuracy: {accuracy:.2f}')

# Print the confusion matrix
conf_matrix = confusion_matrix(y_test, y_pred)
print("Confusion Matrix:\n", conf_matrix)

# Print the classification report
report = classification_report(y_test, y_pred)
print("Classification Report:\n", report)</code></pre>
</div>
</div>
<h3><strong>Conclusion</strong></h3>
<p>In this tutorial, we:</p>
<ol>
<li>Introduced <strong>K-Nearest Neighbors (KNN)</strong> as a simple yet effective classification algorithm.</li>
<li>Used the <strong>Iris dataset</strong> to demonstrate how to implement KNN.</li>
<li>Explored different values of <span class="katex"><span class="katex-mathml">KK</span><span class="katex-html" aria-hidden="true"><span class="base"><span class="mord mathnormal">K</span></span></span></span> to select the best one based on accuracy.</li>
<li>Evaluated the model using various performance metrics.</li>
</ol>
<p>KNN is a versatile algorithm that can be applied to many real-world problems, especially when the dataset is small to moderate in size. Understanding the impact of <span class="katex"><span class="katex-mathml">KK</span><span class="katex-html" aria-hidden="true"><span class="base"><span class="mord mathnormal">K</span></span></span></span> and feature scaling is crucial for effective use of this algorithm.</p>
</div>
</div>
</div>
</div>
</div>
</div>

Web Technology View More