Python Train/Test

Last modified on: October 19, 2024

<h2>Python&nbsp;Train/Test - How to train a <strong>linear regression model</strong> on the training data</h2>
<div class="group/conversation-turn relative flex w-full min-w-0 flex-col agent-turn">
<div class="flex-col gap-1 md:gap-3">
<div class="flex max-w-full flex-col flex-grow">
<div class="min-h-[20px] text-message flex w-full flex-col items-end gap-2 whitespace-normal break-words [.text-message+&amp;]:mt-5" dir="auto" data-message-author-role="assistant" data-message-id="a600d094-27ba-4e2e-9e57-014b4ad72857">
<div class="flex w-full flex-col gap-1 empty:hidden first:pt-[3px]">
<div class="markdown prose w-full break-words dark:prose-invert light">
<p>In machine learning, splitting your data into <strong>training</strong> and <strong>testing</strong> sets is crucial for evaluating a model's performance on unseen data. The <strong>training set</strong> is used to train the model, while the <strong>test set</strong> is used to evaluate how well the model generalizes to new data.</p>
<p>Python&rsquo;s <code>scikit-learn</code> library makes it easy to split data into training and test sets using the <code>train_test_split</code> function.</p>
<hr />
<h2><strong>Tutorial: Train/Test Split in Python</strong></h2>
<h3><strong>1. Import Required Libraries</strong></h3>
<div class="dark bg-gray-950 contain-inline-size rounded-md border-[0.5px] border-token-border-medium relative">
<div class="flex items-center text-token-text-secondary bg-token-main-surface-secondary px-4 py-2 text-xs font-sans justify-between rounded-t-md h-9">
<pre class="language-python"><code>import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score</code></pre>
</div>
</div>
<ul>
<li><code>train_test_split</code>: For splitting data into training and test sets.</li>
<li><code>LinearRegression</code>: A simple linear regression model for demonstration.</li>
<li><code>mean_squared_error</code>, <code>r2_score</code>: Metrics to evaluate model performance.</li>
</ul>
<hr />
<h3><strong>2. Generate Sample Data</strong></h3>
<p>Let's generate some random data to demonstrate how to split the dataset.</p>
<div class="dark bg-gray-950 contain-inline-size rounded-md border-[0.5px] border-token-border-medium relative">
<div class="flex items-center text-token-text-secondary bg-token-main-surface-secondary px-4 py-2 text-xs font-sans justify-between rounded-t-md h-9">
<pre class="language-python"><code># Generating random data
np.random.seed(0)  # For reproducibility
x = 2 * np.random.rand(100, 1)
y = 4 + 3 * x + np.random.randn(100, 1)  # y = 4 + 3x + noise

print("Feature X:\n", x[:5])  # Displaying first 5 rows
print("Target Y:\n", y[:5])</code></pre>
</div>
</div>
<p>Here, we create a simple linear relationship <span class="katex"><span class="katex-mathml">y=4+3x+noisey = 4 + 3x + \text{noise}</span><span class="katex-html" aria-hidden="true"><span class="base"><span class="mord mathnormal">y</span><span class="mrel">=</span></span><span class="base"><span class="mord">4</span><span class="mbin">+</span></span><span class="base"><span class="mord">3</span><span class="mord mathnormal">x</span><span class="mbin">+</span></span><span class="base"><span class="mord text"><span class="mord">noise</span></span></span></span></span>.</p>
<hr />
<h3><strong>3. Splitting Data into Training and Test Sets</strong></h3>
<p>We split the data into <strong>training</strong> and <strong>testing</strong> sets using the <code>train_test_split()</code> function. Typically, the split is 70-80% for training and 20-30% for testing.</p>
<div class="dark bg-gray-950 contain-inline-size rounded-md border-[0.5px] border-token-border-medium relative">
<div class="flex items-center text-token-text-secondary bg-token-main-surface-secondary px-4 py-2 text-xs font-sans justify-between rounded-t-md h-9">
<pre class="language-python"><code># Split data into 80% training and 20% test sets
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2, random_state=42)

print("Training Feature Set Size:", x_train.shape)
print("Test Feature Set Size:", x_test.shape)</code></pre>
</div>
</div>
<ul>
<li><code>test_size=0.2</code>: Reserves 20% of the data for testing.</li>
<li><code>random_state=42</code>: Ensures reproducibility of the split.</li>
</ul>
<hr />
<h3><strong>4. Training a Model</strong></h3>
<p>We&rsquo;ll train a <strong>linear regression</strong> model using the training data.</p>
<div class="dark bg-gray-950 contain-inline-size rounded-md border-[0.5px] border-token-border-medium relative">
<div class="flex items-center text-token-text-secondary bg-token-main-surface-secondary px-4 py-2 text-xs font-sans justify-between rounded-t-md h-9">
<pre class="language-python"><code># Initialize and train the linear regression model
model = LinearRegression()
model.fit(x_train, y_train)

# Predict on the training set
y_train_pred = model.predict(x_train)

# Predict on the test set
y_test_pred = model.predict(x_test)</code></pre>
</div>
</div>
<p>Here, the model is trained on <code>x_train</code> and <code>y_train</code>. After training, we predict the target values for both the training and test sets.</p>
<hr />
<h3><strong>5. Evaluating the Model</strong></h3>
<p>Now, let's evaluate the model&rsquo;s performance on both the training and test data using metrics like <strong>Mean Squared Error (MSE)</strong> and <strong>R-squared (R&sup2;)</strong>.</p>
<h4><strong>Performance on Training Set:</strong></h4>
<div class="dark bg-gray-950 contain-inline-size rounded-md border-[0.5px] border-token-border-medium relative">
<div class="flex items-center text-token-text-secondary bg-token-main-surface-secondary px-4 py-2 text-xs font-sans justify-between rounded-t-md h-9">
<pre class="language-python"><code># Evaluate on training data
mse_train = mean_squared_error(y_train, y_train_pred)
r2_train = r2_score(y_train, y_train_pred)

print(f"Training MSE: {mse_train}")
print(f"Training R-squared: {r2_train}")</code></pre>
</div>
</div>
<h4><strong>Performance on Test Set:</strong></h4>
<div class="dark bg-gray-950 contain-inline-size rounded-md border-[0.5px] border-token-border-medium relative">
<div class="flex items-center text-token-text-secondary bg-token-main-surface-secondary px-4 py-2 text-xs font-sans justify-between rounded-t-md h-9">
<pre class="language-python"><code># Evaluate on test data
mse_test = mean_squared_error(y_test, y_test_pred)
r2_test = r2_score(y_test, y_test_pred)

print(f"Test MSE: {mse_test}")
print(f"Test R-squared: {r2_test}")</code></pre>
</div>
</div>
<p>Comparing the performance on the training and test sets helps you understand if the model is <strong>overfitting</strong> (performing well on training but poorly on test) or <strong>underfitting</strong> (performing poorly on both training and test sets).</p>
<hr />
<h3><strong>6. Visualizing the Results</strong></h3>
<p>Let&rsquo;s plot the original data along with the predictions made by the model.</p>
<div class="dark bg-gray-950 contain-inline-size rounded-md border-[0.5px] border-token-border-medium relative">
<div class="flex items-center text-token-text-secondary bg-token-main-surface-secondary px-4 py-2 text-xs font-sans justify-between rounded-t-md h-9">
<pre class="language-python"><code>import matplotlib.pyplot as plt

# Plotting training data and the fitted line
plt.scatter(x_train, y_train, color='blue', label='Training data')
plt.plot(x_train, y_train_pred, color='red', label='Fitted line (train)')
plt.title("Linear Regression on Training Data")
plt.xlabel("X")
plt.ylabel("Y")
plt.legend()
plt.show()

# Plotting test data and the fitted line
plt.scatter(x_test, y_test, color='green', label='Test data')
plt.plot(x_test, y_test_pred, color='orange', label='Fitted line (test)')
plt.title("Linear Regression on Test Data")
plt.xlabel("X")
plt.ylabel("Y")
plt.legend()
plt.show()</code></pre>
</div>
</div>
<p>This visualization helps you see how well the model fits both the training and test data.</p>
<hr />
<h3><strong>7. Overfitting vs. Underfitting</strong></h3>
<ul>
<li><strong>Overfitting</strong> occurs when the model performs very well on the training data but poorly on the test data, meaning the model has "memorized" the training data and lacks generalization to unseen data.</li>
<li><strong>Underfitting</strong> occurs when the model performs poorly on both training and test data, meaning the model is too simple to capture the underlying patterns in the data.</li>
</ul>
<p>You can detect these problems by comparing the <strong>MSE</strong> and <strong>R&sup2;</strong> scores between the training and test sets.</p>
<hr />
<h3><strong>Conclusion</strong></h3>
<p>In this tutorial, we covered:</p>
<ol>
<li>How to split data into <strong>training</strong> and <strong>test</strong> sets using <code>train_test_split</code>.</li>
<li>How to train a <strong>linear regression model</strong> on the training data.</li>
<li>How to <strong>evaluate</strong> the model on both training and test data using <strong>MSE</strong> and <strong>R-squared</strong>.</li>
<li>The importance of avoiding <strong>overfitting</strong> and <strong>underfitting</strong>.</li>
</ol>
<p>Proper train/test splitting ensures that your model's performance is evaluated on unseen data, providing a better indication of how well it will generalize in real-world applications.</p>
</div>
</div>
</div>
</div>
</div>
</div>

Web Technology View More