Data Loading

To learn how to do machine learning we’re going to need some data to work with. To facilitate learning and experimentation, scikit-learn includes a datasets module containing a number of widely-used toy datasets. Here’s how we could load the (in)famous Iris dataset:

from sklearn import datasets

# Load a dictionary (technically, a Bunch) containing the data
iris = datasets.load_iris()

# 'data' and 'target' contains the feature data and classes, respectively
X, y = iris['data'], iris['target']

X contains feature information for 150 individual Iris flowers drawn from 3 different species. y contains the true class information for all flowers. If we want to inspect the features in a tabular form, we can easily load the data into a pandas DataFrame:

# Here we're importing the pandas package, which we'll use extensively
# for data manipulation. In future sections, we'll put the core imports
# at the top of the notebook, which is the convention in Python.
import pandas as pd

# Initialize a new pandas DataFrame from the X matrix and the feature names
data = pd.DataFrame(X, columns=iris['feature_names'])
data.head()

In principle, we could use the iris dataset (or one of the other datasets bundled with scikit-learn) for many of the examples we’ll work through. But the iris dataset has some limitations—most notably, it’s fairly small (only 150 rows and 4 features), and has nothing to do with real world applications. Instead, we’ll use data that should be of interest to many individuals: house pricing dataset and time-series stock prices. The housing price dataset consists of various house features along with the sales price of the home. The time-series stock price datasets will be harvested from Yahoo finance.

We will first make use of the house pricing dataset to learn the basics of machine learning. We’ll use pandas—the reference data analysis library in Python—to do this. Pandas provides us with a fairly magical read_csv function that can read in almost any kind of tabular data.

# read_csv is a workhorse function that can read almost any kind of
# plain-text format. The returned object is a pandas DataFrame.

all_data = pd.read_csv('data/house_prices.csv', sep=',', index_col=0).reset_index(drop=True)

Representing the data

Once the data have been read in, we can take a look at the first few rows:

# head() display the first few rows of the dataset.
all_data.head()

As we can see, the data are tabular. Every row represents a different house, and every column represents a different variable. In machine learning terminology, we typically refer to the rows and columns as samples and features, respectively. We can thus think of our data as a two-dimensional n (samples) x p (features) matrix. The vast majority of algorithms implemented in the scikit-learn and keras packages expect to receive numerical matrices of this kind as their primary inputs. (Note that some of the columns in our dataset—e.g., “MSZoning” and “LotShape”—contains strings or categorical values, so we need to pre-process these columns). One option would be to recode these columns into a numerical form before we could make proper use of them by defining different levels. The other option would be to just remove them. Since we have 80 features, we will just remove them for now. The original dataset consists of 80 columns and 1460 samples.