Pandas Introduction

Pandas Introduction#

pandas is the standard Python library for working with tabular (structured) data. It provides fast, flexible data structures — primarily Series and DataFrame — that make data manipulation, cleaning, and analysis straightforward.

Tip

If you have ever worked with spreadsheets (Excel, Google Sheets), think of a DataFrame as a programmable spreadsheet.

import pandas as pd

ModuleNotFoundError?

If you get a ModuleNotFoundError: No module named 'pandas', the package is not installed. Install it first:

# With pip
pip install pandas

# With conda (Anaconda)
conda install pandas

Then restart the kernel and run the cell again.

Series vs DataFrame#

A Series is a one-dimensional labeled array — essentially a single column of data with an index.

A DataFrame is a two-dimensional labeled data structure — a table with rows and columns, where each column is a Series.

# A Series
temperatures = pd.Series([22.1, 23.4, 21.8, 24.0], name="Temperature")
temperatures

0    22.1
1    23.4
2    21.8
3    24.0
Name: Temperature, dtype: float64

type(temperatures)

pandas.Series

Creating DataFrames#

DataFrames can be created from Python dictionaries, lists of lists, or many other sources.

From a dictionary#

data = {
    "room": ["Living Room", "Bedroom", "Kitchen", "Office"],
    "temperature": [22.1, 20.5, 23.4, 21.8],
    "humidity": [45.0, 50.2, 38.7, 42.1]
}

df = pd.DataFrame(data)
df
room temperature humidity
0 Living Room 22.1 45.0
1 Bedroom 20.5 50.2
2 Kitchen 23.4 38.7
3 Office 21.8 42.1

From a list of lists#

rows = [
    ["Living Room", 22.1, 45.0],
    ["Bedroom", 20.5, 50.2],
    ["Kitchen", 23.4, 38.7],
    ["Office", 21.8, 42.1]
]

df2 = pd.DataFrame(rows, columns=["room", "temperature", "humidity"])
df2
room temperature humidity
0 Living Room 22.1 45.0
1 Bedroom 20.5 50.2
2 Kitchen 23.4 38.7
3 Office 21.8 42.1

Basic Indexing#

There are several ways to select data from a DataFrame.

Select a single column (returns a Series)#

df["temperature"]

0    22.1
1    20.5
2    23.4
3    21.8
Name: temperature, dtype: float64

Note

Selecting a single column with df["col"] returns a Series.

Select multiple columns (returns a DataFrame)#

df[["room", "temperature"]]
room temperature
0 Living Room 22.1
1 Bedroom 20.5
2 Kitchen 23.4
3 Office 21.8

Note

Selecting multiple columns with df[["col1", "col2"]] returns a DataFrame.

Label-based indexing with .loc[]#

.loc[] selects rows and columns by label (index value and column name).

# Select row with index label 0
df.loc[0]

room           Living Room
temperature           22.1
humidity              45.0
Name: 0, dtype: object

# Select rows 0 to 2, columns "room" and "temperature"
df.loc[0:2, ["room", "temperature"]]
room temperature
0 Living Room 22.1
1 Bedroom 20.5
2 Kitchen 23.4

Position-based indexing with .iloc[]#

.iloc[] selects rows and columns by integer position (like array indexing).

# Select the first two rows, first two columns
df.iloc[0:2, 0:2]
room temperature
0 Living Room 22.1
1 Bedroom 20.5 
# Select a specific cell: row 1, column 2
df.iloc[1, 2]

np.float64(50.2)

Data Types and Info#

Understanding data types is important because certain operations only work with specific types.

df.dtypes

room               str
temperature    float64
humidity       float64
dtype: object

df.info()

<class 'pandas.DataFrame'>
RangeIndex: 4 entries, 0 to 3
Data columns (total 3 columns):
 #   Column       Non-Null Count  Dtype  
---  ------       --------------  -----  
 0   room         4 non-null      str    
 1   temperature  4 non-null      float64
 2   humidity     4 non-null      float64
dtypes: float64(2), str(1)
memory usage: 228.0 bytes

Basic Properties#

# Number of rows and columns
df.shape

(4, 3)

# Column names
df.columns

Index(['room', 'temperature', 'humidity'], dtype='str')

# Index
df.index

RangeIndex(start=0, stop=4, step=1)

Loading a Real CSV File#

Let’s load a real data set to explore. The file flatTempHum.csv contains hourly temperature and humidity measurements from four flats.

Note

The CSV file uses ; as separator, which is common in European data files.

url = "https://raw.githubusercontent.com/retomarek/edap/main/edap/sampleData/flatTempHum.csv"

df = pd.read_csv(url, sep=";")
df.head()
time FlatA_Hum FlatA_Temp FlatB_Hum FlatB_Temp FlatC_Hum FlatC_Temp FlatD_Hum FlatD_Temp
0 2018-10-03 00:00:00 53.0 24.43 38.8 22.40 44.0 24.5 49.0 24.43
1 2018-10-03 01:00:00 53.0 24.40 38.8 22.40 44.0 24.5 49.0 24.40
2 2018-10-03 02:00:00 53.0 24.40 39.3 22.40 44.7 24.5 48.3 24.38
3 2018-10-03 03:00:00 53.0 24.40 40.3 22.40 45.0 24.5 48.0 24.33
4 2018-10-03 04:00:00 53.3 24.40 41.0 22.37 45.2 24.5 47.7 24.30 
df.shape

(17181, 9)

df.dtypes

time              str
FlatA_Hum     float64
FlatA_Temp    float64
FlatB_Hum     float64
FlatB_Temp    float64
FlatC_Hum     float64
FlatC_Temp    float64
FlatD_Hum     float64
FlatD_Temp    float64
dtype: object

Note

The time column is text (dtype object in pandas 2, str in pandas 3). It should be converted to datetime for time-based operations — this is covered in the Datetime Handling chapter.

df.info()

<class 'pandas.DataFrame'>
RangeIndex: 17181 entries, 0 to 17180
Data columns (total 9 columns):
 #   Column      Non-Null Count  Dtype  
---  ------      --------------  -----  
 0   time        17181 non-null  str    
 1   FlatA_Hum   16484 non-null  float64
 2   FlatA_Temp  16484 non-null  float64
 3   FlatB_Hum   16470 non-null  float64
 4   FlatB_Temp  16470 non-null  float64
 5   FlatC_Hum   16497 non-null  float64
 6   FlatC_Temp  16497 non-null  float64
 7   FlatD_Hum   16354 non-null  float64
 8   FlatD_Temp  16354 non-null  float64
dtypes: float64(8), str(1)
memory usage: 1.2 MB

df.describe()
FlatA_Hum FlatA_Temp FlatB_Hum FlatB_Temp FlatC_Hum FlatC_Temp FlatD_Hum FlatD_Temp
count 16484.000000 16484.000000 16470.000000 16470.000000 16497.000000 16497.000000 16354.000000 16354.000000
mean 54.336599 23.786531 49.666697 23.404749 48.895339 23.964698 49.299156 24.134841
std 6.300389 1.273444 6.903822 1.470354 6.487768 1.360014 6.524641 1.308097
min 23.500000 19.100000 25.000000 20.530000 22.500000 21.380000 20.500000 18.120000
25% 51.000000 22.800000 44.700000 22.320000 44.300000 22.880000 44.300000 23.030000
50% 54.800000 23.480000 50.000000 23.000000 49.200000 23.600000 49.000000 23.700000
75% 58.300000 24.600000 54.500000 24.100000 53.800000 24.800000 54.000000 25.200000
max 76.500000 28.880000 73.300000 29.120000 69.800000 28.800000 69.800000 28.030000

What do these statistics mean?

Statistic

Meaning

count

Number of non-missing values

mean

Average value (sum / count)

std

Standard deviation — how spread out the values are

min

Smallest value

25%

25th percentile — 25% of values are below this

50%

Median — the middle value (50th percentile)

75%

75th percentile — 75% of values are below this

max

Largest value

A large difference between mean and 50% (median) indicates skewed data. A high std relative to the mean indicates high variability.

df.columns

Index(['time', 'FlatA_Hum', 'FlatA_Temp', 'FlatB_Hum', 'FlatB_Temp',
       'FlatC_Hum', 'FlatC_Temp', 'FlatD_Hum', 'FlatD_Temp'],
      dtype='str')

# Select only temperature columns
temp_cols = [col for col in df.columns if "Temp" in col]
df[temp_cols].head()
FlatA_Temp FlatB_Temp FlatC_Temp FlatD_Temp
0 24.43 22.40 24.5 24.43
1 24.40 22.40 24.5 24.40
2 24.40 22.40 24.5 24.38
3 24.40 22.40 24.5 24.33
4 24.40 22.37 24.5 24.30 
# Access specific rows and columns
df.loc[0:4, ["time", "FlatA_Temp", "FlatB_Temp"]]
time FlatA_Temp FlatB_Temp
0 2018-10-03 00:00:00 24.43 22.40
1 2018-10-03 01:00:00 24.40 22.40
2 2018-10-03 02:00:00 24.40 22.40
3 2018-10-03 03:00:00 24.40 22.40
4 2018-10-03 04:00:00 24.40 22.37