Tags: pandas, numpy, python, data-analysis, arrays, dataframes

Level: intermediate

TL;DR: Core Python libraries for high-performance array manipulation (NumPy) and tabular data processing (Pandas). Essential for data manipulation, aggregation, and analysis pipelines.

Key concepts

NumPy ndarray is a multidimensional, homogeneous array offering fast vectorized operations.

Pandas DataFrame is a 2D tabular data structure with labeled row and column axes.

Pandas Series is a 1D labeled array capable of holding any data type.

Vectorization eliminates explicit loops, applying operations over arrays for C-level performance.

Broadcasting allows NumPy to perform operations on arrays of different shapes automatically.

Pandas handles missing data natively using standard NaN representations.

Indexes in Pandas provide fast lookups and alignment of data across operations.

Typical questions and ready answers

Q: What is the main difference between a NumPy array and a Python list?A: NumPy arrays are contiguous in memory and contain elements of the same type, making them significantly faster and more memory-efficient.

Q: How do loc and iloc differ in Pandas?A: I use loc for label-based indexing to reference explicit row or column names, and iloc for standard zero-indexed positional mapping.

Q: Why use Pandas over raw NumPy for tabular data?A: Pandas offers labeled columns, heterogeneous data types within the same table, and built-in features for joining and grouping.

Pitfalls and common mistakes

Iterating over DataFrame rows using iterrows() instead of leveraging fast vectorized operations.

Modifying a slice of a DataFrame directly without using .copy(), which triggers a SettingWithCopyWarning.

Ignoring the axis parameter, accidentally applying operations across rows (axis=1) instead of columns (axis=0).

Failing to handle or impute missing values before applying strict mathematical functions in NumPy.

Snippets and commandsPython

```import numpy as np

import pandas as pd

NumPy: create array and reshape

arr = np.arange(10).reshape(2, 5)

Pandas: create DataFrame, filter, and group

df = pd.DataFrame({'A': ['foo', 'bar', 'foo'], 'B': [1, 2, 3]})

filtered_df = df[df['B'] > 1]

grouped_df = df.groupby('A').sum()

```

To review

[ ] Advanced NumPy broadcasting rules

[ ] Pandas window functions and rolling aggregations