pandas: Python Data Analysis and Manipulation Library

What pandas is

pandas is a Python package that provides fast, flexible and expressive data structures for working with "relational" or "labeled" data. Its README describes the goal as being the fundamental high-level building block for practical, real-world data analysis in Python, with the broader ambition of becoming the most flexible open-source data analysis and manipulation tool available in any language. The code lives at github.com/pandas-dev/pandas under the BSD-3-Clause licence, and the documentation is hosted at pandas.pydata.org. The README shows the project is powered by NumFOCUS.

According to the README, work on pandas started in 2008 at AQR, a quantitative hedge fund, and the project has been developed by open-source contributors ever since. The package overview lists the kinds of data it is built for: tabular data with heterogeneously typed columns (as in an SQL table or an Excel spreadsheet), ordered and unordered time series that need not have a fixed frequency, arbitrary matrix data with row and column labels, and any other observational or statistical data set.

The audience is broad: analysts exploring data, engineers writing ETL scripts and pipelines, researchers, and library authors who need a labeled table type. For R users, the docs state that the DataFrame provides everything R's data.frame provides and more.

How pandas works

Series and DataFrame

pandas has two primary data structures. A Series is a one-dimensional labeled array that holds values of a single type. A DataFrame is a general two-dimensional labeled, size-mutable table whose columns can each have a different type. Axes carry semantic meaning, so with tabular data it is more natural to think in terms of the index (rows) and the columns than axis 0 and axis 1.

pandas is built on top of NumPy and is meant to integrate with the wider scientific Python stack. The docs say many of its low-level algorithms have been heavily tuned in Cython, while also cautioning that generalization costs performance, so a tool specialized for one task may be faster.

Mutability and Copy-on-Write

All pandas structures are value-mutable, but not always size-mutable: a Series cannot change length, while columns can be added to a DataFrame. Since pandas 3.0 this is formalised by Copy-on-Write (CoW), which is now the default and only mode. Under CoW, the result of any indexing operation or any method that returns a new DataFrame or Series always behaves as a copy, so the only way to modify an object is to modify that object directly. Internally pandas still uses views where it can and defers the actual copy until one of the objects that share data is modified.

Pluggable storage: NumPy, nullable and PyArrow

Columns are not limited to NumPy arrays. Through pandas' ExtensionArray interface, a Series, Index or DataFrame column can be backed by a pyarrow.ChunkedArray. The PyArrow guide lists the benefits as more extensive data types than NumPy, missing-value (NA) support for every type, faster I/O readers, and interoperability with other dataframe libraries built on the Apache Arrow specification, naming polars and cuDF. Operations on Arrow-backed columns are accelerated with PyArrow compute functions where available, covering numeric aggregations and arithmetic, rounding, logical and comparison functions, string functions and datetime functions.

Key features

• Missing data handling. Missing values are represented as NaN, NA or NaT and handled in floating point as well as non-floating point data; most operations exclude missing values by default.
• Automatic and explicit data alignment. Objects can be aligned to a set of labels explicitly, or pandas aligns Series and DataFrames automatically during computations.
• Group by. Split-apply-combine operations for both aggregating and transforming data, including grouping by several columns into a hierarchical result.
• Indexing and selection. Label-based slicing, fancy indexing and boolean subsetting of large data sets, plus hierarchical (MultiIndex) labeling of axes.
• Time series. Date range generation, frequency conversion and resampling, moving window statistics, time zone localization and conversion, and date shifting and lagging.
• Wide I/O coverage. The I/O guide lists readers and/or writers for CSV, JSON, HTML, XML, Excel, HDF5, Feather, Parquet, Apache Iceberg, ORC, Stata, SAS, SPSS, pickle, SQL and more.
• Multiple CSV engines. read_csv offers a default C engine, a slower Python engine that supports regex separators and skipfooter, and a PyArrow engine that the docs describe as fastest on larger workloads.
• Expressions with pd.col(). Introduced in 3.0, pd.col() builds column expressions that can replace lambda functions in places such as DataFrame.assign, DataFrame.loc and getitem/setitem.
• Optional performance engines. numexpr and bottleneck accelerate certain numerical and nan operations, and numba provides a JIT engine for methods that accept engine="numba".

Getting started

The pandas team officially distributes pandas through conda-forge, PyPI and GitHub (source). The install guide notes that pandas may be installable from other places, such as the Anaconda distribution, but those are not managed by the pandas development team. With pip:

pip install pandas

With conda, from the conda-forge channel, ideally inside a dedicated environment:

conda install -c conda-forge pandas
conda create -c conda-forge -n name_of_my_env python pandas
conda activate name_of_my_env

Optional functionality ships as pip extras. For example, to add the libraries needed to read Excel files:

pip install "pandas[excel]"

Other extras include performance, plot, computation, parquet, feather, iceberg, postgresql, mysql, aws, gcp and all. The install guide strongly encourages the performance extras (numexpr, bottleneck and numba) for large data sets.

The 10-minute guide starts with the customary imports and simple object creation:

import numpy as np
import pandas as pd
s = pd.Series([1, 3, 5, np.nan, 6, 8])
dates = pd.date_range("20130101", periods=6)
df = pd.DataFrame(np.random.randn(6, 4), index=dates, columns=list("ABCD"))
df.head()

Grouping follows the split-apply-combine pattern:

df = pd.DataFrame(
    {
        "A": ["foo", "bar", "foo", "bar", "foo", "bar", "foo", "foo"],
        "B": ["one", "one", "two", "three", "two", "two", "one", "three"],
        "C": np.random.randn(8),
        "D": np.random.randn(8),
    }
)
df.groupby("A")[["C", "D"]].sum()
df.groupby(["A", "B"]).sum()

SQL-style joins use merge:

left = pd.DataFrame({"key": ["foo", "bar"], "lval": [1, 2]})
right = pd.DataFrame({"key": ["foo", "bar"], "rval": [4, 5]})
pd.merge(left, right, on="key")

Reading and writing files is a matter of matching read_* functions and to_* methods. Parquet support needs pyarrow (the default engine) or fastparquet:

df.to_csv("foo.csv")
pd.read_csv("foo.csv")
df.to_parquet("foo.parquet")
pd.read_parquet("foo.parquet")

Any reader can return PyArrow-backed columns with dtype_backend="pyarrow" (here data is any path or buffer read_csv accepts), and individual columns can be created with Arrow types directly:

ser = pd.Series([-1.5, 0.2, None], dtype="float32[pyarrow]")
df_pyarrow = pd.read_csv(data, dtype_backend="pyarrow")

What changed in pandas 3.x

The release notes index lists the 3.0 series (released January 21, 2026) and the 3.1 series. pandas 3.1.0 was released on October 22, 2026 and 3.1.1 on November 4, 2026. At the time of writing the 3.1.2 notes are still dated "November XX, 2026" and the 3.2.0 notes are a template dated "Month XX, 2027", so both are in development rather than released.

pandas 3.0 was a major release that removed a lot of previously deprecated functionality. Its main changes:

• A dedicated string dtype by default. String data is now inferred as the new str dtype instead of NumPy object. It is backed by PyArrow if installed, otherwise by NumPy object storage, can only hold strings or missing values, and always uses NaN as its missing-value sentinel.
• Copy-on-Write everywhere. Every subset or returned object behaves as a copy, chained assignment no longer works, SettingWithCopyWarning is gone, and defensive .copy() calls are no longer needed.
• pd.col() expressions as a shorter alternative to lambdas, for example df.assign(c = pd.col('a') + pd.col('b')).
• Datetime resolution inference. Conversions to datetime64/timedelta64 no longer always produce nanosecond resolution; parsing strings now defaults to microseconds, falling back to nanoseconds when the precision requires it.
• Consistent NaN handling in nullable dtypes. For pyarrow and NumPy-nullable float dtypes, NaN is now treated as equivalent to NA in all cases.
• Python 3.11 and higher are required.

pandas 3.1.0 added DataFrame.select, which returns the specified columns and can compute new ones from pd.col() expressions or callables, and a cond keyword for DataFrame.filter and Series.filter to filter with a boolean mask. It also reworked read_csv: the notes say a single-threaded read with the default C engine is typically two to three times faster than in 3.0, and large files are now read in parallel. Parallel reads apply automatically to local, uncompressed files whose data rows total at least 5 MB, as long as no option requires parsing the file as a whole (such as chunksize, nrows or parse_dates). The mode.max_threads option (default: CPU cores available to the process, capped at 6) controls the thread count; set it to 1 inside applications that already parallelise work:

with pd.option_context("mode.max_threads", 1):
    df = pd.read_csv("large.csv")

Use cases

• Exploratory data analysis. Load a CSV, Excel workbook or Parquet file, inspect it with head(), describe() and dtypes, clean missing values, and summarize with group by and pivot tables.
• ETL and data preparation. Read from SQL databases (through SQLAlchemy or ADBC drivers), files or cloud object storage (through fsspec, s3fs and gcsfs), join and reshape the data, then write it out as Parquet, Iceberg or back to SQL.
• Time series work. Resample second-level data into five-minute buckets, localize and convert time zones, apply business-day offsets and compute rolling statistics; the docs note resampling is extremely common in financial applications.
• Feeding other libraries. pandas is a dependency of statsmodels, and since 3.0 DataFrames can be exchanged with Arrow-based libraries through the Arrow PyCapsule Interface.

How pandas compares

The docs make two direct comparisons. Against NumPy, pandas adds labeled axes, per-column dtypes and the data-wrangling operations above. Against R, the overview positions the DataFrame as a superset of what R's data.frame offers.

The PyArrow guide names polars and cuDF as other dataframe libraries based on the Apache Arrow specification, and pandas' Arrow-backed dtypes and PyCapsule support are aimed at moving data between them. The scale guide also acknowledges that other libraries offer pandas-like APIs with parallel runtimes, distributed memory or clustering, and points to the ecosystem page for options rather than naming a recommendation. Benchmark against your own workloads before switching either way.

Things to know before adopting

• It is an in-memory library. The scale guide states that pandas provides data structures for in-memory analytics, which makes analysing larger-than-memory data sets tricky, and that even data sets that are a sizable fraction of memory become unwieldy because some operations make intermediate copies. Its advice: load only the columns you need (for example columns= with read_parquet or usecols with read_csv), use efficient dtypes such as category for low-cardinality text and downcast numerics with to_numeric (the guide's example cuts memory to a fifth), and process data in chunks when each chunk fits in memory. Chunking works when chunks need little coordination; operations such as groupby are much harder to do chunkwise, and for those the guide suggests switching to a library that implements out-of-core algorithms.
• Chained assignment is gone. Under Copy-on-Write, code like df["foo"][df["bar"] > 5] = 100 never updates df and raises a ChainedAssignmentError warning. Updating a column selected from a DataFrame with an inplace=True method is another form of chained assignment and also stops working. Use loc instead:

  df = pd.DataFrame({"foo": [1, 2, 3], "bar": [4, 5, 6]})
  df.loc[df["bar"] > 5, "foo"] = 100

• Other CoW side effects. to_numpy() on a Series, or on a DataFrame backed by a single array, returns a read-only view; copy it, or set arr.flags.writeable = True at your own risk. Series and DataFrame constructors now copy NumPy arrays by default (pass copy=False to avoid it). The mode.copy_on_write option has no effect, is deprecated and will be removed in pandas 4.0.
• The string dtype breaks some assumptions. Checks like ser.dtype == "object" return False for string columns in 3.x, the dtype can no longer be passed to NumPy functions, None is coerced to NaN, and assigning a non-string value raises TypeError. For code that must run on both 2.x and 3.x, the migration guide recommends pd.api.types.is_string_dtype(ser.dtype) and pd.isna(). A small number of Unicode case-mapping edge cases differ between pyarrow and Python storage (for example uppercasing "ß"), so whether pyarrow is installed can change results.
• Upgrade path. The 3.0 notes recommend upgrading to pandas 2.3 first and making code run without warnings before moving to 3.0. In 2.3, the new string dtype can be tested with pd.options.future.infer_string = True.
• Behaviour changes can arrive in minor releases. For example, in 3.1.0 read_csv with dtype="category" now infers numeric and boolean categories on all engines, so zero-padded identifiers such as account numbers or zip codes lose their leading zeros unless read as str first.
• Optional dependencies are opt-in. Many methods need extra packages (for example read_hdf needs PyTables and to_markdown needs tabulate) and raise ImportError when they are missing. The install guide warns that read_orc may fail with pyarrow installed from PyPI and is not compatible with Windows, and lists the xlrd and pyxlsb Excel readers as deprecated in favour of python-calamine.
• Versioning and deprecation policy. pandas uses a loose variant of semantic versioning: API-breaking changes only in major releases, deprecations introduced in minor releases and enforced in the next major. Following PDEP-17, a new deprecation first emits a PandasDeprecationWarning (hidden by default outside __main__), is promoted to a visible PandasFutureWarning in the final minor release before the next major, and is then removed. Behaviour-changing bug fixes can still ship in minor or patch releases, and features marked experimental are exempt from the policy.
• Python and dependency support. Python version support mirrors the Scientific Python SPEC 0 guideline; pandas 3.0.0 supports Python 3.11 and higher. The current install guide lists NumPy 2.0.2 and python-dateutil 2.9.0 as minimum required versions, with tzdata needed only on Windows and Pyodide.
• Security of some formats. The I/O guide warns that loading pickled data from untrusted sources can be unsafe, and that read_pickle is only guaranteed to read pickles from the current or previous major version (in 3.x, back to 2.0.0).

Project activity

As of November 2026 the repository has about 50,500 stars and more than 20,700 forks. It was created in August 2010, is written mainly in Python and is licensed under BSD-3-Clause. The release notes show a steady cadence: 3.0.0 in January 2026, 3.1.0 in October 2026 and 3.1.1 in early November 2026, with 3.1.2 and 3.2.0 in progress on main.

Source code and issues are at github.com/pandas-dev/pandas, and the documentation, user guide and release notes are at pandas.pydata.org. Security vulnerabilities are reported through the repository's security policy.