Data Handling and Visualization › Performance Optimization
Quantitative FinanceAll chapters Data Handling and Visualization › Performance OptimizationOne section · Sections in this chapter5 / 9 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Pandas and NumPy are highly optimized, but certain operations become bottlenecks with large datasets or real-time systems. Knowing when and how to optimize matters.
NumPy: The Foundation
Pandas commonly stores homogeneous numeric blocks in NumPy arrays. Vectorization applies operations to arrays instead of looping through Python objects. Direct NumPy can reduce labeled-data overhead for suitable homogeneous kernels, but conversion costs, memory layout, dtype, and alternative engines can change the result. Profile the actual workload.
# Extract NumPy arrays from DataFrame
prices_array = multi_prices.values
dates_array = multi_prices.index.values
# Vectorized calculation avoids a Python-level element loop
returns_numpy = np.diff(prices_array, axis=0) / prices_array[:-1]Array shape: (251, 3) First few returns: [[-0.0252544 -0.01601185 -0.03591687] [ 0.01319998 0.02386114 0.01413042] [ 0.01600667 0.00812855 0.01706702]]
One fewer return observation exists per security because returns require two prices. The first few returns form a matrix with one column per simulated security. This direct NumPy expression omits pandas labels and alignment. It may reduce overhead for this homogeneous calculation, but .values can copy data and performance must be measured rather than assumed.
Numba: JIT Compilation for Custom Functions
Numba specializes and JIT-compiles a supported subset of Python and NumPy. Suitable numeric kernels can approach compiled-language performance after warm-up, but support and speed are workload-dependent.
When standard NumPy operations do not express custom loop logic clearly, Numba can JIT-compile supported Python and NumPy operations after inferring concrete types. Unsupported Python features can fail compilation or require code changes.
from numba import jit
@jit(nopython=True)
def calculate_drawdowns(prices):
r"""Calculate drawdown series from price array.
Drawdown measures the percentage decline from a historical peak. At time $t$, the drawdown is computed as:
$$
DD_t = \frac{P_t - \text{Peak}_t}{\text{Peak}_t}
$$
where:
- $DD_t$: drawdown at time $t$ (a non-positive value where more negative numbers indicate larger percentage declines from the peak)
- $P_t$: current price at time $t$
- $\text{Peak}_t$: maximum price observed from the start of the series up to and including time $t$ (where $\text{Peak}_t = \max_{s \leq t} P_s$)
The drawdown is zero when the price equals the within-sample running high, possibly tying an earlier high, and becomes negative when the price falls below the peak. More negative values indicate larger declines. For example, a drawdown of $-0.20$ means the current price is 20% below its historical high. If the peak was $100 and the current price is $80: the drawdown is $(80-100)/100 = -0.20$.
"""
# Float64 arithmetic prevents unsigned integer subtraction from wrapping.
prices = prices.astype(np.float64)
n = len(prices)
if n == 0:
raise ValueError("prices must be nonempty")
for price in prices:
if not np.isfinite(price) or price <= 0:
raise ValueError("prices must contain only finite positive values")
drawdowns = np.empty(n)
peak = prices[0]
for i in range(n):
if prices[i] > peak:
peak = prices[i]
drawdowns[i] = (prices[i] - peak) / peak
return drawdowns
# Apply to our price data
aapl_prices = multi_prices["AAPL"].values
drawdowns = calculate_drawdowns(aapl_prices)Maximum drawdown: -35.88% Current drawdown: -16.08%
The kernels take one-dimensional NumPy arrays and use float64 arithmetic. Casting integer inputs prevents unsigned subtraction from wrapping, but integers that differ beyond float64 precision may become indistinguishable. The minimum drawdown is the largest observed peak-to-trough decline within the supplied sample. The current drawdown measures distance below the running high since the sample began; 0% means the last observation equals that within-sample high, not necessarily an all-time market high. The @jit(nopython=True) decorator requires operations and types that Numba can compile without Python-object fallback. It supports more than only arrays and scalars, but not arbitrary Python semantics.


Let's compare performance between a pure Python implementation and the Numba version:
import time
def calculate_drawdowns_python(prices):
"""Pure Python implementation for comparison."""
prices = prices.astype(np.float64)
n = len(prices)
if n == 0:
raise ValueError("prices must be nonempty")
for price in prices:
if not np.isfinite(price) or price <= 0:
raise ValueError("prices must contain only finite positive values")
drawdowns = np.empty(n)
peak = prices[0]
for i in range(n):
if prices[i] > peak:
peak = prices[i]
drawdowns[i] = (prices[i] - peak) / peak
return drawdowns
# Separate illustrative workload: Gaussian simple-return shocks,
# not the Student-t log-shock data used in the earlier analysis.
large_prices = np.cumprod(1 + np.random.randn(100_000) * 0.02) * 100
# Warm this input signature; compilation cost is excluded from timed calls.
numba_result = calculate_drawdowns(large_prices)
python_result = calculate_drawdowns_python(large_prices)
np.testing.assert_allclose(numba_result, python_result, rtol=1e-12, atol=1e-15)
boundary_prices = np.array([100, 80, 120, 60], dtype=np.uint64)
expected_drawdowns = np.array([0.0, -0.2, 0.0, -0.5])
for kernel in (calculate_drawdowns_python, calculate_drawdowns):
np.testing.assert_allclose(kernel(boundary_prices), expected_drawdowns)
# Time both implementations
start = time.perf_counter()
for _ in range(100):
_ = calculate_drawdowns_python(large_prices)
python_time = time.perf_counter() - start
start = time.perf_counter()
for _ in range(100):
_ = calculate_drawdowns(large_prices)
numba_time = time.perf_counter() - startPure Python: 5.148 seconds Numba JIT: 0.024 seconds Speedup: 213.1x
For this warmed numeric loop in the current environment, the printed multiplier compares 100 calls to the pure-Python and compiled implementations. It is an illustrative local benchmark, not a general Numba guarantee: compilation support, data layout, hardware, versions, and the kernel itself affect both absolute time and speedup. Production decisions require repeated benchmarks in the target environment.
Cython: When You Need More Control
Cython offers another route by translating annotated Python-like code to C. Compared with Numba's type-specialized JIT workflow, Cython can be useful when a project needs explicit C declarations, extension modules, or direct C-library integration. The appropriate tool depends on the kernel and deployment requirements.
# Cython code (would be in a .pyx file)
# cimport numpy as np
# import numpy as np
#
# def calculate_drawdowns_cython(np.ndarray[np.float64_t, ndim=1] prices):
# cdef int n = prices.shape[0]
# cdef int i
# cdef np.ndarray[np.float64_t, ndim=1] drawdowns
# cdef double peak
#
# if n == 0:
# raise ValueError("prices must be nonempty")
# for i in range(n):
# if not np.isfinite(prices[i]) or prices[i] <= 0:
# raise ValueError("prices must contain only finite positive values")
# drawdowns = np.empty(n)
# peak = prices[0]
#
# for i in range(n):
# if prices[i] > peak:
# peak = prices[i]
# drawdowns[i] = (prices[i] - peak) / peak
#
# return drawdownsStart with pandas and NumPy. After profiling, test Numba for supported numeric kernels and Cython when explicit C integration or extension-module control is useful. Benchmark the candidate implementation under the target workload.
When to Optimize
Follow this approach to optimization:
- Write clear code first using pandas and NumPy
- Profile to identify actual bottlenecks with tools like %timeit or cProfile
- Optimize measured hot paths that dominate runtime
- Verify correctness after optimization
Judge optimization against the workload's latency and throughput budget. A 50 ms versus 5 ms difference may be irrelevant for one interactive calculation but material when repeated across a large parameter search. In a live system, insufficient throughput or excessive latency can produce stale decisions or missed service targets; the trading consequence depends on the strategy and execution design.