Data Handling and Visualization › Performance Optimization

Michael BrenndoerferNovember 7, 20258 min read
Quantitative FinanceAll chapters
Part 1 of 15Mathematical Foundations
Chapter 9Data Handling and Visualization › Performance Optimization
Reading section 5 of 9 · Performance Optimization · 8 minSection 5 of 9 · 8 min

Book contents

Section 5 of 9 · 8 min read

Data Handling and Visualization › Performance OptimizationOne section · Sections in this chapter5 / 9 ↓
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Pandas and NumPy are highly optimized, but certain operations become bottlenecks with large datasets or real-time systems. Knowing when and how to optimize matters.

NumPy: The Foundation

Pandas commonly stores homogeneous numeric blocks in NumPy arrays. Vectorization applies operations to arrays instead of looping through Python objects. Direct NumPy can reduce labeled-data overhead for suitable homogeneous kernels, but conversion costs, memory layout, dtype, and alternative engines can change the result. Profile the actual workload.

In[32]:
Code
# Extract NumPy arrays from DataFrame
prices_array = multi_prices.values
dates_array = multi_prices.index.values

# Vectorized calculation avoids a Python-level element loop
returns_numpy = np.diff(prices_array, axis=0) / prices_array[:-1]
Out[33]:
Console
Array shape: (251, 3)
First few returns:
[[-0.0252544  -0.01601185 -0.03591687]
 [ 0.01319998  0.02386114  0.01413042]
 [ 0.01600667  0.00812855  0.01706702]]

One fewer return observation exists per security because returns require two prices. The first few returns form a matrix with one column per simulated security. This direct NumPy expression omits pandas labels and alignment. It may reduce overhead for this homogeneous calculation, but .values can copy data and performance must be measured rather than assumed.

Numba: JIT Compilation for Custom Functions

Just-In-Time (JIT) Compilation

Numba specializes and JIT-compiles a supported subset of Python and NumPy. Suitable numeric kernels can approach compiled-language performance after warm-up, but support and speed are workload-dependent.

When standard NumPy operations do not express custom loop logic clearly, Numba can JIT-compile supported Python and NumPy operations after inferring concrete types. Unsupported Python features can fail compilation or require code changes.

In[34]:
Code
from numba import jit


@jit(nopython=True)
def calculate_drawdowns(prices):
    r"""Calculate drawdown series from price array.

    Drawdown measures the percentage decline from a historical peak. At time $t$, the drawdown is computed as:

    $$
    DD_t = \frac{P_t - \text{Peak}_t}{\text{Peak}_t}
    $$

    where:

    - $DD_t$: drawdown at time $t$ (a non-positive value where more negative numbers indicate larger percentage declines from the peak)
    - $P_t$: current price at time $t$
    - $\text{Peak}_t$: maximum price observed from the start of the series up to and including time $t$ (where $\text{Peak}_t = \max_{s \leq t} P_s$)

    The drawdown is zero when the price equals the within-sample running high, possibly tying an earlier high, and becomes negative when the price falls below the peak. More negative values indicate larger declines. For example, a drawdown of $-0.20$ means the current price is 20% below its historical high. If the peak was $100 and the current price is $80: the drawdown is $(80-100)/100 = -0.20$.
    """
    # Float64 arithmetic prevents unsigned integer subtraction from wrapping.
    prices = prices.astype(np.float64)
    n = len(prices)
    if n == 0:
        raise ValueError("prices must be nonempty")
    for price in prices:
        if not np.isfinite(price) or price <= 0:
            raise ValueError("prices must contain only finite positive values")
    drawdowns = np.empty(n)
    peak = prices[0]

    for i in range(n):
        if prices[i] > peak:
            peak = prices[i]
        drawdowns[i] = (prices[i] - peak) / peak

    return drawdowns


# Apply to our price data
aapl_prices = multi_prices["AAPL"].values
drawdowns = calculate_drawdowns(aapl_prices)
Out[35]:
Console
Maximum drawdown: -35.88%
Current drawdown: -16.08%

The kernels take one-dimensional NumPy arrays and use float64 arithmetic. Casting integer inputs prevents unsigned subtraction from wrapping, but integers that differ beyond float64 precision may become indistinguishable. The minimum drawdown is the largest observed peak-to-trough decline within the supplied sample. The current drawdown measures distance below the running high since the sample began; 0% means the last observation equals that within-sample high, not necessarily an all-time market high. The @jit(nopython=True) decorator requires operations and types that Numba can compile without Python-object fallback. It supports more than only arrays and scalars, but not arbitrary Python semantics.

Out[36]:
Visualization
Line chart of AAPL price and running peak over time.
Simulated AAPL-labelled price series over 252 weekday observations with its within-sample running peak (dashed). Running highs correspond to zero drawdown in the companion panel.
Area and line chart of AAPL drawdown percentage over time with a marker for maximum drawdown.
Simulated drawdown series measured from the within-sample running peak. The marked point is the largest observed peak-to-trough decline in this sample, not a worst-case loss bound.

Let's compare performance between a pure Python implementation and the Numba version:

In[37]:
Code
import time


def calculate_drawdowns_python(prices):
    """Pure Python implementation for comparison."""
    prices = prices.astype(np.float64)
    n = len(prices)
    if n == 0:
        raise ValueError("prices must be nonempty")
    for price in prices:
        if not np.isfinite(price) or price <= 0:
            raise ValueError("prices must contain only finite positive values")
    drawdowns = np.empty(n)
    peak = prices[0]

    for i in range(n):
        if prices[i] > peak:
            peak = prices[i]
        drawdowns[i] = (prices[i] - peak) / peak

    return drawdowns


# Separate illustrative workload: Gaussian simple-return shocks,
# not the Student-t log-shock data used in the earlier analysis.
large_prices = np.cumprod(1 + np.random.randn(100_000) * 0.02) * 100

# Warm this input signature; compilation cost is excluded from timed calls.
numba_result = calculate_drawdowns(large_prices)
python_result = calculate_drawdowns_python(large_prices)
np.testing.assert_allclose(numba_result, python_result, rtol=1e-12, atol=1e-15)
boundary_prices = np.array([100, 80, 120, 60], dtype=np.uint64)
expected_drawdowns = np.array([0.0, -0.2, 0.0, -0.5])
for kernel in (calculate_drawdowns_python, calculate_drawdowns):
    np.testing.assert_allclose(kernel(boundary_prices), expected_drawdowns)

# Time both implementations
start = time.perf_counter()
for _ in range(100):
    _ = calculate_drawdowns_python(large_prices)
python_time = time.perf_counter() - start

start = time.perf_counter()
for _ in range(100):
    _ = calculate_drawdowns(large_prices)
numba_time = time.perf_counter() - start
Out[38]:
Console
Pure Python: 5.148 seconds
Numba JIT:   0.024 seconds
Speedup:     213.1x

For this warmed numeric loop in the current environment, the printed multiplier compares 100 calls to the pure-Python and compiled implementations. It is an illustrative local benchmark, not a general Numba guarantee: compilation support, data layout, hardware, versions, and the kernel itself affect both absolute time and speedup. Production decisions require repeated benchmarks in the target environment.

Cython: When You Need More Control

Cython offers another route by translating annotated Python-like code to C. Compared with Numba's type-specialized JIT workflow, Cython can be useful when a project needs explicit C declarations, extension modules, or direct C-library integration. The appropriate tool depends on the kernel and deployment requirements.

In[13]:
Code
# Cython code (would be in a .pyx file)
# cimport numpy as np
# import numpy as np
#
# def calculate_drawdowns_cython(np.ndarray[np.float64_t, ndim=1] prices):
#     cdef int n = prices.shape[0]
#     cdef int i
#     cdef np.ndarray[np.float64_t, ndim=1] drawdowns
#     cdef double peak
#
#     if n == 0:
#         raise ValueError("prices must be nonempty")
#     for i in range(n):
#         if not np.isfinite(prices[i]) or prices[i] <= 0:
#             raise ValueError("prices must contain only finite positive values")
#     drawdowns = np.empty(n)
#     peak = prices[0]
#
#     for i in range(n):
#         if prices[i] > peak:
#             peak = prices[i]
#         drawdowns[i] = (prices[i] - peak) / peak
#
#     return drawdowns

Start with pandas and NumPy. After profiling, test Numba for supported numeric kernels and Cython when explicit C integration or extension-module control is useful. Benchmark the candidate implementation under the target workload.

When to Optimize

Follow this approach to optimization:

  1. Write clear code first using pandas and NumPy
  2. Profile to identify actual bottlenecks with tools like %timeit or cProfile
  3. Optimize measured hot paths that dominate runtime
  4. Verify correctness after optimization

Judge optimization against the workload's latency and throughput budget. A 50 ms versus 5 ms difference may be irrelevant for one interactive calculation but material when repeated across a large parameter search. In a live system, insufficient throughput or excessive latency can produce stale decisions or missed service targets; the trading consequence depends on the strategy and execution design.