Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

GPTQ › ActOrder: Activation-Based Column Ordering

Michael BrenndoerferJanuary 13, 20264 min read

Language AI HandbookAll chapters

Part 42 of 67Inference Optimization

Chapter 320GPTQ › ActOrder: Activation-Based Column Ordering

Reading section 13 of 17 · ActOrder: Activation-Based Column Ordering · 4 minSection 13 of 17 · 4 min

Section 13 of 17 · 4 min read

GPTQ › ActOrder: Activation-Based Column OrderingOne section · Sections in this chapter13 / 17 ↓

Sections in this chapter17

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The order in which GPTQ processes columns affects the final quantization error. The default left-to-right order treats all weights equally, but weights differ substantially in their importance as measured by the Hessian diagonal. Processing a highly important weight late in the sequence, after many others have already been quantized and their compensation capacity used up, is suboptimal. That important weight receives compensation from fewer remaining weights than it would have if processed earlier.

ActOrder (activation ordering) addresses this by processing columns in decreasing order of their Hessian diagonal values $H_{qq}$ . The intuition is simple: weights with larger $H_{qq}$ have greater impact on the output and should be quantized first, when the maximum number of other weights are still available to absorb compensation. As important weights are quantized early, their errors are distributed among many remaining weights, each absorbing a small fraction. Less important weights are quantized later, when fewer candidates remain for compensation, but this matters less because their errors have smaller impact on the output anyway.

Think of ActOrder as a surgeon strategy for high-stakes operations. A surgical team handles the most necessary, time-sensitive procedures first, while the available support staff is at its fullest. Routine procedures are handled later, when some staff may have moved on, because the consequences of imperfect execution are smaller.

In[20]:

Code

def get_actorder_permutation(H: torch.Tensor) -> torch.Tensor:
    """
    Compute column ordering based on Hessian diagonal.

    Returns permutation that processes largest diagonal elements first.
    """
    diag = torch.diag(H)
    perm = torch.argsort(diag, descending=True)
    return perm

In[21]:

Code

# Demonstrate ActOrder on our sample Hessian
perm = get_actorder_permutation(H)
first_10 = perm[:10].tolist()
last_10 = perm[-10:].tolist()

# Compare Hessian diagonal values
diag_important = H[perm[0], perm[0]]
diag_least = H[perm[-1], perm[-1]]

Out[22]:

Console

First 10 columns to process (most important): [38, 191, 192, 22, 200, 179, 198, 114, 162, 186]
Last 10 columns to process (least important): [62, 67, 208, 61, 78, 81, 232, 222, 230, 168]

Hessian diagonal for most important column: 1183.9625
Hessian diagonal for least important column: 876.6682

The Hessian diagonal values reveal that the "important" columns have much higher sensitivity (larger values) than the least important ones. Processing these sensitive columns first minimizes the accumulation of error in the output.

Out[23]:

Visualization

model.safetensors: reconstructing file:   0%|          |  0.00B /  251MB

model.safetensors: downloading bytes:           |  0.00B

Log-scale line chart of sorted Hessian diagonal values showing a steep initial decline followed by a long low tail across weight indices. — Hessian diagonal values sorted in descending order, plotted on a logarithmic scale. The steep initial drop followed by a long tail is characteristic of realistic transformer activations, where a small fraction of input dimensions carry most of the weight sensitivity. ActOrder exploits this structure by quantizing high-sensitivity weights first.

ActOrder typically improves perplexity by 0.1 to 0.5 points at 4-bit precision, with the benefit more pronounced for smaller models where each individual weight matters more. For large models like LLaMA-65B, the baseline GPTQ without ActOrder already performs well, and the marginal improvement from ActOrder is smaller. For 7B and 13B models, ActOrder is often worth enabling.