Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Code Understanding › Code Implementation: Bug Detection

Michael BrenndoerferFebruary 2, 20263 min read

Language AI HandbookAll chapters

Part 47 of 67Code Generation

Chapter 358Code Understanding › Code Implementation: Bug Detection

Reading section 6 of 13 · Code Implementation: Bug Detection · 3 minSection 6 of 13 · 3 min

Section 6 of 13 · 3 min read

Code Understanding › Code Implementation: Bug DetectionOne section · Sections in this chapter6 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let's implement a simple bug detection system using CodeBERT.

In[9]:

Code

from transformers import RobertaForSequenceClassification, RobertaTokenizer

bug_model_name = "microsoft/codebert-base"
bug_tokenizer = RobertaTokenizer.from_pretrained(bug_model_name)
bug_model = RobertaForSequenceClassification.from_pretrained(
    bug_model_name, num_labels=2
)
bug_model.eval()

In[10]:

Code

def detect_bug_score(code_snippet: str) -> float:
    """Return a defect probability score for a code snippet."""
    inputs = bug_tokenizer(
        code_snippet,
        return_tensors="pt",
        max_length=512,
        truncation=True,
        padding=True,
    )
    with torch.no_grad():
        logits = bug_model(**inputs).logits
    probs = torch.softmax(logits, dim=-1)
    return probs[0, 1].item()

In[11]:

Code

code_pairs = [
    {
        "name": "Off-by-one in loop",
        "buggy": """\
def sum_array(arr):
    total = 0
    for i in range(len(arr) + 1):
        total += arr[i]
    return total""",
        "fixed": """\
def sum_array(arr):
    total = 0
    for i in range(len(arr)):
        total += arr[i]
    return total""",
    },
    {
        "name": "Missing null check",
        "buggy": """\
def get_length(s):
    return len(s)""",
        "fixed": """\
def get_length(s):
    if s is None:
        return 0
    return len(s)""",
    },
]

scores = []
for pair in code_pairs:
    buggy_score = detect_bug_score(pair["buggy"])
    fixed_score = detect_bug_score(pair["fixed"])
    scores.append(
        {
            "name": pair["name"],
            "buggy_score": buggy_score,
            "fixed_score": fixed_score,
        }
    )

Out[12]:

Console

Scenario                      Buggy Score  Fixed Score   Correct?
-----------------------------------------------------------------
Off-by-one in loop                 0.6104       0.6094        yes
Missing null check                 0.5974       0.6025         no

The base CodeBERT model without defect-specific fine-tuning assigns similar scores to both variants. In practice, a fine-tuned checkpoint trained on the Devign or CodeXGLUE defect datasets assigns clearly higher scores to the buggy variants. The structure of the pipeline is the same; only the weights change.

Out[13]:

Visualization

Grouped bar chart with patterned buggy and fixed bars near 0.6 for two scenarios; the first pair is nearly tied and the second ranks the fixed variant slightly higher. — Defect scores from the unfine-tuned base CodeBERT classifier for buggy and fixed variants. All four values cluster near 0.6: the off-by-one pair differs by only 0.001, while the missing-null-check pair is ranked in the wrong direction. The near-equal bars show that the base checkpoint provides little useful defect discrimination.

The chart makes the lack of discrimination visible. An effective detector would show clearly taller buggy bars for every scenario, but these gaps are tiny and the missing-null-check pair is reversed. Defect-specific fine-tuning is what turns the pairwise score gap into a useful signal.