Embedding Evaluation › Evaluation Report
Language AI HandbookAll chapters Embedding Evaluation › Evaluation ReportOne section · Sections in this chapter9 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
A professional embedding evaluation doesn't rely on any single metric. Good practice is to run all available evaluations and report them together, creating a complete picture of what the embeddings do well and where they struggle.
def comprehensive_evaluation_report(
embeddings, similarity_data, analogy_data, downstream_scores
):
"""
Compile a comprehensive evaluation report for a set of word embeddings.
"""
# Intrinsic: word similarity
sim_results = evaluate_word_similarity(embeddings, similarity_data)
# Intrinsic: analogy accuracy
analogy_acc, _, n_skipped_analogy = evaluate_analogies(
embeddings, analogy_data
)
# Vocabulary coverage
all_sim_words = set(w for pair in similarity_data for w in pair[:2])
covered = sum(1 for w in all_sim_words if w in embeddings)
coverage = covered / len(all_sim_words) if all_sim_words else 0
report = {
"vocabulary_size": len(embeddings),
"similarity_spearman": sim_results["correlation"],
"similarity_n_pairs": sim_results["n_pairs"],
"similarity_oov_rate": sim_results["n_skipped"] / len(similarity_data),
"analogy_accuracy": analogy_acc,
"analogy_oov_skipped": n_skipped_analogy,
"downstream_cv_mean": np.mean(downstream_scores),
"downstream_cv_std": np.std(downstream_scores),
"benchmark_coverage": coverage,
}
return report
# Generate comprehensive report
report = comprehensive_evaluation_report(
embeddings, similarity_dataset, test_analogies, cv_scores
)Embedding Evaluation Report ======================================================= Intrinsic Evaluation: Word Similarity (Spearman rho): 0.6480 Pairs evaluated: 12 OOV rate (similarity): 0.0% Analogy accuracy: 33.3% Analogy OOV skipped: 0 Extrinsic Evaluation: Downstream task accuracy: 1.000 ± 0.000 Coverage: Vocabulary size: 21 Benchmark word coverage: 100.0% Note: All metrics should be compared against baselines and alternative embedding models, not evaluated in isolation.
Reading this kind of report correctly requires keeping the big picture in mind. Intrinsic metrics tell you about embedding geometry. The downstream metric tells you about task utility. Coverage tells you about breadth. Bias metrics (not included here but essential) tell you about safety and fairness properties. A complete evaluation report presents all of these dimensions and interprets them jointly rather than picking whichever number looks best.