Code Similarity for Understanding Programming Learning: Surface VS. Conceptual Insights

Abstract

This study investigates how code similarity can be used to understand programming learning behaviors. Rather than focusing on predictive performance, machine learning is employed as an analytical tool to examine how different representations of code reflect learners’ understanding. Using a code clone detection dataset, three types of representations are analyzed: lexical features (TF-IDF), semantic embeddings (CodeBERT), and their combination. The results show that semantic representations outperform lexical features, indicating that surface-level similarity is insufficient to capture deeper conceptual relationships. While feature fusion provides limited gains in linear models, non-linear models achieve better performance, suggesting that programming competence involves complex interactions across multiple aspects of code. Further analysis using confusion matrices reveals a tendency to overestimate similarity, with more false positives than false negatives. This pattern suggests that syntactic similarity may reflect surface imitation rather than true understanding. In contrast, false negative cases highlight semantically equivalent solutions expressed in different ways, indicating conceptual understanding with diverse implementations. Error analysis further shows that discrepancies between lexical and semantic predictions correspond to different learning behaviors. Overall, the findings suggest that code similarity can serve as a lens for distinguishing between surface imitation and conceptual understanding in programming education, highlighting the value of machine learning as a tool for interpreting student learning processes rather than merely improving prediction.