Academic Project Page

“I’m not sure, but.. . ”: Expert Practices that Enable Effective Code Comprehension in Data Science

University of California San Diego
SIGCSE TS 2025
^*Indicates Equal Contribution

Abstract

Data Scientists often read and understand messy and undocumented code that relies on large software libraries. What makes data science experts more effective than novices at this task? To understand expert practices, we conduct a think-aloud study where 4 novice and 5 expert data scientists reasoned about an unfamiliar data analysis scripts with realistic complexity that used the `pandas` library. Our research identified three key practices that distinguish experts from novices. First, experts treat domain knowledge and data atrributes as a critical part of their analysisi, building analysis and interpretation based on this context. Second, they made just much assumptions as novices but always follow up by verifying these assumptions using the dataset and program output. Finally, experts approach codebase with clear goal and interact with the parts that are highly relevant to their goals.

Study Workflow

We conduct the same experiment to all particpants regardless of their level of experts. A pre-written data analytical workbook was being provided to the particiapnts. The workbook was divided into 4 parts and all participants have 10 mins to understand the code in each task independently. Follow up each independent analysis, participants were being presented with a series of itnerview quesiton along with survey rating their performance.

After all the user studies, we asked 2 synonymous data scientsits evaluated particpants performance based on a common rubric :

Analysis and Results

Based on user's self-report score and evaluation scored made by the 2 synonymous data scientsits, expert data scientists genearlly out-perform novices. To further investiage the different practices between novices and experts, we recorded and transcribed participants’ responses for deeper analysis. From their dialogues, we extracted quotes, paraphrased their language, and categorized their responses.

We were able to identify three key trends:

Understand the data as well as the context

Guess and Check mindset

Goal-oriented exploration