“Too little and too late” best describes the user-research effort that goes into many failed software projects. This is also true of the very tools programmers and software developers use to create software. My research takes a human-centered approach to understanding the social and cognitive processes of software developers and development teams, to be able to better design tools to support their activities.
My research so far has dealt with programming-related activities, in various programmer populations–professional software developers, novice programmers (e.g., CS students) and end-user programmers. The latter are people who have no formal programming training, but do some form of programming, often without even knowing that that is what they’re doing (e.g., accountants writing spreadsheet formulas and macros, a home owner configuring home automation systems).
Spreadsheet comprehension and errors
If you ask an auditor’s nightmare at work, he will probably tell you that it is spreadsheet errors. Spreadsheets are excellent in that they bring a lot of “goodness” in its formula language. Anyone can quickly get up to speed to write simple formulas, to accomplish tasks, without thinking about data types, compilation and call stack as professional developers do.
But the same flexibility offered by spreadsheets is also its own arch nemesis: spreadsheets are extremely error-prone and can be pretty hard to read and understand and audit (thanks to the lack of structure in them), and people do all kinds of things with them. See these Dilbert strips to get a sense of this side of the world. So, because we’ve left the floodgates open (and in this case it is actually good), we can’t really put any structure into it now without upsetting at least some fraction of the 500 million spreadsheet users on earth. Even 0.01% of them is a lot of people, and some of them love spreadsheets with a vengeance.
So, what am I doing about it? Well, nothing new — a lot of people (auditors, computer scientists, management researchers, engineers) have been thinking about how to errorproof spreadsheets. I just think we have been looking all the while in the wrong places, and mostly focus on the manifestation of these errors (in formulas and data) without trying to get to the root cause of those errors. My hypothesis is that the root causes are things like poor usability and learnability of spreadsheet tools, poor readability of spreadsheets (because it takes a hella lotta time to make spreadsheets pretty, leave alone understandable!), and indiscriminate reuse of spreadsheets, well past their original intent. Mind you, users are not the problem entirely here, they have work to do, they’re not professional spreadsheet developers. So, I want to study these problems and solve them, and see how well they can help reduce spreadsheet errors / poorly built spreadsheets.
I just finished a study of how people comprehend their coworkers’ spreadsheets on their job, and found way more problems than I’d have ever imagined. Think about it–the world’s largest population is using pointer referencing as the primary referencing mechanism in their program (what else is the A1 and $AA2034 in spreadsheet formulas then?!), and then scrolling back and forth to see what the hell that meant? That’s just the simplest of them! And by the time someone scrolls back, they have already forgotten what they wanted to look for, and then that led to errors (welcome, 15123 instead of 15213).
Anyway, so I’m now prototyping experiences to improve spreadsheet comprehension. I am also surveying all the papers ever written about spreadsheet errors to see what we’ve done, and where we need to go. Unfortunately, that’s all I can talk about for now.
Code tweaking in spreadsheets
One thing every programmer has done is to to copy-paste code from prior projects, or elsewhere in the codebase, or the internet, and then tweak the code to make it work. One population that does this a lot are end-user programmers, especially data analysts and scientists (e.g., a clinical researcher writing code to run statistical tests on her data). But such code tweaking is not easy for end-user programmers who are not proficient in programmer. What if we could help them with that?
We designed TweakIt that helps programmers see what each little piece of source code does. You could select a method call, or a chain of calls, to compare what that bit of code does. It does so by letting a programmer compare what happens without that code, and with that code. For example, consider: str.to_lower().reverse(), a person can click on to_lower() and see what the output of the program is up until that point, and then click on reverse() and see what changes on adding that bit. Our empirical studies found that programmers were more successful, quicker and confident in tweaking code with such support.
We implemented TweakIt for Python + Excel for data analysis code tweaking, but I believe such a live programming + code inspection strategy will also be valuable in IDEs for professional programmers (e.g., picking up a new language or framework), and for computer science education for novice programmers. This was also published at CHI’21.
Variations foraging: the theoretical foundations
For my Ph.D. dissertation, I worked on the theoretical foundations variations foraging. Essentially, I was investigating what it is that happens in people’s heads when they have to work with “stuff” that is all very similar to each other (e.g., very similar versions of code), as in the examples below.
Now, we could do empirical work and try to build a theory from scratch, or see what happens if we take an existing theory, see if it applies, or if not refine it. We did the latter, because, well, “building on others’ shoulders” is good science :-). So, we used Information Foraging Theory (IFT) to see if that will help us understand how people forage among variants. (Variant = each of the copies in the pictures above).
To build the theoretical foundations –or, as I call it, the variations foraging theory–, we my collaborators and I conducted user studies to see how programmers worked with variants, formed hypotheses about what actually was going on in their heads (that made them do what they did) and then built computational models of what goes on in the programmers’ heads. If the model, based on what is going on the programmers’ head, could predict the programmers’ actual code navigations, then we win: our hypotheses are valid. Otherwise also we win — we have more work to do to refine our hypotheses and models.
The second computational model, PFIS-H, is an active manuscript — I hope to finish it up and send it off to a journal in the coming weeks!
Currently, I am looking into how I can take these theoretical foundations to build a variations support tool.
Information seeking in version control history
Developers often go to their project’s version control history looking for answers to questions like “who wrote this code”, “for what requirements is this code here”, “why is this code implemented in this weird way”, to name a few. Unfortunately, information sources like these are also hard to find anything in and I believe we can do much better with their information design.
So, we started off looking at what developers were really doing with their version control history, so that we can identify painpoints and better support those activities. Via interviews and surveys, we did the first study to characterize the whats and hows and whys of developers’ information seeking in version control tools.
The result is what we call the “three-lens model” for software history. The idea here is that there are three parts to software history–uncommitted changes, recent commits and old commits. Developers go to these parts of history with different intents and seek information in very different manners, just like people use different lens to see different kinds of stuff. Tool builders need to keep these three lenses in mind while building version control tools. This paper got an ICSME’15 Best Paper Award. Preprint.
Next, we drilled deeper into the three lens and asked ourselves why questions: why do developers really need the information, why do developers go about finding information the way they do, why are the problems developers face really problems and so on. Theories are very good at answering such why questions and so, we brought in Information Foraging Theory as an analytical framework.
Among the interesting things we found are that: in the old history (archaeological lens), developers foraged in ways traditional IFT has predicted. In the recent history (awareness lens), developers foraged very differently–they spent less cost, to get a high-level (not in-depth) idea of what changed, so that they could make their lives easier in the future (e.g., avoid merge conflicts).
In the area of uncommitted changes (immediate lens), the devil is in committing changes–but in doing so, developers had to consider how to make it easier for future (e.g., split changes to commits, write good commit messages), but often failed at it because different situations and people needed different commit sizes, commit messages, etc. In other words, producers can only produce one thing, whereas consumers might need all kinds of things! (or producer produces something and consumer needs something else!). This is an open problem to investigate in IFT’s cost- value framework.
This work is mostly in my MS thesis. A journal paper is currently under revision.
Supporting end-user programmers
I am under an NDA and not quite allowed to talk about the details of this study. But, I worked on an extension to Calculation View, a new way of editing and working with formulas in spreadsheets. Original Calculation View paper by Sarkar et al.