The ultimate software development tool

Best practices on project management, issue tracking and support

Tag: Unit testing

Using Forensic Psychology to Spot Problems in Your Code – Interview with Adam Tornhill


We interviewed Adam Tornhill, a software architect who combines degrees in Engineering and Psychology to get a different perspective on software. We discuss his book ‘Your Code as a Crime Scene‘, in which he describes using Forensic Psychology techniques to identify high-risk code, defects and bad design. He describes how his techniques are useful by themselves, but can also be used to make other practices like code reviews and unit testing even more effective.

Adam’s Code Maat software is on GitHub.

Content and Timings

  • Introduction (0:00)
  • About Adam (0:31)
  • Detecting Problem Code (1:43)
  • Improving the Software Architecture (3:00)
  • Social Biases and Working in Groups (4:02)
  • Working with Unit Testing and Code Reviews (7:20)
  • Tools and Resources (8:32)

Transcript

Introduction

Derrick:
Adam Tornhill is a Programmer and Software Architect who combines degrees in engineering and psychology to get a different perspective on software. He’s the author of “Lisp for the Web”, “Patterns in C”, as well as “Your Code as a Crime Scene”, in which he describes using forensic techniques to identify defects and bad design in code.

Adam, thank you so much for taking the time to join us today. I really look forward to hearing what you have to say. Why don’t you say a bit about yourself?

About Adam

Adam:
Oh yeah, sure. It’s a pleasure to be here. I’m Adam Tornhill and I’m from Sweden, which is why I have this wonderful accent. I’ve been a programmer for a long time. I’ve been doing this for almost two decades now, and I still love what I do.

Derrick:
I wanted to touch on one of your books, “Your Code as a Crime Scene” where you apply forensic psychology techniques to software development. What made you think to apply techniques from what would seem to be such an unrelated field?

Complexity metrics, the old kind, they just didn’t cut it

Adam:
It actually came a bit surprising to me as well. What happened was some years ago I was in the middle of my psychology studies, working towards my master’s degree and then got hired into architectural roles where I had to prioritize technical debt and identify problematic areas. I found that it was terribly hard to do that because complexity metrics, the old kind, they just didn’t cut it.

When I tried to talk to the different team members and developers, I found out that nobody seemed to have a holistic picture. It was virtually impossible to get that picture out of a large code base.

At the same time, I took a course in criminal psychology. I was just struck by the similarities in the mindset to what we needed to have in the software business. That’s how it all started. I started to think, how can we apply this stuff to software?

Detecting Problem Code

Derrick:
I wanted to jump into a couple of the techniques you cover. How can forensics psychology help us to detect problematic code?

Adam:
The first kind of technique that I would like to introduce, and that’s actually where I started, is a technique I call a Hotspot Analysis. It’s based on a forensic concept called Geographical Offender Profiling.

I find it really fascinating. What Forensic Psychologists do is basically they try to spot patterns in the spatial movement of criminals, to detect our home bases, so we know the area to inspect.

I thought what if we could do the same for software, that would be pretty cool. If we could take a large code base and somehow spot the spatial movement not of criminals, but of programmers, then we could identify the kinds of, the parts of the code that really matter.

That’s where it started, and what I do is, I start to think about where can I find that information, I realized, it was there right in front of us in a version control system. A version control system, basically records every interaction we have with the code.

So I tried to mine source code repositories and then I looked for code with high change frequencies, then I overlaid that with the complexity analysis, which allows us to identify the most complicated code, that is also the code we have to work with often, which is a hotspot. So that’s where it all started. That’s the starting point for the rest of the analysis basically.

Improving the Software Architecture

Derrick:
You also say that you can use your techniques to help improve the architecture of applications. How is that so?

Adam:
First we have to consider what an architecture actually is. A fundamental idea in my book is that software is a living entity. It constantly changes, evolves and sometimes degrades.

So I came to architecture more as a set of guiding principles to help that evolution happen in a successful way. Just to give you a complete example, consider microservices, they seem to be all the rage right now.

So that means that tomorrow’s legacy applications will be microservices. In a microservice architecture, a typical one where each microservice is cohesive and independent, so that’s basically one architectural principle.

What you do is you try to measure and identify violations of that principle. Again I look at the evolution of the codebase in the source code repository. And I try to identify, in that case, multiple services that tended to change at the same time because that’s a violation of your architectural principles. That’s one way to do it.

Social Biases and Working in Groups

If you want to truly improve software, we need to look beyond technique and look at the social side

Derrick:
You also cover how “most studies on groups of people working together find that they perform below their potential”. Why is this and what are some of the problems teams can experience?

Adam:
Social Psychologists have basically known for decades that teamwork comes with a cost. The theory that I come to in my book is called Process Loss. The idea behind process loss, it’s pretty much as how a mechanical machine cannot operate at 100% efficiency all the time, so neither can software teams.

That loss often results in the communication and coordination overhead or perhaps motivation loss. The interesting thing is when you talk to software teams about this, they are aware of the problems. Quite often we mis-attribute it to a technical issue, when in reality we have a social problem.

In one team I worked, the developers, they noticed that they had a lot of bugs in certain parts of the codebase. They thought it was a technical problem. What happened in reality was they were having excess parallel development in those parts of the codebase.

There were multiple programmers working in the same parts of the code all the time. So merely all they did, they said that, they complained a lot about potential merge conflicts. They were really scared to do merges of different feature branches.

When we started to look into it, we identified that what actually happened was that, due to the way they were working, they didn’t really have a merge problem. What they had was basically a problem that the architecture just couldn’t support their way of working.

I think that’s really important to understand that if you want to truly improve software, we need to look beyond technique and look at the social side.

Derrick:
What can we do to avoid such problems?

Adam:
First thing I recommend is always to use something I call Social Hacks. Basically it takes the social situation and tries to tweak some aspect of it, to make social biases less likely because social biases, they are a huge reason why we get process loss.

One of the simple techniques that I recommend is, in every team, assign someone the role of the devil’s advocate. The role of the devil’s advocate is just to take the opposite stance in every discussion to question every decision made. That helps you reduce a bunch of biases because you’re guaranteed that someone will speak up. It also has the nice benefit of making teams more risk averse. There are a bunch of social stuff we can do and I also present a number of analyses that they can apply to investigate and mine social metrics from their codebases.

Derrick:
So what sort of results have you seen from those applying your techniques?

Adam:
I’ve seen that most people seem to use it for their original purpose, which was to identify the code that matters the most for maintenance. I’ve also seen some interesting uses mostly from managers. Managers seem to love the social aspects of the analysis because it makes it possible for them to suddenly reason about something that they couldn’t measure before.

Another interesting area, is a couple of years ago, I worked with a really good test team. These testers they used to do a bit of exploratory testing at the end of each iteration. So what I did was basically I generated a heat map over the complete source code repository, so we can see where most of the development activity was, and we would start to communicate with the testers. They knew where they should focus a little bit of extra energy. That helped us to identify a lot of bugs much earlier.

Working with Unit Testing and Code Reviews

We’re not so much writing code. What we do most of the time is actually making a modification to existing code

Derrick:
How do you see current development practices like unit testing and code reviews, working with those that you’ve described?

Adam:
Techniques that I present, they don’t replace any current practices, save maybe wild guesses and panic near the deadline. But otherwise, they are there to complement what you already do today.

For example, take code reviews. I often use a hot spot analysis to identify code most in need of review. To prioritize reviews as well. Unit testing is interesting, because I actually have a complete chapter in my book about building a safety net around tests. Automated tests are terribly hard to get right in practice. It’s actually something you can do, by again, measuring the evolution of the codebase, looking at application code and testing them both together.

Derrick:
Ultimately you say developers should optimize for understanding in their code, why is this so important?

Adam:
It’s important because we programmers, we’re not so much writing code. What we do most of the time is actually making a modification to existing code.

If you look at the research behind those numbers you will see that the majority of that time is spent trying to understand what the code does in the first place. If you optimize for understanding, we optimize for the most important phase of coding.

Tools and Resources

Derrick:
What are some tools that people use to mine source code data, to use your techniques?

Adam:
When I started out there weren’t any tools really, there were a bunch of academic tools. They were powerful, but the licences were quite limited, you weren’t allowed to use them on commercial projects, they also focused on one single aspect.

I actually had to write my own set of tools. I actually open-sourced all my tools, because I want the readers of my book to have the possibility to try the techniques and stuff.

If you’re interested in this kind of stuff, you can just go to my GitHub account, Adamtornhill, and download the tools and play around with them. I think we’re going to see a lot of stuff happening in this area now.

Derrick:
Beyond your book, are there any resources you can recommend for those interested in learning more about writing maintainable code and improving their code design?

Adam:
My favorite book when it comes to software design has to be ‘The Structure and Interpretation of Computer Programs’. It’s just brilliant and it had a tremendous influence on how I approached software design.

When it comes to coding itself, I would say Kent Beck’s ‘Smalltalk: Best Practice Patterns’. It’s a brilliant book and it takes us way beyond Smalltalk. It’s actually something I recommend for everyone.

And the final book I would like to mention is ‘Working Effectively with Legacy Code’ by Michael Feathers. It’s one of the few books that takes this evolutionary perspective on software, so it’s a great book by a great author.

Derrick:
Adam, thank you so much for joining us today.

Adam:
Thanks so much for having me here, it was a true pleasure.

Working Effectively with Unit Tests: Unit test best practices (Interview with Jay Fields)

In this interview with Jay Fields, Senior Software Engineer at DRW Trading, we discuss his approach to writing maintainable Unit Tests, described in his book ’Working Effectively with Unit Tests’. We cover unit test best practices; how to write tests that are maintainable and can be used by all team members, when to use TDD, the limits of DRY within tests and how to approach adding tests to untested codebases.

For further reading on unit test best practices, check out Jay’s blog where he writes about software development.

Introduction

Derrick:
Jay Fields is the author of Working Effectively with Unit Tests, the author of Refactoring: Ruby Edition, a software engineer at DRW Trading. He has a passion for discovering and maturing innovative solutions. He has worked as both a full-time employee and consultant for many years. The two environments are very different; however, a constant in Jay’s career has been how to deliver more with less. Jay, thank you so much for joining us today. We really appreciate it. Can you share a bit about yourself?

About Jay

Jay:
Thanks for having me. My career has not really been focused in a specific area. Every job I’ve ever taken has been in a domain I don’t know at all, and a programming language, which I don’t really know very well. Starting with joining ThoughtWorks and being a consultant was a new thing for me. I was supposed to join and work on C# and ended up in the Ruby world very quickly. I did that for about five years, and then went over to DRW Trading to do finance, again something I’ve never done, and to do Java, something I had no experience with. That worked okay, and then I quickly found myself working with Clojure. It’s been interesting always learning new things.

Derrick:
Picking up on the book Working Effectively with Unit Tests, most developers now see unit testing as a necessity in software projects. What made you want to write a book about it?

Jay:
I think it is a necessity in pretty much every project these days, but the problem is, I think, really a lack of literature beyond the intro books. You have the intro books that are great, and I guess we have the xUnit Patterns book, which is nice enough as a reference. It’s not very opinionated, and that’s great, we need books like that also. But, if I were to say, I prefer this style of testing, and it’s very similar to Michael Feather’s approach to testing. There’s no literature out there that really shows that. Or, I prefer Martin Fowler’s style of testing, there’s no literature out there for that. I really don’t know of any books that say, “Let’s build upon the simple idea of unit testing, and let’s show how we can tie things together.” You can see some of that in conference discussions, but you really don’t see extensive writing about it. You see it in blog books, and that’s actually how my book started. It was a bunch of blog books that I had over 10 years that didn’t really come together. I thought, if I were to say to someone, “Oh, yeah just troll my blog for 10-year old posts, they’re not really going to learn a lot.” If I could put something together that reads nicely, that’s concise, people can see what it looks like to kind of pull all the ideas together.

Writing Unit Tests in a Team

Derrick:
In the book you say, “Any fool can write a test that helps them today. Good programmers write tests that help the entire team in the future.” How do you go about writing such tests?

Jay:
It’s really tough. I don’t see a lot written about this either, and I think it’s a shame. I think you first have to start out asking yourself, “Why?” You go read an article on unit testing, and you go, “Wow! That’s amazing! This will give me confidence to write software.” You go about doing it, and it’s great because it does give you confidence about the software you’re writing. But the first step I think a lot of developers don’t take is thinking about, “Okay, this is great for writing. It’s great for knowing that what I’ve just written work, but if I come back to this test in a month, am I going to even understand what I’m looking at?” If you ask yourself that, I think you start to write different tests. Then once you evolve past that, you’re really going to need to ask yourself, “If someone comes to this test for the first time, someone goes to this line of code in this test, and they read this line of code, how long is it going to take for them to be productive with that test?” Are they going to look at that test and say, “I have no idea what’s going on here.” Or, is it going to be obvious that you’re calling this piece of the domain that hopefully everyone on the team knows, and if they don’t, it follows a pattern that you can get to pretty easily. I think it’s a lot about establishing patterns within your tests that are focused on team value and on maintenance of existing tests, instead of focused on getting you to your immediate goal.

DRY as an Anti-Pattern

Derrick:
You also mentioned that applying DRY, or don’t repeat yourself, for a subset of tests is an anti-pattern. Why is this?

Jay:
It’s not necessarily an anti-pattern, I think it’s a tradeoff. I think people don’t recognize that merely enough. You’re the programmer on the team. You didn’t write the test. The test is now failing. You go to that test, and you look at it, and you go, “I don’t know what’s going on here. This is not helpful at all. I see some field that’s magically being initialized. I don’t know why it’s being initialized.” At least if you’re an experienced programmer, then hopefully you know to go look for a setup method. But imagine you have some junior guy, just graduated, fantastic programmer. He’s just not really familiar with xUnit frameworks that much. Maybe he doesn’t know that he needs to look for a setup method, so he’s basically stuck. He can’t even help you at that point without asking for help from someone else. DRY’s great. If you can apply DRY on a local scale within a test. It’s fantastic if you can apply it on a global scale, or across the whole suite, so that everybody’s familiar with whatever you’re doing then that’s great too. That helps the team, but if you’re saying that this group of tests within this trial behave differently than these tests up here, you’re starting to confuse people. You’re taking away some maintainability, and that’s fine, maybe you work by yourself so no one’s confused because you did it, but recognize that tradeoff.

When to use TDD

Derrick:
You’re a proponent of the selective use of Test-Driven Development. What type of scenarios are helped by TDD, and when should a developer not apply such techniques?

Jay:
It’s a personal thing. For me, I think I can definitely give you the answer, but I would say everybody needs to try TDD, just try it all the time. Try to do it 100% of the time, and I think you’ll very likely find that it’s extremely helpful for some scenarios, and not for others. Maybe that’ll differ by person, so everyone should give it a try. For me personally, I’ve found when I know what I want to do, if I pretty have a pretty mature idea of what needs to happen, than TDD is fantastic because I can write out what I expect, and then I can make the code work, and I have confidence that it worked okay. The opposite scenario where I find it less helpful is when I’m not quite sure what I want, so writing out what I want is going to be hard and probably wrong. I find myself in what I think is kind of a wasted cycle of writing the wrong thing, making the code do the wrong thing, realizing it’s wrong, writing the new thing that I expect which is probably also wrong, making code do that, and then repeating that over and over, and asking myself, “Why do I keep writing these tests that are not helpful? I should just brainstorm, or play around with the code a little bit, and then see what the test could look like to once I have a good idea of the direction it is going.

Don’t Strive for 100% Test Coverage

Derrick:
You say you’re suspicious of software projects approaching 100% test coverage. Why is this, and why is 100% coverage not necessarily a goal we should all strive for?

Jay:
Earlier on I thought it was a good idea, 100%, because I think a lot of people did. I remember when Relevance used to write in their contracts that they would do 100%, and I thought, “Man, that’s really great for them, their clients.” Then you start to realize you need a test things like, say you’re writing in C#, and you have to automatically generate a set. Do you really want to test that? I think all of us trust that C# is not going to break that functionality in the core language, but if you put that field in there, then you have to test it if you want to give 100% coverage. I think there will be cases where you would actually want to do that. Let’s say you’re using some library, and you don’t really upgrade that library very often, and even though you trust it, maybe you’re writing for NASA or maybe you’re writing for a hospital system – something where if it goes wrong, it’s catastrophic. Then you probably want to write those tests. But if you’re building some web 2.0 start up, not even sure if the company is going to be around in a month, you’re writing a Rails app, do you really want to test the way Rails internals work? Because you have to, if you want 100% coverage. If you start testing the way Rails internals work, you may never get the product out there. You’ll have a great test suite for when your company runs out of money.

Adding Tests to an Untested Codebase

Derrick:
So that’s what’s wrong with too many tests. What about a code base with no tests? How can you approach getting test coverage on an untested code base?

Jay:
I think the focus for me is really return on investment. Do you really need to test everything equally when the business value is not the same? Let’s say for instance, you’re an insurance company. You want to sell insurance policies. So you need a customer’s address, and you need a social security number, probably, to look them up, some type of unique key, and after that, you just want to charge them. Maybe you need their billing details, but you don’t really care if you got their name wrong, you don’t really care if you got their age wrong. There are so many things that aren’t really important to you. As long as you can keep sending them bills and keep getting paid and find the customer when you need to, the rest of the stuff is not as important. It’s nice. Whenever you send them the bill, you want to make sure the name is correct, but it’s not necessary for your software to continue working.

When I’m writing tests, I focus first on the things that are mission critical. If the software can’t succeed without a function working correctly, or a method working correctly, then you probably need some tests around that. After that you start to do tradeoff, basically looking at it and figuring out, “Well, I want to get the name right. If I get the name wrong, what’s the cost?” Well, getting the name wrong I’m not sure if there’s much of a cost other than maybe an annoyed customer that calls up and says, “Can you fix my name?” So, you have maybe have some call center support. I’m guessing that the call center is going to be cheaper than the developer time. So do we want to write a test with a Regex for someone’s name and now we need UTF support, and now we need to support integers because someone put an integer in their name? And you get in to this scenario where you are maintaining the code and the tests. Then maintaining the tests is stopping you from getting a call center call. It’s probably not a good tradeoff. I just look at the return in investment of the tests, and if I have way too many tests, every amount of code needs to be maintained. If I have too many tests, than I need to delete some because I’m spending too much time maintaining tests that aren’t helping me. If I have not enough tests, then it’s very simple, just start to write some more. I guess I tend to do that with whenever a bug comes in for something that’s critical. Hopefully, you caught it before then, but occasionally they get into production, and you write tests around that. I always think to myself, whenever the bug comes in, what was the real impact here?

Common Mistakes with Unit Tests

Derrick:
What are some of the common mistakes you see people making when writing unit tests?

Jay:
I think the biggest one is just not considering the rest of the team, to be honest. It’s really easy to do TDD. You write a test, and then you write the associated code, and you just stop there. Or, you do that, and you then you apply your standard software development patterns, so you say, “How can I DRY this up? How can I apply all the other rules that have been drilled into me that I need to do with production code? What’s really important?” Now if understanding the maintenance, understanding that what will help you write the code doesn’t necessarily mean it’s going to help you maintain the code. What I find myself often doing, actually, is writing the test until I can develop the code, so I know that everything works as I expect, then deleting that test and writing a different test that I know will help me maintain it. I think the largest mistake people make is they don’t think about the most junior member of the team. Think about this very talented junior member on your team that joined not that long ago. Are they going to be able to look at this test and figure out where to go from there? And if the answer’s no, then that might not be the best test you could write for the code.

Derrick:
Beyond your book, can you recommend some resources for developers interested in learning more about writing effective tests?

Jay:
I think that there are great books that help you get started. I think the Art of Unit Testing is a great book to help you get started. There’s the xUnit Patterns book that’s really good. The problem is, I really don’t think there’s much after that. At least I haven’t found much. Kevlin Henney has done some great presentations about test-driven development. I personally really like Martin Fowler’s writing and Michael Feathers’ writing and Brian Marick, but I don’t know of any books. I really think that there’s room for some new books. I think that, hopefully, people will write some more because unit testing’s only going to be more important. It’s not going away, it’s not like people think this is a bad idea. Everybody thinks this is a great idea. They just want to know how to do it better.

Derrick:
Jay, thank you so much for joining us today. It was a pleasure.

Jay:
Yeah, it was great! Thank you very much for your time.

 

For further reading on unit test best practices and software development, check out Jay Field’s blog.