You currently have no automated testing, or broken automated testing, and now you want to get it done right. Perhaps this is a new project, or perhaps it’s an existing project where automation hasn’t been solved yet for one reason or other. Now you’re starting, but how do you give yourself the best possible chance of success? We’re here to talk getting your automated testing right – that is: how to achieve automated testing that adds value to the product and that your developers will appreciate. All of the points in this article are utterly agnostic of specific frameworks and technologies.
Caring
I bang on about this all the time. No matter how many specific suggestions I or others might make, ultimately, you and your team will need to care enough to execute them, and execute them well. If you don’t care, and don’t take care, your tests will be a drain. They are not something you can throw together or maintain as an afterthought. If that’s how you treat them, your reward will be failing tests, late deliveries and complaining developers and testers alike.
You must create and foster a culture that cares about your automated tests. Now that that’s said, let’s talk about some of the specific things you need to care about.
Goals – What is your automation supposed to do?
That’s the question you and we need to pose to know what we need to do. The aims for different teams will vary, but will usually encompass the following high level concerns:
- Catch bugs: “I would like to catch as many bugs as I can, as quickly as I can after developers write code”
- Prove success: “I would like passing automation to be strongly indicative that the software works, and of success”
- Cheap to build: “I don’t want the automation to be a huge outlay on my resources right now”
- Cheap to maintain: “I don’t want the automation to be a huge drain on my resources over time”
In essence, we’re talking about how to deliver real value, without paying an inordinate amount of effort, and especially without disrupting developers’ – or anybody else’s – time.
Hazards and solutions
The first two bullet points suggest that you need people involved in the process who really understand both QA – skilled professionals who understand how to test things – and people who understand the software and software domain. Miss either of these, and you run severe risk of ignoring important test cases and important bugs. It is possible to solve a lot of this with one person doing a good job of wearing multiple hats, but usually it means getting different representatives involved in defining and reviewing test cases.
The second two bullet points, and especially bullet 4, are where this gets much more interesting, and much more dangerous. Bullet 3 says we don’t want to spend lots of time now setting it up. It highlights, possibly, that we don’t want to automate everything right now, that there are things we might accept not automating to save time, and that we might be willing to accept an organic approach where our automated tests grow over time. This is largely solved by having the awareness to choose what tasks we undertake as we go, estimating outlay and accepting some things might stay manual for a time, and or that we implement better processes as we go.
Bullet 4 – ‘Cheap to maintain’ – however, is where things get especially gnarly. This is where the technical debt death lies. This is the bane of teams who don’t have truly skilled technical testing engineers at the helm of their automation frameworks. You absolutely must have skilled engineers involved in your testing. If you don’t, this is where you abandon your new testing suite in 3, 6 or 12 months time because nobody can tell if a test run is good or bad amongst all the failures.
Writing a set of tests that will pass today is easy (relatively speaking). Writing tests that pass forever is hard – but worth it.
Your tests are going to change when your software does. That’s almost a certainty. This means that you’re going to need to engineer them to reduce the impact of those changes, and foster a culture that is willing and able to maintain them.
Technical test engineering
A great deal of test engineering is just regular software engineering. Best practices, patterns, abstractions, helper modules. Pick all your favourite engineering terms and apply them sensibly here. For sure, nobody likes over-engineering, but I’ve seen and heard of more cases of under-engineering on the test side than over. Your test code needs to be treated with the same respect as your production code. How else can you trust it?
However, the developer knowledge alone doesn’t quite get you there, or at least, simply setting a good developer on to write your tests doesn’t solve the problem. Whoever is writing these tests needs to be taking the care to future-proof the tests, rather than thrashing out the code, and there are additional testing-specific considerations. To name just a few:
- At a high level, all your tests should simply be descriptions of what the test does: It should be easy to see and understand what tests are doing. If it isn’t, it can be very difficult to keep track of what tests you do and don’t have. What’s more, lengthy tests full of code and implementation details are a smell that you’re likely repeating code that should be shared. BDD is one of the things that attempts to achieve this, but it can be achieved with regular, well-engineered code too.
- Implementation details are always contained in helper modules/utilities: Shared code and utilities makes it far easier to create working tests. The biggest drain on developers will be failing tests. You need to make it so that if a test fails, you only have to go and fix it in one place, and it’s easy to work out where that is when you inevitably need to debug them.
- Logging Like I said. These tests are going to break. They absolutely must log what they’re doing. Everywhere. All the time. And especially, when things go wrong. Your tests and helper utilities need to dump as much data about what happens throughout a test as they can, in addition to ‘on failure’ – sometimes the cause was earlier in the test, without the prior logging it would be difficult to diagnose many failures. This is another huge reason for having utilities and modules – you don’t have to repeat logging or crash-collection procedures in every single test, which you otherwise would, and new team members automatically get all the value of previous work, rather than having to know that they should implement it in their own tests.
- Any test data/selectors are contained in single places, where they can be updated/replaced conveniently by developers who change the software: Lots of the software updates will be trivial. If you’ve written your tests well, the test updates will be trivial too. Nobody wants to change the same string 500 times in 50 files. Collect “data” into config files, or at least the same place in source code. Consider sharing with production code, even.
- Tests need to be written to pass every time: This is the biggest one. If you don’t set out to achieve this, or won’t commit to maintaining it later, honestly don’t bother starting. One failing test leads to two, leads to three… leads to you throwing your tests away and implementing a new failing framework all over again. A lot of the other points in this article are specifically aimed at trying to help achieve this, but it’s worth mentioning it on it’s own, because there are additional considerations. I wrote another blog post about “sleeps”, my personal most-hated cause of flaky tests, but they’re far from the only one. What if thing X takes longer than expected, what if thing Y doesn’t appear. Your tests and utilities need to be defensive about what they expect to happen, and alert you when they don’t. You’ll need code review processes which aim to anticipate issues in test implementation, and then if something does fail, you absolutely must set yourself up to fix it to always pass.
- Test need to be able to be run conveniently by anybody (either locally, or on-demand remotely): This is in addition to CI. Depending on what software you’re building, local may not be a practical option. In that case, you need to make it easy for anybody to run the tests on the team’s test environment, or ideally, spawn additional test environments. You must expect to get failing tests, and then have people go investigate what happened. A lot of the time that is massively aided by having the ability to spawn variations of the test on-demand, ideally with live debugging. That might mean investing in additional infrastructure or tools to support this use case.
Always forwards
This might sound like a lot so far. It’s a lot of skills, a lot of rules, a lot of hats involved, and a lot of caring. This can be scary to teams setting out on a new project. This is why I mentioned the “organic” approach earlier, to ensure you’re getting best value right now, rather than piling in effort up front and waiting to see a single test pass/fail. The answer? Get CI up and running as soon as possible, with your one passing test, and use it as a smoke for pull requests. You’ll see value immediately. This test should meet the criteria set out above, and from here out, you must not cut back on the technical “quality” of anything you add. If you do, you’ll regret it.
You must cultivate an “always forwards” policy:
“We will add more tests. They will be high quality. We won’t let old tests fail.”
In my experience, with that attitude, you end up with automated tests that developers are happy to have, happy to maintain, and swiftly-delivered features. This kind of test process speeds up development, where badly engineered tests and bad processes slow things down.
Never backwards – No tests fail! No bad code committed!
You really have to take a hard line on this one.
If the company dies if the release doesn’t go out Friday, just maybe you can skip fixing a failure that we understand (and I really hope you’ve investigated it, there might be a real problem!). Even in that case you’ll want to fix those tests as the first thing after the release before starting any new work.
In 99% of cases however, I’d wager you could/should have simply fixed the test before the release, and many successful teams do operate this way. It certainly should be the default position. Once you leave it a day, odds are, you’ll never fix it, and every failing test you accumulate makes it harder to see which failures are real, slows down your progress, and incredibly quickly your tests are too noisy to provide any value. I often liken developing software alongside failing tests to moving in treacle. They will slow down your whole team.
For similar reasons, you must not allow code that falls below the standards set out here and in your code guidelines to be committed to main. Once it is there, odds are, it will never be fixed. Let’s be honest, it won’t. Incomplete code that is otherwise good is fine, but nothing must bring the standard of the tests down. If you don’t have time to write the tests well, you don’t have time to write them. A few rotten tests can endanger the whole suite.
Adding new tests? They must be engineered correctly. This means a good review process, and education of new team members at a minimum.
Is that all?
No, of course not. I couldn’t possibly touch everything in one blog post. Just to name a few, I note that I haven’t mentioned:
The test pyramid (please don’t write all UI tests when you should have lots of unit and integration tests!)
- The test pyramid (please don’t write all UI tests when you should have lots of unit and integration tests!)
- Language considerations
- Hiring
- Other types of tests: Performance/stress
- Other activities: Kick-offs, reviews, reporting
- Pull request process
- CI best practices
- Developers writing tests
- Please tell me what else!
You’ll need to consider all of these. However, I’ve started with the bit I’ve personally seen most wrong, and which trickles down into everything else. Get these things right, and you can typically organically meet those other challenges. In particular, I’ve tried to cover the subjects which result in you being able to keep developers on board with the tests, since you’ll need that to have success.
Summary
In essence, to be successful, we need to foster a culture that understands the care that needs to be taken to develop and maintain working tests. We need to hire/assign people with appropriate skills – both technical and testing – and then the work processes need to give them the time to apply the love that your tests need to keep them running smoothly.
This is not a job that you do and it’s done. Even once you have a working system, there is a slippery slope you must avoid. All the good engineering in the world cannot save you if maintaining tests is not an integral part of your development process. The toll must be paid.
Take care to engineer your tests, fix failures, and keep moving forwards.
