When testing needs a different train of thought
There are many ways to gain confidence in software from exploratory testing to test driven development (TDD). But sometimes when you are faced with a novel problem, an out-of-the ordinary solution is required.
Tyro has been working hard to support the new eftpos Australia contactless cards. After a few months of diligent development work our team of resourceful engineers have produced a working solution. But how do we know it works? Firstly, we always use test driven development.
In this case we ended up with 700 automated functional test cases — that is 10 times more test code than production code. We also produced an Android application that allowed us to simulate a card and record its interaction with our terminal. And of course we have the actual test cards provided by eftpos for end-to-end testing.
But we kept asking ourselves, how can we be sure that this is going to keep working over the course of hundreds of transactions that a terminal will perform every day out with our merchants? In testing parlance, how do we soak test our newborn contactless kernel?
We needed a way of performing a never-ending stream of transactions on the terminal. Unfortunately we can’t just leave a card sitting next to the terminal as the testing required that the card physically approach and leave the contactless reader.
That means we have to move it back and forth. We could have just made one person sit there tapping cards all day, but that’s not the way we work at Tyro. Not only would it be really boring but we would need to make sure they kept changing hands so they didn’t end up with one really muscular arm by the end of it.
After brainstorming as a team we had ideas from the fanciful (a car building robot arm to tap the card for us), to the more mundane (a crowdsourcing testing with a terminal in the kitchen and having employees tap a card every time they made a cup of tea).
The kitchen idea isn’t bad for plan B, but we wanted something a bit more interesting, different, and fun. We needed something that could go around in circles carrying a card past the terminal repeatedly. What could that be?
A drinking bird? A plane? Superman? How about a train? It sounded crazy at first. Our first thought was to ask if any colleagues with kids had something we could use. In the end we didn’t have to ask anyone — a passing developer hearing the magic word ‘train’ piped up and said “I can get you one of those!”
That’s what we like about working in an open-plan office. It sounded too crazy to try explaining to management. Luckily Tyros are encouraged to act on their best judgement rather than waiting around for approval.
A day later with a case of trains, a quick trip to the hobby shop to buy some missing track, and a boot camp on proper model train maintenance — change direction every two hours to ensure even wear on the wheels — we were ready to test.
After assembling the rails and getting the train running with a test card stuck to it, we positioned each of our different terminal models along the track. Our seamless point-of-sale integration made it simple to automatically set the terminals to continually perform transactions. Success.
Assuming we did a transaction for a small enough amount not to require a PIN, we had a non-stop, automated way to perform contactless transactions.
Not only did we increase our confidence in the ability of our contactless kernel to handle a full day’s worth of transactions, but the train also gave us the chance to do some performance testing. The video (above) was taken early on in the testing and you’ll notice how slow the train is going. The problem was one of our older terminals was slowing the whole set-up down. Never to be out-done, a few of our engineers started pushing performance updates directly to the slow terminal and then increasing the speed of the train until we were happy with it.
At Tyro, to keep track of the current engineering work, every team has a sheet of paper with different work areas on it. Every morning all the testers and developers place a dot next to the type of work that best matches what they did on the previous day. Luckily we already had an appropriate section for our work. One more dot next to ‘TRAINing and Learning’.