Everybody has a plan until they get punched in the face

“A failure” can be a result of a software or hardware problem. A fault tolerance design intends to enable the system to continue operating properly in the event of failure. In this post, I’ll talk about a way to treat a special cause of failure – a timeout using a really cool open source framework named Polly.
We prepare for failures, we put “try catch” blocks and check “if(something == null)” every now and then. We think we’ve got it covered and that we are fully protected.
As Mike Tyson said – that’s our plan. However, we all know that something “unpredictable” will always go wrong, and then we’ll be punched in the face.

Let’s say that you have a DB and you query it to get some data – user’s navigation history for example.

One of your users clicked to get his navigation history and waited for almost a minute (yes – that’s a long time) to get an error that crashes the application!

(BTW This not a great error page – read here about the importance of error pages that make your user smile.)
You guessed right – that was a timeout to the DB. Why did it happen? Who cares? It made your user wait and eventually it crashed the application. This is unacceptable!

How can we handle it?

There are two problems in this scenario: the first is that the application crashed and the second is that the user waited a minute for a response.
Let’s start from the crash. The navigation history is a cool and useful feature but it’s not the core of your system. The core feature is the navigation itself. You have to identify the core features of your application and make them work, even if some other features fail.
In this example, the application should have displayed an error message saying something like “History is currently unavailable – try to remember where you went” and allow the user to use its core features any way.
Now let’s deal with the fact that the user had to wait for a whole minute before he saw the error. That’s a horrible user experience. A timeout might happen due to many reasons in any sort of client server communication (no more available connections, waiting for a really long operation to finish or simply a stuck component).
The bottom line is that someday it will definitely happen to you! Furthermore it will cause all your users to wait until they receive the error. A timeout may accrue just once due to a race condition that only a single user will get. However, sometimes the timeout will remain until you manually fix the problem - which can take a while.
You can lower the timeout limit to a second or even 100ms - and indeed no user will wait longer than 100ms - but all the users will wait for 100ms and eventually fail, while choking your servers.
Your goal is to prevent users from waiting at all when you know they will definitely receive a timeout. In other words, if a system component has a timeout,  you don’t want the application to try to communicate with it. You want it to fail fast.

Fail Fast

Fail fast is a concept in fault tolerance systems that is designed to stop flows and normal operations when a possible failure might accrue. Such design will add check points before operations execution, which will check if the operation is “healthy”. Using “health indicators”, the system will know if a certain operation will most likely fail. If an operation isn’t healthy, then there’s no need to execute it and we’d rather call a fallback behavior.

This way, we may give the users a better experience then waiting for an error.
Let’s examine our case: the application shouldn’t try to execute the query unless the connection to the DB is “healthy”. So the system will check the “health indicator” before the execution, and if it returns false then it should immediately turn to the fallback and display the error page without even trying to connect to the DB.
You can take it even further – if such health indicators exist, then why can’t we check them on the application startup? If there’s a problem with the feature, then let’s disable it and even not show it to the user. Perhaps it’s better to make the feature “disappear” then exposing the user to a poor experience. That of course is a pure business decision (it might be a really poor experience if the messages tab in Facebook suddenly “disappears”).
Allowing your system automatically to disable features must be done extremely carefully. You don’t want to disable a feature for all of your users if for some reason a timeout has accrued just once. If it happened 10 times in a row, then most likely there is a real problem and only then should the feature be disabled.
 

Polly

Polly is a really cool and useful open source that helps you create a fault tolerant system using policies for handling exceptions and creating fallback solutions. Let’s see how we can use Polly to handle our timeout scenario and allow your application to fail fast.

NavigationHistoryController has a dependency to the database where the navigation history is stored.
It has a single public method, which provides the history by calling the INavigationHistoryProvider.
We have a try-catch surrounding the provider call. As we know this catch will eventually handle the timeout exception, but it won’t fail fast.

We have created a circuit breaker policy on the timeout exception (error code -2) setting the count to 10 and the policy duration for 10 minutes. In other words, our policy will be triggered if a SqlException will accrue 10 times and in the next 10 minutes every call to the provider will not be executed. BrokenCircuitException will be triggered instead, allowing us to quickly fallback and return an error without waiting for the timeout.

 

Let’s test it

We’ll mock the NavigationHistoryProvider and raise an exception for every single call to it “Get” method.
We’ll call it 10 times and then another time to check if the policy was activated.

Works like a charm. The code is available on GitHub.
 

Conclusion

The applications that we build will have failures. We must recognize that things won’t always fail nicely. We must be prepared for all the possible scenarios and edge cases. The sooner we start thinking about them in our development process – the better.
I encourage you to think about it from the beginning. Start using FDD – Fail-Driven Development.

Best of luck.

Applying TDD in Your Company is More Important than Ever!

I won’t cover the basics of TDD. I do want to talk about the advantages of writing automated tests (unit tests and integration tests) in general, and why applying TDD ,with a little twist, should be a standard in your company.

First, why should you write automated tests?

“In order to test and verify my code?” Daaahh! This is so obvious that it’s not worth mentioning! I think that unit tests and all kind of automated tests have many more benefits than just testing your code.

Better code design – If you have ever written unit tests, you’ll know there is no (pretty) way of testing or mocking static objects and tightly coupled classes which violate the SOLID principles. A testable class is a well-designed class that is loosely coupled from its dependencies, uses dependency injection and rarely has calls to static methods. A testable class is a SOLID class which will improve maintainability, readability and extensibility.

True documentation of the code – You’ve probably written a design paper at least once for a feature or any sort of documentation describing the code. Be honest – is this documentation still relevant? Does it really describe the code that is now in production? Probably not. The automated tests, on the other hand are the true documentation - since if the code breaks the tests, the code will be repaired or the tests will be modified to support the new behavior that has been added. Either way, the tests tell the real story of the class, allowing any other programmer other then you to quickly understand what the class does and how it behaves in different situations.

Killing the fear of change – One of the programmer’s greatest fears is the fear of changing code. You never know which class depends on this specific implementation in a specific scenario. Usually, QA will let you know about it. But not your company QA – your customers (the real QA), who will get hurt and be angry due to your changes. However, if the code is fully covered with automated tests then there’s no fear! You can add, modify and even, so help me god, delete code - and in a matter of a few minutes you’ll know exactly what logic you broke, and why (the tests assertion should be really clear).

A standard for your company

Your company is a business. It has competitors who are breathing down your neck, trying to get hold of your customers. Time and quality are crucial! You must deliver new features fast and with the highest quality so that your customers will be satisfied.

The only way to succeed is by writing automated tests that cover your entire code base and all the user flows. By doing this, you allow every single developer in your company (seniors and juniors) to quickly understand how the code works by reading the related tests, easily changing the code - since it has a great SOLID design – and having immediate feedback if any modification has ruined the existing logic, providing the exact places where it happened. This saves time and money, and should be the standard for every new feature in your product. It should be a fundamental principle for writing code in your company.

Why TDD? Just write the tests afterwards

Why should you use TDD if you can first write all the necessary code, implement the feature and only then write the tests?

Writing tests after implementation is way better than not writing tests at all. However, it might not get you the same effect as I’ve described above. When you are finally done with the implementation, the thing you want the most is to declare the feature as “done” and ship it to production. You probably won’t invest enough time in fully covering the feature with good, meaningful tests. If you encounter something that is fundamental in your implementation that is not testable, you will probably just leave it like that and not change it - leaving it untested.

The last thing is that you are biased toward your implementation: in your opinion, the way you’ve implemented the feature is the right way. Thus you will write tests that support your implementation and not necessarily cover all the edge cases and business goals that the feature was supposed to do.

TDD – the cherry on top

In order to truly enjoy the benefits described above you have to apply TDD -with a little twist:

First, start with an integration test for your feature – a test that will fully test the behavior of your feature. Some might call it “Behavior test” as in BDD. I think that is just semantics.

If you are fixing a bug, your tests should create the environment where this bug is recreated and test that the bug is prevented before you’ve actually fixed it! You want to be sure that you are able to reproduce your bug. Only then can you start actually fixing it by writing the tests first. Same goes with a new feature - first an integration test that checks the behavior of your feature, and only then begin to implement.

This way, you begin with the business goals and stay focused on them while you implement the whole feature.

Another great benefit in starting with the integration test is that you will have to create the setup for the feature. Doing so will make you truly understand how the feature works and on what it depends on. This is crucial before you start to implement, so you won’t make unnecessary mistakes.

It’s not easy: it requires strong testing infrastructure and patience – it’s hard to hold back the lines of code you are just dying to type. But it is so worth it.

If you want to easily write code at your company and not be woken up in the middle of the night to solve a bug in production, just write the tests. Better yet – start with them.