Rhino Mocks vs Moq vs NSubstitute

I seriously believe in TDD. I think that the benefit you get when fully adopting it is un describable. Following it allows you not only to cover most of your code but write testable code from the beginning, which later can be easily modified. But this post is not about TDD (I use it – deal with it!). This post is about what mocking framework is the best from the optional frameworks out there.

I’ve used Rhino Mocks for the last 3 years. Everybody know this framework, most of you probably worked with it in the past (or still do). It gets the job done. But it has different types of mocks that frankly, I never remember when to use stub or mock,  the AssertWasCalled function is not trivial as you would expect.

The main consideration for a software developer who writes code, is choosing a simple mocking framework. I don’t want to “waste time” in writing my tests. I want my tests to be simple and clear without different mock types (Do you really know the difference between strict,  partial, dynamic and stub types of a mock object?). All I want is the ability to derive from a class or implement an interface and then set some custom behavior to fit my test’s needs. The test should be simple, clear and obvious not only to myself but to anyone else.

All the above and my constant quest of finding the most simple way to write code lead me to compare these three frameworks: Rhino Mocks, Moq and NSubstitute (Nuget them).

I’ll show their basic usages, their advantages and disadvantages and of course which one I prefer.

This is the class that I’m going to Test:

   1:   public class SchoolRegistrationServer

   2:   {

   3:       public SchoolDatabase SchoolDatabase { get; set; }

   4:  

   5:       public SchoolRegistrationServer(SchoolDatabase schoolDatabase)

   6:       {

   7:           SchoolDatabase = schoolDatabase;

   8:       }

I receive the Dal abstraction in the constructor so it could be easily mocked.

   1: public class SchoolDatabase

   2:     {

   3:         public virtual DB DB { get; set; }

   4:  

   5:         public virtual bool Add(Student student)

   6:         {

   7:             throw new NotImplementedException();

   8:         }

   9:         

  10:         public virtual bool DoesStudentExist(Student newStudent)

  11:         {

  12:             throw new NotImplementedException();

  13:         }

  14:  

  15:         public virtual string Connect()

  16:         {

  17:             throw new NotImplementedException();

  18:         }

  19:     }

This class is yet to be implemented – TDD or Not. For those of you who ask “why this class doesn’t have an interface” I answer: I don’t put an interface on every class. If it’s not an API contract or a replaceable feature then there is no point adding an interface (How often do you replace your DAL?).

Let’s begin the comparison:

The first thing I want to compare is the initialization of the mock objects:

   1: SchoolDatabase nsubstituteMock = Substitute.For<SchoolDatabase>();

   2: SchoolDatabase rhinoMock = MockRepository.GenerateMock<SchoolDatabase>();

   3: Mock<SchoolDatabase> moqMock = new Mock<SchoolDatabase>();

The main difference is that Moq framework separates the mock from the object that is injected to the tested class.

   1: SchoolDatabase schoolDatabase = moqMock.Object;

The separation is logical – you set custom behavior on the mock object and pass a “ real” object to the tested class. I don’t think that should be a separation since the object that is passed to the tested class is a proxy so there is no point separating them to a mock object and to a “real” class.

Rhino Mocks on the other hand creates a mock that is already a class but they let you choose between 4 different kinds of mocks that you can generate:

• Partial mock – If there’s no stub for a method a partial mock will call the actual implementation method – not the mock.
  
  
• Dynamic mock – If there’s no stub for a method a dynamic mock will return a default value (0 or null).
  
  
• Strict mock - If there’s no stub for a method a strict mock will throw an exception.
  
  
• Mock - Mocks record and verify action that was made on a class, plus they allow you to replace a method behavior with your own and if no stub was made a default value will return.
  
  
• Stub– Same as mock but doesn’t record.

A lot of “nice to have options” but frankly when using GenerateMock (that exposes you almost all the meaningful functionalities) you don’t really need all the others.

NSubsitute introduces you a single method “For” and immediately  creates a mocked object of the object’s type:

  1: SchoolDatabase nsubstituteMock = Substitute.For<SchoolDatabase>();

Smooth.

Next topic is “behavior replacement”":

   1: public bool AddStudent(Student student)

   2: {

   3:     if (SchoolDatabase.Add(student))

   4:     {

   5:         return true;

   6:     }

   7:  

   8:     return false;

   9: }

This the super complex method that I want to test.

   1: [TestMethod]

   2:   public void AddStudent_StudentWasAdded_ReturnsTrue()

   3:   {

   4:       // Arrange

   5:       var studentToAdd = CreateStudent();

   6:      

   7:       // Mocking behavior setup

   8:  

   9:       // Act

  10:       var wasAdded = SchoolServer.AddStudent(studentToAdd);

  11:  

  12:       // Assert

  13:       wasAdded.Should().BeTrue();

  14:   }

This is the super complex test. Now let’s see how every mock allows us to replace the behavior of it’s method.

   1: // Moq

   2: SchoolDatabaseMoqMock.Setup(x => x.Add(studentToAdd)).Returns(true);

   3:  

   4: // Rhino Mocks

   5: SchoolDatabaseRhinoMock.Stub(x => x.Add(studentToAdd)).Return(true);

   6:  

   7: // NSubstitute

   8: SchoolDatabaseNSubtitueMock.Add(studentToAdd).Returns(true);

We can easily see that Rhino Mocks and Moq look almost the same. NSubstitute chooses not to go with the traditional lambda expression syntax, it’s mock exposes the mocked object methods to which we fluently add the return statement. It’s neat but can be a little confusing. Frankly I don’t know what I prefer (it’s a tie).

Next is “Multiple return values”:

In some tests the mocked object will be called several times and every time it should return a different value. My next method tries to connect to the school database, it has 3 attempts after which it fails.

   1: public string ConnectToDB()

   2: {

   3:     var connectionAttemps = 1;

   4:     var maximalConnectionAttemps = 3;

   6:     while (connectionAttemps <= connectionAttemps)

   7:     {

   8:         if (SchoolDatabase.Connect())

   9:         {

  10:             return string.Format("Connected to database after {0}", connectionAttemps);

  11:         }

  12:  

  13:         connectionAttemps++;

  14:     }

  15:  

  16:     return string.Format("Failed to connect to database after {0} attemps", maximalConnectionAttemps);

  17: }

I want to test that the method returns the right output after it managed to connect on the third attempt. I need to setup my mock to return twice “false” and on the third time return “true”:

   1: // Rhino Mocks

   2: SchoolDatabaseRhinoMock.Stub(x => x.Connect())

   3:     .Return(false).Repeat.Once()

   4:     .Return(false).Repeat.Once()

   5:     .Return(true);

   6:  

   7: // Another option

   8: SchoolDatabaseRhinoMock.Stub(x => x.Connect())

   9:     .Return(false).Repeat.Twice()

  10:     .Return(true);

Rhino Mocks allows us to define as many return options as we like, plus we can use it’s cool fluent API. Nice and easy.

   1: // NSubstitute

   2: SchoolDatabaseNSubtitueMock.Connect().Returns(false, false, true);

Next is NSub that allows to override the usual “Return” method and instead of returning just one object we can return as many as we need. The results will return by the order we set them in the mock.

Moq doesn’t allow you to return different values. You have to implement some sort of queue that will withdraw the results by order. This post explains it. Too complicated.

Next it “Recursive mock”:

   1: public string GetConnectionString()

   2: {

   3:     return SchoolDatabase.DB.GetDatabaseDetails();

   4: }

First of all it’s bad practice to allow your object expose methods of it’s properties. However in legacy applications it’s quite common. The problem here is that the “DB” property has no mock so we have no control of the “GetDatabaeDetails” output. Luckily all the three frameworks support the “Recursive mock” meaning that in case the mock has a property that hasn’t been mocked it will automatically mock it.

   1: // Moq

   2: SchoolDatabaseMoqMock.Setup(x => x.DB.GetDatabaseDetails).Returns(expectedConnectionString);

   3:  

   4: // Rhino Mocks

   5: SchoolDatabaseRhinoMock.Stub(x => x.DB.GetDatabaseDetails).Return(expectedConnectionString);

   6:  

   7: // NSubstitute

   8: SchoolDatabaseNSubtitueMock.DB.GetDatabaseDetails().Returns(expectedConnectionString);

The last comparison is method call verification:

   1: public void RegisterStudent(Student newStudent)

   2: {

   3:     var isAlreadyRegistered = SchoolDatabase.DoesStudentExist(newStudent);

   4:  

   5:     if (!isAlreadyRegistered)

   6:     {

   7:         SchoolDatabase.Add(newStudent);

   8:     }

   9: }

I want to test that in case the student doesn’t exist the “Add” method will be called (and on the contrary that it hasn’t called if the student does exist).

   1: // Rhino Mocks            

   2: SchoolDatabaseRhinoMock.AssertWasCalled(x => x.Add(studentToRegister));

   3: SchoolDatabaseRhinoMock.AssertWasNotCalled(x => x.Add(studentToRegister));

   4:  

   5: // Moq

   6: SchoolDatabaseMoqMock.Verify(x => x.Add(studentToRegister),

   7:                              "when student doesn't exist, it should be added to the db");

   8: SchoolDatabaseMoqMock.Verify(x => x.Add(studentToRegister), Times.Never(),

   9:                              "when student doesn't exist, it should be added to the db");

  10:  

  11: // NSubstitute

  12: SchoolDatabaseNSubtitueMock.Received().Add(studentToRegister)

  13: SchoolDatabaseNSubtitueMock.DidNotReceive().Add(studentToRegister);

Rhino Mocks and NSub are pretty straight forward how ever rhino mock’s error messages aren’t clear “...Expected #0, Actual #1. “. Moq on the other hand allows you to add a custom message that will appear if the test fails. NSub has a better error message:

“Expected to receive no calls matching:

    Add(Student)

Actually received 1 matching call:

    Add(Student)”

That’s not as good as the custom error message the Moq exposes but it’s good enough.

 

Conclusion:

So which is the best mocking framework? As all the answers in software development – it depends. It depends on what do you need: if you are going to use multi results return for one method then Moq is probably not a good option, however if you care about good error messages when your tests fail then Rhino Mocks isn’t that good.

I remind you why I started this comparison – I’m interested in a simple framework that allows me easily to mock objects and change their methods behavior, plus obvious error messages. My choice is: NSubstitute!

It is simple, easy to use and  it’s easy to explain to other developers how to use it. It has all the main functionalities and a really simple fluent interface.

I’ll say it again: NSub works for me. Your application may need other functionalities that NSub doesn’t have. That’s fine, just remember to open your eyes and look around. Technology evolves all the time – don’t stick to some framework just because you are used it. Don’t afraid to try.

Good Luck!