Message Driven Systems: Some Rules of Engagement


For the past several months, I’ve been working on an ASP.NET Web API project that uses NServiceBus to deliver events and commands to services that do all of the heavy lifting for the application. The API is the backend for an e-Reader application with multiple mobile front ends, an administration portal and a few other services. The anticipated usage is such that we felt it was appropriate to use the NServiceBus framework so we could scale out the processing of all the information the application produces. In these types of applications, messaging frameworks can be very useful and provides a ton of valuable functionality but they also introduce a level of complexity that is not always easy for a development team to take on.

In this particular case, we have several remote developers that haven’t been as tightly integrated with the team as we would like. They have been disconnected from the rest of the team members for most of the project and it has been harder to get them up to speed on the new technologies that we’ve implemented. In the past few days, I’ve spent some time trying to help one of these team members understand the differences between events and commands, when to use them, how to implement their handlers and what not to do with them. This is the second or third time I’ve had to have this same conversation and I thought it would be beneficial to capture some of the key bullet points here for future reference…both for myself and for others.

What Is An Event?

An event conveys information about something that has already happened. As such, they cannot fail or be rejected. For example, you cannot reject a “user submitted an application” event. The user has, in fact, submitted that application. All the event is doing is communicating that it has happened and giving you, the listener, an opportunity to do something as a result. Here are some other event characteristics.

  1. Events are published, not sent.
  2. Events can be subscribed/unsubscribed to.
  3. Events can be subscribed to by any number of handlers.

What Is A Command?

do-somethingA command, on the other hand, conveys information about something that we wish to happen. Because they have not already happened, commands can fail or be rejected. For example, it is possible that a “create new user application” command could fail or be rejected. For example, we might find that the user already has an application that was submitted less than 30 days ago and that violates the business rules that we have defined for applications. In cases of failure or rejection, we typically have some sort of compensation strategy to handle that scenario. It could be as simple as setting the status of the application to rejected or some other value that conveys that the application creation failed. Here are some other command characteristics.

  1. Commands are sent, not published.
  2. Commands cannot be subscribed/unsubscribed to.
  3. Commands can only be sent to exactly one handler.

What are some benefits of using message driven system?

Because message driven applications decouple the message creator from the consumer(s), they are a good way to build an application that can scale and enforce the separation of concerns, single responsibility and other best practices.

For example, in our application we want to decouple the API from the services that process the data that is submitted by the front end application. This way, we can scale those data processing services up or down depending on the increase/decrease in the amount of information that is being submitted. This allows us to react to temporary spikes or dips it traffic due to special events or the time of day. Because the application we are building targets K-12 students, there is a natural rolling curve of usage that corresponds to the number of students that are in school at different times during the day across different time zones. We need to be able to scale up the processing of information during peak hours and then scale it back down during off hours when most of our users are at home and not using the application.

In addition to this, using a messaging frameworks reinforce the separation of concerns and single responsibility principles of software development. This results in individual components that are easier to implement, test and maintain. For example, instead of having a controller that contains business logic or calls into multiple service to coordinate the fulfillment of the submitted request, we can have a very thin controller that just drops a command on a bus and returns a response to the caller. That command is then delivered to a service that does some work and then publishes an event to notify other services that something has happened. Those other services, in turn, might issue other command(s) that do more work and potentially publish other events…and on and on. These services (command and event handlers) are usually a lot easier to implement, test and maintain because they do a small/focused amount of work.

What are some drawbacks to message driven systems?

All of the benefits described above are not free. They come with added complexity and introduce a different debugging, troubleshooting and support story. The decoupled nature of these systems means that you, as a developer, can’t step into the process and debug the application as easily as you can with a tightly coupled implementation. It also means that the troubleshooting and support stories are slightly more complicated. You now have to look in queues (which are not very user friendly) to hunt down messages and have to come up with compensation strategies for messages that fail and are dropped into an error queue. These are not the typical development and support stories that we are used to. We have to consider this when deciding whether to go with a message driven architecture or a more traditional one.

What are some things to look out for when implementing message driven applications?

There are a few guidelines that we should consider when implementing message driven applications. Following these guidelines will help ensure that the application is easier to develop, test, maintain and support.

Don’t implement complex business logic in event handlers. 

As I mention before, events represent something that has already happened and therefore should not fail or be rejected. This means that we should not have event that include a lot of complex business logic. Instead, event handler should just decide if something else should happened as a result of the event. If so, they should issue the appropriate command(s) to make that happen. You want to avoid building in complex business logic because every additional line of that business logic can cause the event handler to fail. Also, all of that business logic increases the event handler’s reasons for change. This makes the resulting implementation brittle, more difficult to test and maintain.

Don’t use heavy messages with lots of properties. 

Messages should only contain the minimum amount of information necessary to convey the change that has happened or we want to happen. That is, we should not load up our messages with information that we could otherwise pull from somewhere else. Instead, it is usually sufficient to include a timestamp and the id(s) necessary to access any other related information. Everything else that is necessary to process the message can be retrieved from the database or some other persistent store.

bank-tellerIt is sometimes easier to understand this if we think of messages (and messaging systems) in term of interactions that happen in the real world. For example, when you go to the bank to move money from one account to another, you don’t go to one teller and request a copy of all of your bank account information for your two accounts and then walk over to another teller and hand them all of your account information for both accounts, your social security card and driver’s license and ask them to transfer some money from one account into the other. Instead you go to a single teller provide them your ID, the two account numbers and the amount of money to transfer between them. Any other information that the teller might need to process your request (command), such as account balances, can be retrieved using the information you have provided. This way, the teller (command handler) can make decisions based on the actual state of your account and not on the, possibly outdated, information that you provide them. The same is true for our messaging systems. We should only ever include the information that is absolutely necessary to process the event/command and expect that the handler will pull in any other information that it needs from its source of truth.

Don’t write generic command handlers with complex business logic.

I’ve found that one of the most difficult things to get people to do is reign in the impulse to refactor command handlers down to a single generic class that can handle multiple commands. Instead, we should implement several different command handlers that each know how to handle a single command. Sometimes you will end up with multiple command handlers that do almost, or maybe even exactly, the same thing. This is OK! In these cases, we can implement a service that performs the common logic and have each one of the handlers call into that service with its own unique information.

As is the case with event handlers, we also want to minimize the reasons for change in our command handlers. If in the future, we need to change the way one of the related commands is handled, we don’t want to have to go into a generic handler ad one or more “if blocks” or switch statements with the command’s new implementation logic. Not only would this make the command handler difficult to maintain, but it would also make it brittle and more difficult to test. This will only get worse with every other command that subsequently need to be “tweaked” to work differently. If we go with multiple command handlers that only handle a single command, we don’t have the same issues. For every command that we need to change, all we have to do is change its corresponding command handler. If it is implemented using a shared service then we can implementation the command’s new custom logic and stop calling into the shared service. The result is a command handler that is still as maintainable as the original, still easy to test and not as brittle as the generic implementation.

Related Articles:

    Post Footer automatically generated by Add Post Footer Plugin for wordpress.