Three common mistakes of Microservice Design

Building distributed systems can get complicated. So can building a monolithic one, to be fair. But the difference is most of us choose more complexity than we need by going distributed. Any experienced developer or architect will tell you that most people actually don’t need to embrace microservices entirely. All the ones I’ve spoken to suggest a blend of both worlds. And only where absolutely necessary.

But it’s not like my team and I had that wisdom back then. So we jumped in head first. And made every mistake in the book. Here are some of the most egregious.

Too Much Custom Building

Communication between services is a big problem in the land of microservices. Some would say it’s the biggest problem. And it’s all thanks to distributed transactions. In the classic example of an e-commerce application, the process of creating a new order might involve actions that take place in multiple different services, such as the order and customer service. That’d just be one function call in a monolith. But with microservices, it’s not so pretty. You could get into sagas to handle multi-service transactions. But that should send chills down your spine. Sagas aren’t trivial to implement.

Nevertheless, we didn’t know better. So we implemented orchestration-based sagas to solve this problem. Best part was we custom-made the message broker used by each service to communicate and execute these sagas. Built it ourselves using Redis streams and Go. The end result is neat, sure. Super fun to build, too. But in the time it took us to make that, we could’ve made the entire application as a monolith. Startup fails galore.

Too Much Complexity

This one comes down to experience: There are some places you just shouldn’t go, technically speaking. Given a certain project timeline and a certain team with a certain set of capabilities, there are some paths you just shouldn’t explore. And our mistake was we thought we could go anywhere the microservices gods called us.

Which was incorrect. Tragically incorrect. At some point, you realize that just because something sounds feasible in a YouTube keynote, doesn’t mean it’s feasible for your project. If we had placed a cap on the complexity we were willing to handle, we would’ve saved ourselves several unnecessary tangents. I guess another way you could state this mistake was, “we gave ourselves too much time”. If we had a shorter deadline, these tangents would’ve been eliminated automatically.

Then again, there’s a fine line to walk, here. Set the complexity limit too low and you end up flying a plane made up chopsticks. But set it too high and you’ll never leave the runway. In both cases, you’re screwed. And knowing how to cut the pie usually comes from experience. So I suggest taking your best guess and broadcasting it on Medium to attract the attention of wiser engineers who’ll set you straight.

Loose Definitions

Finally, the one thing that would’ve solved most of our issues. At the end of the day, a distributed architecture exists to solve a problem. So you need to know what that problem is before deciding to use it. And you also need to know your problem, in order to determine how good of a match it is. And we knew neither.

Because who spends days on defining the problem at the start? That discipline is rare, especially in an environment where there are things to build immediately. Now, I know that the time you “lose” defining a problem correctly is saved by a more focused implementation. In other words, you spend less time building the wrong things. Way less time. So it’s wise to make sure what you’re building is correct.

However, we didn’t. And our detours didn’t just cost us time and money. They also ended up being pointless. We built and fixed things we didn’t need. And after they were done, we didn’t use them. We might as well have been playing League of Legends in the office during that development time. Probably would’ve boosted morale, too. Hence knowing your problems and solutions very well isn’t a terrible idea.

Then again, where’s the fun in that? The way I see it, you could also waste a bunch of time building the wrong thing, gathering experiences to write articles about in the process, then complain about it all online. Which worked for us. I mean, we still lived to tell the tale.