Building amazing software products is a continuous endeavour, even though the process itself may not be linear. Bugs will always exist and things are most likely to break in production environments.

Building a software product to completion is a false claim because nothing will ever be perfect. But this is an advantage because it allows growth and points the product in the right direction. Bugs are vital in a product's lifecycle.

I came to this realization recently when I was hired under contract to fix some mysterious bugs that were affecting operations at a local startup. The main problem was that the production database had shutdown and failed to restart. Even a manual restart couldn't to work.

The poor startup had already made some losses because the problem had existed for more than two weeks. Their engineering and database teams were puzzled and couldn't figure out any way forward.

How the hell was I going to fix this. The truth is that I had never encountered a problem like this before. I am mostly experienced with the cloud but they were running their stack on a self-hosted solution powered by a Ubuntu 20.04 Server.

All in all here's a breakdown of how I approached this problem.

Examining the technology stack.

The Back End application was built entirely in typescript using Node Js and Express. They were using Postgresql for database and Sequelize library as the ORM. Everything seemed to be in good shape. But why had the database shutdown unexpectedly?

Examining the logs.

To be honest, I found nothing useful in the logs. Browsing Google and Stack Overflow wasted my entire day and still didn't find any solutions to the problem. So I was on my own. At this point I had started to become nervous and frustrated at the same time because an entire startup had put all their hope in me.

Thinking of a productive solution.

Having realized that I was then on my own, I had to think like an engineer.

As a matter of fact, there are so many reasons for a database to shutdown.Such reasons may be known and some of them may never be known. Anything is possible in technology. So I quickly decided not to waste any more time on finding the root cause of the mess.

What did I do instead? I thought of a general solution that would get rid of the the problem and all other similar problems. What could that solution be?

There was a possibility that the reason for the database to shutdown came from the platform / Operating System, who knows! That meant that the solution had make sure that the database somehow runs independent of surrounding environments. Hope you can guess where this is leading us.

Welcome to containerization. Think of it as packing a piece of software together with all it's dependencies and configurations into one box (container). That package can then be redistributed anywhere and anyone using it doesn't have to worry about compatibility with the environment. Hope that made sense.

The absolute game-changer in containerization space is Docker. Docker is easy to use and is found everywhere on the internet. This was my choice without thinking twice hence I installed docker on the system.

I searched DOCKER HUB for a stable, well-maintained container that runs Postgresql and Linux.

Implementing the solution.

Implementing this required running the docker container to be run as a daemon process and mapping the container's PORT to server's local database PORT. The final step was to then map Postgresql's local data directory to a directory inside the container.

As simple as that!

Making the solution even more efficient.

Configured the docker container to restart in case of unexpected shutdowns or server failure.

conclusion.

Hope you learnt something useful from this post. Never be afraid of approaching problems even those you've never seen before.