My Developer Nightmare

John Rush - Feb 6 - - Dev Community

I was having a New Year's Eve party, had a few beers, and went to sleep late.
The next morning I had 100+ unanswered calls and SMS messages on my phone.

I read the message. I couldn't believe what was there, felt like someone punched me in my chest.

[THE BEGINNING]

It was 2011. The world was going crazy about mobile apps. Everyone was convinced the web was ending and native apps were the future.

I was working on a bunch of boring b2b startups until my friend introduced me to a guy, who was about to start a mobile app.

[B2C SUCCESS]

The whole project went really well.
We won the best app of the year award, got into the second-best accelerator in the world, our seed funding, and dream team.
Very good growth on installs. All organic.
But it didn't turn out the way we hoped.

[B2C FAILURE]

We couldn't figure out a way to monetize all this success. Partly we hoped to make an exit, unfortunately, our main rival was acquired instead of us.
The SEED money was running out. But we weren't a good fit for Series A. We were about to close down. Had negative balance and debts.

[PIVOT]

I was very pessimistic about the whole thing, but my co-founder has figured out a chance to survive.
He pitched our app to a corporate client as a Whitelabel solution. They said: cool, we were about to build the same thing from scratch.
But they said: you got 2 months. Launch it on Xmass or no deal.
We shook hands and went on with 60 days of non-stop work. Days and nights.
We slept at the office.

[B2B Success]

Just the day before xmass, we managed to launch it, with tons of bugs which we hot fixed on prod.
The client was happy. They got lots of hype around this launch. They were the first in the world to launch an app as a business.

[WORST DAY]

I celebrated New Year Eve and in the next morning, I wake up to 100s of missed calls and SMS.
I read the first message.
It says: we were hacked. Everything is down.

[DISASTER]

Someone has encrypted the data on our server. So nothing was working and we couldn't do anything.
Back then people ran their own servers, just like we did.
The API, the database, Git, the files...Everything sat on the same server and filesystem. No backups. We simply didn't have time for this. We moved fast, breaking things. Because just 2 months ago we were on the edge of bankruptcy.
From that moment I saw the message, I sat on my chair for 57 hours in a row with just a few trips to the toilet.
I had almost no food, just water. I was trying to fix this.
The client was "firing" us over the phone. If we don't fix it in an hour, we're dead.
It took more than an hour.
The details of these 57 hours are not to be publicized, but on the third day we find a solution, that works and everything goes back up.

[After effect]

Over these 3 days, I literally aged and lost both my hair and my health.
I couldn't sleep well for many months after this.
Every notification on my phone would bring me anxiety.

This was the worst day (or 3 days) of my life.
But it wasn't the last.
Almost every year of my startup life would bring another story like this.
It's not all romantic and joy.
Sometimes it's very hard. Especially when things go well since you have a lot to lose.

[Reflection]

Being a founder is perhaps one of the most difficult jobs on earth because failure often means almost the end of the world at the moment.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .