Waking up to numerous email alerts from our website monitoring tool of numerous failed requests is not a pleasant way to start the day. This blog post is a post-incidence report of the last outage we had on our website, the root cause, how I fix and ways to forestall any future occurrence.

Before the outage, I had previously worked web server to configure how a request is to be sent to our application server. To accomplish this task I wrote a bash to apply the desired action but the command to restart the web server after effecting the desired change somehow failed to execute and also unknowingly failed to catch my watchful eyes.

Upon careful examination, I discovered that the line of code failed to execute due to a typographical error and user's requests to this particular URI failed.

Thankfully the error was reported by the web stack monitoring tool and triggered an alert and the issues were addressed within an hour since the first alert was reported.

key takeaway from this incident is to always test your code and sometimes the error is usually the dev's fault.