This article was first published in BroadcastPro Magazine, August 2020 issue
Nine years ago, I wrote an article in BroadcastPro on “Failing to Prepare: Preparing to Fail”, where I expressed alarm at the lack of Disaster Recovery (DR) and Business Continuity (BC) measures at regional broadcast facilities should a crisis hit. Nine year later, tragically, many regional broadcasters are still struggling with pretty much the same issues and the truth is they could have been better prepared this time.
No more than a handful of broadcasters in the whole MENA region had any BC plans in place for a time such as this and even in those cases, they were limited to certain aspects of their operations. They are typically for backup channels playout, keeping some evergreen material running off a remote server with a basic logo bug until they figure out a solution to their problem.
While some risk scenarios can easily be anticipated, choices were made not to address them. There will be technical failures of broadcast equipment and power supplies, cyber or physical attacks on the broadcasting facilities, telecom and satellite network outages, or natural disasters such as floods or lightning. We have had many incidents over the years impact regional broadcasters and production facilities, both national and commercial, large and small. While everyone gets into firefighting mode when such problems arise, as a regional industry, we have collectively failed to learn any lessons from past instances.
Who, then, is responsible for forecasting and preparing for such events or scenarios? Is it a failure from the technical teams who are typically tasked with initiating new technology investments, budget requests, and requirements specifications? Is it in the finance departments who tend naturally to resist “unnecessary” costs and investments? Or is it the top management’s indecisiveness?
In most cases, we have seen a collective management failure to decide which investments and planning activities should be on their priority lists. Usually, short-to-medium-term projects gain priority over strategic long-term ones, and often, DR and BC plans, and investments in them, are on the bottom of such lists. They do emerge to the surface occasionally when an incident occurs at a broadcaster’s facility, but they quickly plunge back down to the “nice-to-have” list.
An open and honest discussion needs to take place about what a broadcast or production facility’s risk-management priorities should be, and funds should be allocated to address these elements.
Middle- and top-tier broadcasters do spend considerable time, effort, and budgets on fully-redundant production and broadcasting facilities with the infamous target of “No Single Point of Failure”. But they often fail to spend a fraction of that effort and investment in complementing them with DR facilities and capabilities and BC plans, or updating their teams on how to maximise the potential of those investments.
We have been recently involved in a number broadcast projects where we discovered that even when the implemented facilities’ designs had sufficient resilience, redundancy, and backup features, no documented procedures were available for operators and engineers to activate backup and DR scenarios. Although such documentation and procedures were well developed at the time new systems were procured and installed, they are never reviewed, updated, refreshed, or rehearsed. In some cases, we discovered that new staff do not even know that such documents and procedures exist.
One could argue that current disruptions and challenges could never have been anticipated, but this is not really true. If a broadcaster had BC plans to mitigate risks caused by not being able to access their facilities because of a natural disaster, a strike, or any other cause, then such plans could be activated and adapted to operate within the restrictions caused by the pandemic to comply with authorities’ or self-imposed requirements of reduced on-premises staff.
Today’s challenge offers yet another opportunity for broadcasters to prepare themselves for multiple scenarios. They should look at best practices within DR and BC and see how they can be potentially adapted to different situations.
DR and BC plans should be on our top priorities’ list, and we should vociferously advocate with the decision makers to ensure that this is part of the 2021 budget. Otherwise we are preparing to fail yet again.
Hasan R. Sayed Hasan is Managing Director of Master Media, and Chairman of the Arab HDTV and Beyond Group
