Reading the tea-leaves of Big Data

Does Big Data feature as part of your business?  More importantly, if Big Data forms a critical part of your organisation’s enterprise decision making process (or even defines it’s raison d’être if information is your business), then it might be worth shining a light under the covers of Big Data. As a sequel to my earlier article on Big Data in the context of your organisation, I thought it worthwhile to explore a few additional, no less important aspects.

Amidst the enthusiasm and hype associated with Big Data driven analytics, ensuring that your organisation’s key decision makers are correctly reading the  tea-leaves of Big Data is not necessarily a trivial exercise.

Big Data is often associated with online marketing, advertising, cyber-sleuthing or national security. Out every move in cyberspace is being recorded and analysed.

What about Big Data in your organisation?

First to the finishing line in our competitive world

In our globally interconnected world, organisations are under relentlessly competitive pressure to do more with less, in shorter time frames whilst remaining relevant and financially viable.  Access to timely, accurate and relevant data and information is often the lifeblood of agile, adaptable and effective decision making.

Enter Big Data – the weapon of choice in the arsenal of organisations that are looking to accelerate decision making processes (to real-time, if possible) and to improve the accuracy and relevance of the insights offered up by Big Data.

At the heart of Big Data lies a spectrum of potential pitfalls, and making sure your organisation’s key stakeholders that are pinning their hopes on the promise of a rich harvest of value that Big Data will bring are aware of these pitfalls is key to the role of the CIO and the IT leadership cadre. Raising the awareness of these pitfalls is not always a straightforward task. This difficulty is not made any easier when everyone is immersed in a sea of opinions and perspectives, especially when presented by apparently compelling value propositions by Big Data evangelists.

1. Be aware of the politics of Data

The quote “Lies, damned lies, and statistics” that was attributed to Mark Twain, is no less relevant in the world of Big Data. The intrinsic complexity of Big Data has the potential to support the diversity of opinions held by the various stakeholders in your organisation.  Presented with compelling evidence derived from Big Data is no guarantee of consensus amongst key decision makers. The Climate change debate is a classic example of opinion trumping overwhelming hard evidence.

The question to be asked in your own organisation is: What political and related influences are at play in the all-important step of translating the findings of Big Data into actionable decisions and plans?

The degree of political influence is of course, industry and context specific. Spending $5Million investing in Big Data initiatives, only to make the wrong decision is a poor investment indeed!  Big Data has the opportunity of adding to the multiple versions of reality of when, where, how and in what way IT should be used.

2. Don’t forget the science

Organisations that treat Big Data as a magical ‘black box’ where torrents of data flow in, and valuable insights, trends and information flow out, need to be able to validate the veracity of the results.

If you are making important business decisions based on Big Data, then investing the time, resources and effort in validating the magic that occurs inside the ‘black box’ will be well spent.  Even more importantly, when it comes to identifying the risks associated with the architecture, design, logic and processes within the ‘black box’, the degree of due diligence is warranted, and should be commensurate with the importance of the decisions being made.

3. Provenance

Is your Big Data distributed across various IT providers, partner organisations and your internal systems?  If so, ensuring that the provenance of your data is well established and guaranteed is important.   This is especially relevant if your organisation is driving important strategic and operational decisions off Big Data that comprised a blend of your own, internally generated and managed data with the data vacuumed in from third parties such as commercial data wholesalers, business partners.

4. Taxonomy

How well do you understand your organisation’s Big Data taxonomy? The Big Data taxonomy (if you have one that is)  is key to ensuring that key decision makers and IT stakeholders are able to navigate the myriad of choices associated with IT compute and storage architectures, not to mention data and information security and data analytics techniques.

5. Don’t step on the cracks

Big Data relies on a seamless  interaction between its disparate components. Having cracks forming in your underlying Big Data architecture is where you do NOT want to be. By avoiding the hype and ‘me-too’ enthusiasm over the Big Data phenomenon, and by ensuring that the design, implementation and operation of all the moving parts in your Big Data ecosystem is robust, auditable and secure is crucial.  Get the 7 Vs of Big Data right.

6. Mistaking insights for noise and dismissing insights as noise

The process of managing and interpreting spurious data is key to the ensuring the validity of your Big Data.  Excluding that statistical outlier in your  data as it’s thought to be infeasible or highly improbable, may result in a serious, costly error.

7.  Audit?  What audit?

IT audits vary considerably in scope, depth as well as the degree of rigour.   If the IT audit forms part of the annual audit of the organisation itself that focuses on the controls and good governance over the organisation’s finances and related risks and you are now using Big Data to drive mission critical business operations and/or decisions, the obvious question to ask of both your own organisation and that of your Auditors is:  In what way do we need to re-scope our IT, enterprise risk and governance audits, now that we are depending on Big Data?

Are the standards against which we are being audited still appropriate in the face of Big Data? Historically, many, if not most IT and financial auditors have done well when using elementary data analysis tools such as Excel spreadsheets to perform data sampling and analyses. In the realm of Big Data, however, these tools may be inappropriate.   It might be in your best interests to perform a bit of ‘Big Data due diligence’ on your auditor.