Fundamental of Data and so called Big Data

The hot topic that seems to be around every corner these days is “Big Data”.   Most publications work under the premise that everyone already understands Big Data and the value it can bring to the organization.  My experience shows that assumption is not always correct.  Many folks are unclear how to recognize “Big Data” within their particular organizations.  More importantly, folks may not understand the possible business value that can be extracted from it. Without both aspects of understanding, adoption and success of Big Data initiatives will face difficulties.  This blog addresses those two aspects by identifying:

a) Typical “Big Data” examples within organizations

b) Real-world value propositions from harvesting “Big Data”

Simple Definition and Context

Let’s start with establishing a definition and context for “Big Data” since the name alone can be misleading.  Big Data is a reference to the very large scale of data available or being created that cannot be easily handled using traditional processing, methodologies or technologies.  Big Data can relate to structured or unstructured data.  The challenges with Big Data can include how to capture it, the methods for storing it, how to understand it, how to analyze it, how to search it, or how to visualize it and correlate it to something familiar.  Data can be tagged as “Big Data” by evaluating it against the above classifications and by considering at least two factors:

1) The scale and sheer volume of the data in comparison to what is reasonably expected

2) The speed at which it is created or is expected to grow

Since there is a lot of subjectivity in defining “Big Data”, let’s list some real-world examples to solidify the understanding.

Big Data within Organizations

Using the definition above as guidance, a growing number of possible examples may belong in the “Big Data” category. For the sake of being concise, the following represent the types of “Big Data” organizations commonly encounter and the value that can be derived from them.

Call & Contact Details

Definition

Value Proposition

Organizations that offer products and services directly to consumers will have significant call center operations with several platforms to support various methods of customer interaction and engagement.  With each interaction comes information that collectively can be significant in quantity.   While organizations have used some of this data for operational support, many have struggled with maximizing the analytical value.

Re-allocation of agents across centers and queues based on predictive demand planning using diverse criteria.  Improve efficiency on agent utilization and load balancing of resources; enhance customer experience by reducing wait time and abandon rates. Analytics against voice data for accurate tagging of reasons for call, reasons for transfers, etc.  Optimizes understanding of customer satisfaction & response levels and improves call routing. Mining of call characteristics (length, number, source, type, reason) to create stronger correlations for better insight Correlations of contact metrics to recent marketing efforts to measure effectiveness and acceptance. Creating correlations between account/customer details to offers made/accepted to increase upselling & cross-selling opportunities

Transaction Details

Definition

Value Proposition

Industries including financial firms, trading firms and large retailers surmount a continuous stream of transactional detail.  This can include authorization details, trade transactions, and purchases.  Most organizations deal with this vast amount of data by limiting the amount of detail data used during analysis or applying standard practices to summarize and aggregate for business intelligence.

Fraud detection techniques using transactions well beyond individual events; longer time spans, more criteria, un-related events.  Leads to improved risk exposure management. Trending of activity across various time periods to recognize patterns of behavior.  Effective for identifying revenue opportunity or measuring strategy effectiveness. Developing correlations between transactions and customer/account details (demographics, purchase history) to improve marketing strategies. Correlations of transactions to external factors (marketing offers, news events, regional criteria) to understand behavior and measure marketing effectiveness. Analyzing transactions using broad criteria across extensive time periods to improve forecasting accuracy.

Web Clicks & Logs

Definition

Value Proposition

Analyzing details of customers with similar interests and behavioral patterns to maximize the effectiveness of offers made to individual customers.  Will improve and extend sales.Customers and prospects visiting the organization’s web sites have their own distinct behaviors and patterns.  What they click, what they click next, what peaks their current interest, what peaks the interest of other visitors at this same moment are examples of behavior patterns that are valuable to better understand.  Furthermore, each visit brings other interesting criteria such as originating source for visit, geographical tags, SEO tags, etc.   Only a handful of companies are recognizing the strategic value in this treasure chest of information.

Correlating product sales to one another to understand buying patterns, i.e., what other products are bought along with a given product.  Improve offers and up-selling opportunities. Analyzing regional considerations for customers during a given experience to optimize target marketing and ensure relevance of offers. Analyzing click and navigational patterns to improve customer experience, i.e., offer online chat or display tips to improve “stickiness” and overall experience. Correlate traffic, interest and behavior to external factors including media efforts, regional criteria to measure strategy effectiveness.

Other interesting Big Data examples you may encounter include:

Application Logs

Informational, warning, error, monitoring and event messages are continuously produced by software systems, hardware devices and application platforms.  Proactively recognizing potential issues from the patterns can help improve the quality of service and reliability that the IT groups need to ensure.  Furthermore, it can be an element of a good risk mitigation strategy if the services and platforms are a critical part of your business.  This content is often overlooked for the value it possesses.

Social Media

Tweets, Facebook and Google+ posts, blogs & responses have quickly become acceptable means of social interaction between people.  The sheer number of people using these channels creates a “Big Data” problem.  The data and growth is exceptionally large to deal with. The content is text-based and needs to be evaluated in context to derive at the right interpretation. Furthermore the relevance of the content (eliminating noise) is difficult to decipher.  The jury is still in deliberation over the ROI for harvesting this information.  Nonetheless, it’s difficult to ignore.

GPS Trace Records

Equipment, products and personnel are increasingly fitted with GPS technologies that can track every move from point A to point B.  The ability to proactively analyze this movement can lead to supply chain efficiencies, human capital effectiveness, bottom-line cost reduction, fraud mitigation and allow for overall control and continuous visibility.

Technology & Instrument Output

Needless to say, there are countless unique examples within industries.  Utility and communication companies produce incredible amounts of usage details that can be used to manage demand and optimize performance.  Genomics and scientific organizations are deploying technologies producing ever granular bits of potentially important information.

Documents & Other Unstructured Data

Virtually every organization produces an immense amount of unstructured data, or in other words information that does not easily conform to a defined data model.    This can include internal documentation, publications, correspondences, health records, audio recordings, etc.  Not only is this a content management problem but it also requires unique analytical techniques to harvest value from the content.  Increase the scale of it and it now becomes a Big Data challenge.  Businesses can use this data for ensuring compliance, managing risk and achieving more complete records.

Many more organizations will have “Big Data” challenges over the coming years. Some of this can be attributed to their individual growth as a company but much of it is the result of technology advances and outside factors.   It is safe to conclude that all organizations with Big Data will need to take some action to do something valuable with it, at least to remain competitive.

In future blogs I will talk in more specifics about individual approaches, technologies and business application around Big Data.  In the meantime, please feel free to comment below or reach out to me to talk about “Big Data” challenges you are facing.