Driving Costs out with Digital Transformation

If you turn on your TV for more than five minutes these days, you can’t help seeing those great commercials with dramatic music, shimmering lights, and suave spokespeople making sweeping pronouncements about a future made brighter by digital transformation (DT). Those commercials make for amusing viewing, but they don’t provide much in the way of anything but teasers on DT.

One reason might be because DT is such a vast concept—and still somewhat nebulously defined—that it’s hard to fit an adequate description into a 30-second spot crammed between car and golf ball commercials. However, there’s one area of agreement between almost all experts and pundits who tout the virtues of DT: it has enormous potential to drive costs out of your business.

There are three ways DT can help you transform your organization and drive costs out:

  1. Streamlining your IT infrastructure to gain the benefits of technologies such as AI
  2. Automating activities and processes to and free up human resources to concentrate on analysis and value-generating activities
  3. Using AI to become proactive by predicting and mitigating problems before they manifest

Streamline your IT Infrastructure and development processes

The infrastructure and computing power needed to make sense of the deluge of data you face is a rapid drain on resources. You can digitally transform, and streamline, that infrastructure. The cloud is your best option here. Cloud deployments—especially managed-service, intelligent cloud models—enable you to spin up or down when you need to, accessing the storage and analytics resources you need, when you need them.

And you only pay for what you need, so even though you’ll have access to cutting-edge computing technologies, you’ll spread the cost over the value and won’t incur up-front costs that may not ever deliver a return. You can gin up multiple models, try them out, learn, discard them, and move on—without incurring catastrophic costs. What’s more, the costs are not only reduced, they’re transferred from CapEx to OpEx, which improves your bottom line.

Automate repetitive activities and systems

It’s an ugly truth, but people cost money. However, by leveraging digital technologies such as AI and machine learning, you can automate many labor-intensive processes that are currently performed by humans and free those humans up to add their tremendous intellectual value to your workstream.

Customer service is ripe for automation—think chat bots that many banks and retailers are now employing as first-level problem solvers. But there’s so much more. Robotic process automation will revolutionize the back office by automating such diverse processes as invoice processing and sales support. They’ll also transform tech support functions in finance, HR, marketing, sales, etc.

Digital supply networks will also revolutionize the traditional supply chain. There’s almost no limit to automation-ripe functions. Each automation drives costs out of the equation and lets your people contribute their intellectual capital rather than their manual labor.

Become proactive

We’ve all seen those funny commercials on TV where the elevator talks and lets the maintenance guy know there will soon be a problem. It’s amusing, but it’s a simplified version of actual capabilities that are available to help you transform from a reactive organization to a proactive powerhouse that can predict problems before they happen and make plans to solve or mitigate them.

GE pioneered one form of this transformative technology with the concept of digital twinning, where they put sensors in their engine parts and, over time collected and analyzed performance data to enable them to predict how, and when, engines would fail. They created digital engine models (twins) and, based on those models’ performance, were able to create more effective maintenance schedules and significantly reduce costly unanticipated failures.

What wouldn’t you give to know when your products and processes would fail or fall behind, and how? How much could you save if you could develop maintenance schedules for your production equipment based on solid predictive numbers rather than someone else’s set schedule? If you could devise better sales strategies based on more accurate predictions of market conditions and customer behavior? Enough to make it well worth the investment in technologies, such as machine learning, that enable you to make those kinds of predictions.

It’s always the bottom line

These are only a few of the ways to leverage the power of digital transformation to drive costs out of your company. As many activities as your organization engages in, there’s a digital technology to help transform each of those into a more efficient, effective process. As you become more efficient, your costs decrease, and your productivity and margins increase. And that’s what it’s about—increasing those margins to grow your bottom line and shareholder value for long-term success.

Why Your Analytics Program is Struggling

I have a buddy who’s a golfer—really more of a duffer, but he tries hard. He also spends a lot of money on equipment. He truly believes that each new driver or wedge will get him to sub-90. But it never does. To me the answer is obvious. My friend’s problem isn’t equipment; it’s his approach to the game. He doesn’t do what he needs to play better. No lessons, no swing improvement, and not enough structured practice. Eighteen holes on Sunday morning with a cooler won’t do it. But my buddy keeps on spending on equipment, oblivious—and getting poorer by the month.

What’s the point of this? It’s not just a sob story; it’s relevant to a lot of what I see and hear when I talk to clients about their analytics programs. They spend a lot of money year after year, with little improvement in results. They chase after the latest and greatest tools, but they’re not doing the things they need to, to achieve their desired outcomes. What’s worse, analytics tools are seriously more expensive than golf clubs. It’s often a downward spiral that ultimately costs someone a job.

There are two categories of outcomes that will affect the success of your analytics efforts: business outcomes and technology outcomes. Business outcomes are the results of devising and implementing your business strategies. Technology outcomes are the results of how well our technology performs in helping you realize your desired business outcomes. If your business and technology outcomes aren’t well defined, and they aren’t in sync, it won’t matter how many analytics tools you buy, your analytics efforts will struggle.

Business Outcomes

Markets are changing faster than ever, and the future comes at you before you can blink. To survive the changes and capitalize on that future, I believe that business-outcome success should be defined keeping three criteria in mind:

  • How you deal with customers. It’s not enough anymore to satisfy your customers. You must delight them. To delight them, it’s critical to create a sense of greater intimacy between your business and your customers. Customers want to feel as if you know them—what they need, what they want, and—most importantly—what they don’t know they want.
  • How you manage and mitigate risk. There’s no reward without risk, but too much risk leads to disaster. What’s more, the types of risks have multiplied exponentially (fraud, cyber-security, market, economic, political) and it’s essential to evaluate them appropriately and manage them effectively. It’s essential to integrate technology—especially AI—into all aspects of risk management.
  • How you innovate. Innovate—both in terms of what you offer the market and how you operate your business. Social media has changed the market landscape and businesses that don’t constantly innovate die more quickly than ever. Innovation won’t be possible, however, unless your business is operating at its optimal level. To do that, you must constantly innovate your operations to achieve and maintain operational excellence.

Technology Outcomes

The only thing changing faster than the markets these days is the technology being developed to help companies master those markets. Once beyond the reach of many companies, predictive analytics has become table stakes. Architectures have also changed. The cloud is no longer an option, it’s a necessity for flexibility and security. But what kind of cloud? And what’s the relationship between analytics and the cloud? Those are questions that must be addressed and answered in terms of the potential outcomes for different options and combinations. The key to determining what options will work best for your business will be to look at those options through the prism of your business outcomes.

Creating customer intimacy requires massive amounts of data and complex, sophisticated analytics capabilities—think AI and machine learning. Risk management also requires complex analytics and sophisticated analytics, but it requires ironclad security as well. Continual innovation and operational excellence requires it all—big data, sophisticated analytics, and security—but it also requires flexibility in your IT infrastructure. You must be able to spin up or spin down, according to your information or project needs.

There are so many options out there: public, private, and hybrid clouds; intelligent clouds that mingle the cloud with analytics technology (my personal favorite); and machine learning algorithms to augment those analytics. The list is almost endless. However, the technology options you choose must be purchased with the goal of enabling you to meet those high-impact business outcomes that you’ve defined. If you don’t have clear outcomes in mind, and match the technology to those outcomes, the technology is useless—just like my buddy’s latest $500 driver, coupled with that terrible swing, can’t get him a 300-yard drive, no matter how hard he hits it.

2023—The New Intelligent Enterprise

In last week’s blog, I outlined what I thought future companies would look like vis-à-vis their information technology and decision-making capabilities. I believe they’ll be intelligent, connected, and agile. Over the next several weeks, I’m going to dive deeper into how I think companies will realize their futures. And, as I’ve said, that future is not truly five years from now. It is now.

This week I’m going to explore how smart companies will leverage new technologies to become more intelligent. I’ll use the manufacturing sector as a prism for looking at how intelligence will differentiate companies from their competitors and enable them to thrive in a global, competitive network that leaves little margin for error.

On a good day in a production facility, things go smoothly. Everything functions as it should: parts roll in, products roll out, and production managers can breathe a sigh of relief at the end of the day. Good days don’t breed successful companies; they breed complacent ones. True success is bred by how well you deal with the bad days—and solve the problems that created them.

It happens to every company. Machines go down, diagnosis and repairs take critical, revenue-eating time. If the parts are local, the repair is quicker, but if they’re sourced nationally or globally, the company could be looking at loss runs in the millions of dollars. That’s for one production facility.

Most large companies don’t just have one plant; they have several—and those are typically located globally, exacerbating problems. Connectivity between plants is often spotty, and each plant typically comes up with its own solution to problems. Data is collected and created, but so are silos. Intelligence and insight become scarce commodities.

Successful companies will re-imagine their future and deploy analytics augmented with artificial intelligence (AI) across their entire value chain. Why AI? Because machines can be trained to learn, to recognize patterns and trends, to spot changes and exceptions, and suggest alternatives that may not be apparent to humans. Humans can then be presented with deep, relevant data they need to inform their decision-making process.

But there’s another benefit. When companies use AI to create intelligent analytics and gain better decision-making at one node in the value chain, those results can be applied across the entire production and logistics network to make the company smarter and more responsive. More than that, companies can predict problems—and head them off—before they happen.

As an example, let’s look at a global energy producer. The company might have thousands of wells and production facilities in 20 countries. What happens when a well breaks down, when three, or five, or ten wells break down simultaneously? Millions in revenue per day could potentially be lost.

However, using AI, the company can install sensors on their equipment to analyze performance across their production network and predict maintenance failures and repair needs. Downtime can be planned. Decision-makers can schedule production re-routes and revenue losses can be reduced or eliminated.

This example is only one instance where intelligent analytics can be used to control production networks and make smarter, faster decisions. Other uses include:

  • Critical component tracking
  • Delivery variability analysis
  • Predictive maintenance
  • Product and process efficiency and quality analysis
  • Safety improvement
  • Supplier quality analysis

Intelligent analytics will revolutionize how companies handle their data and operations. However, there’s a caveat. Intelligent analytics capabilities must be deployed in weeks, not months. The analytics companies deploy must combine the rich insights of analytics with the promise of data science to generate proof-of-concept models that can show improvements quickly and be propagated throughout the enterprise and generate quick, measurable ROI.

The bottom line here is about the impact to your business outcomes. Intelligent analytics can help you become smarter, faster, and more responsive. It can help you understand not only what’s happening, but why it’s happening—and most critically—what might happen in the future so that you can begin to control that future and make it work for you, not against you, so that you can differentiate yourself from your competitors and own the market.

2023—The Company of the Future

Technological change is happening at a rapid pace, no more so than in information technology. In the past decade, the deluge of big data has wrought a vast transformation of the IT landscape. Predictive analytics has become common-place. New technologies such as artificial intelligence (AI) promise to change the way we approach analytics and decision-making. Additionally, the pressure the gain efficiencies, connect more intimately with suppliers and customers, and increase top and bottom line revenue to maximize shareholder value has made it imperative to respond more quickly to market changes.

So how will companies deal with these pressures? The ones who thrive—not just survive—in the future will have three characteristics: intelligence, connectedness, and agility.

They will be intelligent

Over the past 25 years, business intelligence and analytics technologies have revolutionized business decision-making. However, it’s not enough anymore to simply buy BI or analytics tools, load your data, slice and dice it, and interpret the results. Big data has made traditional tools almost obsolete—especially for large, complex, or multi-national companies. In five years, the analytics technologies in play will have undergone a profound change, and that change is already underway.

Artificial intelligence is the future of analytics. Retailers are already adopting AI technologies for deep learning to manage customer interactions and increase customer satisfaction. In fact, Garner predicts that by 2020, 85% of all customer interactions will be managed by AI technologies.

AI-augmented analytics can be applied to nearly every industry and sector. For example, financial institutions can use it for fraud detection, portfolio optimization, and customer service. Agricultural concerns can use AI to improve research on farming methods, manage crop yields, minimize the use of pesticides, and manage livestock. In healthcare, AI can help doctors diagnose diseases faster and more accurately, help researchers find cures for those diseases, and it can help health systems manage costs and patient populations more effectively while providing optimized care.

They will be connected

The supply chain is dying. It’s being replaced by perpetually-connected, digital supply networks (DSN). A DSN is an ecosystem where information flows to and from who and where it’s needed, when it’s needed, to maximize efficiency throughout the network. Information latency is no longer an issue, because on-demand, near real-time information flow creates almost instantaneous insights that are accessible to all interested and authorized parties.

On a macro level, increased information flow speed provides the insights needed to develop more effective business strategies, which leads to more business opportunities and helps create new strategic advantages.

The benefits of these freed-up, on-demand information flows are enormous. More timely information can help reduce costs by improving efficiency. The back-and-forth data flows can also help companies gain the information they need to make product improvements more quickly, which results in more satisfied customers. More satisfied customers lead to a better reputation, and with that reputation and better information in the hands of management and salespeople, sales effectiveness—and revenues—can soar.

They will be agile

To be sure, AI and DSNs can enable companies to become more agile by gaining the ability to better understand and interpret their environment and become more responsive to market needs. However, the foundation of this agility is the technical infrastructure that supports all these dynamic technologies. If that infrastructure is not cutting-edge, it can’t support innovative technologies. That cutting-edge infrastructure is the cloud.

But not just any cloud. There are many cloud options: public, private, hybrid, and on-premise. However, in the near future, companies that thrive will gain the ultimate flexibility by combining their analytics and cloud technologies into an intelligent cloud, so that their analytics aren’t simply supported by the cloud—they’re integrated with it.

Intelligent clouds contain analytics capabilities and cloud storage, bundled as a managed-services package—on a subscription basis—that can be scaled up or down quickly, at will, depending on your needs. They allow you to deploy the latest, and most sophisticated analytics capabilities, combined with flexible, secure, cloud storage that fits your workload size and operational requirements.

With an intelligent cloud, you get three things:

  1. Your resources are there when you need them.
  2. Peace of mind. Security and management are taken care of for you.
  3. Scale up or down as needed to meet demand quickly.

I titled this blog, “The Company of the Future,” but in reality, that future is now. All these technologies are available today, and forward-thinking companies are implementing them. I believe that in five years, companies that haven’t implemented these technologies—at least to some degree—will be short-changing their future and may end up with a bleak one.

The Virtuous Circle of Machine Learning and Data Quality

You’ve made the decision to engage in a machine learning (ML) initiative to augment your analytics. That’s great; it’s the wave of the future. I’m a huge proponent of ML because I think it gets close to the approach that humans take when they learn—processing all relevant experiences and learning from the outcomes of those experiences. That’s a natural approach that can yield deeper, and more prescient insights than you can get with analytics technology alone.

However, there’s one mistake that I’ve seen companies make that often sours them on ML—and artificial intelligence in general—even though it really has nothing to do with ML technology, or the algorithms used in it: they use low-quality data, so they get erroneous results. The problem is that they often don’t realize that their data quality is poor, so they blame the algorithm for the bad results.

But how do know you can trust your data? It’s a process. You can’t rely on gut. You have to ask hard questions. The first question is, “Where did this data come from?” The second is, “Can I see the data quality stats?” If your data scientist can tell you that it came from your internal data warehouse, and that it’s been cleansed and formatted, and that the data quality stats are acceptable according to your company’s data quality policies, then you can use it.

If the data is from a public source, or from a data gathering effort and it hasn’t been scrubbed and validated, you shouldn’t trust it. Before you use it, insist that it be validated and measured against your data quality policies. If it doesn’t pass the test, insist that it be scrubbed before you use it. You’ll get pushback, but in the end, the cost to clean the data will be worth the insights it provides.

Speaking of quality data, in a virtuous circle, ML can help with your data quality efforts. Specifically, it can help with one of the biggest headaches you face when trying to improve your data quality: matching and de-duplicating data. For example, let’s take say you’re a large financial institution and one smaller bank you deal with is Eastern Community Bank. Across your different business units, this bank may be referred to as ECB, Eastern Bank, Community Bank—you get the point. Whether it’s in systems or spreadsheets, humans are entering the data and they take shortcuts or make mistakes.

Machine learning can help you catch those types of errors and clean up your data, and keep it cleaner going forward. Normally, the process to find and clean this data would be arduous—even with the excellent data quality tools available on the market. However, with ML, the process is simplified, because with the correct data matching algorithms, the machine learns to match data and clean it up as it goes.

Suddenly, the matching process that took weeks, now can be delivered in days. What does that do? Well, let’s say that Eastern Community is failing. What’s your exposure? Instead of waiting weeks for the answers as your analysts pour over spreadsheets, and tease out possible matches in different, unconnected systems across your organization, now you can have the information more quickly, and you can trust that it will be accurate and complete. You can evaluate your exposure and formulate a plan quickly—probably more quickly than your competitor who also has exposure to ECB. How big of an advantage is that?

What’s more with ML, the algorithm learns and gets smarter, so when you feed it the next set of data, it leverages its experience and applies it to that data set. The process repeats itself with each set of data, so it gets quicker over time.

Data matching is only one example of how ML can help with your data quality. There are many more such as error detection and correction, continuous monitoring of data formatting and quality, and automatic enrichment of data without human input—to name but a few. The developments are coming at such a rapid pace, that new uses for ML vis-à-vis data quality are appearing almost every day.

It really is a virtuous circle. Machine algorithms are virtually useless without clean data. It’s the old garbage in equals garbage out axiom. However, if you feed them clean data, they can enhance your insights far beyond your non-ML-augmented analytics can achieve. And, ML algorithms can also help themselves by helping you clean your data. That’s a win-win.

You Need an Intelligent Cloud

According to a recent IDG survey, about 70% of companies have at least one application in the cloud. An additional 43% want to migrate most, or all, of their data workloads and analytics capabilities to the cloud over the next few years. Cloud adoption is growing due to pressure to increase agility and responsiveness to market changes, shrink data center footprints, adapt to changing analytics needs, and move IT costs from CapEx to OpEx.

However, cloud migration is often hindered by concerns about security and the lack of IT staff with experience to manage a cloud environment. As a result, many companies struggle to identify the cloud model that’s right for their goals. One model that can help you meet your goals, regardless of your size or resource capabilities, is intelligent clouds as-a-service.

Intelligent clouds contain analytics capabilities and cloud storage, bundled as a managed-services package—typically on a subscription basis—that can be scaled up or down quickly, at will, depending on your needs. They allow you to deploy the latest, and most sophisticated analytics capabilities, combined with flexible, secure, cloud storage that fits your workload size and operational requirements.

What you get

With an intelligent cloud, your focus is shifted from wrangling your IT infrastructure to leveraging sophisticated analytics capabilities and large data sets, so you can gain faster, deeper insights into your business to improve performance and outcomes. How?

  • You get predictability, both in performance and cost. You pay only for the storage and analytics capabilities you need, and you can scale up or down quickly, depending on your everyday workloads, or any special projects that you might wish to undertake. This gives you resource efficiency and allows you to get more from your technology investments. It’s an awesome capability to have when you’re responding to ever-changing competitive and market conditions.
  • You get peace of mind. You don’t have to worry about your data, or your analytics capabilities. If you have a DBA on staff, that person can manage the interaction between you and your cloud/analytics partner. Otherwise, the infrastructure management, and the analytics capabilities are taken care of—not to mention the security of your data and applications. The entire process is managed for you, to enable you to focus on your business, not your IT infrastructure.
  • You get availability. Because the process is managed for you, and it’s infinitely, immediately scalable depending on your needs, your analytics and storage capabilities are there when you need them, as you need them. Most cloud/analytics vendors will guarantee performance in their SLAs. Can you do that by managing your own infrastructure? Probably not. Also, because your needs can change, and you need ultimate flexibility, most analytics/cloud vendors will offer options to go with public clouds like Amazon Web Services or Microsoft Azure, their own hosted clouds, or a hybrid thereof. It’s the ultimate mix of scalability and flexibility.

What to Look Out For

It’s clear that intelligent clouds as-a-service can help you maximize your efficiency and focus on boosting your analytics outcomes, but there are some things you need to look out for when choosing a vendor partner to manage your intelligent cloud.

  • Performance guarantees. Best-in-class vendor partners will guarantee 99.5%+ availability. That lessens the possibility of downtime for mission-critical applications and gives you the confidence to undertake large-scale projects with big workloads, without worrying about whether you can scale up or have access to your data and apps when you need it.
  • The best vendors will offer sophisticated security with excellent encryption and monitoring capabilities. They shouldn’t ever see or touch your data in the process. The SLA should clearly spell out security measures and guarantees to give you peace of mind as your workload migrates from your control to theirs.
  • Superior customer service and management. Subscription and pricing models should be flexible and customizable to match your needs—now and in the future. Top vendors will offer some sort of easy-to-use portal system where your DBA can perform tasks such as scaling up or down, viewing metrics, starting and stopping the database, scheduling backups, and setting firewall and security parameters. Your SLA should also offer 24/7 customer service, or at least 24-hour service during the work week.

There are almost infinite cloud service models out there, and it’s crucial to your success in producing better analytics outcomes to choose the right one for your business. Intelligent clouds, with their flexibility and performance, can be that right choice for any business, large or small.


Choosing the Cloud Model that’s Right for You

Let’s talk about clouds and analytics. I know there are thousands of blog posts and articles written how you can use the power of the cloud to boost your analytics, and the best deployment models out there. Most of them I read, though, use really technical language and don’t really settle on a bottom line.

There are many definitions of cloud: on-premises, public, private, hybrid—with ever-changing variations on each of those terms, according to the particular cloud service provider. How can you know what’s best for you? It’s simple. What’s right for you is the configuration that lets you worry the least, while getting the best business outcomes.

What’s Out There

On-premises clouds offer the ultimate in security and control. You have to provide the footprint, the physical hardware, security, and IT resources to manage the entire infrastructure. With public clouds, you don’t have to provide the hardware, but you still have to provide resources to manage the process, and there’s the issue of sharing space and possible outages.

Private clouds are dedicated to you, but they’re hosted on a cloud provider’s infrastructure. However, you still have to have a cadre of resources to interact with the cloud provider and manage the process overall. Private clouds are secure, to be sure, but they’re also limiting in that you still need time and planning to ramp up or down.

Managed cloud models are those in which resources at a cloud service provider fully manage your cloud environment and analytics infrastructure. The cloud utilized can either be public or private, or a combination thereof. Managed clouds are great for those companies that don’t have the resources to implement or manage the large data sets and the concomitant infrastructure it takes to perform deep, complex analytics.

Hybrid clouds offer the ultimate combination of all the types I’ve discussed above. With a hybrid model, you can combine managed or self-controlled, on-premises, private, and public cloud deployments in any combination, depending on your needs. You can orchestrate them to work together to meet on-demand data and analytics requirements. This creates a borderless environment that enables you to focus on your analytics without having to worry about where the data you’re accessing resides. You can ramp up or down at will, with minimal or no disruptions. The resources needed on your part are minimal, but more than with managed clouds.

Getting the Right Cloud for Your Business

This sounds simplistic, but the right cloud configuration for you is the one that helps you meet your needs. Not those of your cloud provider, your competition, or anyone else. However, there are certain requirements that any cloud deployment should meet. These are not so much technical requirements as business outcomes—and they should be met regardless of which configuration you choose.

  • Deployment flexibility. Users should have access to the data and analytics capabilities they need, when they need it, and IT resource requirements should fit your level of expertise and desire.
  • Workload optimization. Whatever the size of your data set, the cloud infrastructure and management plan you choose should optimize the storage of, and access to, that data. Access should be seamless and transparent to users. In other words, data access should be lightning quick, and users shouldn’t notice a difference in accessing data, based on where that data resides.
  • Customization based on needs. You need to have the level of control that your resources permit, or that your security needs require. If you want complete control of the infrastructure, security, and management, your choice will be different from that of a company that wants to focus solely on utilization for analytics and is willing to cede control to the cloud provider. Don’t let anyone tell you what you need. You drive the process.
  • The right investment model. Your desire for control and resource allocation will also determine your investment model. If you want to focus solely on analyzing your data, the cloud-as-a service model is right for you. If you don’t want to cede control at all, the on-premise or private cloud is the way to go. If you want flexibility, a hybrid model is your choice. It’s your choice, not the vendor’s.

The bottom line is this: the right choice for you is the one that will best achieve your desired business outcomes, and that’s within your human and monetary resource budget. That’s all that matters. Don’t let anyone tell you otherwise.

Machine Learning–Algorithms and Applications

In my last two posts, I’ve talked about machine learning (ML) and how it can help you get more out of your analytics and data integration efforts. Because ML is not a specific technology, but rather a deep and complex set of mathematical algorithms, it’s important to understand which types of algorithms will help you get the insights you seek from your data—and which will give you insights you didn’t even know you wanted.

I’m going to discuss three very broad categories of ML algorithms: supervised learning, unsupervised learning, and hybrid models that combine elements of the other two. The use of each type is dependent on your goals and your appetite for uncertainty.

Supervised Learning

Supervised learning is the workhorse of ML. It involves training a machine with paired data—a series of inputs where the output is known. Feed the machine enough of these data pairs and it learns which data go together.

For example, if you feed the machine information on the stock market, along with date and economic information, you can construct a relatively accurate predictive model. Of course, it won’t be 100% accurate (humans value and run companies, so there’s irrationality, and therefore unpredictability) but with enough time and data, the model will get really good at predicting the Dow-Jones Average.

You can also build supervised learning models that classify things. For example, researchers can feed the machine population and epidemiological data and build a model of people who are likely to get cancer, heart disease, or diabetes.

Companies can build predictive models of customer segments that are likely to churn, demand forecasts, project outcomes, financial performance—the list goes on and on. The big benefit here is that with ML, you can more accurately predict events or behaviors, and you can devise and implement strategies that capitalize on those models.

Unsupervised Learning

Unsupervised learning is the powerful wildcard of ML, although its power is unfortunately sometimes hindered by its unpredictability. With unsupervised learning, the inputs are known, but the predicted outputs aren’t. Like many humans, the machine learns by trial and error.

Inputs and outputs are paired by experience. Given enough data and time, the algorithm will show you patterns in the data that you would never discover using supervised methods. However, because the outputs aren’t known in advance, it’s often difficult to attest to the validity of the model.

Clustering, is one common technique used for unsupervised learning. It involves grouping set members with common traits together. For example, you can segment customers with similar buying habits or other behaviors. The difficulty lies in knowing whether or not these groups provide useful insights, how many of them should exist, or whether they’re even grouped correctly. You can refine the model over time, but there’s always a level of uncertainty. If you can live with that, though, unsupervised models can provide unique and very valuable insights.

Hybrid Algorithms—The Best of Both Worlds

The best of both worlds are hybrid algorithms that combine elements of both supervised and unsupervised learning methods to couple the relative certainty of supervised learning with the power and novel insight generation of unsupervised learning. One of these so-called hybrid models is reinforcement learning.

You might have heard of this type of algorithm if you’ve read about computers that have trained to beat opponents at games like go, Atari, and chess. Reinforcement learning algorithms basically pair observations and measurements to a prescribed set of actions in the process of trying to achieve and optimize a reward. The computer interacts with its environment in attempt to learn how to master it.

The outcomes aren’t known in advance, but desired outcomes are rewarded. Reinforcement learning can be applied to all sorts of business activities such as risk management, inventory management, logistics, product design. The list is huge. The bottom-line benefit is that reinforcement learning can help you discover the optimal outcomes you seek, and it can reveal outcomes that you didn’t seek, but that you can leverage to optimize your operations.

Two Caveats

There are two caveats to keep in mind with ML. One: it’s easy to let bias creep into ML algorithms, so you must constantly measure your results against your goals, applicable standards, and ethics. Two: it takes a huge volume of clean data to achieve valid, predictable results with ML algorithms. And I mean really, really large data volumes, so it’s a perfect use for all that big data you have, but the data has to be reliable–it’s that old GIGO rule.

I’ve only made a very small scratch on the vast surface of ML here. There are many other techniques—such as anomaly detection to help detect fraud and bolster risk management efforts—that can help you improve your analytics and increase your bottom line. From here, it’s a matter of learning all you can about the technology and selecting what’s right for your business goals and analytics ecosystem. So, what are you waiting for?


Keeping the Humans in Machine Learning

In last week’s blog, I talked about how machine learning (ML) can help with data integration. Much of the feedback I received concerned how to implement ML for data integration, but more broadly within the context of analytics in general. Machine Learning can boost your analytics efforts by helping computer systems learn in a manner that simulates human learning. However, the technology you use to implement ML is not nearly as important as the thought process behind your implementation.

Technically, ML is no different from any other analytics technology. There’s a method to it: ingest the data into a data lake or other large data store, transform it for analysis, build and train a model, and refine that model to help it learn and provide more accurate data for analysis. However, along the way, you’ll run into pitfalls that have nothing to do with technology, but with how the technology is leveraged to gain a deeper understanding of the business, and how—and by whom—the goals for the ML initiative are defined.

There are a few steps you can take to ensure that your ML initiative gets off, and stays, on the right footing. I’ll discuss these steps in terms of ML, but they really apply to any analytics effort.

Let the Business Lead the Way

Technology is useless if it doesn’t provide the outcomes you seek. Usually, when a tech project fails, it’s not the fault of the technology itself. Rather, it’s the fault of the people who defined the goals for it. As with any technology, the outcomes for your ML initiative should always be defined by the business. You can design and train the most intricate ML models, but they’ll be useless if they don’t provide the information that business owners need to do their jobs. Bottom line: actionable insights from ML/analytics initiatives can be gained only if the business leads the efforts.

Communicate the Power of the Technology

Again, the business must lead. Go all out on investing in the technologies you need to integrate ML in your analytics initiative. However, it is critical that business users understand the technology—not the nuts and bolts of the higher math behind it, but the general idea behind what you’re trying to achieve technically, and how you’re going about it at a high level. Communicate the power of the technology—be a champion for it and stress the importance of the business objectives behind it, and how the ML technology will enable you to meet them.

Keep the Dream in Mind

The defining feature of ML technology is how it enhances the predictive capabilities of analytics tools. Combined with emerging prescriptive capabilities of leading analytics software, ML-enhanced analytics can provide you with a powerful roadmap for future success. However, with any project, you’ll always have that person who says, “It can’t be done; we don’t operate that way. It’ll be too hard to change.” Don’t buy it. You can do what you want, if you want it badly enough. Just because you haven’t done it in the past doesn’t mean you can’t do it now. Trust the numbers and commit the resources to make it happen.

Pair Technology and Humans for Optimal Outcomes

As a corollary to what I just said, however, don’t forget the human element. Although ML enables you to go beyond human understanding to get the insights you need to achieve better outcomes, humans still play a large part in any ML/analytics endeavor. The combination of human experience and ML-augmented insights will help you achieve optimal outcomes. Trust the numbers, to be sure, but also trust the human business users who have a deep knowledge of your markets, customers, products, and services. By capitalizing on the connection between humans and machines you can truly achieve the outcomes you seek—and move beyond them to optimized operations and market leadership.

I’d love to hear what you think. Please comment here, connect with me on LinkedIn and Twitter, or email me at anuraag.jain@teradata.com.

Cut Costs and Improve Agility: Automate Data Integration with Machine Learning

Here’s an absurd scenario for you: I decide to build a car. I design it, make a list of all the parts I need, acquire them, and build the car. It turns out great. Then, I decide to build another model of the car—perhaps one with a few more features. So I restart the design process from a totally blank page, make an all-new list of all the parts I need, re-source them from the ground up, and—finally—build the new model. Why would I, or anyone do that? I wouldn’t. It would be a total waste of time, labor, and intellectual capital.

Yet, every day, almost every company does this—not with cars, but with IT projects. For every new information system they design and implement, IT re-performs the process, from the ground up—especially vis-à-vis data integration.

Data integration costs can range upwards of 85% of the total cost of an analytics project. What’s more, probably 65% of the data to be integrated on a new project can be found in a composite of your last five projects. A little quick math tells us that if your project budget is $1 million, $850,000 of that will be data integration costs. Of that, about $550,000 would be money you’d be spending to re-integrate data that has been dealt with on previous projects.

That’s a lot of wasted time, effort, and money. However, for most companies, the problem lies in knowing which data has already been integrated and which is new data that must be assimilated into the new system. Fortunately, there’s a solution: artificial intelligence—specifically machine learning.

Machine learning (ML) is broadly defined as the use of statistical algorithms to enable information to mimic human learning. Thus, theoretically with ML, computers can examine their past experience, parse the data from it, and learn from the experience, on their own.

In using ML for data integration tasks, metadata is the key. Project teams can apply ML algorithms to analyze descriptive, structural, and administrative metadata to gain a clear picture of how data is used and organized throughout the various organizational systems it inhabits.

Using that information, ML algorithms learn about data structures, flows, and needs and can use that information to automate data integration tasks such as:

  • Deducing appropriate data schemas and structures
  • Cataloging data used across applications—both repetitive and unique elements
  • Recommending transformation tasks
  • Mapping metadata elements between applications

Think about the impact of that automation on your business. By enabling you to better understand your data and—more importantly—which of that data is common across applications, you can significantly decrease the time and cost for data integration for new projects.

In turn, because, automation takes over many of the human tasks that can bog down data integration efforts, your human resources are freed up for other tasks—using all your resources more wisely. Also, with the machine taking on the bulk of the effort, human error is greatly reduced, and speed to completion is increased significantly. This can help you get systems up and running faster—not to mention running with all the data they need, without being weighed down by data they don’t.

In the end, automating your data integration tasks will increase user productivity and reduce the time you spend on building workflows for integration. The result? You’ll be able to improve the accuracy of your integration efforts and significantly cut costs. You’ll have the data you need to improve your agility and response time to market conditions and changes, which can ultimately have positive impact your business outcomes. What’s not to like about that?

I’d love to hear what you think. Please comment here, connect with me on Linkedin and Twitter, or email me at anuraag.jain@teradata.com.