Category: Blogs

During multiple meetings this week (online, obviously), the same challenge came up: companies and their customers are extremely poor at precisely defining what the desired outcome is that they’re looking to accomplish. At first blush, every person that I meet claims to know exactly what he or she is looking to achieve, but when push comes to shove and the individual is asked to define this more precisely, the lack of specificity rapidly starts to become apparent.

This is, of course, far from the first time that I’m exposed to this and the interesting thing is that there’s a variety of words that companies will use to cover up the lack of specificity. Words like customer value, quality, speed, reliable and robust are often used as generic terms used to look good but that prove to be void of any real meaning when investigated in more depth.

The challenge is that not being clear on your desired outcome causes all kinds of problems for organizations. One is that prioritization in the company tends to be driven by the loudest customer. Customers tend to create a lot of noise about something that they happen to care about at the moment, especially if it’s not working well. The consequence is that they contact the company with messages of deteriorating quality. If you haven’t defined what you mean with quality and you don’t have metrics to follow up what the current level is, it’s very easy to simply go along with the customer opinion and allocate significant resources to addressing this potential quality problem. The opportunity cost of this is enormous as the proclaimed problem might be a minor thing and it might well be that using the resources elsewhere would have been much better for the company.

A lack of precise definition of desired outcomes creates several issues inside the organization as well. First, it may easily lead to teams whose efforts are mutually conflicting due to different interpretations of what success looks like. For instance, teams looking to improve customer value by developing a simpler, more intuitive user interface will run into conflict with those that seek to improve customer value by developing additional features or more configuration and variants of existing features. The latter will need UI real estate to expose the user to the additional functionality and options that will easily undo the efforts of the first.

Another aspect, which I’ve written about before, is the inefficiency of development that’s not grounded in precisely, or rather quantitatively, defined customer value. For a company allocating 5 percent of its revenue to R&D and an average FTE cost of 120 k€/year, an agile team of 8 people working in 3-week sprints costs around 60 k€ per sprint but has to generate 1.2 M€ (60 k€/0.05) in business value per sprint. Vague statements about quality, speed, reliability and value do not help organizations to accomplish this outcome.

Concluding, many organizations fall short in precisely defining the desired outcome for initiatives, projects, products, and so on. Resources are allocated because of loud customers, historical patterns, ingrained behaviors or internal politics, not because these resources are providing the highest RoI for the company. In our research, we work with companies on value modeling to overcome these challenges and help them get on the way of precisely and quantitatively defining desired outcomes. Contact me if you want to know more.

All of Europe, as well as the rest of the world, is in the throes of the coronavirus and some people I meet are comparing it to the start of one of those apocalyptic movies where the world is going to hell in a handbasket. Of course, nobody can ignore the human suffering that comes from getting sick and potentially even dying from COVID-19. It is terrible and my heart goes out to those that are affected.

The current actions, including avoiding large gatherings, quarantining geographical areas, closing schools and recommendations concerning washing your hands, limiting travel and so on all make sense. However, these actions will only slow the rate of infection in order to decrease the overloading of the healthcare systems (even more) by spreading the cases out over time.

For the companies I work with, the impact has been unprecedented. All travel that’s not absolutely business critical has been suspended. Visitors (and researchers) are no longer welcome. Employees are asked to work from home wherever possible.

The dark COVID-19 clouds can have a silver lining for companies. The disruption of normal work practices offers a great opportunity to review and evaluate all processes and activities in the company and between the company and its ecosystem partners. This allows us to determine which processes that earlier required physical interaction can also be accomplished by virtual means. Similarly, we can review all activities and processes that were performed by humans by virtual means but that can be automated and be achieved without any human involvement. Finally, we may even identify activities that are obsolete and that were only conducted for historical reasons and we can shut those down.

The second benefit that companies can seek to exploit is a step function change in the adoption of digital practices. In many cases, we either figure out how to do things remotely and virtually using digital means or it doesn’t get done at all. This includes holding meetings, managing daily standups for agile teams and running the operations of systems. In fact, I could easily see the efficiency of companies significantly improve due to the forcing function provided by the current situation.

Third, many organizations are built on the principle of aligning all the gears in the machinery as closely as possible in order to optimize operational efficiency. The consequence tends to be very brittle systems that are unable to withstand adverse circumstances. Situations as we’re experiencing now force us to rethink things in terms of risk management and should encourage you and your organization to think in Taleb’s antifragility principle where systems get stronger in the face of detrimental developments.

Fourth, although the short-term economic damage is significant, the removal of business travel and working from home does provide some breathing room in the lives of many professionals. This space can be used to think strategic rather than tactical, to reflect and introspect and to find smarter ways to accomplish the same if not better outcomes.

Concluding, COVID-19 is terrible from a humanitarian, as well as from an economic perspective. Although the short-term impact on revenue and margins is undeniably bad for most companies, we can use these circumstances to realize at least some benefits that may help our companies going forward. This is important as this is not the end. In the same way as the Spanish flu in the early 1900s, SARS in 2002 and swine flu in 2009 weren’t the end. As the poet Rumi wrote centuries ago, this too shall pass.

One of the well-known struggles of every company I work with is to combine innovation with efficiency-oriented operations. This is the classic problem of ambidexterity: the company needs to deliver on today’s revenue and margins while securing its future. The problem is not that companies aren’t aware of the challenge but that they lack the tools or mechanisms to achieve the balance in a good way. This is in some ways surprising as many reams of paper have been written about the challenge, ranging from books to research papers and from blog posts to company presentations.

The key challenge is that the two systems of operation are fundamentally different. In operations, we use the classical assembly-line way of thinking by chopping up the work in chunks, giving it to different people and functions and having handovers between them. The focus is on how each step in the process can be executed at the lowest cost. On the sales side, the focus is on tactics to make customers that have a known and established need for the product to choose us over competitors and to buy now rather than later. This leads to waterfall processes, hierarchical organizations, transactional business models and a relentless focus on efficiency. Resource allocation is based on operational needs and investments sustaining innovation typically have a predictable RoI that follows a Gaussian probability curve.

Innovation, on the other hand, is concerned with finding new opportunities. This means experimenting with new concepts to existing customers and, potentially, selling existing concepts to new customers. As we look to test as many concepts as possible against the lowest cost per experiment and, at the same time, the concept has to cover all relevant aspects including technology, engineering, business model and support, the nature of innovation requires empowered cross-functional teams that engage in continuous relationships with potential customers. The RoI of this type of innovation tends to follow a power function, meaning that the RoI of the most successful innovation is higher than the return of all other innovation initiatives combined. By its very nature, innovation is a high-risk, high-return game.

Many companies go wrong in clearly separating the two operating systems, resulting in a host of issues around innovation. As the majority of resources are on the operations side of the business, innovation efforts tend to be evaluated based on criteria driven by an operations mindset. This tends to lead to two extremes. The first is where any innovative idea is killed before it has had any chance to prove itself. As innovation is, by definition, breaking the existing rules of the game, it’s obvious that the opinions inside the company will seek to kill the idea unless it’s protected. At the other end of the spectrum, there are situations where an innovative concept needs to be fully developed and ready for production before we’re even willing to show it to a customer. This leads to the [minimal viable elephant](https://bits-chips.nl/artikel/are-you-building-a-minimal-viable-elephant/) I wrote about earlier.

Combining innovation and operation is hard. In my experience, there are at least two tools that are useful in this context, ie the three horizons model and unstructured time. The three horizons model (attributed to McKinsey) divides the business of the company into three categories. Horizon 1 are the mature cash cow products that generate the majority of the revenue. The model says that this horizon should receive 70 percent of all resources in the company but also that individual products in this category should receive a resource allocation that’s 5-10 percent below the growth rate. That means that if the product grows at 5 percent per year, it may still mean that the available resources are flat or shrink every year.

Horizon 2 is concerned with the proven, rapidly growing products that aspire to become future H1 products. This part of the business should receive 20 percent of the resources and resource allocation to individual products can and likely should grow faster than revenue growth in order to accelerate things.

Horizon 3 is concerned with new innovation concepts that are unproven. This part of the business should receive 10 percent of the resources and these resources are allocated to running experiments with customers with the intent of finding viable options. In [earlier columns](https://janbosch.com/blog/index.php/2017/02/05/innovation-is-hard-work/), I’ve described how, in my experience, this should be organized.

The important part is to realize that any organization needs to allocate at least 10 percent of the total resources to innovation. One way to allocate those resources is by offering employees unstructured time for innovation. So, everyone who wants to can use 10 percent of their time for innovation efforts without any approval required from any manager. These hours can be used to build teams of people that all use their 10 percent to explore a particular innovation. In [this article](https://janbosch.com/blog/index.php/2017/10/13/the-end-of-innovation-as-we-know-it/), I provide more details.

Concluding, combining innovation and operation is difficult but also essential to ensure the longevity of the company. We need to survive today and ensure a viable future. Although everyone understands this conceptually, my experience is that in most companies, these two systems of operation are conflicting, with the investment in innovation often being the victim. So, think about this: for all the talk about innovation, does your organization really conduct innovation, protect it and grow new businesses out of its innovation efforts? Or is it just ‘feel good’ talk?

As part of the research in Software Center, I work with dozens of companies in the software-intensive embedded systems space on a variety of topics. One of these topics is the development of new products. Having worked with online companies, as well as startups, I’ve become indoctrinated with Steve Blank’s ideas and the “lean startup” concepts. One of the key tenets is that you validate with customers every step of the way. In fact, you seek to minimize the amount of R&D investment between customer proof points. The second tenet is to only rely on what customers say when you absolutely need to, but whenever possible rely on measuring their behavior.

In the embedded systems industry, for some reason, companies are extremely reticent to validate with customers before the entire product has been built. Over the years, I’ve heard a great variety of reasons as to why this is. The main ones include: “We can’t show customers anything but a complete product”, “We’ll damage the company brand if we show an early prototype”, “This idea is so good that they’ll buy it if we only build it”, “Experimentation with customers makes us look like idiots because it looks like we don’t know”, “This stuff is secret and we don’t want to tip off the competition”, “It’s so hard to organize this across the company as I need to coordinate with everyone.”

The consequence of this is that companies tend to build, as one of my colleagues quipped, minimal viable elephants (MVEs) instead of minimal viable products (MVPs). When I confront people with this and we get past the ‘excuses’, it seems to me that there are at least three fundamental causes to this phenomenon.

First, most of the companies established themselves and became successful by building mechanical and electronic products. Because of a variety of reasons, not the least the need to establish manufacturing facilities, the design processes for atom-based products have traditionally been very waterfall and specification based. With the advent of additive manufacturing and rapid prototyping hardware facilities, this is, of course, changing, but the traditional approaches are still widespread. It’s important to realize that building innovative digital offerings requires a fundamentally different process than building physical products. In fact, using the traditional process is a recipe for disaster as you’re flying blind and base your decisions on the beliefs of you and the others at your company – beliefs that are almost certainly wrong.

Second, especially in new product development, complicated internal political processes and games tend to be part of the equation. As a consequence, the attention shifts from the customer and the business ecosystem to the internal political landscape. The folks involved in the development of the new product often have to compromise with various forces in the company. This causes functionality to be added or changed independently of actual customer feedback but based on the opinions of HIPPOs. In the worst case, this results in new products that, for all practical means, are a Swiss army knife that can do many small things but doesn’t solve the one key problem that initiated the product development in the first place.

Third, many think of new product development as a one-shot opportunity that we have to get right on the first try. This, of course, is driven by the difficulty of changing mechanical and electronic products after the start of manufacturing. However, digital offerings are akin to a conversation: constant updates are cheap, allow you to measure how customers respond and, in fact, are expected by many customers.

Building digital offerings that include mechanical and electronic components as well requires a different view on the priorities and process. First, establish problem/solution fit by simply spending ample time with intended customers, well before any R&D effort. Second, establish product/market fit by presenting the intended solution concept to customers and validate the fit, as well as their willingness to pay. Third, build the scrappiest, fastest, smallest realization of the product that allows for small-scale validation. Here you transition from measuring what customers say to what they actually do. Fourth, build a slightly more advanced prototype of the product that can be validated on a larger scale.

Of course, each of these steps is conducted iteratively and you only proceed to the next step when the feedback that you received from customers justifies it. In practice, it often means that the mechanical and electronic parts of the product are over-dimensioned in terms of specifications in order to allow for a larger ‘design space’ of opportunities for the digital part.

Concluding, innovating digital offerings is hard for embedded systems companies and the result often is a ‘minimal viable elephant’ that sees no customer feedback until after the start of manufacturing. Instead, focus extensively on collecting customer feedback throughout the entire innovation process and on customer behavior wherever possible. In the end, it’s the customer who decides whether you’ll be successful.

This week, for the umpteenth time, I met a team in the process of putting a new product in the market, telling me that they were so customer centric. What they meant was that during development, they’d talked to a number of potential customers and some of the employees had used prototypes. For those that read my articles on a regular basis, it will come as no surprise that I disagree with this view. There are at least three aspects that I think are critical to be truly customer centric: choose your customer, measure before you build and focus on behavior, not words.

First, there’s a fundamental difference between making customers happy and making happy customers. Many product teams with which I’ve interacted have vague and conflicting notions of who their target customer is. That makes it very likely that within the team, efforts are focused on different aspects of the product and that feedback from any individual viewed as a potential customer is immediately incorporated as the absolute truth.

To be customer centric, the first step has to be to define who you want your customer to be. Products that try to be everything to everyone will fail. The challenge of defining the customer is that you need to say no to categories of customers and it feels like you’re limiting the market of your product. In addition, team members often have different opinions and engaging in constructive conflict is extremely difficult in many corporate cultures. However, if you don’t know for whom you’re building the product, you can’t be successful.

Second, investment decisions in R&D are almost always based on opinions and qualitative feedback from a very small customer sample that almost certainly isn’t representative of the actual customer base. The mechanism to avoid this is to measure customer response – “signal” – before you even invest a single dime into building anything. Of course, the lean startup movement and Steve Blank’s books have been advocating this for decades, but in industry, this is still far from adopted.

The general feedback that I receive in response to the suggestion to test product concepts and potential features with customers is threefold. First, we don’t want to expose our customers to bad, immature, unfinished products as it will hurt our brand and our customers’ perception of us as a company. Second, what we’re doing is unique and we need to keep it a secret as long as possible so that our competitors don’t get wind of our plans. Third, marketing likes to surprise the market with a bombarding of messaging and if we go out early for testing purposes, everyone knows what’s coming and marketing can’t do its job.

All of these reasons are orders of magnitude less relevant than putting a product in the market that nobody wants and, with that, wasting all your R&D resources. Every investment in R&D, whether it’s new features in existing products or entirely new products, needs to be driven by a clear signal from the market.

Third, product teams put way too much stock into what customers say. Both in B2C and B2B contexts, most new products introduced to the market fail. In the fast moving customer goods (FMCG) space, the failure rate is, according to some studies, above 85 percent. This is despite the fact that every one of these products had extremely positive feedback from potential customers. Decisions in product teams should, to the largest extent possible, be based on measuring what customers do, rather than on what they say. For all kinds of reasons, customers will be much more positive in their verbal responses than in their actual behavior.

Developing new products is hard as you’re dealing with lots of uncertainty, internal stress and negative feedback from the rest of the organization. This can easily lead to the product team ending up in an X-Files style “I want to believe” situation where any feedback not supporting the current direction is filtered out. To be successful, however, you need to maintain the positive drive in the team while being crystal clear on who your customer is, be data driven in your decision-making and focus on customer behavior. Be customer centric, but do it for real!

During the last months, I’ve met with several companies that had an interesting common denominator: they were all building a new platform to replace a legacy platform and things weren’t going so well. The legacy platform often is decades old and has functionality in it that’s the result of hundreds of person-years of effort. And it typically forms the basis for a significant part, if not the entire, product portfolio.

When the new platform effort is initiated, the organization accepts that it will take some time (years) to build it to a point that it can replace the legacy platform. As stalling the product portfolio is unacceptable, there will be a continuous flow of development effort into the old platform to realize the features that are needed right now. This, of course, happens in parallel to the new platform effort. The consequence is that the goal for the new platform becomes a moving target as the legacy platform is continuously adding new functionality that the new platform will need to add as well in order to achieve feature parity.

There are three possible outcomes in this scenario. The first is also the worst: the new platform effort forever fails to achieve feature parity. As a consequence, no product line is willing to move over to it. At some point, there’s a change of management, the new platform effort is reviewed and the track record to date leads to an obvious conclusion: the new platform is canceled. The result is that the company is stuck with the old platform and has lost years of time and hundreds of person-years of R&D effort without any business benefit.

Second, the platform effort starts to be used by one product line and becomes the de-facto platform for only a part of the portfolio. For product management, the choice between building functionality for a product that’s here now or a potential product in the future very often falls in favor of the existing product. When this happens continuously, it results in the platform becoming increasingly optimized for the specific product line and getting further and further behind for the other product lines. This could be viewed as a way for one product line to hijack general R&D resources for its own purposes, which makes it very attractive as a strategy in companies with a strong focus on teams delivering in their own area of responsibility without maintaining responsibility for the success of the company overall.

The third situation is where senior R&D and business leaders understand these patterns and force the organization to abandon the legacy platform and adopt the new one. This often leads to significant tension as product lines are asked to put out new products with reduced feature sets. The discussion typically leads to many requests to perform “small” maintenance efforts on the old platform in response to customer requests and other, highly viable, reasons. In general, many in the R&D organization will fight for staying on the old platform as long as possible. In the worst case, the legacy platform will never be sunsetted and the platforms continue to exist next to each other for years or even decades.

The fundamental reason behind these scenarios is a mechanical mindset by leadership. The notion of having generations of platforms may make sense in mechanics and electronics (although these platforms would benefit from continuous evolution as well), but software is different. Our research shows that there’s no correlation between system age and the level of technical debt, meaning that new systems and old ones have similar amounts of requirements for technical debt management.

Rather than thinking in terms of platform generations, the focus should shift to a continuously improving platform with a constant flow of refactoring effort keeping it current from a technology and architectural perspective. Instead of allowing an existing platform to sink in the morass of commodity and irrelevance, organizations should look to continuously invest some percentage (10-25 percent) into the platform to ensure it never has to be replaced by a next generation.

Concluding, building new platforms to replace existing ones is a highly risky R&D initiative fraught with risks and challenges. Instead, commit to continuously investing a part of your R&D resources in refactoring your existing platform. The best platform is one that never has to be replaced because it’s always current and up to date. Because you committed to keeping it that way!

 

Having worked with companies on the use of AI, I’ve noticed an interesting pattern: although most of the attention is spent on algorithms, data storage infrastructure, training and evaluation of applications, the hardest part very often seems to be coming up with a promising concept in the first place. When exploring promising concepts, many start to realize that taking the resultant ML model from the prototyping phase to real deployment is a major challenge that requires changes in existing customer engagement models, product architectures, ways of working, the data collected and often even legal constraints.

Exploring promising concepts, of course, requires exploring both the potential business benefits and the expected cost for introducing a machine or deep-learning model in a product, solutions or service. However, my observation is that many struggle quite a bit with coming up with potential concepts that exploit the benefits that ML/DL models provide.

Earlier, we introduced the HoliDev model, which distinguishes between requirements-driven, outcome-driven and AI-driven development and claims that each type of development has its own characteristics. AI-driven development thrives where, on the one hand, there’s sufficient data available for training and operations and, on the other hand, we’re looking to solve an inference problem that’s particularly hard to solve without the use of ML/DL techniques as there’s no clear algorithmic approach. In general, we focus on three main characteristics that provide the key preconditions for a successful AI concept, ie removing hardcoded responses, using ignored data and revisiting negative RoI use cases.

First, in situations where the system response is hardcoded, there can be a significant benefit to providing a response to each request based on the available information. The obvious example is in the online advertising space where companies like Google and Facebook are constantly looking to create more accurate profiles of users in order to serve more relevant ads, rather than showing people a random ad. AI models can, especially when a good algorithmic approach is lacking, provide better responses by training based on available data.

Second, there are numerous situations where available data simply is ignored as humans haven’t been able to detect patterns in it and consequently follow a mathematical approach to solve a particular problem. An interesting example can be found in control systems where several companies are working to complement or replace traditional P, PI, PD and PID controllers with AI models. The reason being that traditional controllers operate based on a theoretical model of how a system is supposed to behave in response to control signals. In practice, no real-world system responds completely in accordance with the theory and AI models can improve the quality of control by taking all data into account.

Third, the most difficult case is where the cost of collecting data for human interpretation has had a negative return on investment as the effort required to benefit from the data was too high. With the decreasing cost of sensors, computing resources and communication, however, more and more cases exist where collecting the data for use by an AI model is actually becoming profitable.

It’s in this category where the most rewarding AI business cases can be found. One well-known example is sentiment analysis in social media. The amount of data in social media vastly outweighs the ability of even large teams of people to keep track of the sentiment around, for instance, a product or a company and consequently people didn’t even try. With the emergence of ML approaches, however, it becomes entirely feasible to have real-time dashboards of the state of the sentiment and companies use these insights for decision-making.

Concluding, for all the focus on AI algorithms, data and training, one of the most challenging activities remains the identification of interesting business cases and evaluating the feasibility and desirability of each case. I’ve discussed three categories of cases that can provide inspiration for identifying your AI business case.

During the last weeks, I’ve experienced multiple situations where an organization (industrial or academic) simply doesn’t have a business strategy or a strategy concerning a key area for their business. When probed and questioned on the strategy, I’ve observed at least three types of responses.

First, leaders in the company say that there **is** a strategy and that I’m wrong in claiming otherwise. Although I’ve been wrong many times in my life, to me a strategy should provide clear guidance on what tasks and opportunities should be prioritized over others and, above all, what we shouldn’t spend time, energy and resources on. A strategy that fails to specify what we shouldn’t do, to paraphrase Michael Porter, is no strategy.

Second, the company admits that the strategy is high-level and not operational but defends itself by claiming that its key success in the market is to be customer focused and, consequently, it needs to respond to the requests from customers rather than set its own course. Obviously, this is a fallacy as it causes companies to fall into the “make customers happy” trap. It’s impossible to satisfy everyone. Rather, you need to choose what kind of customers you want and then focus on making them happy. This, of course, is a strategic choice.

Third, especially in new areas where the company has no established business, leaders claim that it’s impossible to formulate a strategy as nobody knows how the market will unfold. This, however, causes them to become the plaything of more proactive, strategic competitors who will dictate how the market will establish itself. It’s important to avoid an Alice in Wonderland situation where by not knowing what you want, any direction is equally good.

Although these responses are understandable and human, they lead to a number of serious problems for the company. There are at least three that I’ve witnessed over the years.

First, the company acts tactically and opportunistically. Due to the lack of a clear strategy, individuals at all levels in the organization take tactical decisions that provide them with the most benefit in the short term without considering the long-term consequences. This results in an accumulation of architectural, technical and process debt in the organization, as well as in the relationships with customers and other ecosystem participants, which over time causes enormous disadvantages due to reduced business agility, unreasonable expectations by others, as well as numerous other consequences.

Second, there’s a significant risk that different teams in the company pursue opposing local strategies and consequently nullify each other’s efforts, causing the company to expend, sometimes significant, resources without any business benefit. Burning resources without generating business benefits obviously is the fastest road to bankruptcy.

Third, even if none of the above effects occur at your organization, all employees will, at any point in time, still have way more work than they could possibly hope to accomplish during work hours. In the absence of a clear strategy, individuals randomly prioritize tasks based on personal preferences, expediency and other factors. So, the “strategy in use” will become whatever everyone feels like. In practice, this tends to lead to people doing what they did yesterday, meaning the company gets stuck in the past and fails to evolve and respond to changes in the market.

Concluding, developing and communicating a clear and actionable strategy that represents tangible choices is a critical tool in aligning large groups of people. The alternative is to micromanage everyone, which will cause you to lose your best people as nobody likes being told how to do their job. A successful strategy defines a clear what and why and leaves it to individuals and teams to figure out how.

In recent columns, I’ve been sharing my view on the quality of the data that many companies have in their data warehouses, lakes or swamps. In my experience, most of the data that companies have stored so carefully is useless and will never generate any value for the company. The data that actually is potentially useful tends to require vast amounts of preprocessing before it can be used for machine learning, for example. As a consequence, in most data science teams, more than 90 percent of all time is spent on preprocessing the data before it can even be used for analytics or machine learning.

In a paper that we recently submitted, we studied this problem for system logs. Virtually any software-intensive system generates data capturing the state and significant events in the system at important points in time. The challenge is that, on the one hand, the data captured in logs is intended for human consumption and, consequently, contains a high variability in the structure, content and type of the information for each log entry. On the other hand, the amount of data stored in logs often is phenomenally large. It’s not atypical for systems to generate gigabytes of data for even a single day of operations.

The obvious answer to this conundrum is to use machine learning to derive the relevant information from the system logs. This approach experiences a number of significant challenges due to the way logs are generated. Based on our research in literature and company cases, we identified several challenges.

Due to the nature of data generation, the logs require extensive preprocessing, reducing the value. It’s also quite common that multiple system processes write into the same log file, complicating time series analysis and other machine learning techniques assuming sequential data. Conversely, many systems generate multiple types of log files and establishing a reliable ground truth requires combining data from multiple log files. These log files tend to contain data at fundamentally different levels of abstraction, complicating the training of machine learning models. Once we’re able to apply machine learning models to the preprocessed data, interpretation of the results often requires extensive domain knowledge. Developers are free to add new code to the system that generates log entries in ad-hoc formats. The changing format of log files complicates the use of multiple logs for training machine learning models as the logs aren’t necessarily comparable. Finally, any tools built to process log files, such as automated parsers, fail unpredictably and are very brittle, requiring constant maintenance.

We studied the problem specifically for system logs, but my experience is that our findings are quite typical for virtually any type of automated data generation. Although this is a huge problem for almost all companies that I work with and enormous amounts of resources are spent on preprocessing data to get value out of it, it’s a losing battle. The amount of data generated in any product, by customers, across the company, and so on, will only continue to go up. If we don’t address this problem, every data scientist, engineer and mathematician will soon be doing little else than preprocessing data.

The solution, as we propose in the paper, is quite simple: rather than first generating the data and then preprocessing it, we need to build software to generate data in such a format that preprocessing isn’t required at all. Any data should be generated in such a way that it can immediately and automatically be used for machine learning. Preferably without any human intervention.

Accomplishing this goal is a bit more involved than what I can outline in this post, but there are a number of key elements that I believe are common for any approach aiming to achieve this. First, all data should be numerical. Second, all data of the nominal type (different elements have no order nor relationship to each other) should be one-hot encoded, meaning that the elements are mapped to a binary string as long as the number of element types. Third, data of the ordinal type can use the same approach or, in the case of non-dichotomous data, use a variety of encodings. Fourth, interval and ratio data needs to be normalized (mapped to a value between 0 and 1) for optimal use by machine and deep-learning algorithms. Five, where necessary, the statistical distribution of the data needs to be mapped to a standard Gaussian distribution for better training results.

Accomplishing this at the point of data generation may require engineers and developers to interact with data scientists. In addition, it calls for alignment across the organization, which hasn’t been necessary up to now. However, doing so allows companies to build systems that can fully autonomously collect, train and retrain machine learning models and deploy these without any human involvement (see the figure).

System logging for machine learning

Concluding, most data in most companies is useless because it was generated in the wrong way and without proper structure, encoding and standardization. Especially for the use of this data in training machine learning models, this is problematic as it requires extensive amounts of data preprocessing. Rather than improving our data preprocessing activities, we need to generate data in a way that removes the need for any preprocessing at all. Data scientists and engineers would benefit from focusing on how data should be generated. Rather than trying to clean up the mess afterwards, let’s try to not create any mess to begin with.

During the big data era, one of the key tenets of successfully realizing your big data strategy was to create a central data warehouse or data lake where all data was stored. The data analysts could then run their analyses to their hearts’ content and find relevant correlations, outliers, predictive patterns and the like. In this scenario, everyone contributes their data to the data lake, after which a central data science department uses it to provide, typically executive, decision support (Figure 1).

Figure 1: Everyone contributes their data to the data lake, after which a central data science department uses it to provide, typically executive, decision support.

 

Although this looks great in theory, the reality in many companies is, of course, quite a bit different. We see at least four challenges. First, analyzing data from products and customers in the field often requires significant domain knowledge that data scientists in a central department typically lack. This easily results in incorrect interpretations of data and, consequently, inaccurate results.

Second, different departments and groups that collect data often do so in different ways, resulting in similarly looking data but with different semantics. These can be minor differences, such as the frequency of data generation, eg seconds, minutes, hours or days, but also much larger differences, such as data concerning individual products in the field vs similar data concerning an entire product family in a specific category. As data scientists in a central department often seek to relate data from different sources, this easily causes incorrect conclusions to be drawn.

Third, especially with the increased adoption of DevOps, even the same source will, over time, generate different data. As the software evolves, the way data is generated typically changes with it, leading to similar challenges as outlined above. The result is that the promise of the big data era doesn’t always pan out in companies and almost never to the full extent that was expected at the start of the project.

Finally, to gain value from big data analytics requires a strong data science skillset and there simply aren’t that many people around that have this skillset. Training your existing staff to become proficient in data science skills is quite challenging and most certainly harder than providing machine learning education to engineers and developers.

Many in the industry believe that artificial intelligence applications, and especially machine and deep-learning models, suffer from the same challenges. However, even though both data analytics and ML/DL models are heavily based on data, the main difference is that for ML/DL, there’s no need to create a centralized data warehouse. Instead, every team, business unit or product organization can start with AI without any elaborate coordination with the rest of the company.

Each business unit can build its own ML/DL models and deploy these in the system or solution for which they’re responsible (Figure 2). The data can come from the data lake or from the local data storage solutions, so you don’t even need to have adopted the centralized data storage approach before starting with using ML/DL.

Figure 2: Each business unit can build its own ML/DL models and deploy these in the system or solution for which they’re responsible.

Concluding, AI is **not** data analytics and doesn’t require the same preconditions. Instead, you can start today, just using the data that you have available, even if you and your team are just working on a single function or subsystem. Artificial intelligence and especially deep learning offer amazing potential for reducing cost, as well as for creating new business opportunities. It’s the most exciting technology that has reached maturity in perhaps decades. Rather than waiting for the rest of the world to overtake you, start using AI and DL today.