About the cognitive biases in the analyst’s job (that actually exist in everyday life)
You, as an analyst, can be arbitrarily into the data collection and analysis methods, however it is very difficult to avoid the “great emotional randomizer”: in any case you are passing the data and analysis results through yourself, so you make decisions based on emotions, unclosed Gestalts, previous experience. Thus, you expose data to the cognitive biases (CB).
CB is a scientific concept, meaning systematic deviations in behavior, perception and thinking caused by the subjective prejudices and stereotypes, social, moral and emotional reasons, failures in the processing and analysis of information.
I want to emphasize those cognitive biases which may lead to the erroneous decisions in the field of analytics. Let me give some examples but it’s you to decide whether you are prone to them or not, to deal with them, or to leave as it is.
Confirmation bias stands for a tendency to search for or interpret information in a way to confirm the available pre-conception. Perhaps this is the most common CB among analysts. When conducting analysis, the analyst generates in advance some ideas on the results of this analysis (bets by himself on the result), and then interprets the results of the analysis on the basis of these ideas, sometimes adjusting the results to the answer.
E.g.: week retention of your product after the launch was unexpectedly low (for example, let’s take 5%). Shortly before the start, you offered to change the icon, but no one agreed. As a result, you analyze the reasons for low retention, and begin to see that people are not interested in your product at once, from the first event, from the tutorial. You go to the department of development and set the task to change the icon. And in fact, the reasons may be totally different: from the unsuccessful design of the entire application or inappropriate setting — to the unclear tutorial or just poor balance of the game.
A particular case of the confirmation bias can be called anchoring — feature of the adoption of numerical solutions by the person causing irrational bias of responses towards the number, that already was on the mind before making a decision. You just watched the movie “47 Ronin” and it really got under your skin. So much that the number 47 just etched in your brain. Among the numbers of the huge spreadsheet you see only those records in which the numerical entry is “47”, not paying attention to other metrics. Or, say, you start offering to add to the game AK-47 and the character of the Hitman game series (Agent 47).
Curse of knowledge: when better-informed people find it extremely difficult to think about problems from the perspective of lesser-informed people. Application developers tend to overestimate how much other people (including their precious users) are versed in their applications.
It’s so obvious that here it is necessary to press the green button and that the tank is to jump! No, it’s not obvious. In your head you already have an understanding, the appropriate structures and templates are formed, you look at your product every day (as well as in your night dreams), but the user sees everything for the first time. What should be done for the user to understand the product as good as you do? That’s right! To explain him everything
And in order to understand whether the user perceives that you are giving to him, there are several ways:
- Play-tests. Great invention of mankind, which is just necessary in order to get rid of the curse of knowledge. Hear what users are saying, listen to feedback from each user, and you will understand your product better;
- Analysis of user profiles. You may see the sequence of actions of each user, without conducting expensive individual play-tests.Modern analytical systems allow you to see the product through the eyes of the certain real people.
Another example of the curse of knowledge can be called communication between analyst and other departments. The analyst speaks his own language and suggests that everyone understands him. Believe me, in fact, most of your colleagues do not know what is ARPU and LTV, and they certainly never heard of the models of ARMA and ARIMA.
E.g.: once I had to present a mathematical model of predicting to the city officials. I spoke with passion about how well this regression model works, about how long we were developing and testing it. And then I was asked, “We’re a progressive city, why do you have a regression model?”
Illusion of transparency. People overestimate others’ ability to understand them. Question is solely about the communication — are you sure that you were understood correctly? Are you sure that you have understood correctly? Maybe it makes sense to talk more?
Let’s assume you’re a producer and you’ve just invented a new cool game. You have gathered a meeting and say “We should make a game about Super Sonic the Hedgehog! Well, you all sure played it. Let’s do exactly the same!”
And you leave in full confidence that the game designer have understood you. Now please go to the website and answer the question, which of those games did you have in mind?
Retrospective bias. Talking about choice-supportive bias (tendency to remember your choices to be more right than they were when they were made) and hindsight effect (the tendency to see past events as being predictable at the time those events happened). Both biases could be described by the phrase “I-knew-it-all-along”.
A striking example: a flurry of statements from the representatives of the gaming industry, who began to say that they invented this game long time ago, immediately after the successful release of the game Pokemon Go.
In order to avoid such biases, I recommend to conduct a detailed analysis of all changes in the project, preferably from its very start. You’ll know how each change affected the project, which metrics it has changed, how it worked to the bottom line (most often — revenue, at least — the audience). Basing subsequent decisions on the analysis of previous results, you will be able to more accurately select the most effective hypothesis of several options. And no “Well, I told you!” is not a ride.
There is one more similar CB — contribution effect. This tendency to overestimate the value of the object, in the creation of which you were involved. To avoid it, again, be guided by the statistics, keep on hand a clear log of all changes and it’s detailed analysis.
Texas sharpshooter fallacy — selection or adjustment of the hypothesis after the data is collected, making it impossible to test the hypothesis fairly. I think you all know the example when the shooter shoots at the barn, and then, in the place where the most holes are, draws the target. When talking about this CB, very often the Nostradamus is mentioned, whose prophecies are adjusted to the occurred events.
In analytics such a fallacy is found everywhere. Let’s say you make revenue forecasting model and test it on the same data, that were used for teaching. If it’s done by ignorance or by selfish intent — that is another question. But the fact is that the results of such testing would be excellent, although the model may well prove to be unworkable.
Survivorship bias — on one group (“survivors”) there is lots of data, and on the other group (“dead”) — virtually none. And researchers are trying to find common features among the “survivors”, forgetting that not the less important information is hidden among the “dead”.
Most often this fallacy is illustrated by an excellent example: “Rumors about the mind and kindness of dolphins are based on the stories of the tired swimmers, that were pushed to the coast, but we are unable to hear the stories of those who were pushed in the other direction”.
If talking about the analytics, it is incorrect to draw conclusions only on the basis of user data, which is already active in your project. It is also necessary to take into account those who left the project. It is possible, that they also could give you enough information for the development of the project. Here is a good example of how to optimize the user activation on the basis of a comparison of the “survivors” and the “dead”.
Selective perception It lies in the fact that as the basis for the analysis not all the data is taken, but only some part selected beforehand. Of course, such an analysis would be irrelevant, though it would look good enough. The analyst just would not talk about the data he did not take for analysis.
Let’s say you release a beta version of the project, collect feedback, but pay attention only to the positive reviews (well, because those who wrote negative reviews, they do not understand anything and are just trolls, isn’t that right?). The result is that much of the feedback passes by, and the product does not change the way it should have. By the way, a similar situation appeared in the series “Silicon Valley”.
Surprisingly, the persistence — is also CB, and it lies in continuing to work on what has already lost its value.
Let’s say you are still waiting that the product will take off only by the means of the viral traffic. The months are passing by and it still does not take off, and you’re still waiting that almost right now, right now.
Information bias — the tendency to seek information even when it cannot affect action.
Often the role of the analyst in the company is wide enough, and he is entrusted both quantitative and qualitative research. The analyst feels quite at ease: if there is plenty of time, he may Google the information needed. It may happen that in solving some issue, the analyst may purely instinctively take a few days for research and RnD. But before you go on a journey through the depths of the Internet for a few days, ask yourself this question: are you definitely sure you can’t begin to address the problem now? Are you sure to find the information that will help you in the following analysis and pay these days? If it’s a definite “yes”, then certainly go for exploring.
“Well traveled road” effect. Decisions are made in favor of the more familiar and studied technology, even if it may not be effective.
Let’s say, you were involved only in the web projects before, and then decided to enter the mobile market. And you immediately begin to imagine how much new you as an analyst and the developers will have to explore. So you insist on sticking to the web and not meddling in this terra incognita.
Very similar CB is zero-risk bias — preference for reducing a small risk to zero over a greater reduction in a larger risk. People are afraid of medical complications more than of the disease itself. How to deal with this bias? If there is a choice, then collect the data, be as much unbiased as possible and not be guided by the personal preferences.
On the other side, there is bandwagon effect — the tendency to do things because many other people do the same. For each of us there are people whom we look up to. And it may often be the case that we begin to imitate them, to use the same tools and techniques. Or, once learned about the new method you start to see it everywhere (it is known as the Baader-Meinhof Phenomenon) and now seek to use it anywhere, even if it is not quite reasonable.
That’s my case. I learned about the factor analysis and the method of principal component analysis, I was impressed with them, especially with the pictures of the transition from one space to another:
Asking around friends and analysts revealed that it was long used by them, though I was the one to know nothing about it before. And I began to use this method everywhere, even if it was not required. The method of principal component analysis is needed to reduce the dimension (e.g. you have 100 variables, and you need to leave only 5 of them), but I used it even when I had only three variables on hand, and no reduction in the dimension was required.
The next CB makes sense to use for your own benefit. Discounts revaluation — the name speaks for itself. People often fall victim to marketers and buy what was they do not need, but at a discount.
How to take advantage? Just try to make a small discount for your goods, virtual or real — it does not matter. It may often be the case that a short-term small discount raises the revenue greatly.
In my practice, I have seen a very high price elasticity of demand, particularly in the virtual goods. Sometimes a small (let’s say 10%) discount could give a strong (let’s say, 50%) increase in demand. Experiment with the goods for the offer, with the size of the discounts, and I’m sure that you find an optimal strategy of sales, maximizing revenue and minimizing risks.
Finally, a couple of CB, peculiar only to analysts:
- clustering illusion — tendency of seeing phantom patterns;
- illusory correlation — inaccurately perceiving a relationship between two unrelated events
Can not be said that the correlation doesn’t imply causation, and if between two objects there is a correlation, it does not mean that a change to one will inevitably lead to a change in another.
And that’s not all cognitive biases that exist! I very much want you to know what kind of biases there are, that they are fairly ubiquitous, and it is possible that you identify some of them in your work, and this is an excellent first step. In general, I recommend to periodically review these lists and wonder whether you have any biases mentioned in them. It’s like going to the dentist every six months.
The main thing is not to be overzealous. After all, the continuous search of CBs — it is also a kind of cognitive bias.
The analyst, like other professionals, should have a cool head. The analyst should be as impartial as possible. You should rely on the statistical significance, and not on the templates and emotions!
Bonus tip for those who read to the end: Now that you are aware of many CBs, you know that they are common for everyone, so you have a competitive advantage: when presenting the next project in the course of argument, in order to increase the probability of making the decision needed, try to use a suitable CB but be careful: let it be only for the benefit of the project.
Vasiliy Sabirov, ex-Lead Analyst @ devtodev
Check our latest news on web and mobile app analytics here.
