41. INTRODUCTION TO HYPOTHESIS TESTING

OBJECTIVE

Verify whether two (or more) groups are significantly different from each other, usually by comparing their means or medians.

DESCRIPTION

Generally speaking, statistical hypothesis testing concerns all the techniques that test a null hypothesis versus an alternative hypothesis. Although it also includes regressions, I will only focus on the testing performed on samples.

There are three main steps in hypothesis testing:

• -          Definition: identify the problem, study it, and formulate hypotheses;
• -          Experiment: choose and define the data collection technique and sampling method;
• -          Results and conclusion: check the data, choose the most appropriate test, analyze the results, and make conclusions.

DEFINITION

The first step in hypothesis testing is to identify the problem and analyze it. The three main categories of hypothesis testing are:

• -          to test whether two samples are significantly different; for example, after conducting a survey in two hotels of the same hotel chain, we want to check whether the difference in average satisfaction is significant or not;
• -          to test whether a change in a factor has a significant impact on the sample by conducting an experiment (for example to check whether a new therapy has better results than the traditional one);
• -          to test whether a sample taken from a population truly represents it (if the population’s parameters, i.e. the mean, are known); for example, if a production line is  expected to produce objects with a specific weight, it can be checked by taking random samples and weighting them. If the average weight difference from the expected weight is statistically significant, it means that the machines should be revised.

After defining and studying the problem, we need to define the null hypothesis (H₀) and alternative hypothesis (H_a), which are mutually exclusive and represent the whole range of possibilities. We usually compare the means of the two samples or the sample mean with the expected population mean. There are three possible hypothesis settings:

• -          To test any kind of difference (positive or negative), the H₀ is that there is no difference in the means (H₀: μ = μ₀ and H_a: μ ≠ μ₀);
• -          To test just one kind of difference:
• o   positive (H₀: μ ≤ μ₀ and H_a: μ > μ₀);
• o   negative (H₀: μ ≥ μ₀ and H_a: μ < μ₀).
• 
  

EXPERIMENT

The sampling technique is extremely important; it must be certain that the sample is randomly chosen (in general) and, in the case of an experiment, the participants must not know in which group they are placed. Depending on the problem to be testing and the test to be performed, different techniques are used to calculate the required sample size (check www.powerandsamplesize.com, which allows the calculation of the sample size for different kinds of tests).

RESULTS AND CONCLUSIONS

Once the data have been collected, it is necessary to check for outliers and missing data (see 36. INTRODUCTION TO REGRESSIONS) and choose the most appropriate test depending on the problem studied, the kind of variables, and their distribution. There are two main approaches to testing hypotheses:

• -          The frequentist approach: this makes assumptions on the population distribution and uses a null hypothesis and p-value to make conclusions (almost all the methods presented here are frequentist);
• -          The Bayesian approach: this approach needs prior knowledge about the population or the sample, and the result is the probability for a hypothesis (see 42. BAYESIAN APPROACH TO HYPOTHESIS TESTING).

DEPENDENT VARIABLE	SAMPLE CHARACTERISTICS (INDEPENDENT VARIABLES)					CORRELATION
	1 SAMPLE	2 SAMPLES		SAMPLES > 2
		INDEPENDENT	DEPENDENT	INDEPENDENT	DEPENDENT
DICHOTOMOUS	Test of proportions		McNemar test	X²	Cochran's Q	Phi coefficient, contingency tables
CATEGORICAL	X²	X²
ORDINAL	X²	Mann‒Whitney U test	Wilcoxon signed-rank test	Kruskal‒Wallis test, Wilcoxon rank sum test	Scheirer‒Ray‒Hare test (two-way), Friedman test (one-way)	Spearman’s correlation
INTERVAL OR RATIO	One-sample z-test or t-test	Two-sample t-test	Paired t-test	One-way ANOVA	Repeated measure ANOVA	Pearson’s correlation
		Two-way ANOVA

Summary of Parametric and Non-parametric Tests

Tests usually analyze the difference in means, and the result is whether or not the difference is significant. When we make these conclusions, we have two types of possible errors:

-          α: the null hypothesis is true (there is no difference) but we reject it (false positive);

-          β: the null hypothesis is false (there is a difference) but we do not reject it (false negative).

Possible outcomes of hypothesis testing	NOT REJECT NULL HYPOTHESIS	REJECT NULL HYPOTHESIS
THE NULL HYPOTHESIS IS TRUE	1-α	Type I error: α
THE NULL HYPOTHESIS IS FALSE	Type II error: β	1-β

Possible Outcomes of Hypothesis Testing

The significance of the test depends on the size of α, that is, the possibility of rejecting the null hypothesis when it is true. Usually we use 0.05 or 0.01 as a critical value and reject the null hypothesis when α is smaller than the p-value (the critical value representing the probability, assuming that the null hypothesis is true, of observing a result at least as extreme as the one that we have (i.e. the actual mean difference).

It is important to remember that, if we are running several tests, the likelihood of committing a type I error (false positive) increases. For this reason we should use a corrected α, for example by applying the Bonferroni correction (divide α by the number of experiments).[1]

In addition, it is necessary to remember that, with an equal sample size, the smaller the α chosen, the larger the β will be (false negative).

If the test is significant, we should also compute the effect size. It is important not only whether the difference is significant but also how large the difference is. The effect size can be calculated by dividing the difference between the means by the standard deviation of the control group (to be precise, we should use a pooled standard deviation, but it requires some calculation). As a rule of thumb, an effect size of 0.2 is considered to be small, 0.5 medium, and above 0.8 large. However, in order contexts the effect size can be given by other statistics, such as the odds ratio or correlation coefficient.

Confidence intervals are also usually calculated to have a probable range of values to derive a conclusion in which there is, for example, 95% confidence that the true value of the parameter is within the confidence interval X‒Y. The confidence interval reflects a specific interval level; for example, a 95% interval reflects a significance level of 5% (or 0.05). When comparing the difference between two means, if 0 is within the confidence interval, it means that the test is not significant.

ALTERNATIVE METHODS

In the following chapters I will present several methods for hypothesis testing, some of which have specific requirements or assumptions (type of variables, distribution, variance, etc.). However, there is also an alternative that we can use when we have numerical variables but are not sure about the population distribution or variance. This alternative method uses two simulations:

• -          Shuffling (an alternative to the significance test): we randomize the groups’ elements (we mix the elements of the two groups randomly, each time creating a new pair of groups) and compute the mean difference in each simulation. After several iterations we calculate the percentage of trials in which the difference in the means is higher than the one calculated between the two original groups. This can be compared with the significance test; for example, if fewer than 5% of the iterations indicate a larger difference, the test is significant with α < 0.05.
• 
  
• -          Bootstrapping (an alternative to confidence intervals): we resample each of our groups by drawing randomly with replacement from the groups’ elements. In other words, with the members of a group, we recreate new groups that can contain an element multiple times and not contain another one at all. An alternative resampling method would be to resample the original groups in smaller subgroups (jackknifing). After calculating the difference in means of the new pairs of samples, we have a distribution of means and can compute our confidence interval (i.e. 95% of the computed mean differences are between X and Y).

---

[1] There are also other methods that can be more or less conservative, for example the Šidák correction or the false discount rate controlling procedure.

Thank you...

I just wanted to thank all of you for downloading my eBook, you have been so many!!!

I hope you'll find it useful!

Thank you again!

Alberto

67. S-CURVE LIFE CYCLE ANALYSIS

OBJECTIVE

Identify the maturity of a product or service and forecast the demand for the next periods.

DESCRIPTION

The assumption behind this model is that usually a product has a life cycle that follows an S-shaped curve with three main phases (see 12. PRODUCT LIFE CYCLE):

• -          Emergent phase: this is characterized by a low number of firms, low revenues, and usually zero or negative margins;
• -          Growth phase: the margins are increasing rapidly (for a while, but less in the last part of the growth phase), as well as the number of firms;
• -          Mature phase: the global revenues are increasing at a far lower rate; both the margins and the number of firms are decreasing. At this point the product can enter into a decline phase, for example if a newer substitute product is introduced or if the demand is decreasing.

As mentioned in chapter 12. PRODUCT LIFE CYCLE, it is important to identify the phase of the product or service (this analysis also applies to different product levels – brand, product line, product category, etc.). For this purpose we should identify the trend in the number of firms, revenues, and margins. However, we can complement this analysis with a statistical approach that will help us to forecast the future growth.

If our product is in the emergent phase or just in the development phase, we do not have enough data to estimate the S-curve, so we should use the data of products with similar characteristics and analyze their product life cycle curves. It is important to make good assumptions about the market saturation level and the differences between our product and the similar product that can affect the growth rates in the three phases. The more data we obtain after the launch of our product, the more precisely we can compare it with the S-curve of similar products and, thus, the more we can adjust our assumptions.

If we estimate that our product is already in the growth phase, besides comparing it with similar products, we can build our S-curve using for example a logarithmic linear estimate (see the template) and forecast future sales. As shown in the figure below, the sales data for each period (years[1]) are inserted into the tthe able then transformed to estimate a linear trend. In the last row of the table, the logarithmic transformation is reversed and the sales are projected (see the graph on the right in the graphs).

Sales Data and Forecast Using Log Transformation

If we think that the S-curve does not fit the forecast properly, we can change our assumptions about the market saturation level (which will change the log-transformed data and thus our forecast). In the template the R² is calculated and we can use it as a measure of how well the forecast curve fits the actual sales data. However, remember that our assumptions about market saturation levels are more important than reaching a “perfect” fit of the forecast curve. For example, we can reach a higher R² but with an improbably high market saturation level. In fact, there are different factors that can affect the shape of the curve: the economic situation, competitors’ strategies, the entry of new substitute products, new fashions, and so on.

Log Transformed Data and Linear Trend Line (on the Left); Real Sales Data and Forecasted Sales (on the Right)

Finally, in the case that our product is already in the mature phase, this model will just project a slightly increasing or stable level of sales. However, it cannot predict or estimate whether or when our product will pass into the decline phase. This can be realized only when the product has already started to decline. Anyway, when it becomes apparent that our product is in the mature phase, our strategic actions will focus on optimization and “life extension.” On one side we have to optimize what we can obtain with this product and its market by reaching its full potential (pricing optimization, loyalty programs, product bundling, advertising, promotions, etc.). However, sometimes this is not enough either to continue to increase sales or to avoid declining. In this case we should consider actions that are able to “push upward” the S-curve, extending the life cycle of the product: product evolution, innovation, new markets, new use of the product, and so on.

Finally, it is important to remember that this is not a precise forecasting method, since this is not its main goal. With this method we can project the medium- and long-term product growth potential, but, for a more accurate forecast, this method has to be complemented with more precise methods, especially for the short-term forecast.

TEMPLATE

Download the S-Curve Life Cycle Analysis Template

---

[1] Usually the life cycle of a product takes years to reach maturity, but, if we have a shorter life cycle, we can use deseasonalized monthly data (for example using a moving average).

27. CUSTOMER LIFETIME VALUE 1 - Principles and Calculation

OBJECTIVE

Estimate the lifetime value of a customer or group of customers.

DESCRIPTION

Customer lifetime value is an indicator that represents the net present value of a customer based on the estimated future revenues and costs. The main components of this calculation are:

• -          Average purchase margin (revenue – costs);
• -          Frequency of purchase;
• -          Marketing costs;
• -          Discount rate or cost of capital.

There are several ways to calculate it, and different formulas have been proposed. The most difficult part is to estimate customers’ retention (in contractual settings) or repetition and to estimate the monetary amount that a customer will spend in the future. It is important to remember that CLV is about the future and not the past, which is why using past data of a customer is not the best method for calculating CLV. A good practice is to segment customers and estimate the retention and spending patterns based on similar customers. Then, the following formula can be applied:

• CLV = customer lifetime value
• MC = yearly marginal contribution, that is to say the total purchase revenue in a year minus the unit costs of production and marketing
• R = retention rate (yearly)
• D = discount rate
• CA = cost of acquisition (one-time cost spent by the company to reach a new customer)

Customer Lifetime Value of Different Customers

The discount rate can be the average cost of capital for the company or the related industry, and it is used to depreciate the value of future benefits to estimate what they are worth today. With this formula we can estimate the CLV of a single customer or a segment of customers. In the case of estimating it at the individual level, we should use the retention rate (r) of similar customers, for example customers who buy similar products, or more sophisticated techniques, for example cluster analysis.

When we define the value of a customer or a group of customers, we can make decisions concerning the level of attention, the investment in marketing and retention costs, or the amount that we can spend (cost of acquisition) to attract customers with a similar CLV.

Even though it is quite difficult to estimate, we have to consider that the CLV formula does not take into account the value generated by referrals. Although some formulas have been proposed,[1] this calculation is seldom used due to the lack of information. In fact, to calculate the customer referral value, we need information about the advocates and the referred customers, and for the latter we should be able to distinguish those who would have made the purchase anyway (without the referral). As a proxy we can use the NPS (see 29. NET PROMOTER SCORE^® (NPS^®)) combined with other information from surveys, such as asking whether a customer has been referred and how much the referral has affected the purchase.

The market’s historical data is the main source of information (at the individual level, usually from CRM systems), but it can be enriched with survey data, for example concerning the likelihood of repeating the purchase or recommending the product.

TEMPLATE

Download the CLV Template

---

[1] https://hbr.org/2007/10/how-valuable-is-word-of-mouth

26. RFM MODEL (Recency, Frequency, Monetary Value)

OBJECTIVE

Estimate the lifetime value of a customer or group of customers.

DESCRIPTION

This is probably the simplest model for the estimation of customers’ value. In spite of its simplicity, it is also famous for its reliability, which is based on three variables:

• -          Recency: the more recent the purchase or interaction, the more inclined the client is to accept another interaction;
• -          Frequency: the more times a customer purchases, the more valuable he or she is to the company;
• -          Monetary value: the total value of a customer also depends on the amount spent in a given period.

Customers’ Value Calculated by an RFM Model

Usually, these three variables are transformed into comparable indicators (for example into a “0 to 1” indicator) and summed up to obtain a total value indicator. We can also define different weights for each indicator.

TEMPLATE

Download the RMF Model Template