What is statistical significance? Determining the significance of the impact

I have no clarity in understanding how the significance of READY-made context-sensitive behavioral chains is determined. As I understand it, the behavioral chain is some kind of BRAIN activity. behavioral context is a pattern of behavior in a given state of the environment. the state of the environment is monitored by sensory organ receptors. in order to determine the significance of a behavioral context, it is necessary to obtain the expected result of behavior in a given state of the environment, and before launching the behavioral chain for execution. for this to happen, by now the image of behavioral options in a certain range of environmental states containing what is being tracked by the senses at the moment must ALREADY be active in the brain. So?

>>a behavioral chain is some kind of BRAIN activity

No. This is a sequence of links responsible for more elementary actions in the program of the entire chain. As the individual links are sequentially active, the entire program begins to be executed by separate subroutines.

>>behavioral context is a pattern of behavior under a given state of the environment

In one chain, individual links may have branches into other chains so that under some conditions activity continues along one chain, and under other conditions - along other chains. This is the contextual nature of program execution depending on conditions.

>>to determine the significance of a behavioral context, it is necessary to obtain the expected result of behavior under a given state of the environment

Each link already has some kind of significance attached to it - as a result of working out a given link in the chain under certain conditions. This significance can only be assessed by conscious attention to this chain. Without awareness, significance plays a permissive (positive significance) or prohibitive (negative) role. If, under these conditions, a negative significance is associated with a link, then further activity of the chain stops.

Conscious attention can scan the chain without performing actions (blocking them) and receive significance, including the final link that predicts the outcome of the action.

When constructing a regression model, the question arises of determining the significance of the factors included in the regression equation (1). Determining the significance of a factor means clarifying the question of the strength of the factor’s influence on the response function. If, in the course of solving the problem of checking the significance of a factor, it turns out that the factor is insignificant, then it can be excluded from the equation. In this case, it is considered that the factor does not have a significant impact on the response function. If the significance of the factor is confirmed, then it is left in the regression model. It is believed that in this case the factor has an influence on the response function that cannot be neglected. Solving the question of the significance of factors is equivalent to testing the hypothesis that the regression coefficients for these factors are equal to zero. Thus, the null hypothesis will have the form: , where is the subvector of the dimension vector (l*1). Let's rewrite the regression equation in matrix form:

Y = Xb+e,(2)

Y– vector of size n;

X- size matrix (p*n);

b is a vector of size p.

Equation (2) can be rewritten as:

,

Where X l and X p - l - matrices of size (n,l) and (n,p-l), respectively. Then the hypothesis H 0 is equivalent to the assumption that

.

Let's determine the minimum of the function . Since under the corresponding hypotheses H 0 and H 1 = 1 - H 0 all parameters of a certain linear model are estimated, the minimum under the hypothesis H 0 is equal to

,

whereas for H 1 it is equal

.

To test the null hypothesis, we calculate statistics , which has a Fisher distribution with (l,n-p) degrees of freedom, and the critical region for H 0 is formed by 100*a percent of the largest values ​​of F. If F F cr - the hypothesis is rejected.

The significance of factors can be checked using another method, independently of each other. This method is based on the study of confidence intervals for the coefficients of the regression equation. Let us determine the variances of the coefficients, The values ​​are the diagonal elements of the matrix . Having determined estimates of coefficient variances, confidence intervals can be constructed for estimates of regression equation coefficients. The confidence interval for each estimate will be , where is the tabular value of the Student's criterion for the number of degrees of freedom with which the element was determined and the selected significance level. A factor with number i is significant if the absolute value of the coefficient for this factor is greater than the deviation calculated when constructing the confidence interval. In other words, the factor with number i is significant if 0 does not belong to the confidence interval constructed for this coefficient estimate. In practice, the narrower the confidence interval at a given significance level, the more confident we can be about the significance of the factor. To check the significance of a factor using the Student's test, you can use the formula . The calculated t-test value is compared with the table value at a given significance level and the corresponding number of degrees of freedom. This method of checking the significance of factors can be used only if the factors are independent. If there is reason to consider a number of factors dependent on each other, then this method can only be used to rank factors according to the degree of their influence on the response function. The test of significance in this situation must be supplemented by a method based on the Fisher criterion.

Thus, the problem of checking the significance of factors and reducing the dimension of the model in the case of an insignificant influence of factors on the response function is considered. Further here it would be logical to consider the issue of introducing additional factors into the model, which, according to the researcher, were not taken into account during the experiment, but their impact on the response function is significant. Let us assume that after the regression model has been selected

, ,

the task arose to include additional factors x j in the model so that the model with the introduction of these factors takes the form:

, (3)

where X is a matrix of size n*p of rank p, Z is a matrix of size n*g of rank g and the columns of matrix Z are linearly independent of the columns of matrix X, i.e. matrix W of size n*(p+g) has rank (p+g). Expression (3) uses the notation (X,Z)=W, . There are two possibilities for determining estimates of the newly introduced model coefficients. First, you can find the estimate and its dispersion matrix directly from the relations

At the end of our collaboration, Gary Klein and I finally came to an agreement on the main question posed: when should we trust an expert’s intuition? We are of the opinion that it is still possible to distinguish meaningful intuitive statements from empty ones. This can be compared to analyzing the authenticity of an object of art (for an accurate result, it is better to start not with examining the object, but with studying the accompanying documents). Given the relative immutability of the context and the ability to identify its patterns, the associative mechanism recognizes the situation and quickly develops an accurate forecast (decision). If these conditions are met, the expert's intuition can be trusted.
Unfortunately, associative memory also gives rise to subjectively valid but false intuitions. Anyone who has followed the development of a young chess talent knows that skills are not acquired immediately and that some mistakes along the way are made with complete confidence in oneself being right. When assessing an expert's intuition, one should always check whether he or she has had sufficient chances to learn environmental cues - even when the context remains unchanged.
In a less stable, unreliable context, the judgment heuristic is activated. System 1 can provide quick answers to difficult questions by replacing concepts and providing coherence where there should be none. As a result, we get an answer to a question that was not asked, but it is quick and quite plausible, and therefore capable of slipping through the lenient and lazy control of System 2. Let’s say you want to predict the commercial success of a company and you think that this is what you are evaluating, whereas in fact In fact, your assessment is based on the energy and competence of the company's management. The substitution occurs automatically - you don’t even understand where the judgments that your System 2 accepts and confirms come from. If a single judgment is born in the mind, it may be impossible to subjectively distinguish it from a significant judgment made with professional confidence. This is why subjective conviction cannot be considered an indicator of the accuracy of the forecast: judgments-answers to other questions are expressed with the same conviction.
You might be surprised: how come Gary Klein and I didn’t immediately think of evaluating expert intuition depending on the constancy of the environment and the expert’s training experience, without looking at his belief in his words? Why didn't you find the answer right away? This would be a useful remark, since the decision was looming before us from the very beginning. We knew in advance that the significant intuitions of fire brigade leaders and nurses were different from the significant intuitions of the stock market analysts and specialists whose work Meehl studied.
It is now difficult to recreate what we devoted years of work and long hours of discussion, endless exchanges of drafts and hundreds of emails. Several times each of us was ready to give up everything. However, as always happens with successful projects, once we understood the main conclusion, it began to seem obvious from the very beginning.
As the title of our article suggests, Klein and I argued less often than we expected and made joint decisions on almost all important points. However, we also discovered that our early disagreements were not just intellectual. We had different feelings, tastes and views about the same things, and over the years they have changed surprisingly little. This is clearly manifested in the fact that each of us finds it entertaining and interesting. Klein still winces at the word “distortion” and rejoices when he learns that some algorithm or formal technique produces a delusional result. I am inclined to see rare errors in algorithms as a chance to improve them. Again, I rejoice when a so-called expert utters predictions in a context with zero credibility and gets a well-deserved beating. However, for us, in the end, intellectual agreement became more important than the emotions that divide us.

Hypotheses are tested using statistical analysis. Statistical significance is found using the P-value, which corresponds to the probability of a given event assuming that some statement (null hypothesis) is true. If the P-value is less than a specified level of statistical significance (usually 0.05), the experimenter can safely conclude that the null hypothesis is false and proceed to consider the alternative hypothesis. Using the Student's t test, you can calculate the P-value and determine significance for two data sets.

Steps

Part 1

Setting up the experiment

Define your hypothesis. The first step in assessing statistical significance is to choose the question you want to answer and formulate a hypothesis. A hypothesis is a statement about experimental data, their distribution and properties. For any experiment, there is both a null and an alternative hypothesis. Generally speaking, you will have to compare two sets of data to determine whether they are similar or different.

• The null hypothesis (H 0) typically states that there is no difference between two sets of data. For example: those students who read the material before class do not receive higher grades.
• The alternative hypothesis (H a) is the opposite of the null hypothesis and is a statement that needs to be supported by experimental data. For example: those students who read the material before class get higher grades.

1. Set the significance level to determine how much the data distribution must differ from normal for it to be considered a significant result. Significance level (also called α (\displaystyle \alpha )-level) is the threshold you define for statistical significance. If the P-value is less than or equal to the significance level, the data is considered statistically significant.

   Decide which criterion you will use: one-sided or two-sided. One of the assumptions in the Student t test is that the data is normally distributed. The normal distribution is a bell-shaped curve with the maximum number of results in the middle of the curve. Student's t-test is a mathematical method of testing data that allows you to determine whether data falls outside the normal distribution (more, less, or in the “tails” of the curve).

   • If you are not sure whether the data is above or below the control group values, use a two-tailed test. This will allow you to determine significance in both directions.
   • If you know in which direction the data might fall outside the normal distribution, use a one-tailed test. In the example above, we expect students' grades to increase, so a one-tailed test can be used.

2. Determine sample size using statistical power. The statistical power of a study is the probability that, given the sample size, the expected result will be obtained. A common power threshold (or β) is 80%. Analyzing statistical power without any prior data can be challenging because it requires some information about the expected means in each group of data and their standard deviations. Use an online power analysis calculator to determine the optimal sample size for your data.

   • Typically, researchers conduct a small pilot study that provides data for statistical power analysis and determines the sample size needed for a larger, more complete study.
   • If you are unable to conduct a pilot study, try to estimate possible averages based on the literature and other people's results. This may help you determine the optimal sample size.

   Part 2

   Calculate Standard Deviation

   1. Write down the formula for standard deviation. The standard deviation shows how much spread there is in the data. It allows you to conclude how close the data obtained from a certain sample are. At first glance, the formula seems quite complicated, but the explanations below will help you understand it. The formula is as follows: s = √∑((x i – µ) 2 /(N – 1)).

      • s - standard deviation;
      • the sign ∑ indicates that all data obtained from the sample should be added;
      • x i corresponds to the i-th value, that is, a separate result obtained;
      • µ is the average value for a given group;
      • N is the total number of data in the sample.

   2. Find the average in each group. To calculate the standard deviation, you must first find the mean for each study group. The mean value is denoted by the Greek letter µ (mu). To find the average, simply add up all the resulting values ​​and divide them by the amount of data (sample size).

      • For example, to find the average grade for a group of students who study before class, consider a small data set. For simplicity, we use a set of five points: 90, 91, 85, 83 and 94.
      • Let's add all the values ​​together: 90 + 91 + 85 + 83 + 94 = 443.
      • Let's divide the sum by the number of values, N = 5: 443/5 = 88.6.
      • Thus, the average for this group is 88.6.

   3. Subtract each value obtained from the average. The next step is to calculate the difference (x i – µ). To do this, subtract each value obtained from the found average value. In our example, we need to find five differences:

      • (90 – 88.6), (91 – 88.6), (85 – 88.6), (83 – 88.6) and (94 – 88.6).
      • As a result, we get the following values: 1.4, 2.4, -3.6, -5.6 and 5.4.

   4. Square each value obtained and add them together. Each of the quantities just found should be squared. This step will remove all negative values. If after this step you still have negative numbers, then you forgot to square them.

      • For our example, we get 1.96, 5.76, 12.96, 31.36 and 29.16.
      • We add up the resulting values: 1.96 + 5.76 + 12.96 + 31.36 + 29.16 = 81.2.

   5. Divide by sample size minus 1. In the formula, the sum is divided by N – 1 due to the fact that we do not take into account the general population, but take a sample of all students for evaluation.

      • Subtract: N – 1 = 5 – 1 = 4
      • Divide: 81.2/4 = 20.3

   6. Take the square root. After you divide the sum by the sample size minus one, take the square root of the value found. This is the last step in calculating the standard deviation. There are statistical programs that, after entering the initial data, perform all the necessary calculations.

      • In our example, the standard deviation of the grades of those students who read the material before class is s =√20.3 = 4.51.

   Part 3

   Determine significance

   1. Calculate the variance between the two groups of data. Before this step, we looked at an example for only one group of data. If you want to compare two groups, you should obviously take data from both groups. Calculate the standard deviation for the second group of data, and then find the variance between the two experimental groups. The variance is calculated using the following formula: s d = √((s 1 /N 1) + (s 2 /N 2)).

Professional analysts pay a lot of attention to statistical significance, and that's a good thing. However, statistical significance is only one aspect of a good analysis.

Testing for statistical significance involves making a number of assumptions and determining the likelihood that the results obtained would occur if the assumptions were correct. Testing for statistical significance will help ensure that the data is not misleading. It will show from a mathematical point of view whether the difference is significant enough. Sometimes differences that seem significant are not so, and sometimes small differences turn out to be significant. Statistical testing will ensure that the conclusions drawn are correct.

An entire discipline has been created based on testing. In the business world it is known as the approach "test and learn" (test and learn), covering basic experimental concepts taught in statistics courses. In a test-and-learn environment, the experiment is designed so that you can measure the effects of using one or more options and determine which one will work best.