## APPENDIX V

Excerpted from Lois Plunkert, Job Openings Pilot Program: Final Report (Washington, D.C.: U.S. Department of Labor, Bureau of Labor Statistics, Office of Employment Structure and Trends, March 1981).

[from "Chapter VII. Estimates from Pilot Survey Data"]

Standard Error. There are two types of errors possible in an estimate based on a sample survey--sampling and nonsampling. Sampling errors occur because observations are made only on a sample, not on the entire population. Nonsampling errors can be attributed to many sources, e.g., inability to obtain information about all cases in the sample, definitional difficulties, differences in the interpretation of questions, inability or unwillingness to provide correct information by respondents, mistakes in recording or coding the data obtained, and other errors of collection, response, coverage, and estimation for missing data. Nonsampling errors also occur in complete censuses. The accuracy of a survey result is determined by the joint effects of sampling and nonsampling errors....

The standard error or sampling error presented in the tables is a measure of the variation among the estimates from all possible samples. A random group method of variance estimation was used. As derived, the estimated standard errors include part of the effect of the nonsampling errors.

Sampling errors in these tables are given in absolute terms. That is, sampling errors are in terms of the number of job openings. The sampling estimate and an estimate of its standard error can be used to construct interval estimates with a prescribed confidence that the interval includes the average result of all possible samples. To illustrate, if all possible samples were surveyed under essentially the same conditions, and a estimate and its estimated standard error were calculated from each sample, then:

1. Approximately 68 percent of the intervals from one standard error below the estimate to one standard error above the estimate would include the average value of all possible samples.

2. Approximately 90 percent of the intervals from 1.6 standard errors below the estimate to 1.6 standard error above the estimate could include the average value of all possible samples.

3. Approximately 95 percent of the intervals from two standard errors below the estimate to two standard errors above the estimate would include the average value of all possible samples.

For example, the estimated number of job openings in manufacturing in Massachusetts in March 1979 was 22,108 with a standard error 1,727. An approximate 95 percent confidence interval (i.e., plus or minus 2 standard errors) is from 20,381 to 23,835.

The standard errors for cells in which small units are concentrated are generally quite high since 1) small establishments had infrequent job openings and 2) these establishments had large weights. Often the openings in the smallest of these cells (such as the smaller detailed occupations) were reported on just two or three schedules.

Relative error. The ratio of the standard error to the estimate, or the relative error, is especially useful in judging the reliability of estimates. Generally, the relative error decreases as 1) response rate increases, 2) the percent of the universe in the sample increases, and 3) the number of job openings increases. Hence, the relative error were lower overall in Massachusetts and Utah where the response rates were the highest and a greater percent of the universe was in the sample.

The relative error is also lower when reported data are similar among establishments. Certain industries, such as construction and some service industries, have extreme fluxuations among reporting units because hiring often occurs in spurts. This results in very high relative errors in these industries.

The delineation of an "acceptable" relative error depends on what use is being made of the data. If one wants to get a "feel" for the general level of job openings at a point in time, an estimate with a relative error of 20 percent or more could be useful information. On the other hand, if the estimates are to be used in comparison with other statistics of a similar magnitude, a much smaller relative error might be required....

It should be emphasized that relative errors for most of the detailed estimates are quire high and therefore these estimates should be used with caution. The emphasis in the pilots was to provide data needed to develop a full-scale survey, not to produce estimates. The large relative errors indicate that the sample must be much larger in order to provide publishable estimates.

[from "Chapter VIII. Implications for National Sample Design"]

One major objective of the Job Openings Survey Pilot Test was to determine how large a sample is necessary in order to produce reliable job opening statistics. This section of the report discusses the reliability of job opening statistics as measured in the pilot test and, based on the pilot test results, the reliability criteria for a national job openings survey is explained. A number of assumptions are then made in order to propose a sample size for a national survey.

A. Establishing the Reliability Guidelines

The first step in determining a required sample size is to establish reliability guidelines. The sample size necessary for the survey is very sensitive to any change in the desired reliability criteria. Careful consideration should be given to the reliability level stated, and to determine the impact it may have on proposed uses of the data. The sample size developed later in this section assumes that a relative standard error (RSE) of less than 10 percent will be adequate for estimates of the number of job openings by occupational division by State. (The 10 percent RSE will not apply in those instances where the occupational division has fewer than 500 vacancies.) Using this as a reliability guideline, the pilot test experiences will be reviewed and implications for a full-scale survey will be discussed.

The estimates and the standard errors from the JOS March 1979 pilot test results can be used to illustrate the type of accuracy which will be achieved given the reliability guidelines. It should be noted that the 10 percent RSE criteria has not been achieved with the sample used in the JOS pilot test. In fact, in Florida in March 1979, the RSE for the total number of job openings was 14%, with major occupational division RSE's ranging form 11% to 81% with a median of 40%. Massachusetts' RSE's were somewhat lower; 5% on total, 10-100% on occupational division; with a median of 28%. However, using the 10% criteria for a full-scale survey, we would expect the State total estimates to have considerably better than 10% RSE's (probably in the range of 1-3%) based on JOS pilot data. The estimates by size class for the State total could have estimates considerably more reliable than 10% with the exception of the smallest size classes (0-3, 4-9) which should have approximately 10% RSE's. In more detailed occupational groups the RSE will depend more on the number of job openings -- if there is a substantial number of openings, say 500 or more, then there should be a RSE of 10% or less.

If this is considered adequate accuracy for estimates of level, then one should look at the accuracy of estimates of change. If decisions are going to be made based on the estimates of change, then the accuracy should be sufficient for detecting change. In the pilot test results for Phase 1, the statistical significance of change in the level of job openings between quarters was not being detected due not only to the small sample size, but also to the fact that the overall number of job openings was not changing that much. The estimate of job openings rate in Massachusetts ranged from 2.5 to 2.9 percent between March and September, 1979. The rate in Florida ranged from 2.3 to 2.7 percent, and in Texas and Utah the rate ranged form 2.2 to 2.8 percent. Some of the changes in Phase II were more dramatic and therefore were detected. Roughly speaking, changes of a magnitude of 0.2 percent should be detectable at the state level under the proposed reliability criteria. Correspondingly, changes in the job openings rate of a major occupational division of a magnitude of 0.8 percent should be detectable.

The above discussion is not intended to imply that the reliability criteria is not rigid enough, but is meant to present the implications for the reliability criteria. It should be noted that the effect of nonsampling errors has not been discussed. Their effect should also be considered in the design, and in particular the effect of nonresponse.

B. Necessary Sample Size for a National Job Openings Survey

The preliminary analysis of job opening statistics from the pilot test yielded results which contradicted some of the initial assumptions concerning vacancy distributions. Only an estimated 17% of all establishments had one or more vacancies. Also, employees in certain occupational divisions are found only in a small percentage of many industries. Even when the percentage of firms having one or more employees in an occupational division was large, the standard errors were higher than expected. In order to reduce the RSE to the desired 10% level for the pilot test survey design, it is estimated that the example size would have to be 9 times larger than the pilot test sample size. However, it is believed that additional research on our pilot test data into stratification criteria and allocation within strata can greatly improve the efficiency of the sample. Preliminary analysis of Massachusetts data suggested that a properly allocated size class stratification could yield at least a 10% reduction in the standard error for a given sample size; an additional 30% reduction in standard error of the estimate of total vacancies could be achieved with an appropriate allocation by industry division.

Based on BLS experience in gains due to stratification, it will be assumed that further stratification by 2.3 or 4 digit SIC would reduce the standard error beyond 40% to approximately 50%. Hence, it will be assumed that the pilot sample size in a state such as Massachusetts will need to be increased by about 4 1/2 times in order to achieve the accuracy desired.

A proposed sample size of 5400 establishments in Massachusetts assumes that the response rates in a full scale survey will be at least 75%. The sample size needed in Massachusetts, which is the 10th largest State in terms of the total number of establishments, will be assumed to be the average sample size per State for 50 States and the District of Columbia. While there was not enough evidence from the pilot test to assure that a 75% response rate could be achieved for all States in the initial survey period, the design described below may be able to achieve an initial response rate of greater than 75% and continue to achieve a high response rate in subsequent quarterly surveys....