Recommendation E.421
SERVICE QUALITY OBSERVATIONS ON A STATISTICAL BASIS
1	Definitions
1.1	service observation
F: observation de la qualité de service
S: observación de la calidad del servicio
	Monitoring to obtain a complete or partial assessment of the quality of telephone calls, excluding test calls.
1.2	manual observation
F: observation manuelle
S: observación manual
	Monitoring of telephone calls by an observer without using any automatic data–recording machine.
1.3	automatic observation
F: observation automatique
S: observación automática
	Monitoring of telephone calls without an observer.
1.4	semi–automatic observation
F: observation semi–automatique
S: observación semiautomática
	Monitoring of telephone calls using equipment which records some data automatically. For example, equipment in which information, such as exchange being observed, number dialled by the subscriber, metering pulses and time of call, is recorded automatically on some means suitable for data processing. The observer merely has to key in a code indicating the condition observed.
2	Relative merits of manual, automatic and semi–automatic observations
2.1	The three methods mentioned above in §§ 1.2, 1.3 and 1.4 are not exclusive; for example, automatic observations may be used to supplement observations taken by an operator. It was considered in 1968 that the need for automatic observations would increase in view of the heavy cost associated with manual or semiautomatic observations on the rapidly expanding international network. It was also considered that automatic observations would not entirely supersede observations taken by an observer within the foreseeable future.
	The relative merits of the three methods can be assessed as follows:
2.2	Manual observation
	Provides all the data required in Tables 1/E.422 and 1/E.423.
	Observations can be carried out with a minimum of equipment.
	Observations can permit the detection of a number of abnormalities which cannot be detected automatically, e.g. very poor speech transmission (item 5.2 of Table 1/E.422), or difficulty with audible tones encountered in the international service (item 6.4 of Table 1/E.422).
2.3	Semi–automatic observation
	Provides all the data required in Tables 1/E.422 and 1/E.423.
	There is a saving in staffing costs compared with manual observation.
	Greater accuracy compared with manual observation is possible due to the fact that there is an automatic recording of the number dialled, the time of the call, etc.
	It is possible for the observer to devote greater attention to the more critical conditions being checked during observations of calls.
	The results are produced in a form suitable for subsequent mechanized analysis.
	Owing to the reduction of costs it is possible to obtain a larger sample for the same expenditure.
	Semi–automatic equipment may be converted, during certain hours of the day, to automatic operation.
2.4	Automatic observation
	Operating cost is minimum (staff reduction).
	Continuous observation is possible.
	It is possible to have a larger sample.
	Human error is eliminated.
	Automatic processing of data is facilitated.
	Conversational privacy is ensured.
	Control of the time at which observations are made is facilitated.
	Some of the differences between internal and external automatic observations are given below:
2.4.1	Internal automatic observations can be made in the switching centre itself, on the incoming side or the outgoing side or in between, according to the way the switching centre is engineered:
a)	Only line signals, such as seizure, answer, etc. can be monitored, and also register signals as long as they do not pass through the exchange in an end–to–end signalling procedure.
b)	Signals received are only monitored if the exchange itself operates correctly in that respect.
c)	Item b) applies also to outgoing signals. If there is a fault in the exchange it can happen that signals have not been sent in the appropriate way without the exchange being aware of it.
	More information on this type of observation technique is given in Recommendation E.425.
2.4.2	External automatic observations are made by means of monitoring equipment which is supervising the traffic on incoming or outgoing lines:
–	All signalling signals can be monitored.
–	The detection of tones, speech and data is possible if advanced equipment is used.
–	This observation technique provides all the data required in Table 2/E.422 and Table 2/E.423.
–	The application is very flexible and can be used instead of manual or semi–automatic observation techniques.
3	Time of observations
	The results of all observations taken over the whole day should be recorded in Table 1/E.422 or Table 2/E.422.
	In the case where observations are not taken over the whole day the observation period is recorded under the heading “Time of observations” and should include the three busiest hours of the day.
4	Observation access points
4.1	Observations for Table 1/E.422 or Table 2/E.422 should be carried out from access points as close as possible to the outgoing international exchange.
	The following access points can be considered:
For definitions of test access points see Recommendation M.700. See also Recommendation M.110.
i)	outgoing relay set of an international circuit (“exchange” side), i.e. international circuit access point1);
ii)	incoming relay set of a national circuit;
iii)	link circuits of the international exchange.
	Observations will be made only while the call is being set up, and a few seconds after the called subscriber's reply.
	When the circuit access point is used for observation of international calls it is possible that the service quality of the international exchange may not be checked by either international or national observation programmes.
	Preferably, and where technically feasible for the most complete results, observations for Table 1/E.422 should be carried out as close as possible to the international exchange on the national side. This would be more representative of service to the subscriber, and allows observation of call failure at the outgoing international exchange. Where it is not possible to make the distinction between failures in the outgoing international exchange, and failures beyond this exchange, or where there is a meaningful advantage in doing so, observations should be taken on the outgoing side.
	It is necessary to state in Table 1/E.422 or Table 2/E.422 the access point where the observations have been made, as observations obtained at each one of the three access points mentioned above are not comparable.
4.2	Observations for Table 1/E.423 must be carried out from access points on the operators' positions.
5	Number of observations
5.1	Service observing programmes should be established in such a manner that statistical results obtained be as reliable as practicable bearing in mind the cost of obtaining large samples.
5.2	According to the studies carried out by the CCITT in 1964–1968, the quantities shown below are considered the minimum quantities to provide a general indication of the quality of service.
5.2.1	Table 1/E.422
	The minimum number of observations per outgoing circuit group for Table 1/E.422 should be 200 per month when more than 20 circuits are included in a group, 200 per quarter when there are between 10 and 20 circuits in a group and 200 per year if there are less than 10 circuits in a group.
5.2.2	Table 1/E.423
	The minimum number of observations for Table 1/E.423 should be 200 per quarter when there are more than 20 circuits in the group, 200 per semester when there are between 10 and 20 circuits and 200 per year when there are less than 10 circuits in the group.
5.2.3	Transit traffic
	Where an outgoing circuit group also carries transit traffic it is desirable to obtain data for each destination country reached via this circuit group. In principle, the number of observations for each destination should be obtained as indicated above. To accomplish this, one should use for each destination country its corresponding number of erlangs and derive from these erlangs a theoretical number of circuits.
	However, where only a very small amount of traffic is handled, e.g. less than 5 erlangs, each Administration may wish either to make a smaller number of observations or (e.g. in case of no complaints) no observations at all and rely on the information obtained at the transit exchange.
5.3	The number of observations specified above will provide a general indication of results on quality of service in certain broad categories. Administrations may desire more accurate results especially for the individual categories in Table 1/E.422.
	Attention is drawn to Table 1/E.421 which gives the number of observations required to obtain a certain degree of accuracy.
TABLE 1/E.421
Expected percentage rate 	Number of observations of a random sample required to predict with 95% confidence the true percentage of failure with an accuracy of:	
of failure	± 25%	± 30%	± 35%	± 40%	± 45%	± 50%	
  2	 3136	 2178	 1600	 1225	 1030	 880	
  4	 1536	 1067	  784	  600	  500	 440	
  6	 1003	  696	  512	  392	  330	 290	
  8	  736	  511	  376	  288	  245	 215	
 10	  576	  400	  294	  225	  195	 170	
 12	  469	  326	  239	  183	  150	 132	
 14	  393	  273	  201	  154	  128	 112	
 16	  336	  233	  171	  131	  112	  98	
 18	  292	  202	  149	  114	   95	  80	
 20	  256	  178	  131	  100	   85	  70	
 30	  149	  104	   76	   60	   50	  42	
 40	   96	   67	   50	   38	   30	  24	
 50	   64	   44	   33	   25	   20	  16	

Annex to Table 1/E.421
Examples of use of Table 1/E.421
	Example 1 – It is estimated from previous results that a particular type of failure occurs on about 4% of calls. If it is required to confirm, with 95% confidence, that the existing failure rate is between 3% and 5% (i.e. ± 25% of 4%), then observations must be made on a random sample of 1536 calls.
	Example 2 – For an expected failure rate of 2%, observations must be made on a random sample of about 1200 calls (1225 in the table) to predict, with 95% confidence, that the true percentage is between 1.2% and 2.8% (i.e. ± 40% of 2%). This means that when 200 observations are taken over a period it is necessary to take the “rolling average” of conditions over six periods. The rate of failure for a number of categories important from the maintenance point of view is expected to be about 2%.
	Example 3 – After observations have been taken and the rate of failure in the sample has been calculated, the table may be used in a “backward” direction to give a rough indication of the accuracy of the result.
	Suppose that out of a sample of 1000 observations, there were 29 failures due to cause “X” and 15 failures due to cause “Y”. The rates of failure in the sample due to X and Y, respectively, are then 2.9% and 1.5%. From the table, it is apparent from this sample of 1000 calls that the true rate of failure due to X has an accuracy of about ± 35% (i.e. is between 1.9% and 3.9%), and that due to Y has an accuracy of about ± 50% (i.e. is between 0.8% and 2.3%).
6	Exchange and analysis of the results of observations
6.1	Exchange of the results of observations
	The following periodicities are proposed for the exchange of results between Administrations:
	Table 1/E.422 or Table 2/E.422 – a monthly exchange is desirable;
	Table 1/E.423 or Table 2/E.423 – a quarterly exchange is desirable.
	Nevertheless, in the case of small groups of circuits (less than 20 circuits) the information should be exchanged after 200 observations have been made but never later than one year in any case; attention is drawn to the fact that less than 200 observations are of little value.
	Results of observations will be reported without delay:
–	to the Administrations and the network analysis point of the country where observations are carried out;
–	to the Administrations and the network analysis point of the other country (including transit Administrations and their network analysis point when involved).
	The benefits to be derived from service observations tend to decrease if there is any increase in the time taken to make the results available to those who can take action to bring about an improvement. The results of service observations according to Tables 1/E.422 and 1/E.423 should therefore be made available to the Administration in the countries of destination as soon as possible after completion of the observation period and in any case within six weeks.
6.2	Analysis of observation results
	An analysis of the results should be carried out in the country of origin as well as in the country of destination.
	Some Administrations have found it useful to distribute to other Administrations concerned, service observation statistics in the form of graphs.