In general, when a QTG test is run using Engineering Data as the source, the tolerances are reduced from those applicable to tests run against flight test data. As you rightly point out, a value of 20% of the flight test tolerances is stated as being the guideline, so that +/- 3 knots airspeed becomes +/- 0.6 knots, etc. These reduced tolerances can make a test very difficult to match, even when the mathematical models from the aircraft manufacturer have been perfectly implemented by the simulator manufacturer when building the FSTD. EASA recognises this problem and states some of the possible reasons for this in Appendix 1 to AMC1 FSTD(H).300, paragraph (a).(6).

The 20% ‘guideline’ was set by international agreement back in 2005, and the FAA applied this criteria from the first edition of 14 CFR Part 60 from July 2008. However, right from the start the industry struggled to meet these 20% of flight test tolerances, so the FAA responded in 2009 using a Guidance Bulletin which increased the allowable values to +/- 40% of the flight test tolerance values. EASA, on the other hand, have left the +/-20% in CS-FSTD(A), including Issue 2 (2016), and CS-FSTD(H), although they may increase this to 40% for both fixed-wing and rotary-wing FSTDs in the next issue of the regulations. So, there is currently a disparity between the FAA and EASA.

As for whether the regulators (both FAA and EASA) might accept test results which do not meet the 20% (EASA) and 40% (FAA) tolerances, it is, as you point out, quite possible for the failed test to be accompanied by a rationale as to why the reduced tolerances cannot be met. BUT, in my experience the FSTD Operator would have to have a very good case for this to be accepted – it is quite possible, but that does not mean it is simple! Incidentally, this situation can also apply to tests based on flight test data tolerances – a rationale can be supplied for any test which may require further explanation, but the Regulator can still challenge that rationale if they think that the reasons given for the failed test are not adequate.

The running of QTG tests is not just to satisfy a regulatory requirement, it is meant to help ensure that the FSTD continues to meet the performance it was qualified to.  In that perspective, QTG checks should be used to quickly verify a FSTD’s performance as part of its ongoing maintenance & quality checking.   In addition, and contrary to common belief, QTG tests are not the sole criteria used to establish a FSTDs performance, they are just spot checks.  For example, QTG tests do not verify the performance of systems, navigation or atmospheric conditions.

From a regulatory perspective, although not clearly defined in the past, it has always been the intention to run all of the QTG tests sequentially over a 1 year period.  It was also recommended that for things that may go out of tolerance more easily, such as control loading, motion and visual,  some of the tests should be run on a quarterly basis unless you can prove through your quality and compliance monitoring system that your FSTD does not change over long periods of time.

Also, don’t forget that it was always an intention that all QTG validation tests should also be run manually (pilot flown) over a period of time (4 – 5 years), as automatic tests cannot be considered as truly end-to-end with the pilot in the loop.  

EASA rules are quite specific;

Reference CS-FSTD (A) page 47 of 174  – FSTD Recurrent Qualification Basis (9) (ii)
“The FSTD operator should run the complete QTG, which includes validation, functions & subjective tests, between each annual evaluation by the competent authority. As a minimum, the QTG tests should be run progressively in at least four approximately equal three-monthly blocks on an annual cycle. Each block of QTG tests should be chosen to provide coverage of the different types of validation, functions & subjective tests. Results should be dated and retained in order to satisfy both the FSTD operator as well as the competent authority that the FSTD standards are being maintained. It is not acceptable that the complete QTG is run just prior to the annual evaluation. 

You do not say in which flight control axis the control forces were too low, if it was the collective then the friction adjustment might not be representative of the aircraft.  If it was the cyclic forces, which are normally very small, we suggest you might set up two files, one to data from the Fokker (target) plots and the other to those requested by the pilots and then run with the subjective pilot forces until you have had a significant number of pilots to comment on the forces.

The normal situation is that the Fokker plots are taken without vibration and when there is vibration, as in the helicopter, then the running friction is lower and the hysteresis force loop smaller. Another issue is whether the pilots validating the results were true test pilots or those with significant experience on the aircraft type and that the a/c data was recorded correctly to provide the target values. It is not unusual for there to be variations between individual aircraft although a 30% difference is unlikely if the data aircraft had been properly maintained.

We assume that the data was collected by yourselves or by contract and was not supplied by the aircraft manufacturer. In either case it would be wise to obtain additional data test cases in order to explain whether there was any discrepancy in their values.

Having considered all of this and still being unable to resolve the issue, we would suggest that the matter be discussed with the Authority prior to submitting the QTG for validation and we cannot emphasis strongly enough the need to discuss this issue with the relevant NAA ahead of time.

A 30% change in control force would need some very careful explanation. It’s more than likely that there’s an error somewhere in the model and not necessarily in the control system.

It might be because the A/C data is not detailed enough, meaning you either have not recorded enough parameters or the parameters do not have a high enough resolution to see exactly what the A/C is doing. Regardless, if the sim inputs are accurately following the A/C and the model is good, then the sim outputs should follow within tolerances. However, there are often differences between simulator tests and real world data due to wind velocity etc. as you state. Even with the correct control inputs the response can be divergent in long period tests. One testing method to overcome this is to use a closed loop feedback system to maintain the profile; deviations outside the tolerance parameters can be axplained as a note to the test result. See Vol I of RAeS Handbook for examples of these conditions & the explanations provided for the differences. The amount of acceptable deviation on the inputs we are afraid is down to “good engineering judgement”. Any systematic error across multiple tests should be clearly explained (or justified) in a rationale. (We assume that you are discussing the parameters that are those mandated for the test in the Table of Validation and not those additional parameters listed in the RAeS Handbook, which are listed to assist in any fault analysis, and in which deviations might be acceptable).

It is common to have some reference values offset at the start of a test but are you asking if you can offset the tolerance centre point for the whole plot so that would effectively mean instead of +/- 5% (or whatever) it becomes +8/-3% (or whatever)? That would be unacceptable in so much as we believe the tolerance is specific to a piece of A/C data at a specific time.

There are two documents that define standards for commercial helicopter simulators; FAA Advisory Circular 120-63 “Helicopter Simulator Standards”, and the European Joint Aviation Authorities (JAA) document JAR-STD 1H “Helicopter Full flight Simulators”.

The referenced documents require visual systems to contain the following features:

  • Three specific airport models for use an operator’s typical route flying procedures.
  • “Capability” to present ground and air hazards.
  • Sufficient scene content to conduct training scenarios in the operator’s approved training program.
  • An “operator” in this context refers to commercial entities such as an airline or corporate operator of fixed or rotary wing aircraft.

Neither document specifies a minimum or maximum quantity of moving ground and air hazards that must be displayed in a visual scene.

Operators comply with the FAA/JAA requirements for moving models (e.g., trucks, planes, etc.) with a single airborne hazard that can be selected (by the instructor) to intrude on the host airplane from any direction or altitude. Ground hazards can be selected to appear at designated locations on the airport runways, taxiways or terminal ramps. The maximum number of moving ground objects that we have seen used in a current generation visual system is four (4), although we’re sure more are possible.

Visual systems that we have seen on helicopter simulators contain many more moving ground objects than those found in a typical commercial simulator. However, we are not aware of any document describing a specified quantity of objects to appear simultaneously in a visual scene (database). Decisions on required quantity of objects and scene content are usually a determined by the user’s training scenarios and capability of the visual system technology to meet the specified requirement.

Evans and Sutherland, a major visual system supplier, have a product called “Harmony” undergoing testing on a network of six helicopter simulators which can be operated together. It was due to the need to network the simulators that the initial demonstration was 350 models, the capability to demonstrate 500 models displayed simultaneously, or are selectable through a lesson plan menu, is now available.

Joe DePaola – Managing Partner, Training Technology International Ltd.

3.m.1 states that daylight, twilight and night is required “as applicable for the level of qualification sought” which is somewhat redundant as 3.m.4 states that only a Level C & D device requires daylight, 3.m.5 states that only a Level C & D device require twilight, and 3.m.6 states the Levels A, B, C & D require night capability.  Appendix 8 to AMC1 FSTD(A).300 – Table 1 is more clear.  However, it is possible that somebody wishes to conduct a training task on a Level A & B device that requires daylight or twilight capability, and in that case the standards cover that possibility.

We understand that “Special Category” was included (probably by the UK CAA) to cover cases where FSTD’s were in service and considered good devices, but had not been evaluated to an established standard previous to the introduction of Levels A/A(G), B/(BG), C/C(G) and D/D(G), i.e. the initial Qualification predates the use of a formal standard. In the UK for example that would mean any device qualified prior to the adoption of CAP 435.(circa 1989).  The UK CAA advises that “We actually only oversee one such device, a S-61 helicopter FFS from ancient history that actually has no QTG at all. It has plenty of capability but is limited in what training credits it can deliver due to age etc.”.  We are not sure there are any other devices out there designated as a S/C but regardless these will obviously be retired in due course.