Proficiency testing and interlaboratory comparisons in food microbiology: what the standards require and how to read a z-score

What proficiency testing is and how it differs from other comparisons

A laboratory with validated methods and trained analysts can carry a bias that nobody detects from the inside. The way to see it is to compare its results with those of other laboratories analyzing the same material. ISO/IEC 17043:2023 defines an interlaboratory comparison as the organization, performance and evaluation of measurements or tests on the same or similar items by two or more laboratories in accordance with predetermined conditions. Proficiency testing (PT) is a particular case: it evaluates the performance of each participant against pre-established criteria by means of an interlaboratory comparison.

Not every comparison evaluates the laboratory. An interlaboratory study to obtain the performance characteristics of a method, such as those of the ISO 16140 series described in validation and verification of microbiological methods, measures the method and not the participant.

What ISO/IEC 17025 requires

Clause 7.7 of ISO/IEC 17025:2017 requires a procedure for ensuring the validity of results. Subclause 7.7.1 brings together the internal measures (reference materials, controls, replicates, comparison between analysts) and 7.7.2 adds the external view: the laboratory must monitor its performance by comparison with the results of other laboratories, through proficiency testing or other interlaboratory comparisons, where available and appropriate. The general framework of accreditation is in ISO/IEC 17025 accreditation in food laboratories.

The ILAC-P9:01/2024 policy, applied by accreditation bodies that are signatories to the ILAC mutual recognition arrangement, turns that requirement into concrete rules:

• Participation is considered mandatory when proficiency testing is available, appropriate and judged necessary according to the laboratory's risk analysis.
• The laboratory must have a participation plan, and the accreditation body evaluates it before granting accreditation to check that it covers the scope representatively.
• When performance is unsatisfactory, the body evaluates the evidence of prompt and appropriate corrective actions.

Level and frequency of participation

No one takes part in a scheme for every combination of method, analyte and matrix. ILAC-P9, following guidance document EA-4/18 G:2021, proposes grouping the scope into sub-disciplines of technical competence: combinations of technique, property and product that are equivalent enough for the result of one to represent the others. Different techniques rarely fit in the same area; different products can, if the matrices are of an equivalent nature. The minimum frequency per area is set by the laboratory according to risk: number of analyses, staff turnover and experience, stability of the analyte and the matrix, robustness of the method and the end use of the data. The guidance cites food safety among the areas requiring a high level of assurance.

The three standards behind proficiency testing in microbiology

ISO/IEC 17043:2023: provider competence

It sets the requirements for the competence and impartiality of proficiency testing providers. The second edition, of May 2023, replaced the 2010 edition, which according to ILAC-P9 remained valid until May 2026. For the participant, the practical question is whether the provider is accredited to this standard by a body that is a signatory to the ILAC arrangement, which is the recognized way of demonstrating its competence.

ISO 13528:2022: the statistics

It describes the statistical methods for designing schemes and analyzing their data, and gives recommendations for interpreting results, both quantitative and qualitative. The third edition is from August 2022 and has an amendment published in 2026. Here are the methods for setting the assigned value (the reference value of the round), the standard deviation for proficiency assessment (σ_pt, which defines how much is tolerated) and the performance scores.

ISO 22117:2019: what is specific to microbiology

It replaced the technical specification ISO/TS 22117:2010 and was confirmed at its systematic review. It establishes requirements and guidance for organizing proficiency testing schemes in the microbiology of food and beverages, animal feed, environmental samples from food and feed production and handling, and primary production; also of water used in food production or considered food by national legislation. It covers the technical organization of the scheme and the statistical treatment of results, and is intended to be used together with ISO/IEC 17043 and ISO 13528: it only adds what microbiology additionally needs. One example is preparation of the material: a 2021 scheme in India for Listeria monocytogenes and Salmonella in freeze-dried shrimp checked the homogeneity and stability of the item during storage (up to 28 days) and transport (up to 6 days) before shipping it.

How to read a z-score

The most widely used score is the z-score: z = (x_i − x_pt) / σ_pt, where x_i is the laboratory's result, x_pt the assigned value and σ_pt the standard deviation for proficiency assessment. The usual interpretation, taken from ISO 13528 and included in the 2021 Eurachem guide, is:

• |z| ≤ 2.0: satisfactory performance, no signal.
• 2.0 < |z| < 3.0: questionable performance, warning signal.
• |z| ≥ 3.0: unsatisfactory performance, action signal.

An illustrative example: if the assigned value of a count is 4.20 log CFU/g, σ_pt is 0.25 log and the laboratory reports 4.85 log CFU/g, z = 0.65 / 0.25 = 2.6: questionable. What a CFU represents and how a count is interpreted is explained in what CFU are.

Two caveats. First, the z-score depends on σ_pt: the larger it is, the smaller z is and the better performance appears, so it is worth checking whether that value makes sense for the use the laboratory makes of its results. Second, the z-score does not take into account the uncertainty of the assigned value. If that standard uncertainty is less than 0.3·σ_pt, it is considered negligible; if not, the provider must use another way of setting the assigned value, scores that incorporate it, such as z′ or zeta (ζ), or inform the participants. The relationship between these scores and the laboratory's own uncertainty is covered in limit of detection and measurement uncertainty.

Qualitative tests: when the result is detected or not detected

Most pathogen analyses are presence or absence tests, and there it makes no sense to calculate means or standard deviations. ISO 13528 provides for setting the assigned value from the known preparation of the item, from reference materials or by expert judgment, and performance is evaluated with criteria of correct results on positive and negative samples. The 2024 proficiency test of the EU reference laboratory for Salmonella, whose general design is based on ISO 22117, shows how it is done:

• A total of 68 national reference laboratories took part, analyzing with EN ISO 6579-1 fabric swabs contaminated with S. Typhimurium at different levels and uncontaminated samples.
• The criterion for good performance required at least 2 of 3 positives at the high level (37.5 CFU per sample), at least 2 of 4 at the low level (4.5 CFU per sample) and 0 of 3 positives in the negatives.
• Sixty-five laboratories met it. One failed at the low level, but demonstrated its capability with extra-low-level samples and its rating changed to moderate performance. Two others were not evaluated because the parcel took 25 days to arrive and survival of the bacterium could not be guaranteed.

That last point teaches something: a bad result can be due to the item and not the laboratory. In the two rounds in India mentioned above, about 16% of the laboratories in each round had unsatisfactory results, almost all due to false positives. The authors pointed to incorrect reconstitution of the sample or insufficient time between reconstitution and analysis, operational errors, cross-contamination and failures in the identification or confirmation of typical colonies as probable causes. The latter connects with what is explained in presumptive vs. confirmed result. In France, the national reference laboratories organized a common scheme for official laboratories between 2019 and 2021, with an average of 65 participants per year.

What to do with a questionable or unsatisfactory result

The Eurachem guide is clear on the principle: every unsatisfactory result must be investigated and documented, and the evaluation is recorded even if it is decided not to take action. A single bad result does not prove that the laboratory is incompetent, but repeated poor performance indicates a serious problem, and the laboratory should then consider ceasing to offer that test until it is resolved. The investigation is managed within the laboratory's nonconforming work procedure. A practical order:

• Review the report. Confirm that the published results are those that were submitted and understand how the score was calculated.
• Rule out clerical errors. Transcription, labeling, decimals or units. If they recur, the problem lies in the management system, not the technique.
• Review the technical side. Storage and reconstitution of the item, internal control data for the run, equipment, reagents and calibration, environmental conditions and data processing. In microbiology, that includes the performance testing of media described in quality control of culture media according to ISO 11133 and, if PCR was used, the controls explained in PCR controls.
• Review the scheme. Matrix differences from routine samples, deterioration of the item, levels outside the method's range, inappropriate assigned value or σ_pt, poorly chosen comparison group or provider errors.
• Close the loop. Analyze the cause with the raw data, the complete round, previous rounds and internal control; plan the corrective action, implement and record it, and verify its effectiveness, ideally with a new participation.

For questionable results, Eurachem suggests defining when to investigate them: for example, with repeated questionable scores or consecutive results with the same sign of bias.

How to get the most out of it

The item should be treated as a routine sample: same method, same workflow and the analysts who process the usual samples. It is advisable to choose schemes whose matrix and level resemble routine work and to report the method used; in the 2024 European proficiency test, twenty laboratories also reported the results of a second method.

How TAAG kits relate to proficiency testing

TAAG does not organize proficiency tests or produce materials for them. But a laboratory that uses a TAAG kit in its routine must cover that method in its participation plan. Two examples from the catalog are Ampliora 1.1 Salmonella spp. (V-SF97, real-time PCR with fluorescent probes, Mila design), with a declared sensitivity of 1 CFU/25 g and validated matrices of raw chicken thigh, frozen beef burger, longanicilla sausage, mortadella and surface, and Specio 2.3 Listeria spp. and L. monocytogenes (V-SF04, real-time PCR with melting curve), with a declared sensitivity of 1 CFU/25 g in food and declared matrices of food, beverages and living and inert surfaces. Both are qualitative: a positive is presumptive until confirmed. The catalog records no third-party certifications for these kits, so a good result in a proficiency test demonstrates the laboratory's performance with that method, but does not replace the validation against the reference method required by the ISO 16140 series to use it instead. If the scheme's matrix is not among those declared, it is advisable to confirm the scope with TAAG before participating.