Auditing The Measurements, Too

24 March 2017

So, you’ve been assigned to audit the TL 9000 measurements! Check out this blog post for guidance and tips on auditing this unique process.

TL 9000 – The two-part standard

Anyone who knows anything about the TL 9000 standard knows that it has two parts: the Requirements Handbook (RHB) and the Measurements Handbook (MHB). TL 9000 is unique among management system standards in its requirement that every certified organization submit prescribed monthly measurements to its Measurements Repository System (MRS).

The organization can then turn around and obtain aggregate Performance Data Reports (PDRs) from the MRS which allow it to compare its performance against its industry peers – in other words, a free benchmarking service, which is a big part of the TL 9000 value proposition.

Why both parts need to be audited

In order to successfully calculate and submit its measurements, per the requirements of the MHB, an organization needs to establish a measurements process. And the measurements process, like any of its other processes, needs to be internally audited regularly in order for there to be confidence that it conforms to requirements, achieves its objectives and can identify and take advantage of improvement opportunities. Your certification body audits it every year, without fail: so should you.

Auditing the measurements process has always been an implicit requirement of the standard. It has now been made explicit in TL 9000:2016 (R6), clause 9.2.2.C.1 Internal Audit Program Planning: “The internal audit program shall include all applicable Requirements and Measurements Handbook requirements.”

So, you've been assigned to audit the measurements

In one sense, it’s like auditing any other process. You need to examine the inputs to the process, and the activities that are performed to transform them into outputs. You need to look at the methods being applied, the competence of the performers, the adequacy of the tools used, and whether or not the desired results are being achieved. But there’s no denying that the measurements process is a unique auditing assignment. How do you get your arms around it?

First off, you need to have at least a basic understanding of the TL 9000 measurements themselves. In principle, you should be able to get that from a study of the current MHB R5.0; but you’ll be much better prepared if you’re able to take QuEST Forum’s measurements E-Learning.

The basic course which covers the common measurements (as well as the hardware ones) will give you a solid foundation. Then, if you can take the additional course or courses which cover any other measurements your organization has to submit, you’ll be that much farther ahead. Get the details from the TL 9000 E-Learning page.

Three measurements steps, three measurements uses

Think “3 and 3” when you’re auditing the measurements process, because there are a couple of key things which you need to verify, each of which has three elements.

1. The organization must:

collect the data for its measurements;
validate the measurement data; and,
submit the measurements to the MRS.

It’s the validate step that’s most often overlooked, or implemented only weakly. It’s not good enough for one and the same person to pull together the data, calculate the measurements, and shoot them in to the MRS. Everyone knows the value of a second set of eyes. The MHB doesn’t tell you how to perform the validation: that’s up to the organization to do, in the way that makes the best business sense. But it’s got to be done some way or other.

2. The organization should use its measurements in three ways:

internally, to inform management and to drive continual improvement;
externally, to share with customers and, potentially, to drive joint improvement initiatives; and,
of course, to submit to the MRS.

The TL 9000 measurements must be shared with the organization’s leadership, and they must be compared with the industry benchmarks provided in the PDRs. Sharing the measurements with your customers is a nice-to-have; but most third party auditors will tell you that it’s a rare organization whose customers are actually interested in seeing those measurements.

Two measurements checklists

Checklists are always helpful to an auditor, especially when you’re auditing a process for the first time. There are a couple that are helpful for the measurements.

The first is not really a checklist at all: it’s simply the MHB’s clause 3.5.2 Organization Responsibilities. This clause’s fifteen bullets capture the core requirements of the measurements process: everything else is measurement-specific counting rules. Cover these fifteen points and you’re halfway there.

The other is provided as a guide by QuEST Forum: a Measurements Handbook Checklist. It’s a classic audit checklist with suggested questions and associated “look for” tips, all cross-referenced to the related MHB R5.0 clauses. It also provides guidance on when to raise a nonconformity or an OFI, what to include in the audit report, and more.

Eleven measurements process fails

If you encounter any of the following during your measurements process audit, then you should be raising a nonconformity.

1. No documented process (or processes) for the collection and validation of measurement data.
2. No real validation of measurement data.
3. Failure to retain a copy of the submitted data.
4. Failure to retain Data Submission Receipts (DSRs).
5. No investigation of measurement advisories.
6. Failure to resubmit data when required due to a previous incorrect submission.
7. Failure to check the last 24 months’ worth of data when correcting submission errors.
8. Failure to follow a counting rule for a submitted measurement.
9. Failure to report on the measurements to the organization’s leadership.
10. Failure to compare the measurements to PDR benchmark data.
11. Failure to provide TL 9000 certified suppliers with any data they need to calculate their measurements.

Auditing the measurements themselves

This is where the rubber hits the road, of course: verifying the correct submission of the actual data. Select a sample of measurements and of months as seems appropriate. (But make sure that you cover all the measurements regularly.) Now, for each given measurement and month, check the following:

Data Submission Receipt (DSR) – Should reflect successful, error-free submission. Note whether any advisories were identified: were they investigated?
Data Submission Confirmation (DSC) (if data was submitted online) or Data Submission Template (if data was uploaded) – Note the actual values that were submitted.
Source data – Compare the submitted values against the source data. Check also any intervening stages (e.g., if data was pulled from the source into a spreadsheet, and then entered from the spreadsheet into the MRS). Source data may have come from any number and types of sources: a spreadsheet (e.g., of customer complaints), an ERP (e.g., Oracle), a CRM (e.g., SalesForce), a home-grown tool of some sort, or information from customers (e.g., about outages).
Verify that the counting rules were applied correctly in the calculation of the submitted data based on the source data.
Was data resubmitted for this measurement and month? Or submitted late? Investigate the reasons. There may be an OFI.

Four measurement process audit tips

1. Make sure you’re using the correct revision of the Product and Service Category Tables (PCT) as a reference. The PCT (Tables A-1 through A-7) document is updated frequently; so, in fact, the tables in the back of MHB R5.0 have become obsolete! At the time of writing, there are two current revisions of the PCT, R5.3 and R5.4, against either of which an organization may submit its measurements. But an earlier month’s data may have been submitted against an earlier PCT revision, which may have had different definitions for, say, a given product category’s normalization units (NUs) or measurements applicability.

2. Check how the organization defines a Problem Report (PR) for NPR, FRT and OFR purposes. Make sure the definition in the MHB’s Glossary is being respected: if the customer does not expect an investigation, then it’s not a Problem Report.

3. Make sure the organization is looking at the correct dates in connection with Problem Reports. For NPR, it’s the date the PR was received that they’re interested in; but, for FRT and OFR, it’s the date the PR was due to be resolved and the date it was actually resolved that matter.

4. How does the organization establish the Customer Requested Date (CRD) and Supplier Promised Date (SPD) for OTD purposes? And make sure the organization doesn’t initiate any changes to the CRD: only the customer is permitted to do that.

Author: Rick Hill, TL 9000 Program Manager NQA USA