O'PEEP'S ATTRIBUTE AGREEMENT ANALYSIS EXPLANATION
Does the bathroom need cleaning, yes or no? To you it looks sparkling clean. (So cleaning? No!) But your spouse says “The bathroom is disgusting, Honey!” (Cleaning? Yes!).
So, how to find out whether your bathroom needs cleaning? You want to measure something that you cannot measure with a gage. You need to get to results like “yes”/”no” OR “super clean”/ “more or less clean”/ ”disgusting”. In such a case, the attribute agreement analysis will tell you how good is your measurement system.
Attribute Agreement Analysis is a Gage R&R Study for discrete data.
Work with Attribute Agreement Analysis if you aim at doing a Gage R&R (Measurement System Analysis) for a test/measurement that does NOT lead to results you could gage with a measurement instrument. You need it if the results are qualitative. Examples:
The criteria for the assessment is definable, e.g. by working with an expert or working under very good conditions. For example if high concentration and a very good visual capability is required, the inspection is done by a very reliable appraiser after the break.
To work with Attribute Agreement Analysis you must be able to set a standard.
What do you need to find out? Accuracy or Precision? Reproducibility or Repeatability?
Understand how well your measurement system comes to the same results as your “standard”.
Expresses how precisely your different appraisers agree regarding their results and how precisely one and the same appraiser comes to the same results when measuring multiple times.
Total %R&R describes the level of agreement within/ between appraisers and a standard (if available)
Total %R&R must be 100% (this sounds very high but since there are only 30 parts here and the study conditions are different from reality every result below 100% will lead to a measurement system not worth trusting)
If Total %R&R is less than 100% investigate the results to see if this is a problem of repeatability, reproducibility or both
O’Peep wants to evaluate how well his quality control department assesses the taste of his famous O’Gin. He wants to know whether the appraisers in his Quality Control have the same exquisite sense of taste as he does. That gives him the Accuracy.
He would also like to find out about the Precision in his Quality Control.
So he takes 3 of his appraisers and makes them test 30 samples three times. To make it easier for us he has elaborated this sophisticated system to assess the Gin: “Very good”, “Okay” or “Very bad”.
He records the assessment given by the three appraisers in this table to easily compare them with the standard:
From this, he observes the following results and calculates accuracy and precision:
Using a statistical tool to run a Gage R&R test he finds that the Reproducibility and Repeatability parameters are not met at 100%. This shows a clear lack of precision with the appraisers.
O’Appraiser 2 for instance, only assessed 17 out of 30 samples correctly across the three trials. And they all agreed only on 12 samples 100% of the time.
Want to learn more about Precision and Gage R&R? Click here
NUMBER OF CORRECT APPRAISALS VS. STANDARD
Is O’Peep happy about an accuracy of 90% when it comes to judging on the taste of his Gin? Good enough always depends on what you need to achieve.
Even though an accuracy of more than 90% seems to be quite good, the total lack of precision shows that this 90% is not trustworthy and that further measures are needed to improve the performance of the appraisers. This can be done through training and clearer guidance on how to assess the Gin.