Ensuring the quality of a reference set is important for the successful use of SNOMED CT. It is crucial that the content of the reference set references the concepts which represent the meaning of the data it is to be linked to. In addition, it is important that the human interpretation of the concepts referred to aligns with the logical definition provided by SNOMED CT. Even when using reference sets for less patient safety critical purposes, the quality of the reference set needs to be sufficient to be trusted to serve its purpose.
Reviewing a reference set is important throughout the reference set development process, however at least three types of validation should be emphasized, see illustration below.
Gliffy Diagram name reviewStages
Quality assurance stages
Reviewing a reference set before testing it in the context of its usage is important in order to correct the most distinctive errors, such as content gaps or disagreements about selection of concepts. Additionally, reviewing a reference set is important, because it is much easier to make major adjustments as early as possible. In addition, the adaptability of the reference set increases the higher the quality of the reference set, according to the perspective of the subset users. The level of quality assurance needed will of course be dependent on the practical use case of the reference set.
The checks that should be done as part of the review process include:
The people involved with reviewing the reference set should be by a combination of domain experts and SNOMED CT experts.
SNOMED International does not recommend ONE specific approach for review of reference sets, but blinded approaches are typically a feasible approach to achieve high quality reference set. Alternatively, or additionally, a combination of review methods can be recommendable to ensure a reliable and usable review process. Ideally, reference sets should be reviewed iteratively until the reviewers are satisfied. Examples of the approaches to take are illustrated in the following table.
|Single author single reviewer||In this approach one person author the reference set, i.e. determines the content to be included, while another person reviews the selected reference set members. The reviewer may also add comments against the proposal and suggest alternative content. In a really small team, the author and reviewer may be the same person. However this is not ideal, and it is highly recommended to include two or more people in the review process.|
Using this approach the reference set development work is divided evenly between two or more authors of the reference set.
This approach can be very efficient if time is short, because you can spread both the development and the review load between more than one person – however, reviewing someone else’s material may not always as effective (in terms of quality) as the dual blinded approach.
|Dual blind review with an adjudicator|
This approach is useful in a slightly bigger team than required for the single mapper single reviewer approach.
This approach includes following steps:
This approach can produce higher quality reference sets, because each author is independently cross-checking each other's material without being biased by the decisions of the other author. While this approach takes longer during the development phase (because both authors need to develop a draft of the whole reference set ), the review phase can be a lot quicker (because the adjudicator only needs to check the parts that the authors disagreed on).
Validation workshops is workshops dedicated to review and validate the design and/or content of a reference set. In these workshops the content or uncertainties are discussed in details, or test-persons are asked to prioritize and assess specific subset members etc. The participants may have had a chance to review the reference set individually prior to the workshop to prepare questions and comments for discussion. The number of people in the workshop and their roles should be considered and selected dependent of the format and the scope of a specific workshop.
This approach is time consuming which should be acknowledged already in the planning stage. However, this approach may also be rewarding. Workshops often give rise to detailed discussions or unplanned discussions of relevant issues, but at the sametime workshops provide an opportunity of increased ownership and participation among the participants, which may have a positive effect on the adoption of the reference set. It is recommended to plan these workshops in detail and to include a set of workshops. The number of workshops necessary depends on the size of the reference set, and how the feedback sharing is conducted.
In addition to the abstract assurance of the subset in isolation, there may be a requirement for a level of testing to be undertaken in healthcare systems, and tested under the exact circumstances of intended use. This may be undertaken by releasing a technology preview targeted at specific bodies for feedback. An impact assessment and, most importantly, safety testing will need to take place upon deployment of the subset into systems, particularly where significant changes have taken place.