GMDN Mapping

Overview

Information on the GMDN mapping project.

• ModuleId: 466707005
• RefsetId: 467614008

Setting Up

The following steps were used to create the projects:

• Load the terminology
  • The data is available in the repo (see GMDN folder):
    • git.ihtsdotools.org:ihtsdo/ihtsdo-mapping-tool-data.git
• Create and configure the project
  • Including users, scope definition, reports and QA checks.
  • INCLUDES
    • 707732003 |Device material (physical object)|
  • EXCLUDES
    • n/a
• Load the previously published map records.
  • NOTE: the scope for "device" was computed for 20150131 and each SNOMED device not mapped to GMDN was added as a blank entry, allowing this scope to be reliably used going forward.
  • After this, the effectiveTime = lastModified
• Compute workflow

Monthly GMDN Update

The data files for GMDN get updated monthly and should be loaded in a timely fashion.  To get the data files onto the prod mapping server the following process is used:

1. INSERT NEW PROCESS HERE
2. Navigate to the mapping tool (e.g. https://mapping.ihtsdotools.org).
   1. Log in as "mappingadmin"
   2. Change to the GMDN project
   3. Find the GMDN project in the "map projects" accordion
   4. Update the destination terminology version the latest version (e.g. 17_1)
   5. Save the map project.
       
3. Concepts from the updated GMDN version should now be available to mappers.

Helpful Admin Commands

Remove

cd ~/code/admin/remover
mvn install -PMapRecords -D$rc -Drefset.id=467614008
mvn install -PTerminology -D$rc -Dterminology=GMDN -Dversion=latest

Load

Get maintenance window

cd ~/code/admin/loader
mvn install -PGMDN -Dinput.dir=/home/ihtsdo/data/GMDN/16_10 -Dterminology=GMDN -Dversion=17_1 -D$rc
cd ~/code/admin/loader
set file = /home/ihtsdo/data/doc/release/20160731/gmdn/der2_sRefset_SimpleMapSnapshot_INT_20160731.txt
/bin/rm -f /tmp/x.txt
perl -ne '@_=split/\t/; print if $_[4] eq "467614008"' $file > /tmp/x.txt
mvn install -PSimpleMapRecords -D$rc -Dinput.file=/tmp/x.txt -Dmember.flag=true -Drecord.flag=false

Compute

cd ~/code/admin/loader
mvn install -PComputeWorkflow -D$rc -Drefset.id=467614008

Release

cd ~/code/admin/release
mvn install -PBeginRelease -D$rc -Drefset.id=467614008 -Dtest.mode.flag=true
mvn install -PRelease -D$rc -Drefset.id=467614008 -Doutput.dir=. -Dtime=20170131 -Dmodule.id=466707005 -Dtest.mode.flag=true
mvn install -PFinishRelease -D$rc -Drefset.id=467614008
mvn install -PBeginEditingCycle -D$rc -Drefset.id=467614008

Loading "Human Readable" View

Similar approach to ICD10 but using the "simple" map loader.  e.g.

cd ~/code/admin/loader
set file = ~/data/GMDN/*2016*txt
grep -v 900000000000207008 $file | perl -pe 's/467614008/P467614008/;' >! x.txt
mvn install -PSimpleMapRecords -Drun.config=/home/ihtsdo/config/config.properties -Dinput.file=x.txt -Dmember.flag=false -Drecord.flag=true >&! mvn.log

Derivative Release Process

1. Determine the source terminology version (SNOMEDCT), the destination terminology version (GMDN) and the effective time for the release.  The effective time and the source terminology version will be the same release date as the immediately previous SNOMED International release (e.g. 20XX0731 or 20XX0131).  The destination version should include the last GMDN content that got included in the International Edition before the content cut off date (for July release, this would be early May, e.g. 17_5).  Examples below are based on SNOMEDCT 20170731 to GMDN 17_5 release. 
2. Stop editing on Prod
3. Check Prod database for no tracking records on the project.
4. Clone the final prod backup to the release server.
5. Begin next editing cycle on the Prod server.
6. Compute Workflow on Prod.

7. On release server, stop the server

   supervisorctl stop mapping-rest

8. On release server, remove simple map refset members with GMDN refsetId. 

   //in DB, first remove simplemap refset members
   DELETE from simple_map_refset_members where refsetId = 467614008;

9. On release server, load previous release active simple map refset members (der2_sRefset_SimpleMapActiveSnapshot_INT_20170131.txt) with GMDN refsetId from previous (e.g.20170131) release.  Andrew will provide this file.  If you use the SimpleMapSnapshot file instead, you first need to remove the inactive rows.  awk '$3 == "1" { print $_ }' initialFile.txt > outputFile.txt

   sudo su - mapping-rest
   cd /opt/mapping-admin/loader
   perl -ne '@_=split/\t/; print if $_[4] eq "467614008"' /home/ihtsdo/data/doc/release/20170131/gmdn/der2_sRefset_SimpleMapActiveSnapshot_INT_20170131.txt > /tmp/x.txt
   mvn install -PSimpleMapRecords -Drun.config=/opt/mapping-rest/config.properties -Dinput.file=/tmp/x.txt -Dmember.flag=true -Drecord.flag=false
   exit

10. Remove SNOMEDCT 20170131.

11. Add SNOMEDCT 20170731  (this took 2 1/2 hours on the release server)
    
    

    cd /opt/mapping-service-admin/admin/loader
    mvn install -PRF2-snapshot -Drun.config=/opt/mapping-service/conf/config.properties -Dterminology=SNOMEDCT -Dversion=20170731 -Dinput.dir=/home/ihtsdo/data/SNOMEDCT/SnomedCT_InternationalRF2_PRODUCTION_20170731T120000Z/Snapshot > /opt/mapping-service/logs/loadSnomed20170731.log

12. Add GMDN 17_05.  (OK if later versions remain in the db as long as the project references the correct release version)

    cd /opt/mapping-service-admin/admin/loader
    mvn install -PGMDN -Dinput.dir=/home/ihtsdo/data/GMDN/17_5 \
      -Dterminology=GMDN -Dversion=17_5 -Drun.config=/opt/mapping-service/conf/config.properties > /opt/mapping-service/logs/loadGmdn17_05.log

13. Reindex (is this necessary?)

    cd /opt/mapping-service-admin/admin/lucene 
    mvn install -PReindex -Drun.config=/opt/mapping-service/conf/config.properties > /opt/mapping-service/logs/reindex.log

14. Start the server 

    supervisorctl start mapping-rest

15. Update the project to point to the SNOMEDCT and GMDN release versions.
16. Compute Workflow.  Watch log at  tail -f /var/log/tomcat.log.
17. Begin release.  (Use effectiveTime 20170731 given example dates above)  ModuleId is 466707005.
18. Check QA report
19. Have mappers fix any errors.
20. Compute Workflow
21. Check database for no tracking records on the project.
22. Begin release
23. Process release
24. Copy ActiveSnapshot file back to Prod
25. On Prod, run Preview Finish
26. Check Release Finalization report
27. Run Finish Release.
28. Question: we loaded sources onto the release server.  What needs to be loaded into prod?

NOTE: No need to create 'Published SNOMED to GMDN'.  GMDN project is private.

Troubleshooting

ERROR - UUID-quintuples count does not match refset member count for SimpleMap

If during release it throws the following error: "ERROR - UUID-quintuples count does not match refset member count for SimpleMap", check database for duplicate entry for map.

mysql> select *, count(*) as ct from simple_map_refset_members where refsetId='467614008' group by refsetId, concept_id, mapTarget having ct>1;
+----------+--------+---------------------+-------+-----------+-------------+--------------------------------------+--------------------+-----------+-----------+------------+----+
| id | active | effectiveTime | label | moduleId | terminology | terminologyId | terminologyVersion | refSetId | mapTarget | concept_id | ct |
+----------+--------+---------------------+-------+-----------+-------------+--------------------------------------+--------------------+-----------+-----------+------------+----+
| 10882752 | 0 | 2018-01-31 00:00:00 | NULL | 466707005 | SNOMEDCT | a4c5d7d5-3990-52d1-96f4-6a5887dd295c | latest | 467614008 | 32608 | 14431694 | 2 |
+----------+--------+---------------------+-------+-----------+-------------+--------------------------------------+--------------------+-----------+-----------+------------+----+

If a duplicate is identified, find the duplicate entries:

mysql> select * from simple_map_refset_members where concept_id=14431694 and refsetId='467614008';
+----------+--------+---------------------+-------+--------------------+-------------+--------------------------------------+--------------------+--------------------+-----------+------------+
| id | active | effectiveTime | label | moduleId | terminology | terminologyId | terminologyVersion | refSetId | mapTarget | concept_id |
+----------+--------+---------------------+-------+--------------------+-------------+--------------------------------------+--------------------+--------------------+-----------+------------+
| 10882752 | 0 | 2018-01-31 00:00:00 | NULL | 466707005 | SNOMEDCT | a4c5d7d5-3990-52d1-96f4-6a5887dd295c | latest | 467614008 | 32608 | 14431694 |
| 10885727 | 1 | 2018-01-31 00:00:00 | NULL | 466707005 | SNOMEDCT | ef5c6b29-c874-5b5c-891e-5dd510866e06 | latest | 467614008 | 61449 | 14431694 |
| 10890543 | 0 | 2017-01-31 00:00:00 | NULL | 466707005 | SNOMEDCT | 67c1c3cd-38c7-5ee1-83a8-588020f7c7d5 | latest | 467614008 | 32608 | 14431694 |
+----------+--------+---------------------+-------+--------------------+-------------+--------------------------------------+--------------------+--------------------+-----------+------------+

Here we have two inactive rows where mapTarget=32608.  Remove the older row, and it should resolve the issue.

mysql> delete from simple_map_refset_members where id=10890543;

References/Links

• n/a