Overview

Documentation on the ICD-O mapping project.

Information on the ICDO mapping project.

Helpful Admin Commands

Remove

cd ~/code/admin/remover
mvn install -PMapRecords -D$rc -Drefset.id=446608001
mvn install -PTerminology -D$rc -Dterminology=ICDO -Dversion=16_1

Load

Get maintenance window
cd ~/code/admin/loader
mvn install -PICDO -Dinput.dir=/home/ihtsdo/data/ICDO -Dterminology=ICDO -Dversion=16_1 -D$rc
cd ~/code/admin/loader
set file = /home/ihtsdo/data/ICDO/der2_sRefset_IcdoMorphSimpleMapSnapshot_INT_20170131.txt
/bin/rm -f /tmp/x.txt
# Keep only morphology codes (with '/')
perl -ne '@_=split/\t/; print if $_[4] eq "446608001"' $file | grep '/' > /tmp/x.txt
mvn install -PSimpleMapRecords -D$rc -Dinput.file=/tmp/x.txt -Dmember.flag=true -Drecord.flag=false

Compute

cd ~/code/admin/loader
mvn install -PComputeWorkflow -D$rc -Drefset.id=446608001

Release

cd ~/code/admin/release
mvn install -PBeginRelease -D$rc -Drefset.id=446608001 -Dtest.mode.flag=true
mvn install -PRelease -D$rc -Drefset.id=446608001 -Doutput.dir=. -Dtime=20170731 -Dmodule.id=900000000000207008 -Dtest.mode.flag=true
mvn install -PFinishRelease -D$rc -Drefset.id=446608001
mvn install -PBeginEditingCycle -D$rc -Drefset.id=446608001

Release with Alpha/Beta Iteration

1.) Get maintenance window. Take server down 
2.) cd ~/code/admin/remover
3.) mvn install -PTerminology -Drun.config=/home/ihtsdo/config/config.properties -Dterminology=SNOMEDCT -Dversion=latest
4.) cd ~/code/admin/loader
5.) mvn install -PRF2-snapshot -Drun.config=/home/ihtsdo/config/config.properties -Dterminology=SNOMEDCT -Dinput.dir=/home/ihstdo/data/xSnomedCT_InternationalRF2_ALPHA_20170731T120000Z/Snapshot
6.) bring server back up 
7.) DELETE from simple_map_refset_members where refsetId = 446608001; 
8.) set file = /home/ihtsdo/data/ICDO/der2_sRefset_SimpleMapSnapshot_INT_20170131.txt
# Keep only morphology codes (with '/')
perl -ne '@_=split/\t/; print if $_[4] eq "446608001"' $file | grep '/' > /tmp/x.txt
mvn install -PSimpleMapRecords -D$rc -Dinput.file=/tmp/x.txt -Dmember.flag=true -Drecord.flag=false
9.) rerun compute workflow, begin, and process release steps

Loading "Human Readable" View

TBD

Updating Published Project

Similar approach to ICD10 but using the "simple" map loader.  e.g.

cd ~/code/admin/remover
mvn install -PMapRecords -D$rc -Drefset.id=P446608001
cd ~/code/admin/loader
set file = ~/data/doc/release/20170131/icdo/*ActiveSnapshot*txt
grep 446608001 $file | perl -pe 's/446608001/P446608001/;' > x.txt
mvn install -PSimpleMapRecords -Drun.config=/home/ihtsdo/config/config.properties -Dinput.file=x.txt -Dmember.flag=false -Drecord.flag=true >&! mvn.log

Meeting Notes

Dev Requested

Details

Computing Scope

Use the attached transitive_closure.pl script.  The file icdoScope.txt is attached (1558 entries).  A file of those unmapped codes is also attached.

cd Snapshot/Refset/Map
grep 446608001 *Simple* | grep '/' | perl -ne '@_=split/\t/; print if $_[2];' | cut -f 6 | sort -u -o icdo.txt
./transitive_closure.pl ../../Terminology/*_Relationships*txt out.txt
sort -t\ 	 -k 1,1 -o out.txt out.txt
join -j 1 -o 2.2. icdo.txt out.txt > scope.txt
sort -u -o icdoScope.txt scope.txt icdo.txt
/bin/rm f icdo.txt out.txt scope.txt

Preparing the Data

In Lieu of obtaining offical ICD-O-3.1 files from WHO, we proceeded with ICDO morphology data from NCI Metathesaurus (available from NCI in the US).  As we understand it, this is the correct ICDO version to map to and contains the real data.

Steps for processing the data:

 

At such time as official files are obtained from WHO and are in a different format, work will be done (as part of warranty/maintenance) to either convert that format into suitable ClaML or a new loader will be created for the native data format.

Production Deployment

  1. Load the ICDO-3-1 data
  2. Create and configure the project (with scope definition based on icdoScope.txt "plus descendants".
    1. Use "REVIEW" workflow
    2. Configure Nicki/Donna as users
    3. Add basic productivity reports.
  3. Load the prior version ICDO map (as starting point)  - both records and members.
  4. Begin editing cycle
  5. Compute workfow
  6. Create a "published" form of the project as well

Here's the process in code:

service tomcat7 stop
# 1. load ICDO-3-1 data
cd ~/code/admin/loader
set file = ~/data/ICDO/icdo-3-1.xml
mvn install -PClaML -D$rc -Dterminology=ICDO -Dversion=3_1 -Dinput.file=$file >&! mvn.log
 
# 2. do in mapping tool
service tomcat7 start
...
service tomcat7 stop



# 3. Load the prior version ICDO map
cd ~/code/admin/loader
set file = ~/data/ICDO/der2_sRefset_IcdoMorphSimpleMapSnapshot_INT_20160131.txt
mvn install -PSimpleMapRecords -D$rc -Dinput.file=$file -Dmember.flag=true -Drecord.flag=true >&! mvn.log
 
# 4. Begin editing cycle
cd ~/code/admin/release
mvn install -PBeginEditingCycle -D$rc -Drefset.id=446608001 >&! mvn.log


# 5. Compute workflow
cd ~/code/admin/loader
mvn install -PComputeWorkflow -D$rc -Drefset.id=446608001 >&! mvn.log
 
# load published project data
cd ~/code/admin/loader
set file = ~/data/ICDO/der2_sRefset_IcdoMorphSimpleMapSnapshot_INT_20160131.txt
perl -pe 's/446608001/P446608001/;' $file >! x.txt
mvn install -PSimpleMapRecords -Drun.config=/home/ihtsdo/config/config.properties -Dinput.file=x.txt -Dmember.flag=false -Drecord.flag=true >&! mvn.log

service tomcat7 start

References/Links