Please Note: This service will be unavailable on 20 July from 06:00 UTC till 18:00 UTC for essential maintenance.

Overview

Information on the ICD11 mapping project.

  • ModuleId: 123456789
  • RefsetId: icd11RefsetId

Documentation

Deployment Server Details

Software installed

  • csh, git, maven

Uploading files

  • Log in sftp as you
  • Upload files to home dir
  • Log in ssh as you
  • move files to /tmp
  • sudo su
  • su -s /bin/bash mapping-service
  • cd /opt/mapping-data/ICD11
  • cp /tmp/*txt .

Making a Release

  • Log into prod-mapping-icd11.ihtsdotools.org
  • Become the root user:

    # this will require having sudo access, then enter your password
    sudo su
  • Become the "mapping-service" user

    su -s /bin/bash mapping-service
  • Go to the admin directory for running a release

    cd /opt/mapping-data/code/admin/release
  • Make a release

    time=20171201
    mvn install -PRelease -D$rc -Drefset.id=icd11RefsetId -Doutput.dir=. -Dtime=$time -Dmodule.id=900000000000207008 -Dtest.mode.flag=true
    # the output will be a file like the following
    der2_iisssccRefset_ExtendedMapActiveSnapshot_INT_20171107.txt

Important details

  • Log in as your user (requires key to be configured)
  • User should have "sudo" rights
  • The server runs as the "mapping-service" user. 
    • For a shell as this user

      sudo su
      su -s /bin/bash mapping-service
  • App details
    • star/stop server "supervisorctl start/stop mapping-service"
      • look at /etc/supervisor/conf.d/mapping-service.conf for details
      • /opt/mapping-service/lib/mapping-rest.war
    • Admin tools in: /opt/mapping-service-admin/admin (run as mapping-service user)
      • /opt/mapping-service-admin/admin/lucene/mvn install -PReindex -Drun.config=/opt/mapping-service/conf/config.properties
    • Data: in /opt/mapping-data
      • conf - configuration files
      • indexes - lucene indexes
    • Logs - /var/log/supervisor
      • STDOUT log:  /var/log/tomcat.log
      • mapping-service-stderr---supervisor-79oJH0.log
      • mapping-service-stdout---supervisor-6Jv0gZ.log
    • Instead of tomcat we use webapp-runner.jar which is a tomcat wrapped up. Allows us to package a known tomcat/webapp package.
    • It is installed to /opt/mapping-service with data in /opt/mapping-data and admin in /opt/mapping-admin
    • a Jenkins job exists allowing you to install wrt updating code or what ever  - see below
  • Jenkins Deploy - https://jenkins.ihtsdotools.org/view/Mapping/job/deploy-to-prod-mapping-icd11/
    • This just fires off an ansible process
    • Check /var/log/tomcat.log to see what it's doing (while you run jenkins job)

Details to configure:

Data files for this project are in /home/ihtsdo/data/ICD11

  • icd11Concepts.txt - faked from the "linearization" file to be oriented around "entityId" while showing the codes.
  • icd11ParChd.txt - faked from the "linearization" file to expres relationships between codes that have obvious par/chd relationships (as devined by the textual codes but not the entity codes)
  • icd11Map.txt - the computed map (from the algorithm)
  • icd11MapNotes.txt - the computed map notes (from the algorithm)

These steps were taken to set up the project

  • Manually create and configure the project as mapping admin
    • SNOMED to ICD11 Pilot - top level metadata for project
    • Map Leads selected
    • No map specialists
    • Map relations configured
    • Map age ranges configured
    • Scope concepts configured (starter set minus procedures)
  • Remake "SNOMED to ICD11" as a copy.
    • Scope definition configured (ICD10 scope, minus starter set, minus procedures)


Helpful Admin Commands

A cycle of redeploying maps involves running the algorithm to generate the "icd11Map.txt" and "icd11MapNotes.txt" files.  Assuming everything described above is already configured, the task here is simply to replace the data.  This is done with the "reload.csh" script in /home/ihtsdo/data/ICD11/reload.csh. 

#!/bin/csh -f

set rc = run.config=/opt/mapping-service/conf/config.properties
if ($#argv != 1) then
  echo "usage: $0 <refsetId>"
  echo "  icd11RefsetId"
  echo "  icd11RefsetId-b"
  exit 1
endif
#set refsetId = icd11RefsetId-b
set refsetId = $1
if ($refsetId != "icd11RefsetId" && $refsetId != "icd11RefsetId-b") then
    echo "ERROR: unexpected value for refsetId: $refsetId"
    exit 1
endif

set adminDir = /opt/mapping-service-admin/admin

echo "---------------------------------------------------"
echo "Starting ...`/bin/date`"
echo "---------------------------------------------------"
echo "refsetId: $refsetId"
echo ""

curl "https://prod-mapping-icd11.ihtsdotools.org/" | grep 'Bad Gateway' > /tmp/x.$$
if ($status == 0 && `cat /tmp/x.$$ | wc -l` == 0) then
    echo "STOP the server before running this (as root)"
    echo "    supervisorctl stop mapping-service"
    exit 1
endif

# Stop server
#sudo supervisorctl stop mapping-service

echo "  Extract icd11Map.txt into LOW, MED, HIGH"
grep 'FINAL CATEGORY LOW' icd11MapNotes.txt | cut -f 6 | sort -u -o low.txt
grep 'FINAL CATEGORY MEDIUM' icd11MapNotes.txt | cut -f 6 | sort -u -o medium.txt
grep 'FINAL CATEGORY HIGH' icd11MapNotes.txt | cut -f 6 | sort -u -o high.txt
grep 'FINAL CATEGORY NO MAP' icd11MapNotes.txt | cut -f 6 | sort -u >> high.txt

/bin/rm -f icd11Map.{low,medium,high}.txt
perl -ne 'BEGIN {open(I,"low.txt"); while(<I>){chop; $m{$_}=1;} close(I);} @_=split/\t/; print if $m{$_[5]}; ' icd11Map.txt > icd11Map.low.txt
perl -ne 'BEGIN {open(I,"medium.txt"); while(<I>){chop; $m{$_}=1;} close(I);} @_=split/\t/; print if $m{$_[5]}; ' icd11Map.txt > icd11Map.medium.txt
perl -ne 'BEGIN {open(I,"high.txt"); while(<I>){chop; $m{$_}=1;} close(I);} @_=split/\t/; print if $m{$_[5]}; ' icd11Map.txt > icd11Map.high.txt

/bin/rm -f icd11MapNotes.{low,medium,high}.txt
perl -ne 'BEGIN {open(I,"low.txt"); while(<I>){chop; $m{$_}=1;} close(I);} @_=split/\t/; print if $m{$_[5]}; ' icd11MapNotes.txt > icd11MapNotes.low.txt
perl -ne 'BEGIN {open(I,"medium.txt"); while(<I>){chop; $m{$_}=1;} close(I);} @_=split/\t/; print if $m{$_[5]}; ' icd11MapNotes.txt > icd11MapNotes.medium.txt
perl -ne 'BEGIN {open(I,"high.txt"); while(<I>){chop; $m{$_}=1;} close(I);} @_=split/\t/; print if $m{$_[5]}; ' icd11MapNotes.txt > icd11MapNotes.high.txt

echo "  Replace notes for maps ...`/bin/date`"
perl -ne 'BEGIN {open(I,"icd11MapNotes.txt"); while(<I>){@_=split/\t/; $m{$_[5]}=1;} close(I);} @_=split/\t/; print unless $m{$_[5]}; ' notes/icd11MapNotes.txt > fix
/bin/mv -f fix notes/icd11MapNotes.txt
cat icd11MapNotes.txt >> notes/icd11MapNotes.txt

# map remover augmented to remove map records for concept ids in a specified input file
echo "  Removing old maps ...`/bin/date`"
cd $adminDir/remover
mvn install -PMapRecords -D$rc -Drefset.id=$refsetId -Dinput.file=/opt/mapping-data/ICD11/icd11Map.txt >&! /tmp/mvn.log
if ($status != 0) then
    cat /tmp/mvn.log | sed 's/^/     /'
    echo "ERROR removing map records"
    exit 1
endif

echo "  Adding new maps and notes ...`/bin/date`"

echo "    add MEDIUM records as EDITING_DONE ...`/bin/date`"
cd $adminDir/loader
mvn install -PMapRecords -D$rc -Dinput.file=/opt/mapping-data/ICD11/icd11Map.medium.txt \
  -Dmember.flag=false -Drecord.flag=true -Dworkflow.status=EDITING_DONE >&! /tmp/mvn.log
if ($status != 0) then
    cat /tmp/mvn.log | sed 's/^/     /'
    echo "ERROR installing maps"
    exit 1
endif
echo "    add MEDIUM notes ...`/bin/date`"
cd $adminDir/loader
mvn install -PMapNotes -D$rc -Dinput.file=/opt/mapping-data/ICD11/icd11MapNotes.medium.txt >&! /tmp/mvn.log
if ($status != 0) then
    cat /tmp/mvn.log | sed 's/^/     /'
    echo "ERROR installing map notes"
    exit 1
endif

echo "    add HIGH as PUBLISHED ...`/bin/date`"
cd $adminDir/loader
mvn install -PMapRecords -D$rc -Dinput.file=/opt/mapping-data/ICD11/icd11Map.high.txt \
  -Dmember.flag=false -Drecord.flag=true >&! /tmp/mvn.log
if ($status != 0) then
    cat /tmp/mvn.log | sed 's/^/     /'
    echo "ERROR installing maps"
    exit 1
endif
echo "    add HIGH notes ...`/bin/date`"
cd $adminDir/loader
mvn install -PMapNotes -D$rc -Dinput.file=/opt/mapping-data/ICD11/icd11MapNotes.high.txt >&! /tmp/mvn.log
if ($status != 0) then
    cat /tmp/mvn.log | sed 's/^/     /'
    echo "ERROR installing map notes"
    exit 1
endif

# before doing workflow stuff with HIGH, compute workflow
echo "  Compute workflow ...`/bin/date`"
cd $adminDir/loader
mvn install -PComputeWorkflow -D$rc -Drefset.id=$refsetId >&! /tmp/mvn.log
if ($status != 0) then
    cat /tmp/mvn.log | sed 's/^/     /'
    echo "ERROR running compute workflow"
    exit 1
endif

echo "    convert HIGH to QA ...`/bin/date`"
cd $adminDir/loader
mvn install -PAdHoc -D$rc -Drefset.id=$refsetId -Dmode=icd11 -Dinput.file=/opt/mapping-data/ICD11/icd11Map.high.txt >&! /tmp/mvn.log
if ($status != 0) then
    cat /tmp/mvn.log | sed 's/^/     /'
    echo "ERROR running adhoc step"
    exit 1
endif

# compute workflow again when finished assigning to QA
echo "  Compute workflow ...`/bin/date`"
cd $adminDir/loader
mvn install -PComputeWorkflow -D$rc -Drefset.id=$refsetId >&! /tmp/mvn.log
if ($status != 0) then
    cat /tmp/mvn.log | sed 's/^/     /'
    echo "ERROR running compute workflow"
    exit 1
endif


# Start server -
#sudo supervisorctl start mapping-service

echo "---------------------------------------------------"
echo "Finished ...`/bin/date`"
echo "---------------------------------------------------"
echo ""
echo "REMEMBER to restart the server = supervisorctl start mapping-service"







 

  • No labels