Release Process

Overview

Documents the release process details.  There is an admin page already that indicates how to run the mojo.  See Release Processing Tool for more information.

Assumptions

• Source terminology is SNOMED CT
• Release is in RF2 delta format
• QA of overall release is done by an external system (though project-specific validation rules are processed here)
• That editing was done in conjunction with a "drip feed" system that brought emerging content into the system as it was being developed, so it could be mapped.
• That the mapping files are part of the source terminology release

Details

The mapping tool supports a release process that takes into account end-of-editing cycle activities, some quality assurance, production of data files, and upgrading terminology versions when the time comes.  The release processing is specifically tailored to the needs of IHTSDO and maintenance of the RF2 complex and extended map refsets that are part of the SNOMED CT distribution.

 TODO: flowchart

The steps are:

• Upgrade to latest version - there are two forms of this and it assumes that the version of the map and the "source terminology" are coordinated.  This is to ensure the expected release state of "in scope" concepts is properly represented for workflow and release state computations.  Given processing of "daily build", innaccuracies can build up over an editing cycle.
  • Load a definitive delta release and update the version of all terminology components to latest version.

  • Remove and reload an entire definitive snapshot release. (NOTE: it may be desirable to do this FIRST, before "begin release"). 

    # first prepare an edition that contains map files from the previous release 
    # along with the next release beta files with the right structure
    service tomcat7 stop
    cd ~/code/admin/remover
    mvn install -PTerminology -Drun.config=/home/ihtsdo/config/config.properties -Dterminology=SNOMEDCT -Dversion=latest >&! mvn.log
    cd ~/code/admin/loader
    mvn install -PRF2-snapshot -Drun.config=/home/ihtsdo/config/config.properties -Dterminology=SNOMEDCT -Dversion=latest -Dinput.dir=/home/ihtsdo/data/snomedct-20160131 >&! mvn.log
    
    service tomcat7 start

  • Double check that complex and extended map files have entry counts matching the files used to load.  One ICD9CM record for (5755759c-4d9b-57d0-b4a9-f20b19326194) gets lost because the concept id it is connected to does not exist in the concepts file. This entry has to get manually added back in to the final "snapshot" if it is really desired.  If delivering a delta this is not a problem.

  • Recompute workflow to ensure all in scope concepts are mapped

    service tomcat7 stop
    cd ~/code/admin/loader
    mvn install -PComputeWorkflow -Drun.config=/home/ihtsdo/config/config.properties -Drefset.id=447562003,447563008 >&! mvn.log 
    service tomcat7 start

• Finish all editing - assumes drip feed has been processing along the way
• Disable automations (e.g. cron jobs).
• "Begin" a release
  • Verify all "in scope" concepts are mapped
  • Validate all existing map records against current rules

  • Remove map records for concepts that are retired, deleted, or out of scope. 

    service tomcat7 stop
    cd ~/code/admin/release
    mvn install -PBeginRelease -Drun.config=/home/ihtsdo/config/config.properties -Drefset.id=447562003 >&! mvn.log
    mvn install -PBeginRelease -Drun.config=/home/ihtsdo/config/config.properties -Drefset.id=447563008 >&! mvn2.log
    service tomcat7 start

  • NOTE: iterate through the process above with -Dtest.mode.flag=true until there are no errors.  This is needed for the "Release QA" report to be properly generated (because the non test mode issues a rollback() on any errors).

    • Cases marked as "Not marked ready for publication" will need to be manually removed with "Batch delete" as the map admin user.
      
      

• Generate release files
  • Create an RF2 delta file
  • Create human readable file

  • Create module dependency file (if needed)

    service tomcat7 stop
    set time=20160131
    cd ~/code/admin/release
    mvn install -PRelease -Drun.config=/home/ihtsdo/config/config.properties -Drefset.id=447562003 \
      -Doutput.dir=. -Dtime=$time -Dmodule.id=449080006
    mvn install -PRelease -Drun.config=/home/ihtsdo/config/config.properties -Drefset.id=447563008 \
      -Doutput.dir=. -Dtime=$time -Dmodule.id=900000000000207008
    service tomcat7 start

• It may be desirable to load release files into static, read-only projects that are public and can serve as a "human readable" form of the published map. For example,

  service tomcat7 stop
  # remove prior map records
  cd ~/code/admin/remover
  mvn install -PMapRecords -Drun.config=/home/ihtsdo/config/config.properties -Drefset.id=P447563008 >&! mvn.log
  # load latest active snapshot
  # Add a "P" to the front of the refset id, as refset ids have to be unique across projects.
  cd ~/code/admin/loader
  set file=/home/ihtsdo/data/doc/release/20160131/icd9cm/der2_iissscRefset_ComplexMapActiveSnapshot_INT_20160131.txt
  perl -pe 's/447563008/P447563008/;' $file >! x.txt
  perl -ne 'chop; chop; @_=split/\t/; $_[12]= $_[11]; print join "\t",@_; print "\r\n";' x.txt >! xx.txt 
  mv xx.txt x.txt
  mvn install -PMapRecords -Drun.config=/home/ihtsdo/config/config.properties -Dinput.file=x.txt >&! mvn.log
  
  # remove prior map records
  cd ~/code/admin/remover
  mvn install -PMapRecords -Drun.config=/home/ihtsdo/config/config.properties -Drefset.id=P447562003 >&! mvn.log
  # load latest active snapshot
  # Add a "P" to the front of the refset id, as refset ids have to be unique across projects.
  cd ~/code/admin/loader
  set file=/home/ihtsdo/data/doc/release/20160131/icd10/der2_iisssccRefset_ExtendedMapActiveSnapshot_INT_20160131.txt
  perl -pe 's/447562003/P447562003/;' $file >! x.txt
  mvn install -PMapRecords -Drun.config=/home/ihtsdo/config/config.properties -Dinput.file=x.txt >&! mvn.log
  service tomcat7 start
  
  

  NOTE: this process maintains the history of the previously published map sets which may wind up being a lot of data over time.  An alternative would be to manually remove the map records for the P447563008 and P447562003 map sets from map_records and map_records_AUD (and connected tables), load the new ones, and rebuild the map record indexes.  Currently map_records_AUD has about 2 million entries, so another 250K per release may not be a big deal.  Though periodically this could be done to keep the size of the table reasonable.
  
  

• "Finish" a release
  • Create complex map refset entries for all delta entries (replacing any there with the same identifiers)
  • Save the date of a clean DB so the state can be recovered later if needed - for 20160131 - 11/27/2015

  • Mark all map records for the project that are READY_FOR_PUBLICATION as PUBLISHED.

    service tomcat7 stop
    cd ~/code/admin/release
    mvn install -PFinishRelease -Drun.config=/home/ihtsdo/config/config.properties -Drefset.id=447562003 \
     -Dinput.file=/home/ihtsdo/data/doc/release/20160131/icd10/der2_iisssccRefset_ExtendedMapActiveSnapshot_INT_20160131.txt >&! mvn.log
    mvn install -PFinishRelease -Drun.config=/home/ihtsdo/config/config.properties -Drefset.id=447563008 \
     -Dinput.file=/home/ihtsdo/data/doc/release/20160131/icd9cm/der2_iissscRefset_ComplexMapActiveSnapshot_INT_20160131.txt >&! mvn2.log
    service tomcat7 start

• Begin editing cycle 

  • When ready to start editing for the next cycle, this marks the map project "editing cycle begin date".

    service tomcat7 stop
    cd ~/code/admin/loader
    mvn install -PBeginEditingCycle -Drun.config=/home/ihtsdo/config/config.properties -Drefset.id=447562003 >&! mvn.log
    mvn install -PBeginEditingCycle -Drun.config=/home/ihtsdo/config/config.properties -Drefset.id=447563008 >&! mvn2.log
    service tomcat7 start

• Remove old reports

  • When starting a new editing cycle, it may be useful to remove reports more than a year old to keep the overall size of accumulated reports reasonable.

    service tomcat7 stop
    # set date one year ago, e.g. 20140615
    set date = 20140615
    cd ~/code/parent/admin/remover
    mvn install -PReports -Drun.config=/home/ihtsdo/config/config.properties Drefset.id=447562003 -Dend.date=$date >&! mvn.log
    mvn install -PReports -Drun.config=/home/ihtsdo/config/config.properties Drefset.id=447563008 -Dend.date=$date >&! mvn.log
    service tomcat7 start

• Re-enable automations (e.g. cron jobs)
  • Make sure to check the setting for the drip feed to make sure it is current.  Choose most current snapshot version under:
    https://mgr.servers.aceworkspace.net/apps/ihtsdo-archiva/repository/all/org/ihtsdo/intl/release/process/wb-release-process/
  • Make sure to update the SNOMEDCT_VERSION setting in the loadDelta.csh script so that it can properly compute the "retire removed concepts" algorithm.  Set it to the corresponding release version of this release.

Creating Custom Release Processing

If the process above is not needed or desired, alternative release processing can be built as a custom development task.  Things to consider are:

• The release format
• Whether the release is part of the source/target terminologies of the map
• Whether the mapping versioning itself is coordinated with the source/target terminology versioning
• Whether map records were built completely from scratch or were generated from terminology data
• Whether a "delta" release is needed - e.g. a release that compares to the previous release and publishes only the changes.

These things are very project specific and are not the same for all projects.

Custom release processing can be implemented via an outside package, or in the mapping-custom project (see Custom Release Process).

References/Links

• n/a