Release Process

Overview

Documents the release process details.  There is an admin page already that indicates how to run the mojo.  See Release Processing Tool for more information.

Assumptions

• Source terminology is SNOMED CT
• Release is in RF2 delta format
• QA of overall release is done by an external system (though project-specific validation rules are processed here)
• That editing was done in conjunction with a "drip feed" system that brought emerging content into the system as it was being developed, so it could be mapped.
• That the mapping files are part of the source terminology release

Details

The mapping tool supports a release process that takes into account end-of-editing cycle activities, some quality assurance, production of data files, and upgrading terminology versions when the time comes.  The release processing is specifically tailored to the needs of IHTSDO and maintenance of the RF2 complex and extended map refsets that are part of the SNOMED CT distribution.

 TODO: flowchart

The steps are:

• Upgrade to latest version - there are two forms of this and it assumes that the version of the map and the "source terminology" are coordinated.  This is to ensure the expected release state of "in scope" concepts is properly represented for workflow and release state computations.  Given processing of "daily build", innaccuracies can build up over an editing cycle.
  • Load a definitive delta release and update the version of all terminology components to latest version.

  • Remove and reload an entire definitive snapshot release. (NOTE: it may be desirable to do this FIRST, before "begin release").  IMPORTANT NOTE: when reloading data for the current release, reload mapping files from the PRIOR release as they are used for comparison (e.g. replace the complex/extended maps that come with this release). TODO: need code for this.

    # first prepare an edition that contains map files from the previous release 
    # along with the next release beta files with the right structure
    service tomcat7 stop
    cd ~/code/admin/remover
    mvn install -PTerminology -Drun.config=/home/ihtsdo/config/config.properties -Dterminology=SNOMEDCT -Dversion=latest >&! mvn.log
    cd ~/code/admin/loader
    mvn install -PRF2-snapshot -Drun.config=/home/ihtsdo/config/config.properties -Dterminology=SNOMEDCT -Dversion=latest -Dinput.dir=/home/ihtsdo/data/snomedct-20160131 >&! mvn.log
    
    service tomcat7 start

  • Double check that complex and extended map files have entry counts matching the files used to load.  One ICD9CM record for (5755759c-4d9b-57d0-b4a9-f20b19326194) gets lost because the concept id it is connected to does not exist in the concepts file. This entry has to get manually added back in to the final "snapshot" if it is really desired.  If delivering a delta this is not a problem.

  • Recompute workflow to ensure all in scope concepts are mapped

    set refsetId=447562003
    service tomcat7 stop
    cd ~/code/admin/loader
    mvn install -PComputeWorkflow -Drun.config=/home/ihtsdo/config/config.properties -Drefset.id=$refsetId >&! mvn.log 
    service tomcat7 start

• Finish all editing - assumes drip feed has been processing along the way
• Disable automations (e.g. cron jobs).
• "Begin" a release
  • Verify all "in scope" concepts are mapped
  • Validate all existing map records against current rules

  • Remove map records for concepts that are retired, deleted, or out of scope. 

    # run this and check "Release QA" report for the project until there are no errors
    set refsetId=447562003
    cd ~/code/admin/release
    mvn install -PBeginRelease -Drun.config=/home/ihtsdo/config/config.properties -Drefset.id=$refsetId -Dtest.mode.flag=true >&! mvn.log
    
    # Once there are no errrors
    set refsetId=447562003
    service tomcat7 stop
    cd ~/code/admin/release
    mvn install -PBeginRelease -Drun.config=/home/ihtsdo/config/config.properties -Drefset.id=$refsetId >&! mvn.log
    service tomcat7 start

  • NOTE: iterate through the process above with -Dtest.mode.flag=true until there are no errors (this can be done while server is still running).  This is needed for the "Release QA" report to be properly generated (because the non test mode issues a rollback() on any errors).

    • Cases marked as "Not marked ready for publication" will need to be manually removed with "Batch delete" as the map admin user.
      
      

• Generate release files
  • Create an RF2 delta file
  • Create human readable file

  • Create module dependency file (if needed)

    set time=20160731
    set refsetId=447562003
    set moduleId=449080006
    service tomcat7 stop
    cd ~/code/admin/release
    mvn install -PRelease -Drun.config=/home/ihtsdo/config/config.properties -Drefset.id=$refsetId \
      -Doutput.dir=. -Dtime=$time -Dmodule.id=$moduleId >&! mvn.release.log
    service tomcat7 start

• It may be desirable to load release files into static, read-only projects that are public and can serve as a "human readable" form of the published map. For example,

  set refsetId=447562003
  service tomcat7 stop
  # remove prior map records
  cd ~/code/admin/remover
  mvn install -PMapRecords -Drun.config=/home/ihtsdo/config/config.properties -Drefset.id=P$refsetId >&! mvn.log
  # load latest active snapshot
  # Add a "P" to the front of the refset id, as refset ids have to be unique across projects.
  cd ~/code/admin/loader
  set file=/home/ihtsdo/data/doc/release/...path to Snapshot_INT_....txt
  perl -pe 's/'$refsetId'/P'$refsetId'/;' $file >! x.txt
  perl -ne 'chop; chop; @_=split/\t/; $_[12]= $_[11]; print join "\t",@_; print "\r\n";' x.txt >! xx.txt 
  
  mv xx.txt x.txt
  mvn install -PMapRecords -Drun.config=/home/ihtsdo/config/config.properties -Dinput.file=x.txt >&! mvn.log
  
  

  NOTE: this process maintains the history of the previously published map sets which may wind up being a lot of data over time.  An alternative would be to manually remove the map records for the P447563008 and P447562003 map sets from map_records and map_records_AUD (and connected tables), load the new ones, and rebuild the map record indexes.  Currently map_records_AUD has about 2 million entries, so another 250K per release may not be a big deal.  Though periodically this could be done to keep the size of the table reasonable.
  
  

• "Finish" a release
  • Create complex map refset entries for all delta entries (replacing any there with the same identifiers)
  • Save the date of a clean DB so the state can be recovered later if needed - for 20160131 - 11/27/2015

  • Mark all map records for the project that are READY_FOR_PUBLICATION as PUBLISHED.  This also attempts to reload the map records with the definitive data.  We are faking it here with an empty file because generally SNOMED will need to be reloaded anyway after it is final and the reloading step is merely for computation of the delta for the NEXT release.  Given that publication files are in SNOMED data loaded, this is unnecessary.

    set refsetId=447562003
    service tomcat7 stop
    cd ~/code/admin/release
    /bin/rm -f x.txt
    touch x.txt
    mvn install -PFinishRelease -Drun.config=/home/ihtsdo/config/config.properties -Drefset.id=$refsetId \
     -Dinput.file=x.txt >&! mvn.log
    service tomcat7 start

• Begin editing cycle 

  • When ready to start editing for the next cycle, this marks the map project "editing cycle begin date".

    set refsetId=447562003
    service tomcat7 stop
    cd ~/code/admin/release
    mvn install -PBeginEditingCycle -Drun.config=/home/ihtsdo/config/config.properties -Drefset.id=$refsetId >&! mvn.log
    service tomcat7 start

• Remove old reports

  • When starting a new editing cycle, it may be useful to remove reports more than a year old to keep the overall size of accumulated reports reasonable.

    service tomcat7 stop
    # set date one year ago, e.g. 20141215
    set refsetId=447562003
    set date = 20141215
    cd ~/code/admin/remover
    mvn install -PReports -Drun.config=/home/ihtsdo/config/config.properties -Drefset.id=$refsetId -Dend.date=$date >&! mvn.log
    service tomcat7 start

• Re-enable automations (e.g. cron jobs)
  • Make sure to check the setting for the drip feed to make sure it is current.  Choose most current snapshot version under:
    https://mgr.servers.aceworkspace.net/apps/ihtsdo-archiva/repository/all/org/ihtsdo/intl/release/process/wb-release-process/
  • Make sure to update the SNOMEDCT_VERSION setting in the loadDelta.csh script so that it can properly compute the "retire removed concepts" algorithm.  Set it to the corresponding release version of this release.

Creating Custom Release Processing

If the process above is not needed or desired, alternative release processing can be built as a custom development task.  Things to consider are:

• The release format
• Whether the release is part of the source/target terminologies of the map
• Whether the mapping versioning itself is coordinated with the source/target terminology versioning
• Whether map records were built completely from scratch or were generated from terminology data
• Whether a "delta" release is needed - e.g. a release that compares to the previous release and publishes only the changes.

These things are very project specific and are not the same for all projects.

Custom release processing can be implemented via an outside package, or in the mapping-custom project (see Custom Release Process).

References/Links

• n/a