Python autoProcessing wrapper for SAVU

For information about how to create a json config file to use with the autoProcessing wrapper, please visit this site:

Introduction

The autoProcessing wrapperis a python class included in the I13analysis module. It aims to create an easy to use interface between the users and SAVU. Changing of parameters in SAVU config files is automated by the interface and several methods allow tailoring individual reconstructions as required.

autoProcessing uses a combination of SAVU calls (for most data processes) and local reconstruction for the centering and entropy metrics calculation (used in automatic COR determination). Please make sure not to start any processes on the data acquisition machine. In addition, autoProcessing also searches for intermediate SAVU data to be reused to minimize the processing time. All intermediate data is written to the <visitID>/tmp folder and only center-finding and final reconstructions are written to the <visitID>/processing folder.

Calling parameters

The following calling parameters are used to configure the autoProcessing

parameter	alias	datatype	description
help	-h, --help		Use this to display all available options and help text on the command line
json file name	-jsonfile, -j	string (file path)	Path for the json config file which includes all the beamtime meta data. This is a mandatory argument. However, if omitted the script will look in the current working directory and if there is exactly one file with a .json suffix, this will be loaded.
objecive number	-objective, -o	integer	The objective number to be used for the distortion correction.The default is <8> which corresponds to no distortion correction.
centre of reconstruction	-cor, -c	float	The centre of reconstruction, if a manual input is required. This value will be replaced by the automatic centering if the latter is called.
slice number	-slice, -s	integer	The slice number to be used for testing. If not specified, the value from the original SAVU configuration file will be used.
stepping width	-dx	float	Step width for the automatic centering. Defaults to 0.1
range of center finding	-range	float	The range argument specifies which range is to be scanned for the final COR determination. The search range is [x0 - range, x0 + range]. The default range is 2.0
SAVU big	--big		Use this optional argument to call SAVU with the "BIG" keyword (running on com14 instead of com10). This allows you to start more than two parallel processes.
redo all calculations	-redo, -r		Use this flag to force re-running all preliminary SAVU steps. Required for example after changing the slice number.
scan numbers		integers	the number of the scans to be processed. Add as many scans as required, the will be worked on in a queue. Please note that an exception during runtime will stop the whole queue.

Configuration and starting up

Usage of predefined scripts

The usage of scripts is the most convenient way. It is good practice to copy the script files and json file in the <visitID>/processing/auto_processing folder and work in there.

To prepare the terminal, go to your visit directory and load the required python libraries:

[fedID@machine ~]$ cd /dls/i13/data/YEAR/VISITID/processing/auto_processing
[fedID@machine auto_procesing]$ source loadPython

This step only needs to be performed once (but in each new console or shell).

To start a process, use for example:

python ap_full.py -j VISITID.json -o 1 -cor 1266 -dx 0.5 -range 5 XXXXXX YYYYYY

on-the fly scripting

to use autoProcessing from within a script, you need to specify the calling parameters (all as string). You need to load the I13analysis module, which can e found at /dls_sw/i13/scripts/Malte.

import sys
sys.path.append('/dls_sw/i13/scripts/Malte')
import I13analysis
args = ['-j', '/dls/i13/data/2018/cm19664-3/processing/auto_processing/cm19664-3.json', '-o', '1', 'XXXXXX']
ap = I13analysis.autoProcessing(args)

Script examples:

All these examples can be found at /dls_sw/i13/scripts/Malte/auto_processing/exampleNew

Full autoprocessing: ap_full.py

This script will start a VO-centering to find a good starting point for COR refinement, then do a refinement using Donath's entropy metric before starting the full reconstruction.

ap_full.py

import sys
sys.path.append('/dls_sw/i13/scripts/Malte/')
import I13analysis

ap = I13analysis.autoProcessing(sys.argv[1:])
for scan in ap.scans:
    print '\n\n%s: Processing scan #%i.' % (I13analysis.GetTimeString(), scan)
    ap.setScan(scan)
    ap.runVoCentering()
    ap.autoTomopyRecAndMetrics(maskData = True)
    ap.finalReconstruction()
    print '\n%s: Processing of scan %i complete.' % (I13analysis.GetTimeString(), scan)

COR refinement and final reconstruction around a given COR start value: ap_guessCOR.py

This script will skip the VO-centering and use the value specified with "-cor XXX.XX" as a starting point for refinement.

ap_guessCOR.py

ap = I13analysis.autoProcessing(sys.argv[1:])
for scan in ap.scans:
    print '\n\n%s: Processing scan #%i.' % (I13analysis.GetTimeString(), scan)
    ap.setScan(scan)
    ap.runSinoCreation()
    ap.autoTomopyRecAndMetrics(maskData = True)
    ap.finalReconstruction()
    print '\n%s: Processing of scan %i complete.' % (I13analysis.GetTimeString(), scan)

Manual center finding: ap_manualCORfinding.py

This script will take the manual input "-cor XXX.XX" and reconstruct in the vincinity (determined by the "-range X.X" and "-dx Y.Y" parameters) without starting a full reconstruction.

import sys
sys.path.append('/dls_sw/i13/scripts/Malte/')
import I13analysis

ap = I13analysis.autoProcessing(sys.argv[1:])
for scan in ap.scans:
    print '\n\n%s: Processing scan #%i.' % (I13analysis.GetTimeString(), scan)
    ap.setScan(scan)
    ap.manualTomopyCenterSearch()
    print '\n%s: Processing of scan %i complete.' % (I13analysis.GetTimeString(), scan)