Tomo Reconstruction FAQ

Questions:

How to determine the width, length and bit depth of TIFF images?
How to reconstruct just a sub-set of slices using recon_arrayxml.py?
How to reconstruct a region of interest using recon_arrayxml.py?
How to check the status of a SGE job?
How to delete a SGE job?
How to extract TIFF images from a given HDF5 dataset accompanied by a NeXus scan file?
How to check the status of all SGE jobs being run by all users?
How to find out which modules have been loaded (added) into the shell environment?

Answers:

The width, length and bit depth of TIFF images can be determined in many different ways. For example, the Linux command tiffinfo can be used to print information about the input TIFF file(s) in the following way:
tiffinfo [options] input1.tif input2.tif ...
Red Hat Enterprise Linux 7

Note, machines which have been upgraded to RHEL7 may not have tiffinfo on the path by default.
The easiest way to put it onto your path is to module load python, as our default Anaconda Python includes libtiff.
For example,
Linux command
```
tiffinfo 06029.tif
```
displays the following summary information:
Linux output from tiffinfo
```
TIFFReadDirectory: Warning, 06029.tif: unknown field with tag 65000 (0xfde8) encountered.
TIFF Directory at offset 0x146d308 (21418760)
  Image Width: 4008 Image Length: 2672
  Bits/Sample: 16
  Sample Format: unsigned integer
  Compression Scheme: None
  Photometric Interpretation: min-is-black
  Samples/Pixel: 1
  Rows/Strip: 2672
  Planar Configuration: single image plane
  Make: Unknown
  Model: Unknown
  Tag 65000: 727733745.824557
```
It is sometimes desirable to reconstruct a sub-set, or a slab, of slices rather than the complete set. The recon_arrayxml.py script is capable of accomplishing this task with the help of the F- and L-options.
For example,
Linux command
```
python2.6 /dls_sw/i13/software/gda/config/tomography_scripts/recon_arrayxml.py -I settings.xml -o ../../reconstruction/564/ -C 2000.5 -F 1299 -L 1300
```
will reconstruct a slab consisting of exactly 1 slice (=1300-1299), namely that indexed 1299.
The ROI element included in your settings.xml file needs to be modified in the following way:

<ROI>
        <Type info="Standard, Rectangle">Rectangle</Type>
        <Xmin>580</Xmin>
        <Xmax>3000</Xmax>
        <Ymin>1225</Ymin>
        <Ymax>1725</Ymax>
        <OutputWidthType info="Standard, Given">Standard</OutputWidthType>
        <OutputWidth>500</OutputWidth>
        <Angle>0.0</Angle>
</ROI>

where the above values of Xmin, Xmax, Ymin, and Ymax need to be changed to reflect your particular reconstruction needs. Incidentally, these ROI coordinates can be determined, for example, by viewing the original projection images in ImageJ.

The information about all current SGE jobs can be obtained with the help of the qstat command. The state of a successfully submitted job is indicated by the status code qw or r. If a job encountered a problem, then its status code is Eqw or Rr. The possible status codes are:

d(eletion)

E(rror)

h(old)

r(unning)

R(estarted)

s(uspended)

S(uspended)

t(ransferring)

T(hreshold)

w(aiting)

For a more detailed explanation of all possible job states, see the table displayed below:

Category	SGE Letter Code	Description
Pending
	qw	Job is in the queue and waiting to be executed.
	hqw	Job is in the queue and waiting to be executed but it is not elligible for execution due to a hold request or because it is waiting for completion of all the jobs in its job dependecies.
	hRqw	Job is in the queue and waiting to be executed after being Restarted but it is not elligible for execution due to a hold request or because it is waiting for completion of all the jobs in its job dependecies.
Running
	r	Job is running.
	t	Job is transferring and about to be executed.
	Rr	Job is running after being Restarted.
	Rt	Job is transferring after being Restarted.
Suspended
	s ts	Job is suspended.
	S tS	The queue containing the job is Suspended and therefore the job is also suspended.
	T tT	At least one suspend Threshold of the queue containing the job has been exceeded and therefore the job has been suspended as a consequence.
	Rs Rts RS RtS RT RtT	Job is in the Suspended category after being Restarted.
Error
	Eqw Ehqw ERqw	Job in the Pending category is in the Error state due to its job properties and therefore could not be executed.
Deletion
	dr dt dRr dRt ds dts dS dtS dT dtT dRs dRts dRS dRtS dRT dRtT	Job in the Running or Suspended category which has been marked for deletion via the qdel command.

If a SGE job is in the Eqw state, then the following command:

Linux command

qstat -j job_id | grep "error reason"

can be used to diagnose the errors (please replace the job_id argument with the identifier of your particular job).

The owner of a SGE job can delete it with the help of the qdel command:
Linux command
```
qdel job_id 
```

The nxs2tiff command can be used to extract TIFF images from a given HDF5 dataset accompanied by a NeXus scan file. One can use this command to extract all or any subset of images from this dataset.

Linux commands

module load tomography; nxs2tiff -h

outputs a brief description of the command's arguments:

Linux output

>nxs2tiff -h
    

     Welcome to the DLS compute cluster

     For MPI jobs, please use 'module load openmpi'.

     If using a different OpenMPI installation,
     or manually specifying path to OpenMPI, option
     '-mca orte_forward_job_control 1'
     must be added to mpirun to ensure cluster functionality.

     Please report any issues to linux.manager@diamond.ac.uk

Loading 64-bit scipy, version 0.10.0
/dls_sw/apps/tomopy/tomopy/src/tifffile.py:2170: UserWarning: failed to import _tifffile.decodepackbits
  warnings.warn("failed to import %s" % module_function)
/dls_sw/apps/tomopy/tomopy/src/tifffile.py:2170: UserWarning: failed to import _tifffile.decodelzw
  warnings.warn("failed to import %s" % module_function)
/dls_sw/apps/tomopy/tomopy/src/tifffile.py:2170: UserWarning: failed to import _tifffile.unpackints
  warnings.warn("failed to import %s" % module_function)

Usage: nxs2tiff.py [options] file output_folder

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -b BEGIN, --begin=BEGIN
                        projection to begin from
  -s SLICES, --slices=SLICES
                        number of projections to process
  -d DATASET, --dataset=DATASET
                        path to the dataset to read from
  -k KEYS, --image_keys=KEYS
                        path to the image_keys dataset
  -i DIM, --slice_dim=DIM
                        dimension to slice the data in to create the
                        projections
  -f FORMAT, --format=FORMAT
                        format for the output files
  -r, --robert          output for robert
  --scan=SCAN           scan number if needed (for roberts extraction)
>

The simplest way to use nxs2tiff is to follow the following 3-argument pattern:

Linux command

nxs2tiff -d /nexus/internal/path/to/dataset /path/to/nexus/file /path/to/output/directory

Please note that the above command requires the use of absolute paths. The default filename format for all extracted files is: p_%05d.tif (c.f. the f-option above).

Please also note that the default value for the d-option is /entry1/tomo_entry/instrument/detector/data, so there is normally no need to include this option unless the tomo entry is absent from your Nexus scan file, or is malformed, or if you would like to extract images from a dataset stored in a location different from this default path.

For example, the command given below extracts all images (flat fields, dark fields, and sample projections) in the TIFF format, and then saves the entire sequence of these images in a common directory:

Linux command

nxs2tiff -d /entry1/instrument/pco1_hw_hdf/data /dls/i13/data/2013/cm5937-3/raw/23912.nxs /dls/i13/data/2013/cm5937-3/processing/raw/23912/projections

Please note that the first input path, corresponding to the d (standing for data or dataset) option, is not a path to an HDF5 file but an internal path to a desired dataset inside your NeXus scan file (incidentally, the path to the latter file should be used as the second argument). This internal path can be verified in the usual way using hdfview:

If your Nexus scan file contains the relevant image-key data, then it is possible to extract and save your images in 3 separate directories depending on whether they are flat fields, dark fields, or sample projections. This can be done with the help of the r and scan options. For example, the command given below:

Linux command

nxs2tiff -r --scan 23912 /dls/i13/data/2013/cm5937-3/raw/23912.nxs /dls/i13/data/2013/cm5937-3/processing/raw

extracts sample-projections, flat-field (ff), and dark-field (df) images to <output_folder>/<scan>/projections, <output_folder>/ff1<scan>/projections and <output_folder>/df<scan>/projections, respectively.

Image type	Path to destination directory relative to the location of <output_folder>	Absolute path to destination directory	Comment(s)
sample projection	/23912/projections	/dls/i13/data/2013/cm5937-3/processing/raw/23912/projections	The extracted projection images are indexed from 0. For example, if default filename format, p_%05d.tif, is used to execute nxs2tiff, then they are named p_00000.tif, p_00001.tif, p_00002.tif, etc.
flat field	/ff123912/projections	/dls/i13/data/2013/cm5937-3/processing/raw/ff123912/projections	The extracted flat-field images are indexed from 0. For example, if default filename format, p_%05d.tif, is used to execute nxs2tif, then they are named p_00000.tif, p_00001.tif, p_00002.tif, etc.
dark field	/df23912/projections	/dls/i13/data/2013/cm5937-3/processing/raw/df23912/projections	The extracted dak-field images are indexed from 0. For example, if default filename format, p_%05d.tiff, is used to execute nxs2tif, then they are named p_00000.tif, p_00001.tif, p_00002.tif, etc.

Please also note that the default value for the k-option is /entry1/tomo_entry/instrument/detector/image_key, so there is normally no need to include this option unless the tomo entry is absent from your Nexus scan file, or is malformed, or if you would like to utilise image keys stored in a location different from this default path.

To extract a subset of images, please use the b and s options.

When the nxs2tiff command is executed, the terminal output should contain the following message:

Linux output

Your job job_id ("run_nxs2tiff.sh") has been submitted

Please note that nxs2tiff runs the above job on the DLS cluster (one can monitor its progress in the usual way using qstat). The nxs2tiff command does not return until this job is completed. When the job is completed with success, one should see the following final message:

Linux output

Job job_id exited with exit code 0.

If there are any errors, the message reads as follows:

Linux output

Job job_id exited with exit code 1.

The errors can be diagnosed with the help of the relevant qsub log files, which can be found in /dls/tmp/tomopy. Please examine your latest run_nxs2tiff.sh.e* style file, found in that directory.

To view the status of all SGE jobs being run by all users, please execute the following qstat command for all users:
Linux commands
```
qstat -u "*"
```
or, equivalently,
Linux commands
```
qstat -u \*
```
To find out which modules have been loaded (added) to the shell environment, please execute:
Linux commands
```
module list
```
As usual, for more information about the module command and its various useful switches and sub-commands, please execute
Linux commands
```
man module
```
to display the on-line manual pages on module.