Page Comparison

Questions:

1. How to determine the width, length and bit depth of TIFF images?
2. How to reconstruct just a sub-set of slices using recon_arrayxml.py?
3. How to reconstruct a region of interest using recon_arrayxml.py?
4. How to check the status of a SGE job?
5. How to delete a SGE job?
6. How to extract TIFF images from a given HDF5 dataset accompanied by a NeXus scan file?
7. How to check the status of all SGE jobs being run by all users?
8. How to find out which modules have been loaded (added) into the shell environment?

...

Answers:

1. Anchor
   A1 A1

   The width, length and bit depth of TIFF images can be determined in many different ways. For example, the Linux command tiffinfo can be used to print information about the input TIFF file(s) in the following way:

   tiffinfo [options] input1.tif input2.tif ...

   Info
   title Red Hat Enterprise Linux 7
   Note, machines which have been upgraded to RHEL7 may not have tiffinfo on the path by default. The easiest way to put it onto your path is to module load python, as our default Anaconda Python includes libtiff.

   For example,

   Code Block
   title Linux command
   tiffinfo 06029.tif

   displays the following summary information:

   Code Block
   title Linux output from tiffinfo
   TIFFReadDirectory: Warning, 06029.tif: unknown field with tag 65000 (0xfde8) encountered. TIFF Directory at offset 0x146d308 (21418760) Image Width: 4008 Image Length: 2672 Bits/Sample: 16 Sample Format: unsigned integer Compression Scheme: None Photometric Interpretation: min-is-black Samples/Pixel: 1 Rows/Strip: 2672 Planar Configuration: single image plane Make: Unknown Model: Unknown Tag 65000: 727733745.824557

2. Anchor
   A2 A2

   It is sometimes desirable to reconstruct a sub-set, or a slab, of slices rather than the complete set. The recon_arrayxml.py script is capable of accomplishing this task with the help of the F- and L-options. 
   For example,

   Code Block
   title Linux command
   python2.6 /dls_sw/i13/software/gda/config/tomography_scripts/recon_arrayxml.py -I settings.xml -o ../../reconstruction/564/ -C 2000.5 -F 1299 -L 1300

   will reconstruct a slab consisting of exactly 1 slice (=1300-1299), namely that indexed 1299.
   
   

3. Anchor
   A3 A3

   The ROI element included in your settings.xml file needs to be modified in the following way:

   
   

   <ROI>
           <Type info="Standard, Rectangle">Rectangle</Type>
           <Xmin>580</Xmin>
           <Xmax>3000</Xmax>
           <Ymin>1225</Ymin>
           <Ymax>1725</Ymax>
           <OutputWidthType info="Standard, Given">Standard</OutputWidthType>
           <OutputWidth>500</OutputWidth>
           <Angle>0.0</Angle>

   </ROI>

   
   

   where the above values of Xmin, Xmax, Ymin, and Ymax need to be changed to reflect your particular reconstruction needs. Incidentally, these ROI coordinates can be determined, for example, by viewing the original projection images in ImageJ.  
   
   

4. Anchor
   A4 A4

   The information about all current SGE jobs can be obtained with the help of the qstat command. The state of a successfully submitted job is indicated by the status code qw or r.If a job encountered a problem, then its status code is Eqw or Rr. The possible status codes are:

   
   d(eletion)

   E(rror)

   h(old)

   r(unning)

   R(estarted)

   s(uspended)

   S(uspended)

   t(ransferring)

   T(hreshold)

   w(aiting)
    

    For a more detailed explanation of all possible job states, see the table displayed below:

   
   

   Category	SGE Letter Code	Description
   Pending
   	qw	Job is in the queue and waiting to be executed.
   	hqw	Job is in the queue and waiting to be executed but it is not elligible for execution due to a hold request or because it is waiting for completion of all the jobs in its job dependecies.
   	hRqw	Job is in the queue and waiting to be executed after being Restarted but it is not elligible for execution due to a hold request or because it is waiting for completion of all the jobs in its job dependecies.
   Running
   	r	Job is running.
   	t	Job is transferring and about to be executed.
   	Rr	Job is running after being Restarted.
   	Rt	Job is transferring after being Restarted.
   Suspended
   	s ts	Job is suspended.
   	S tS	The queue containing the job is Suspended and therefore the job is also suspended.
   	T tT	At least one suspend Threshold of the queue containing the job has been exceeded and therefore the job has been suspended as a consequence.
   	Rs Rts RS RtS RT RtT	Job is in the Suspended category after being Restarted.
   Error
   	Eqw Ehqw ERqw	Job in the Pending category is in the Error state due to its job properties and therefore could not be executed.
   Deletion
   	dr dt dRr dRt ds dts dS dtS dT dtT dRs dRts dRS dRtS dRT dRtT	Job in the Running or Suspended category which has been marked for deletion via the qdel command.

     

   If a SGE job is in the Eqw state, then the following command:

     

   Code Block
   title Linux command
   qstat -j job_id | grep "error reason"

     

   can be used to diagnose the errors (please replace the job_id argument with the identifier of your particular job).  

     

5. Anchor
   A5 A5

   The owner of a SGE job can delete it with the help of the qdel command:

     

   Code Block
   title Linux command
   qdel job_id

   
   
   

6. Anchor
   A6 A6

   The nxs2tiff command can be used to extract TIFF images from a given HDF5 dataset accompanied by a NeXus scan file. One can use this command to extract all or any subset of images from this dataset.

   Code Block
   title Linux commands
   module load tomography; nxs2tiff -h

   outputs a brief description of the command's arguments:

   Code Block
   title Linux output
   >nxs2tiff -h Welcome to the DLS compute cluster For MPI jobs, please use 'module load openmpi'. If using a different OpenMPI installation, or manually specifying path to OpenMPI, option '-mca orte_forward_job_control 1' must be added to mpirun to ensure cluster functionality. Please report any issues to linux.manager@diamond.ac.uk Loading 64-bit scipy, version 0.10.0 /dls_sw/apps/tomopy/tomopy/src/tifffile.py:2170: UserWarning: failed to import _tifffile.decodepackbits warnings.warn("failed to import %s" % module_function) /dls_sw/apps/tomopy/tomopy/src/tifffile.py:2170: UserWarning: failed to import _tifffile.decodelzw warnings.warn("failed to import %s" % module_function) /dls_sw/apps/tomopy/tomopy/src/tifffile.py:2170: UserWarning: failed to import _tifffile.unpackints warnings.warn("failed to import %s" % module_function) Usage: nxs2tiff.py [options] file output_folder Options: --version show program's version number and exit -h, --help show this help message and exit -b BEGIN, --begin=BEGIN projection to begin from -s SLICES, --slices=SLICES number of projections to process -d DATASET, --dataset=DATASET path Selectto the dataset to read from -k KEYS, --image_keys=KEYS path Selectto the image_keys dataset -i DIM, --slice_dim=DIM Dimensiondimension to slice the data in to create the projections -f FORMAT, --format=FORMAT format for the output files -r, --robert output for robert --scan=SCAN scan number if needed (for roberts extraction) >

   
   The simplest way to use nxs2tiff is to follow the following 3-argument pattern:

   Code Block
   title Linux command
   nxs2tiff -d /nexus/internal/path/to/dataset /path/to/nexus/file /path/to/output/directory

   Please note that the above command requires the use of absolute paths. The default filename format for all extracted files is: p_%05d.tif (c.f. the f-option above).

   Please also note that the default value for the d-option is /entry1/tomo_entry/instrument/detector/data, so there is normally no need to include this option unless the tomo entry is absent from your Nexus scan file, or is malformed, or if you would like to extract images from a dataset stored in a location different from this default path.

   For example, the command given below extracts all images (flat fields, dark fields, and sample projections) in the TIFF format, and then saves the entire sequence of these images in a common directory:

   Code Block
   title Linux command
   nxs2tiff -d /entry1/instrument/pco1_hw_hdf/data /dls/i13/data/2013/cm5937-3/raw/23912.nxs /dls/i13/data/2013/cm5937-3/processing/raw/23912/projections

   Please note that the first input path, corresponding to the d (standing for data or dataset) option, is not a path to an HDF5 file but an internal path to a desired dataset inside your NeXus scan file (incidentally, the path to the latter file should be used as the second argument). This internal path can be verified in the usual way using hdfview:

   

   If your Nexus scan file contains the relevant image-key data, then it is possible to extract and save your images in 3 separate directories depending on whether they are flat fields, dark fields, or sample projections. This can be done with the help of the r and scan options. For example, the command given below:

   Code Block
   title Linux command
   nxs2tiff -r --scan 23912 /dls/i13/data/2013/cm5937-3/raw/23912.nxs /dls/i13/data/2013/cm5937-3/processing/raw

   extracts sample-projections, flat-field (ff), and dark-field (df) images to <output_folder>/<scan>/projections, <output_folder>/ff1<scan>/projections and <output_folder>/df<scan>/projections, respectively.

   Image type	Path to destination directory relative to the location of <output_folder>	Absolute path to destination directory	Comment(s)
   sample projection	/23912/projections	/dls/i13/data/2013/cm5937-3/processing/raw/23912/projections	The extracted projection images are indexed from 0. For example, if default filename format, p_%05d.tif, is used to execute nxs2tiff, then they are named p_00000.tif, p_00001.tif, p_00002.tif, etc.
   flat field	/ff123912/projections	/dls/i13/data/2013/cm5937-3/processing/raw/ff123912/projections	The extracted flat-field images are indexed from 0. For example, if default filename format, p_%05d.tif, is used to execute nxs2tif, then they are named p_00000.tif, p_00001.tif, p_00002.tif, etc.
   dark field	/df23912/projections	/dls/i13/data/2013/cm5937-3/processing/raw/df23912/projections	The extracted dak-field images are indexed from 0. For example, if default filename format, p_%05d.tiff, is used to execute nxs2tif, then they are named p_00000.tif, p_00001.tif, p_00002.tif, etc.

   Please also note that the default value for the k-option is /entry1/tomo_entry/instrument/detector/image_key, so there is normally no need to include this option unless the tomo entry is absent from your Nexus scan file, or is malformed, or if you would like to utilise image keys stored in a location different from this default path.

   To extract a subset of images, please use the b and s options.
   
   

   When the nxs2tiff command is executed, the terminal output should contain the following message:

   Code Block
   title Linux output
   Your job job_id ("run_nxs2tiff.sh") has been submitted

   
   Please note that nxs2tiff runs the above job on the DLS cluster (one can monitor its progress in the usual way using qstat). The nxs2tiff command does not return until this job is completed. When the job is completed with success, one should see the following final message:

   Code Block
   title Linux output
   Job job_id exited with exit code 0.

   
   

   If there are any errors, the message reads as follows:

   Code Block
   title Linux output
   Job job_id exited with exit code 1.

   The errors can be diagnosed with the help of the relevant qsub log files, which can be found in /dls/tmp/tomopy. Please examine your latest run_nxs2tiff.sh.e* style file, found in that directory.

   
   

7. Anchor
   A7 A7

   To view the status of all SGE jobs being run by all users, please execute the following qstat command for all users:

   Code Block
   title Linux commands
   qstat -u "*"

   or, equivalently,

   Code Block
   title Linux commands
   qstat -u \*

8. Anchor
   A8 A8

   To find out which modules have been loaded (added) to the shell environment, please execute:
   
   

   Code Block
   title Linux commands
   module list

   
   As usual, for more information about the module command and its various useful switches and sub-commands, please execute

   
   

   Code Block
   title Linux commands
   man module

   
   to display the on-line manual pages on module.

...