Page Comparison

Questions:

...

Answers:

Anchor

	A1
	A1

The width, length and bit depth of TIFF images can be determined in many different ways. For example, the Linux command tiffinfo can be used to print information about the input TIFF file(s) in the following way:

tiffinfo [options] input1.tif input2.tif ...

Info

title	Red Hat Enterprise Linux 7

Note, machines which have been upgraded to RHEL7 may not have tiffinfo on the path by default.

The easiest way to put it onto your path is to module load python, as our default Anaconda Python includes libtiff.

For example,

Code Block

title	Linux command

tiffinfo 06029.tif

displays the following summary information:

Code Block

title	Linux output from tiffinfo

TIFFReadDirectory: Warning, 06029.tif: unknown field with tag 65000 (0xfde8) encountered.
TIFF Directory at offset 0x146d308 (21418760)
  Image Width: 4008 Image Length: 2672
  Bits/Sample: 16
  Sample Format: unsigned integer
  Compression Scheme: None
  Photometric Interpretation: min-is-black
  Samples/Pixel: 1
  Rows/Strip: 2672
  Planar Configuration: single image plane
  Make: Unknown
  Model: Unknown
  Tag 65000: 727733745.824557

Anchor
A2
A2
It is sometimes desirable to reconstruct a sub-set, or a slab, of slices rather than the complete set. The recon_arrayxml.py script is capable of accomplishing this task with the help of the F- and L-options.
For example,
Code Block
title Linux command
python2.6 /dls_sw/i13/software/gda/config/tomography_scripts/recon_arrayxml.py -I settings.xml -o ../../reconstruction/564/ -C 2000.5 -F 1299 -L 1300
will reconstruct a slab consisting of exactly 1 slice (=1300-1299), namely that indexed 1299.
Anchor
A3
A3
The ROI element included in your settings.xml file needs to be modified in the following way:

<ROI>
        <Type info="Standard, Rectangle">Rectangle</Type>
        <Xmin>580</Xmin>
        <Xmax>3000</Xmax>
        <Ymin>1225</Ymin>
        <Ymax>1725</Ymax>
        <OutputWidthType info="Standard, Given">Standard</OutputWidthType>
        <OutputWidth>500</OutputWidth>
        <Angle>0.0</Angle>
</ROI>

where the above values of Xmin, Xmax, Ymin, and Ymax need to be changed to reflect your particular reconstruction needs. Incidentally, these ROI coordinates can be determined, for example, by viewing the original projection images in ImageJ.

Anchor

	A4
	A4

The information about all current SGE jobs can be obtained with the help of the qstat command. The state of a successfully submitted job is indicated by the status code qw or r.If a job encountered a problem, then its status code is Eqw or Rr. The possible status codes are:

d(eletion)
E(rror)
h(old)
r(unning)
R(estarted)
s(uspended)
S(uspended)
t(ransferring)
T(hreshold)
w(aiting)

For a more detailed explanation of all possible job states, see the table displayed below:

Category SGE Letter Code Description
Pending


qw Job is in the queue and waiting to be executed.

hqw Job is in the queue and waiting to be executed but it is not elligible for execution due to a hold request or because it is waiting for completion of all the jobs in its job dependecies.


hRqw
Job is in the queue and waiting to be executed after being Restarted but it is not elligible for execution due to a hold request or because it is waiting for completion of all the jobs in its job dependecies.
Running


r Job is running.

t Job is transferring and about to be executed.

Rr Job is running after being Restarted.

Rt Job is transferring after being Restarted.
Suspended


s
ts
Job is suspended.

S
tS
The queue containing the job is Suspended and therefore the job is also suspended.

T
tT
At least one suspend Threshold of the queue containing the job has been exceeded and therefore the job has been suspended as a consequence.


Rs
Rts
RS
RtS
RT
RtT
Job is in the Suspended category after being Restarted.
Error


Eqw
Ehqw
ERqw
Job in the Pending category is in the Error state due to its job properties and therefore could not be executed.
Deletion


dr
dt
dRr
dRt
ds
dts
dS
dtS
dT
dtT
dRs
dRts
dRS
dRtS
dRT
dRtT
Job in the Running or Suspended category which has been marked for deletion via the qdel command.

If a SGE job is in the Eqw state, then the following command:

Code Block
title Linux command
qstat -j job_id | grep "error reason"

can be used to diagnose the errors (please replace the job_id argument with the identifier of your particular job).

Anchor
A5
A5
The owner of a SGE job can delete it with the help of the qdel command:

Code Block
title Linux command
qdel job_id

Anchor

	A6
	A6

The nxs2tiff command can be used to extract TIFF images from a given HDF5 dataset accompanied by a NeXus scan file. One can use this command to extract all or any subset of images from this dataset.
Code Block
title Linux commands
module load tomography; nxs2tiff -h
outputs a brief description of the command's arguments:
Code Block
title Linux output
>nxs2tiff -h Welcome to the DLS compute cluster For MPI jobs, please use 'module load openmpi'. If using a different OpenMPI installation, or manually specifying path to OpenMPI, option '-mca orte_forward_job_control 1' must be added to mpirun to ensure cluster functionality. Please report any issues to linux.manager@diamond.ac.uk Loading 64-bit scipy, version 0.10.0 /dls_sw/apps/tomopy/tomopy/src/tifffile.py:2170: UserWarning: failed to import _tifffile.decodepackbits warnings.warn("failed to import %s" % module_function) /dls_sw/apps/tomopy/tomopy/src/tifffile.py:2170: UserWarning: failed to import _tifffile.decodelzw warnings.warn("failed to import %s" % module_function) /dls_sw/apps/tomopy/tomopy/src/tifffile.py:2170: UserWarning: failed to import _tifffile.unpackints warnings.warn("failed to import %s" % module_function) Usage: nxs2tiff.py [options] file output_folder Options: --version show program's version number and exit -h, --help show this help message and exit -b BEGIN, --begin=BEGIN projection to begin from -s SLICES, --slices=SLICES number of projections to process -d DATASET, --dataset=DATASET Selectpath to the dataset to read from -k KEYS, --image_keys=KEYS path Selectto the image_keys dataset -i DIM, --slice_dim=DIM Dimensiondimension to slice the data in to create the projections -f FORMAT, --format=FORMAT format for the output files -r, --robert output for robert --scan=SCAN scan number if needed (for roberts extraction) >

The simplest way to use nxs2tiff is to follow the following 3-argument pattern:
Code Block
title Linux command
nxs2tiff -d /nexus/internal/path/to/dataset /path/to/nexus/file /path/to/output/directory
Please note that the above command requires the use of absolute paths. The default filename format for all extracted files is: p_%05d.tif (c.f. the f-option above).
Please also note that the default value for the d-option is /entry1/tomo_entry/instrument/detector/data, so there is normally no need to include this option unless the tomo entry is absent from your Nexus scan file, or is malformed, or if you would like to extract images from a dataset stored in a location different from their this default path.
For example, the command given below extracts all images (flat fields, dark fields, and sample projections) in the TIFF format, and then saves the entire sequence of these images (each named ) in a common directory:
Code Block
title Linux command
nxs2tiff -d /entry1/instrument/pco1_hw_hdf/data /dls/i13/data/2013/cm5937-3/raw/23912.nxs /dls/i13/data/2013/cm5937-3/processing/raw/23912/projections
Please note that the first input path, corresponding to the d (standing for data or dataset) option, is not a path to an HDF5 file but an internal path to a desired dataset inside your NeXus scan file (incidentally, the path to the latter file should be used as the second argument). This internal path can be verified in the usual way using hdfview:
If your Nexus scan file contains the relevant image-key data, then it is possible to extract and save your images in 3 separate directories depending on whether they are flat fields, dark fields, or sample projections. This can be done with the help of the r and scan options. For example, the command given below:
Code Block
title Linux command
nxs2tiff -r --scan 23912 /dls/i13/data/2013/cm5937-3/raw/23912.nxs /dls/i13/data/2013/cm5937-3/processing/raw
extracts sample-projections, flat-field (ff), and dark-field (df) images to <output_folder>/<scan>/projections, <output_folder>/ff1<ff1<scan>/projections and <output_folder>/df<df<scan>/projections, respectively.
Image type
Image path
Path to destination directory relative to the location of <output_folder>
Image absolute path
Absolute path to destination directory Comment(s)
sample projection
/23912/projections
/dls/i13/data/2013/cm5937-3/processing/raw/23912/projections
flat field
/ff123912/projections
/dls
The extracted projection images are indexed from 0. For example, if default filename format, p_%05d.tif, is used to execute nxs2tiff, then they are named p_00000.tif, p_00001.tif, p_00002.tif, etc.
flat field
/ff123912/projections
/dls/i13/data/2013/cm5937-3/processing/raw/ff123912/projections
dark
The extracted flat-field images are indexed from 0. For example, if default filename format, p_%05d.tif, is used to execute nxs2tif, then they are named p_00000.tif, p_00001.tif, p_00002.tif, etc.
dark field
/df23912/projections
/dls/i13
/data/2013/cm5937-3/processing/raw/df23912/projections
/data/2013/cm5937-3/processing/raw/df23912/projections
The extracted dak-field images are indexed from 0. For example, if default filename format, p_%05d.tiff, is used to execute nxs2tif, then they are named p_00000.tif, p_00001.tif, p_00002.tif, etc.
Please also note that the default value for the k-option is /entry1/tomo_entry/instrument/detector/image_key, so there is normally no need to include this option unless the tomo entry is absent from your Nexus scan file, or is malformed, or if you would like to utilise image keys stored in a location different from their this default path.
To extract a subset of images, please use the b and s options.

When the nxs2tiff command is executed, the terminal output should contain the following message:
Code Block
title Linux output
Your job job_id ("run_nxs2tiff.sh") has been submitted

Please note that nxs2tiff runs the above job on the DLS cluster (one can monitor its progress in the usual way using qstat). The nxs2tiff command does not return until this job is completed. When the job is completed with success, one should see the following final message:
Code Block
title Linux output
Job job_id exited with exit code 0.

If there are any errors, the message reads as follows:
Code Block
title Linux output
Job job_id exited with exit code 1.
The errors can be diagnosed with the help of the relevant qsub log files, which can be found in /dls/tmp/tomopy. Please examine your latest run_nxs2tiff.sh.e* style file, found in that directory.

Anchor
A7
A7
To view the status of all SGE jobs being run by all users, please execute the following qstat command for all users:
Code Block
title Linux commands
qstat -u "*"
or, equivalently,
Code Block
title Linux commands
qstat -u \*
Anchor
A8
A8
To find out which modules have been loaded (added) to the shell environment, please execute:

Code Block
title Linux commands
module list

As usual, for more information about the module command and its various useful switches and sub-commands, please execute

Code Block
title Linux commands
man module

to display the on-line manual pages on module.

Versions Compared

Old Version 55

New Version Current

Key

Questions:

Answers:


	hRqw	Job is in the queue and waiting to be executed after being Restarted but it is not elligible for execution due to a hold request or because it is waiting for completion of all the jobs in its job dependecies.
Running

The extracted projection images are indexed from 0. For example, if default filename format, p_%05d.tif, is used to execute nxs2tiff, then they are named p_00000.tif, p_00001.tif, p_00002.tif, etc.
flat field	/ff123912/projections	/dls/i13/data/2013/cm5937-3/processing/raw/ff123912/projections

The extracted flat-field images are indexed from 0. For example, if default filename format, p_%05d.tif, is used to execute nxs2tif, then they are named p_00000.tif, p_00001.tif, p_00002.tif, etc.
dark field	/df23912/projections	/dls/i13