path: root/doc/man/partialator.1

.\"
.\" partialator man page
.\"
.\" Copyright © 2012-2020 Deutsches Elektronen-Synchrotron DESY,
.\"                       a research centre of the Helmholtz Association.
.\"
.\" Part of CrystFEL - crystallography with a FEL
.\"

.TH PARTIALATOR 1
.SH NAME
partialator \- scaling and post-refinement of partial reflections
.SH SYNOPSIS
.PP
.B partialator
\fB-i\fR \fIinput.stream\fR
\fB-o\fR \fIoutput.hkl\fR
\fB-y\fR \fIpointgroup\fR
[\fBoptions\fR] \fB...\fR
.PP
.B partialator
\fB--help\fR

.SH DESCRIPTION
\fBpartialator\fR merges reflections by scaling and post refinement, accounting
for the partialities of the reflections.  That means that it models the geometry
of diffraction for each pattern (crystal orientation, unit cell parameters,
X-ray bandwidth and so on) and attempts to optimise the geometrical parameters
to make the fully integrated intensities calculated using the model agree as
closely as possible between the many patterns.

See \fBUSAGE CASES\fR below for examples of commands to merge reflections in different
ways, for example with and without partiality or scaling.

In addition to the output reflection list, \fBpartialator\fR will write a file called partialator.params which contains the scaling factors determined for each crystal.  After each iteration, a file will be written called pgraph-iter\fIn\fR.dat which contains the observed and calculated partialities for a randomly chosen set of "free" reflections which were not included in the refinement.

\fBpartialator\fR will also write datasets merged using two halves of the total data using the same filename as you give for \fB-o\fR or \fB--output\fR, with \fB1\fR and \fB2\fR appended.  For example, \fBpartialator.hkl1\fR and \fBpartialator.hkl2\fR.  Compare these two files with each other to calculate figures of merit such as Rsplit and CC1/2.  See the section on \fBCUSTOM DATASET SPLITTING\fR for more discussion on this topic.

.SH OPTIONS
.PD 0
.IP "\fB-i\fR \fIfilename\fR"
.IP \fB--input=\fR\fIfilename\fR
.PD
Give the name of the input stream.

.PD 0
.IP "\fB-o\fR \fIfilename\fR"
.IP \fB--output=\fR\fIfilename\fR
.PD
Give the name of the output file.  The default is \fB--output=partialator.hkl\fR.

.PD 0
.IP "\fB-y\fR \fIpointgroup\fR"
.IP \fB--symmetry=\fR\fIpointgroup\fR
.PD
Merge according to symmetry \fIpointgroup\fR.

.PD 0
.IP "\fB-n\fR \fIn\fR"
.IP \fB--iterations=\fR\fIn\fR
.PD
Run \fIn\fR cycles of scaling and post refinement.

.PD 0
.IP \fB--no-scale\fR
.PD
Disable the scaling part of the refinement calculation.

.PD 0
.IP \fB--no-Bscale\fR
.PD
Disable the Debye-Waller part of the scaling calculation.

.PD 0
.IP \fB--no-pr\fR
.PD
Disable the orientation/physics model part of the refinement calculation.

.PD 0
.IP \fB--no-deltacchalf\fR
.PD
Disable rejection based on deltaCChalf.

.PD 0
.IP "\fB-m\fR \fImodel\fR"
.IP \fB--model=\fR\fImodel\fR
.PD
Specify the partiality model.  See the list below for possible choices.

.PD 0
.IP "\fB-j\fR \fIn\fR"
.PD
Run \fIn\fR analyses in parallel.

.PD 0
.IP \fB--polarisation=\fItype\fR
.PD
Specify the polarisation of the incident radiation.  \fItype\fR can be \fBhoriz\fR or \fBvert\fR to indicate 100% polarisation of the electric field in the horizontal plane or vertical plane respectively.  Setting \fItype\fR to \fBnone\fR completely disables the polarisation correction (see the note below).  Alternatively, \fItype\fR can be a direction followed by a percentage polarisation fraction.  For example, \fB45deg90\fR means that 90% of the radiation is polarised with its electric field in a direction 45 degrees from horizontal, and \fB10deg100\fR means that all the radiation is polarised at 10 degrees from horizontal.  The angle is specified clockwise from horizontal as viewed along the beam direction, i.e. as shown by \fBhdfsee\fR.  The beam is unpolarised when the fraction is 50% (equal parts of the radiation have their electric field in the specified plane).  If the polarisation fraction is 100%, it can be omitted.  For example \fB10deg\fR or \fBhoriz\fR.

Note that \fB--polarisation=none\fR is not the same as, for example, \fB--polarisation=vert50\fR.  In the first case, the polarisation correction will be completely disabled.  In the other case, the incident beam will be unpolarised, but the polarisation of the diffracted radiation will still be corrected for (the factor of (1+cos^2(2theta))/2 or, equivalently, (2-sin^2(2theta))/2).

The default is \fB--polarisation=horiz\fR.

.PD 0
.IP \fB--no-polarisation\fR
.PD
Synonym for \fB--polarisation=none\fR.

.PD 0
.IP \fB--max-adu=\fR\fIn\fR
.PD
Include reflections only if their peak values were less than \fIn\fR.  That means, \fIn\fR is the saturation value of the detector.  The default is infinity, i.e. no cutoff.

.PD 0
.IP \fB--min-res=\fR\fIn\fR
.PD
Merge crystals only if they diffract to beyond \fIn\fR Angstroms resolution.  The default is infinity, i.e. all crystals are included.  The resolution is taken from the diffraction_resolution_limit line in the stream.

.PD 0
.IP \fB--min-measurements=\fR\fIn\fR
.PD
Include a reflection in the output only if it appears at least least \fIn\fR times.  The default is \fB--min-measurements=2\fR.

.PD 0
.IP \fB--push-res=\fIn\fR
.PD
Merge reflections which are up to \fIn\fR nm^-1 higher than the apparent resolution limit of each individual crystal.  \fIn\fR can be negative to merge \fIlower\fR than the apparent resolution limit.   The default is \fB--push-res=inf\fR, which means no resolution cutoff at all.

.PD 0
.IP \fB--start-after=\fR\fIn\fR
.PD
Ignore the first \fIn\fR crystals in the input.  The default is \fB--start-after=0\fR, i.e. start at the beginning.

.PD 0
.IP \fB--stop-after=\fR\fIn\fR
.PD
Stop processing after \fIn\fR crystals have been successfully read.  The default is \fB--stop-after=0\fR, which means to process all the patterns from the start point to the end of the input (see \fB--start-after\fR).

.PD 0
.IP \fB--no-free\fR
.PD
Disable cross-validation (for testing only).

.PD 0
.IP \fB--custom-split=\fIfilename\fR
.PD
Read a list of filenames, event IDs and dataset IDs from \fIfilename\fR.  See the section on \fBCUSTOM DATASET SPLITTING\fR below.

.PD 0
.IP \fB--max-rel-B=\fIn\fR
.PD
Reject crystals if the absolute values of their relative Debye-Waller ("B") factors are more than \fIn\fR square Angstroms.  The default is \fB--max-rel-B=100\fR.

.PD 0
.IP \fB--output-every-cycle\fR
.PD
Write out the per-crystal parameters and reflection lists after every cycle of refinement, instead of only at the end.  The intermediate reflection lists and parameter filenames will be prefixed with \fBiter\fIN\fB_\fR (note the trailing underscore), where \fIN\fR is the iteration number.  If you use \fB--custom-split\fR, intermediate results will also be output for each custom dataset.

.PD 0
.IP \fB--no-logs\fR
.PD
Do not write the extensive log files needed for plotting contour maps and spectrum graphs.  This makes the process a lot faster, but you probably do want these logs to check that post-refinement is working reasonably.

.PD 0
.IP "\fB-w\fR \fIpg\fR"
.PD
Set the apparent symmetry of the crystals.  The ambiguity operator will be determined by comparing this to the actual symmetry.
.IP
If you prefer (or the scenario demands it), you can specify the ambiguity operator directly using \fB--operator\fR.

.PD 0
.IP \fB--operator=\fR\fIop\fR
.PD
Specify the indexing ambiguity operator.  Example: \fB--operator=k,h,-l\fR.
.IP
If you prefer, you can specify the ambiguity operator by specifying the apparent symmetry using \fB-w\fR.

.PD 0
.IP \fB--force-bandwidth=\fIbw\fR
.IP \fB--force-radius=\fIR\fR
.IP \fB--force-lambda=\fIl\fR
.PD
Set the X-ray bandwidth, initial profile radius or wavelength for all crystals before proceeding, overriding the values from the stream.  Bandwidth is given as a fraction, i.e. \fB--force-bandwidth=0.0013\fR means 0.13 percent (approximate FWHM).  Radius is given in nm^-1.  Wavelength is given in Angstroms.

.SH PARTIALITY MODELS

The available partiality models are:

.IP \fBxsphere\fR
.PD
The volume of intersection between a sphere centered on each reciprocal lattice point and the part of reciprocal space excited by the Ewald sphere taking into account the finite bandwidth of the incident X-rays.  A "source coverage factor" is included to take into account the spectral brightness of the effective source for the reflection.  The X-ray spectrum is modelled as a super-Gaussian with exponent 1.5, and the overlap integral is evaluated numerically.

This model is the same as that described in Acta Cryst. D71 (2015) p1400.

.IP \fBunity\fR
.PD
Fix all partialities at 1.

.SH USAGE CASES

.IP "Merging without scaling, partialities or post-refinement:"
.PD
partialator -i \fImy.stream \fR-o \fImy.hkl\fR -y \fImypointgroup \fB--model=unity --iterations=0\fR

.IP "Merging without partialities or post-refinement, but with scaling:"
.PD
partialator -i \fImy.stream \fR-o \fImy.hkl\fR -y \fImypointgroup \fB--model=unity --iterations=1\fR
.IP
(Use a higher number of iterations to increase the accuracy of scaling, but at a cost of more CPU time and possibly more rejected crystals)

.IP "Merging with partialities, but without post-refinement and without scaling:"
.PD
partialator -i \fImy.stream \fR-o \fImy.hkl\fR -y \fImypointgroup \fB--model=xsphere --iterations=0\fR

.IP "Merging with partialities, with scaling but without post-refinement:"
.PD
partialator -i \fImy.stream \fR-o \fImy.hkl\fR -y \fImypointgroup \fB--model=xsphere --iterations=1 --no-pr\fR

.IP "Merging with partialities, post-refinement and scaling:"
.PD
partialator -i \fImy.stream \fR-o \fImy.hkl\fR -y \fImypointgroup \fB--model=xsphere --iterations=1\fR
.IP
(Use a higher number of iterations to increase the accuracy of scaling and post-refinement, but at a cost of more CPU time and possibly more rejected crystals)

.IP "Merging with partialities and post-refinement, but without scaling:"
.PD
This would be a strange thing to want to do, however:
.IP
partialator -i \fImy.stream \fR-o \fImy.hkl\fR -y \fImypointgroup \fB--model=xsphere --iterations=1 --no-scale\fR
.IP
(Use a higher number of iterations to increase the accuracy of post-refinement, but at a cost of more CPU time and possibly more rejected crystals)

.SH CUSTOM DATASET SPLITTING
When performing a time-resolved experiment (for example), it is preferable to ensure that the data for all time points has been processed identically.  Rather than processing each time point independently with separate runs of partialator, it is better to process them all together and do the splitting into time points just before the final output.  Consider, for example, the case of simple scaling (without a B factor): when merging independently, the resulting datasets would probably end up with different overall scaling factors.  When comparing the results, you would need to take this difference into account.  In practice, most programs can do that job easily, but what about if a B factor is included?  And what if partialities are included - how unique is the solution?

With \fBpartialator --custom-split\fR, you can provide a separate text file containing a list of filenames, event numbers and \fIdataset names\fR, one event (detector frame) per line, with the fields separated by any number of spaces, commas or tabs.  For each unique \fIdataset name\fR, a separate reflection list will be output.  All crystals will be refined together, but they will be merged according to the dataset names you give.  The parameters (scaling factors, partialities etc) determined during the joint refinement will be applied.  For each dataset, a separate pair of split half-datasets will also be written, allowing you to calculate figures of merit such as Rsplit and CC1/2 for each one.

If the overall output filename (given with \fB-o\fR or \fB--output\fR) were \fBmerged.hkl\fR, then a dataset named \fIdataset\fR would be written to \fBmerged-\fIdataset\fB.hkl\fR.  The corresponding half-datasets would be written to \fBmerged-\fIdataset\fB.hkl1\fR and \fBmerged-\fIdataset\fB.hkl2\fR.

Note that the filenames and event names must match \fBexactly\fR what is written into the stream as the \fBImage filename\fR and \fBEvent\fR, taking into account options such as \fBindexamajig --prefix\fR and \fB--basename\fR.  You should therefore check that the numbers of crystals in each dataset, which will be written on the terminal by partialator, match your expectations and that no patterns have been "lost".  There is no requirement for every event in the list to appear in the stream, nor for every event in the stream to belong to one of the datasets.  If an event is listed for more than one dataset, the results are "undefined".

If you do not have event IDs for your data, i.e. if you have one detector frame per input file, simply leave out the event IDs from the custom split file.

Finally, note that the main and all custom split datasets, and also all the half-datasets, are subject to --min-measurements.

.SH AUTHOR
This page was written by Thomas White.

.SH REPORTING BUGS
Report bugs to <taw@physics.org>, or visit <http://www.desy.de/~twhite/crystfel>.

.SH COPYRIGHT AND DISCLAIMER
Copyright © 2012-2020 Deutsches Elektronen-Synchrotron DESY, a research centre of the Helmholtz Association.
.P
partialator, and this manual, are part of CrystFEL.
.P
CrystFEL is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
.P
CrystFEL is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more details.
.P
You should have received a copy of the GNU General Public License along with CrystFEL.  If not, see <http://www.gnu.org/licenses/>.

.SH SEE ALSO
.BR crystfel (7),
.BR indexamajig (1),
.BR process_hkl (1),
.BR partial_sim (1)