path: root/ChangeLog

Changes in this development version
-----------------------------------

- New tutorial added (doc/articles/tutorial.rst)
- Add Dockerfile and container CI setup (Silvan Schön)
- Added scripts/install-indexers
- Fixed peak coordinates when loading a stream (#66)
- indexamajig: fixed --wait-for-file (#65)
- indexamajig: re-implement and improve --profile
- indexamajig: Return value 5 if no files could be processed
- indexamajig: Fix harvest file JSON syntax (#68)
- indexamajig: Put MsgPack peaks through validation procedure
- indexamajig: Fix ZMQ request-reply sequence
- indexamajig: Refuse to overwrite a stream
- indexamajig: Avoid trouble with negative peaks in prediction refinement
- indexamajig: Add ASAP::O and Seedee interfaces
- indexamajig: Improve shutdown at end of data
- partialator: Use default event ID when none given in --custom-split
- Dropped support for pre-2015 stream formats
- Recognise '.nx5' as HDF5
- GUI: Add custom icons for tasks
- GUI: Optionally show resolution rings
- GUI: Use sbatch/scancel/scontrol instead of SLURM API
- GUI: Honour fixed profile radius for single frame indexing (#61)
- GUI: Better handling of error-filled indexamajig logs
- GUI: Fix stream re-scan logic (#64)
- GUI: Fix scrolling on Wayland
- GUI: Add 'desktop' file (fixes window icons on Wayland)
- GUI: Correct a mis-labelled field (Jerome Kieffer)
- Build system: Allow building without HDF5
- Build system: Disable OpenCL by default


CrystFEL version 0.10.1, 14 January 2022
----------------------------------------

- GUI: Proper overlines for reflection indices
- GUI: Allow import of multiple streams at once
- GUI: Fix for image panning after change of beam centre
- GUI: Fix for pixel value display at high zoom with "slanty" pixels
- GUI: Addition of checkbox "Rescan streams wihen changing frame"
- GUI: Crystal and project names will be added to MTZ
- GUI: Copy environment from parent process for SLURM jobs
- GUI: Addition of SLURM time limit field
- Build system: Auto-compilation of dependencies when using Meson
- Build system: Fix detection of OpenCL and SLURM
- Overhaul Doxygen API docs
- Geometry files: Wavelength (or photon energy etc) must be specified
- Geometry files: Experiment-space directions of fs/ss vectors must be specified
- Geometry files: Removal of rail direction
- Geometry files: Addition of mask_edge_pixels
- indexamajig: Switch from pixel distance to fraction of cell axis, when pairing peaks
- indexamajig: Add --data-format, implement HDF5s from online data source
- indexamajig: Shut down cleanly on SIGUSR1 (useful for online data)
- indexamajig: Generate per-frame profiling data
- MTZ export: Split into "plain" and "Bijvoet" modes of operation
- XDS export: Bug fix (missing 'S'!)
- compare_hkl: Bug fix for FoM reflection pair selection
- partialator: Handle spaces in custom split file
- partialator: Gracefully handle zero crystals in split dataset


CrystFEL version 0.10.0, 16th August 2021
-----------------------------------------

- New GUI ('crystfel') added, replacing 'hdfsee' which has been removed
- Added ZeroMQ interface for streaming data processing
- Added Meson build system
- get_hkl can now write XDS and MTZ directly
- Removed geoptimiser
- Removed wide-bandwidth features
- Added 'parameter harvest files' to indexamajig and partialator
- Better handling of static masks with multi-event files
- Added 'FromFile' indexing
- Added scripts/display-hdf5 and scripts/make-csplit
- New DataTemplate API
- New Stream API, allowing random access and better recreation of processing
- Bug fix for asdf indexing, vastly increasing speed and success rate
- Big speed improvement for TakeTwo
- Speed improvement for detector pixel coordinate transformation
- Speed improvement for UnitCell
- Speed improvement for creation of bad pixel maps from bad regions
- Fixed a large memory leak in cell_tool
- Fix large number of "rejected" reflections
- Support for units in geometry file distances (eg 'clen = 43 mm')
- Support for wavelength and electron accelerating voltage in geometry file
- Added detector_shift_x/y to geometry file
- Added peak_list_type to geometry file
- Added flag_{lessthan,morethan,equal} to geometry file
- Added peak_list to geometry file (no more --hdf5-peaks)
- Added requirement for fs/ss bad regions to name a panel
- indexamajig: Added --no-image-data and --no-mask-data
- partialator, process_hkl: Handle multiple input streams
- partialator: Reject crystals with large profile radii
- indexamajig: Added --wavelength-estimate and --camera-length-estimate
- indexamajig: Added --max-indexer-threads
- Figure of merit calculations moved to libcrystfel
- Removed badrow culling
- compare_hkl: Removed --intensity-shells


CrystFEL version 0.9.1, 22nd June 2020
--------------------------------------

- indexamajig: Fixed default unit cell tolerance (was 1.5 rad, now 1.5 deg)
- Fixed bug in unit cell reduction/comparison
- Improved the sigma estimate for very weak reflections
- Fixed bug when indexing with rhombohedral R cells
- indexamajig: Added --pinkIndexer-max-refinement-disbalance
- Fixed two problems when reading metadata values from HDF5
- partialator: Gracefully handle negative and NaN sigma values
- scripts/transfer-geom: Now only writes coffset for panels which had
  coffset values before
- Fixed/improved pinkIndexer documentation and error messages
- Fixed build with zlib < 1.2.3.5 (missing gzbuffer routine)
- Fixed build with HDF5 >= 1.12
- scripts/mtz2hkl, cif2hkl, hkl2hkl: Added missing header


CrystFEL version 0.9.0, 6th February 2020
-----------------------------------------

- Introduction of cell_tool
- Addition of PinkIndexer interface
- Handle CBF files natively, not via CBFlib
- Add Spectrum API
- New symmetry operation/cell transformation parser
- New unit cell comparison algorithm based on Niggli reduction
- New prediction calculation which works for electrons and wide bandwidth
- indexamajig: Removed --no-cell-combinations
- indexamajig: Improvements to indexing result check ("--check-peaks"),
  improving results with multi-lattice indexing
- indexamajig: Add --zmq-msgpack
- indexamajig: Allow individual tolerances for all six parameters
- indexamajig: Return non-zero exit status if no patterns were processed
- indexamajig: Rename --min-gradient to --min-squared-gradient
- indexamajig: Status message updates
- indexamajig: Bandwidth now specified in geometry file (not --fix-bandwidth)
- indexamajig: Individual tolerances for each cell parameter
- indexamajig: Remove 'rescut' integration flag
- cell_explorer: Allow enforcement of Bravais lattice type
- ambigator: Use pre-defined resolution bins if limits not given
- geoptimiser: Use geometry from stream
- partialator: Many stability improvements to post-refinement
- partialator: Parallelise deltaCChalf calculation
- partialator: Add --model=offset for monochromatic synchrotron data
- partialator/process_hkl: Allow arbitrary polarisation direction and degree


CrystFEL version 0.8.0, 19th December 2018
------------------------------------------

- Add support for peakfinder9
- Add support for XGANDALF indexer
- Switch to CMake build system
- Handle 16bpp and gzipped CBF files
- Introduce make_pixelmap as a core CrystFEL program
- indexamajig: Add --wait-for-file
- indexamajig: Clean up temporary folders on clean exit
- indexamajig: Name and shame slow processing stages
- indexamajig: Warn if a frame takes more than 10 minutes to process
- Multiple fixes for XDS indexer interface
- Fix for indexing centered cells with Mosflm without prior cell parameters
- partialator: Fixes and improvements to post-refinement
- partialator: Add deltaCChalf rejection test
- Add scripts: euxfel-train-analysis, find-multiples and transfer-geom
- detector-shift: Click to choose different center
- check_{peak_detection,near_bragg}: Warn if stream is older than image
- Some fixes for peakfinder8
- turbo-index-lsf: Match turbo-index-slurm as closely as possible
- sum-peaks: Accept "-" for stdin
- geoptimiser: Fix calculation of displacement error
- check_hkl: Avoid NaN result for L-test with --zero-negs
- compare_hkl: Fix an edge case in calculation of anomalous figures of merit
- Handle numbers (e.g. "k-2h") in symmetry operator specifications (e.g.
  ambigator)
- Use GTK3 if available


CrystFEL version 0.7.0, 17th May 2018
--------------------------------------

- indexamajig: Auto-detection of indexing methods
- indexamajig: New options system - removes confusing "comb","bad","retry" etc.
- Most programs now use the symmetry in the reflection files, allowing "-y"
  options to be omitted.
- Ginn partiality model was added.
- partialator: Switch to gradient-free minimiser.
- geoptimiser: Fixes for rotation and new features for AGIPD.
- Speed and accuracy improvements for TakeTwo indexing (Helen Ginn).
- process_hkl, partialator and ambigator now include the audit information from
  the stream in their output files.
- pattern_sim: Write orientation and lattice basis vectors into HDF5 file.
- hdfsee: Add event ID and peaks/preds/rings to exported image filenames.
- indexamajig --serial-start was added (useful when using turbo-index).
- indexamajig --overpredict was added (needed for post-refinement)
- pattern_sim --flat was added (new reflection profile option)
- cell_explorer: Now shows the number of matching cells on terminal.
- Felix options were simplified.
- scripts/sum-hdf5-files and eiger-badmap were added


CrystFEL version 0.6.3, 24th July 2017
--------------------------------------

- indexamajig --peaks=peakfinder8 was added (Valerio Mariani, Oleksandr Yefanov)
- CBF files are now supported as input.
- TakeTwo indexing was incorporated (Helen Ginn)
- indexamajig --min-peaks and --no-non-hits-in-stream were added
  (basic hitfinder functionality)
- indexamajig: Hung worker processes will now be detected and killed.
- Peak locations from HDF5 or CXI files are now offset by 0.5,0.5 (needed for
  peaks from Cheetah and previously neglected).
- Detector panels no longer need to be perpendicular to the X-ray beam.
- Detector "rail" direction ("camera length" axis) no longer needs to be along
  the X-ray beam direction.
- Lattice parameters are now checked after prediction refinement as well as before.
- Multi-event HDF5 files can now contain peak lists.
- The peak list location can now be given in geometry file (instead of --hdf5-peaks)
- The number of detector units per photon (rather than per eV) can now be
  specified in the geometry file.
- indexamajig: Small changes to how peaks from HDF5 files are checked.
- compare_hkl --fom=d1sig and d2sig were fixed (Wolfgang Brehm).
- compare_hkl --min-measurements was added.
- New polarisation correction.
- Reflection data files now contain audit information (CrystFEL version number
  and command line parameters).
- Improvements to enumeration of events in multi-event files
- scripts/detector-shift: Show a heat map, handle different panel resolutions
  correctly
- Add scripts/move-entire-detector, scripts/split-by-mask,
  scripts/peakogram-stream and scripts/sum-peaks.
- scripts/turbo-index was renamed to turbo-index-lsf, and turbo-index-slurm
  was added (Steve Aplin).
- An example geometry file for Eiger was added.
- Multiple bug fixes for asdf indexing
- cell_explorer: Fixed invisible Greek letters on some systems
- Mask paths with fewer placeholders than data paths (e.g. some SACLA files)
  are now handled properly.
- cell_explorer: Added "Save cell" function.
- pattern_sim: Multiple bug fixes.
- geoptimiser: Multiple bug fixes.
- partialator: Fixes for some edge case bugs.
- indexamajig: Fix files left open when using XDS indexing.
- {check,compare}_hkl: Add warnings for potentially silly option choices.
- Remove "data slab" from detector data handling (simplifies internal data
  structures a lot).
- Python scripts were made compatible with Python 3.


CrystFEL version 0.6.2, 21st March 2016
---------------------------------------

- partialator: Stability improvements to scaling algorithm, default --push-res
  value changed from zero to infinity (no resolution cutoff by default).
- partialator: Added --custom-split, --max-rel-B and --output-every-cycle
- Added "retry" indexing method to improve indexing rate.
- Per-pixel saturation values can now be used (needed for CSPAD dual-gain mode).
- Fixed a regression with indexing using MOSFLM introduced near 0.6.1.
- indexamajig: Fixed a bug which occasionally caused corrupted stream output
  ("Failed while reading reflections").
- indexamajig: gradient fitting of background now optional and OFF by default.
- indexamajig: Added --profile for speed monitoring.
- indexamajig: Added "felix" indexing method.
- scripts/ave-resolution: Show resolution histogram.
- scripts/turbo-index: Added a missing line
- hdfsee: Improve quality of display by rounding pixel coordinates.
- hdfsee: Can now safely save new geometry file over old one.
- Target unit cell is now written to the stream.
- pattern_sim: Fixed bugs in --gpu and --powder.
- compare_hkl: Avoid double counting of reflections when calculating anomalous
  figures of merit.
- geoptimiser: Major refactoring
- ambigator: fixes for a few problem when using very large numbers of correlation
  coefficients or patterns.
- Some crashes and hangs in asdf were fixed.
- The "random" partiality model was improved.
- Prototype for prepare_indexing() changed, otherwise only additions were made
  to the API.


CrystFEL version 0.6.1, 3rd August 2015
---------------------------------------

- partialator: big improvements, now works well on most data sets.
- indexamajig: prediction refinement added.
- "mosflm" indexing method now uses prior cell information when used with
  mosflm version 7.2.0 or later.
- "asdf" indexing method added, which needs no external program.
- process_hkl --even-only and --odd-only were added.
- External bad pixel masks can now be used (see "mask_file" in geometry file).
- indexamajig: Can now use --int-diag with -j n where n>1
- Image data filters (e.g. --noise-filter) were fixed.
- indexamajig sandbox was redesigned and made more robust.
- compare_hkl now uses a B-factor for scaling the datasets.
- pattern_sim: spectrum normalisation added.
- scripts/ave-resolution and scripts/create-frame-number added.
- scripts/plot-predict-refine and scripts/plot-radius-resolution added.
- scripts/detector-shift added.
- scripts/turbo-index added.
- "reax" and "grainspotter" indexing methods removed.
- Many other small but important bug fixes and improvements to data quality.


CrystFEL version 0.6.0, 23rd February 2015
------------------------------------------

- The geometry specification was extended to allow multiple events per HDF5
  file as well as different panels in different HDF5 locations.
- geoptimiser, list_events and whirligig were added.
- CrystFEL unit cell files were introduced.
- Beam parameter files are no longer used.
- indexamajig auto-determination of prediction parameters was added.
- indexamajig --fix-profile-radius, --fix-bandwidth and --fix-divergence were
  added.
- pattern_sim --beam-bandwidth, --photon-energy and --nphotons were added.
- partial_sim --beam-bandwidth, --photon-energy, --beam-divergence and
  --profile-radius were added.
- Event serial numbers are now written in streams.
- indexamajig -e/--image was removed.
- process_hkl --stat and --min-cc were added.
- render_hkl --highres and --no-axes were added.
- partialator --max-adu was added.
- 2D profile fitting (--integration=prof2d) was fixed.
- CSPAD geometry file examples were improved.
- compare_hkl --fom=d1sig and d2sig were added.
- pattern_sim now simulates detector saturation.
- Compression was switched off when saving HDF5 files.
- The units conversion for the resolution limit written into the stream was
  fixed.
- The pre-merging Poission error calculation was improved.
- A bug in pattern_sim's handling of symmetry was fixed.
- {check,compare}_hkl now report the min/max bin ranges.


CrystFEL version 0.5.4, 5th September 2014
------------------------------------------

- hdfsee calibration mode, ADSC export and scrollbars were added.
- Bad regions can now be specified as fs/ss as well as x/y.
- "--version" options were added to everything.
- partial_sim --background was added.
- partialator --reference, --geometry and --beam were removed.
- ambigator --corr-matrix was added.
- indexamajig --peak-radius and --highres were added.
- Handling of bad pixels and overlapping reflections was improved.
- ambigator now updates the lattice basis vectors as well as the indices.
- pattern_sim can now simulate all point groups on the GPU.
- Unit cell output on the command line now uses Angstroms instead of nm.
- Beam/crystal properties and mean camera length were added to the stream.
- scripts/gen-sfs-expand was added.
- Stream reading/writing API was changed and improved.
- libcrystfel was made compatible with C++.
- get_symop(), get_matrix_name() mark_resolution_range_as_bad() were added.
- Prototypes changed for find_panel_number(), *_prepare() (all indexing
  methods), find_intersections(), image_feature_closest(),
- {get,set}_{refineable,scalable}() were removed.
- integrate_peak() was removed from the exposed API.


CrystFEL version 0.5.3a, 14th April 2014
----------------------------------------

- A problem with ambigator's option processing was fixed.


CrystFEL version 0.5.3, 9th April 2014
--------------------------------------

- ambigator and cell_explorer were added.
- get_hkl -w was fixed, and the option --no-need-all-parts added.
- pattern_sim was overhauled: the ability to simulate a finite bandwidth was
  restored, and a SASE or two-colour spectrum can now be simulated.
- All programs which use a random number generator now use a proper random
  number generator instead of random().
- partial_sim --really-random and --images were added.
- pattern_sim --template and --no-fringes were added.
- check_hkl --ltest and --wilson were added.
- process_hkl --max-adu and --min-res were added.
- compare/check_hkl --lowres and- -highres were added.
- The "rescut" integration flag was added.
- The stream format was changed to include peak heights and mean backgrounds.
- scripts/fg-graph and truncate-stream were added.
- record_image(), random_quaternion(), random_flat(), flat_noise(),
  gaussian_noise() and poisson_noise() now take an additional gsl_rng argument.
- validate_peaks() now takes an additional check_snr argument.


CrystFEL version 0.5.2, 4th December 2013
-----------------------------------------

- Reflection sigmas were improved for negative intensities, and the behaviour
  of "process_hkl --min-snr" was rationalised.
- Peak centering ("--integration=rings-cen") was fixed.
- XDS default behaviour was fixed to prevent an unexpected axis permutation.
- Incorrect handling of monoclinic C cells was fixed.
- The counting of saturated peaks was fixed.
- hdfsee now needs the beam file (use option -m).
- A more than 2 year old bug was fixed in RefList.
- indexamajig --int-diag was added.
- render_hkl --res-ring was added.
- compare_hkl --fom=Rano and Rano/Rsplit was added.
- check_hkl --shell-file was added.
- get_hkl --cutoff-angstroms was added.
- partialator was extensively worked on.
- scripts/split-indexed was added.
- scripts/Rsplit_surface and clean-stream.py were added (Fedor Chervinskii).
- A locale-related problem with hdfsee was fixed.
- Histogram, update_partialities_2(), hdfile_is_scalar() and show_matrix() were
  added to the libcrystfel API.
- integrate_all() was changed to incorporate IntDiag.
- show_matrix_eqn() was simplified, removing a parameter.


CrystFEL version 0.5.1, 4th July 2013
-------------------------------------

- New integration framework in indexamajig - see "--integration" option.
- The "Lorentz factor" was added to partial_sim (and partialator).
- CCano was added as a figure of merit to compare_hkl.
- Anomalous correlation ratio (CRDano) was added as a figure of merit to
  compare_hkl (Lorenzo Galli).
- XDS indexing was improved.
- The calculation of sigma(I) in process_hkl was made more numerically stable.
- The polarisation correction in process_hkl was fixed.
- The display of feature names (i.e. Miller indices) in hdfsee was fixed.
- partialator was extensively worked on.
- A compilation failure (when FFTW3 was unavailable) was fixed.
- docs/stream-format.txt was added.
- scripts/create-xscale was fixed (sigma(I) values were wrong).
- Warnings were added to scripts which MUST be modified before use.
- polarisation_correction() was added to the API.


CrystFEL version 0.5.0, 19th April 2013
---------------------------------------

- The internal architecture has been changed to accommodate multiple crystals
  per diffraction pattern.
- The stream format has changed as a result of the above.
- The indexing logic has reworked to make it easier to control what information
  is used for indexing and to allow the user to try more indexing options in
  turn.
- Saturated peaks and integrated reflections are now counted and the number
  included in the stream.
- GrainSpotter indexing has been added.
- XDS indexing has been added.
- The "--nshells" argument was added to check_hkl.
- The "--median-filter" argument was added to indexamajig and hdfsee.
- The "--filter-cm" argument was removed from indexamajig and hdfsee.
- check-near-bragg and check-peak-detection can now be given extra command-line
  options which will be passed through to hdfsee.
- An error in the solid angle calculation was fixed, affecting simulation with
  pattern_sim (Chun Hong Yoon).
- Lattice type, centering and unique axis information is now included in the
  stream.


CrystFEL version 0.4.3, 10th January 2013
-----------------------------------------

- compare_hkl was reworked to simplify the available figures of merit, add CC*,
  and to be consistent between the overall and resolution shell values.  A new
  option, --intensity-shells, was also added.
- Unit cells can now be handled in any setting.
- Indexamajig will now examine the unit cell given on the command line, even if
  the final cell will not be checked (i.e. if --cell-reduction=none).  The
  lattice type will then be given to MOSFLM, which can help a lot in finding the
  right solution.
- Indexamajig now records the number of saturated reflections in the stream.
- Indexamajig now processes the peaks obtained using "--peaks=hdf5" to check for
  bad detector regions, peaks too close to panel edges, etc.
- New options "--integrate-saturated", "--use-saturated" and "--integrate-found"
  were added to indexamajig.
- A bug was fixed which caused systematically absent (due to centering)
  reflections to be predicted and integrated.
- The colour scaling in hdfsee was improved.
- powder_plot was moved to the extra programs repository.
- The configure script now checks for HDF5.
- Forbidden (due to centering) reflections are now taken into account by
  check_hkl when calculating the completeness.
- The speed of pattern processing was increased significantly (in many cases) by
  avoiding an unnecessary iteration.
- A crystfel.pc file was added, allowing use of "pkg-config --cflags crystfel"
  and so on.
- The new option "--min-measurements" was added to process_hkl.
- The wavelength can now be specified in the beam file as an HDF5 path or as an
  explicit number, just like the camera length.  There is no longer any
  "fallback" to a nominal value (Chun Hong Yoon).


CrystFEL version 0.4.2, 2nd October 2012
----------------------------------------

- A data quality regression, introduced between versions 0.3.0 and 0.4.0, was
  fixed.
- A speed regression, introduced at the same time as the data quality
  regression, was also fixed (approximately a factor of 2 speed increase with
  indexamajig).
- Centered unit cells are now handled correctly during unit cell reduction.
- The same colour scale is now used for every panel by hdfsee.
- "get_hkl -w" was fixed.
- A spurious "-p" option was removed from scripts/gen-sfs-ano.
- Option handling across all programs was fixed to prevent confusion.


CrystFEL version 0.4.1, 29th August 2012
----------------------------------------

- Some compiler warnings with newer versions of GCC were fixed.
- Bad pixels are now shown in brown by hdfsee.
- Bad pixels are now ignored when calculating the range of the image for the
  colour scale in hdfsee.
- A crash when there were too many unit cell candidates was fixed.
- Small typos were fixed in a warning message and pattern_sim(1).
- "indexamajig --image" was fixed.


CrystFEL version 0.4.0, 31st July 2012
--------------------------------------

- Performance increase when using indexamajig on multi-core hardware (Chun Hong
  Yoon).
- "max_adu" field was added to geometry file format.
- Added masking of inner ("peak") regions of all peaks during the background
  calculation (Kenneth Beyerlein).
- Bandwidth and subsampling was restored to the GPU version of pattern_sim.
- Spot prediction and associated partiality calculations were improved.
- "profile_radius" field was added to beam file format.
- "compare_hkl --shells" now uses separate denominators in each resolution bin
  for the R-factor calculation.
- Removed fields "peak_sep" and "integr_radius" in geometry files.
- Bug fix for process_hkl --histogram (Lorenzo Galli).
- A bug was fixed in find_panel().
- Improvements were made to the build system.
- "indexamajig --min-integration-snr" was fixed.
- Fixed "process_hkl --start-after" and "--stop-after".
- "compare_hkl --sigma-cutoff" added.
- Shell resolutions in Angstroms added to the shells files of check_hkl and
  compare_hkl.
- Fixed small compilation problem with Cairo < 1.6.
- The tolerances for the cell reduction procedure, "ltl" in match_cell(), is now
  "const float *ltl" instead of just "float *ltl".
- The "peak_sep" field of "struct panel" was removed, and "max_adu" added.
- "struct beam_parameters" now contains a new field: "profile_radius".
- "write_detector_geometry()" was added to detector.h.
- "hdf5_write_image()" was added to hdf5-file.h.


CrystFEL version 0.3.0, 14th March 2012
---------------------------------------

- Initial release of CrystFEL.