This is CrystFEL 0.10.10.10.1

7 files changed, 196 insertions, 228 deletions

diff --git a/CMakeLists.txt b/CMakeLists.txt
index df68ad18..965c7119 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -1,7 +1,7 @@
 cmake_minimum_required(VERSION 3.6)
-set(CRYSTFEL_SHORT_VERSION 0.10.0)
+set(CRYSTFEL_SHORT_VERSION 0.10.1)
 set(CRYSTFEL_VERSION ${CRYSTFEL_SHORT_VERSION})
-set(CRYSTFEL_API_VERSION 14)
+set(CRYSTFEL_API_VERSION 15)
 project(crystfel VERSION ${CRYSTFEL_VERSION} LANGUAGES C)
 
 cmake_policy(SET CMP0074 NEW)
diff --git a/ChangeLog b/ChangeLog
index addf5ee2..e1b8d1a9 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,32 @@
+CrystFEL version 0.10.1, 14 January 2022
+----------------------------------------
+
+- GUI: Proper overlines for reflection indices
+- GUI: Allow import of multiple streams at once
+- GUI: Fix for image panning after change of beam centre
+- GUI: Fix for pixel value display at high zoom with "slanty" pixels
+- GUI: Addition of checkbox "Rescan streams wihen changing frame"
+- GUI: Crystal and project names will be added to MTZ
+- GUI: Copy environment from parent process for SLURM jobs
+- GUI: Addition of SLURM time limit field
+- Build system: Auto-compilation of dependencies when using Meson
+- Build system: Fix detection of OpenCL and SLURM
+- Overhaul Doxygen API docs
+- Geometry files: Wavelength (or photon energy etc) must be specified
+- Geometry files: Experiment-space directions of fs/ss vectors must be specified
+- Geometry files: Removal of rail direction
+- Geometry files: Addition of mask_edge_pixels
+- indexamajig: Switch from pixel distance to fraction of cell axis, when pairing peaks
+- indexamajig: Add --data-format, implement HDF5s from online data source
+- indexamajig: Shut down cleanly on SIGUSR1 (useful for online data)
+- indexamajig: Generate per-frame profiling data
+- MTZ export: Split into "plain" and "Bijvoet" modes of operation
+- XDS export: Bug fix (missing 'S'!)
+- compare_hkl: Bug fix for FoM reflection pair selection
+- partialator: Handle spaces in custom split file
+- partialator: Gracefully handle zero crystals in split dataset
+
+
 CrystFEL version 0.10.0, 16th August 2021
 -----------------------------------------
 
diff --git a/announcement-0.10.0 b/announcement-0.10.0
deleted file mode 100644
index 75122e68..00000000
--- a/announcement-0.10.0
+++ /dev/null
@@ -1,43 +0,0 @@
-Dear CrystFEL users and interested crystallographers,
-
-CrystFEL version 0.10.0 has been released!
-
-This is a huge update to CrystFEL, adding a completely new graphical user
-interface (GUI) which makes it much easier to interact with your data, guiding
-you through the processing steps.  To give a preview of the GUI and help you
-get started, I've made a video tutorial to accompany this release:
-https://www.desy.de/~twhite/crystfel/presentations.html
-
-Electron diffraction is fully supported in this version of CrystFEL.  Amongst
-other things, you can now specify an electron accelerating voltage (instead of
-an X-ray photon energy) in the geometry file.  Combined with the 'diffractem'
-pre-processing software developed by the Max Planck Institute for the Structure
-and Dynamics of Matter, you have a complete solution for processing data from
-serial electron diffraction crystallography.  For more details, see:
-https://gitlab.desy.de/thomas.white/crystfel/-/blob/master/doc/articles/electrons.rst
-
-In addition, I've taken advantage of the extended quiet time to work on many
-internal parts of CrystFEL.  The most noticable benefit of this is that several
-parts of CrystFEL are now much faster.
-
-Some command-line parameters and geometry file directives have changed, but you
-should find that the error messages will direct you towards what needs to be
-updated.
-
-See the release notes and the ChangeLog for full details of the other
-improvements.
-
-Release notes for version 0.10.0:
-https://www.desy.de/~twhite/crystfel/relnotes-0.10.0
-
-Change log:
-https://www.desy.de/~twhite/crystfel/changes.html
-
-CrystFEL website:
-https://www.desy.de/~twhite/crystfel
-
-Thanks for all of your past and future feedback and contributions, and
-your continued use of CrystFEL.  As ever, please contact me directly if
-you would prefer to be unsubscribed from this mailing list.
-
-Tom
diff --git a/announcement-0.10.1 b/announcement-0.10.1
new file mode 100644
index 00000000..a0817dfd
--- /dev/null
+++ b/announcement-0.10.1
@@ -0,0 +1,26 @@
+Dear CrystFEL users and interested crystallographers,
+
+CrystFEL version 0.10.1 has been released!
+
+This release is primarily a bug-fixing update to CrystFEL.  There have been
+improvements around the entire suite, fixing several problems which came up in
+version 0.10.0.  Some new features have been added as well.  Please read the
+release notes for a summary of the most important changes:
+https://www.desy.de/~twhite/crystfel/relnotes-0.10.1
+
+Refer to the change log for a more comprehensive list of improvements:
+https://www.desy.de/~twhite/crystfel/changes.html
+
+Thanks for all of your past and future feedback and contributions, and your
+continued use of CrystFEL.  As always, there is much more information on the
+CrystFEL website: https://www.desy.de/~twhite/crystfel
+
+You can unsubscribe from this mailing list by simply sending me a short
+message. Alternatively, you can now subscribe and unsubscribe via the web
+interface: https://lists.desy.de/sympa/info/crystfel-announce
+
+With very best wishes,
+
+Tom
+
+
diff --git a/meson.build b/meson.build
index 7a5b6ed3..604b70a2 100644
--- a/meson.build
+++ b/meson.build
@@ -1,11 +1,11 @@
 # Meson file for CrystFEL
 project('crystfel', 'c',
-        version: '0.10.0',
+        version: '0.10.1',
         license: 'GPL3+',
         meson_version: '>=0.55.0',
         default_options: ['buildtype=debugoptimized'])
 
-libcrystfel_api_version = 14
+libcrystfel_api_version = 15
 
 add_project_arguments('-DHAVE_CONFIG_H', language: 'c')
 
diff --git a/relnotes-0.10.0 b/relnotes-0.10.0
deleted file mode 100644
index 2ebeebe6..00000000
--- a/relnotes-0.10.0
+++ /dev/null
@@ -1,181 +0,0 @@
-CrystFEL - Data processing for serial crystallography
------------------------------------------------------
-
-Release notes for version 0.10.0
-
-Copyright © 2012-2021 Deutsches Elektronen-Synchrotron DESY,
-                      a research centre of the Helmholtz Association.
-
-See AUTHORS as well as individual source code files for full details of contributors.
-
-CrystFEL is free software: you can redistribute it and/or modify it under the
-terms of the GNU General Public License as published by the Free Software
-Foundation, either version 3 of the License, or (at your option) any later
-version.
-
-CrystFEL is distributed in the hope that it will be useful, but WITHOUT ANY
-WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
-PARTICULAR PURPOSE.  See the GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License along with
-CrystFEL.  If not, see <http://www.gnu.org/licenses/>.
-
-
-Overview
---------
-
-The most important changes in this version of CrystFEL are:
-
-- New graphical user interface
-
-- Support for electron diffraction
-
-- Temporary removal of geoptimiser and pink beam support
-
-- Improved MTZ and XDS output
-
-- Big speed improvements
-
-- Improved options for bad pixel masking
-
-See below for information about each of these changes.
-
-In addition, there were many bug fixes and other improvements.  See the
-"ChangeLog" or the changes page on the CrystFEL website for details.
-
-
-New graphical user interface
-----------------------------
-
-This release adds a new graphical user interface to CrystFEL.  The old image
-viewer "hdfsee" has been removed entirely.  To start the GUI, simply run
-"crystfel" at the command line.  The GUI guides you through the stages of
-processing a dataset, from importing image data all the way to exporting
-merged data.  To help you get started, there is a video introduction:
-
-https://www.desy.de/~twhite/crystfel/presentations.html
-
-The "old" command-line way of processing data is still available.  The only
-difference is that you should now use the GUI to view the results graphically,
-instead of hdfsee and the old "check-near-bragg" and "check-peak-detection"
-programs.  Simply run the GUI, giving the stream filename as an argument, for
-example: "crystfel results-output.stream".
-
-
-Support for electron diffraction
---------------------------------
-
-This release adds support for serial crystallography using electrons.  For more
-information, see doc/articles/electrons.rst in the CrystFEL distribution:
-https://gitlab.desy.de/thomas.white/crystfel/-/blob/master/doc/articles/electrons.rst
-
-
-Temporary removal of geoptimiser and pink beam support
-------------------------------------------------------
-
-A lot of internal re-organisation and cleaning-up has taken place in this
-release cycle.  Due to a lack of development resources, two things fell by the
-wayside in order to make the release in a reasonable timescale: this release
-does not include "geoptimiser", nor support for wide bandwidth ("pink-beam")
-diffraction.
-
-But don't worry!  The next release will include an improved replacement for
-geoptimiser.  For the time being, note that the data formats used by CrystFEL
-have not changed much, so an older CrystFEL version (e.g. 0.9.1) can still be
-used for fine-grained geometry refinement in almost all cases.
-
-If you previously installed CrystFEL 0.9.1, you should find that geoptimiser
-from the old installation is still present and working.  Therefore, you might
-not even notice the difference!
-
-Features for processing data acquired with wide bandwidth radiation will be
-developed further if there is demand.
-
-
-Improved MTZ and XDS output
----------------------------
-
-This CrystFEL version can export directly to MTZ (CCP4) or XDS-ASCII format.
-Export to MTZ and XDS format can also be done via the "Export" function of the
-new CrystFEL GUI.  Alternatively, use get_hkl to convert the file, for example:
-
-$ get_hkl -i my-data.hkl -o my-data.mtz --output-format=mtz -p my-data.cell
-
-This replaces the old "create-mtz" and "create-xscale" scripts, and the
-resulting files are more consistent with the expectations of other software.
-
-
-Big speed improvements
-----------------------
-
-Several significant speed improvements have been made.  In particular, these
-affect 'asdf' and 'TakeTwo' indexing, unit cell transformations and loading
-image data.  You should notice a greatly reduced number of "timeouts" during
-indexing.
-
-
-Improved options for bad pixel masking
---------------------------------------
-
-More ways to mask bad pixels have been added.  You can now use one or more of
-the following directives in the geometry file to flag pixels with values lower
-than, equal or more than a given value: "flag_lessthan", "flag_morethan",
-"flag_equal".
-
-In addition, this version of CrystFEL is better able to handle the situation
-where a single bad pixel mask applies to multiple frames in a "multi-event"
-data file.  For example: with data in HDF5 format, if the location of the bad
-pixel mask contains fewer "placeholder" characters ("%") than the image data,
-only the required number of placeholders will be filled in when calculating the
-location of the mask.  This means that you can do something like this:
-
-data = /data/image/%/frames/%/pixel_values
-mask = /data/image/%/bad_pixel_mask
-
-In this case, one "bad_pixel_mask" would apply to everything under each set of
-"frames".
-
-
-API changes
------------
-
-The following changes have been made to the libcrystfel API.
-
-Additions:
-	- DataTemplate API (data_template_*, see datatemplate.h)
-	- Figure of merit API (fom_*, see fom.h)
-	- Stream API (stream_*, see stream.h)
-	- Image API (image_*, see image.h)
-	- 'struct detgeom' and associated API (detgeom_*)
-	- str_peaksearch and parse_peaksearch
-	- intcontext_new, integrate_rings_once, intcontext_free
-	- write_to_mtz, write_to_xds, libcrystfel_can_write_mtz
-	- parse_indexing_methods
-	- base_indexer_str
-	- default_method_options
-	- cell_print_oneline
-	- has_unique_axis
-	- lowest_reflection
-	- colscale_lookup (replaces 'render_scale')
-	- crystal_copy_deep
-	- Miscellaneous utility functions: compare_double, safe_strdup,
-	  load_entire_file, file_exists, filename_extension,
-	  set_log_message_func, el_v_to_lambda, CLEAR_BIT, convert_int,
-	  convert_float, ELECTRON_MASS
-
-Removals:
-	- 'struct detector' and associated API
-	- 'struct event' and associated API
-	- 'struct hdfile' and associated API (hdf5-file.h)
-	- 'struct beam_params'
-	- All direct interfaces to indexing engines, e.g. xds_prepare
-	- render_scale (renamed to colscale_lookup)
-	- stderr_lock
-	- {get,set}_panel (replaced by {get,set}_panel_number)
-	- INDEXING_DEBUG (use INDEXING_FILE instead)
-
-Changes:
-	- cell_get_* and cell_has_parameters (changes to 'const' qualifiers)
-	- struct image: removed 'parent', 'p', 'rx,ry,rz', added 'pn'
-	- estimate_peak_resolution (added 'det' arg)
-	- setup_indexing (added lots of new args)
diff --git a/relnotes-0.10.1 b/relnotes-0.10.1
new file mode 100644
index 00000000..d7388eea
--- /dev/null
+++ b/relnotes-0.10.1
@@ -0,0 +1,137 @@
+CrystFEL - Data processing for serial crystallography
+-----------------------------------------------------
+
+Release notes for version 0.10.1
+
+Copyright © 2012-2022 Deutsches Elektronen-Synchrotron DESY,
+                      a research centre of the Helmholtz Association.
+
+See AUTHORS as well as the individual source code files for full contributor details.
+
+CrystFEL is free software: you can redistribute it and/or modify it under the
+terms of the GNU General Public License as published by the Free Software
+Foundation, either version 3 of the License, or (at your option) any later
+version.
+
+CrystFEL is distributed in the hope that it will be useful, but WITHOUT ANY
+WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
+PARTICULAR PURPOSE.  See the GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License along with
+CrystFEL.  If not, see <http://www.gnu.org/licenses/>.
+
+
+Overview
+--------
+
+This release is primarily a bug-fixing update to CrystFEL, and there have been
+improvements around the entire suite.  The most important changes are detailed
+below.  See the ChangeLog or the Git history for a comprehensive list of all
+changes.
+
+
+Graphical user interface
+------------------------
+
+SLURM jobs started from the GUI will now inherit the entire environment of the
+GUI, including the values of 'PATH', 'HDF5_PLUGIN_PATH' and other variables.
+This matches the default behaviour of 'sbatch' and should make it easier to set
+up the environment correctly.  The text entry field entitled 'Search path for
+executables' for SLURM jobs has been removed.  Simply make sure that any extra
+locations (e.g. to find Mosflm, DirAx or XDS) are added to 'PATH' before
+starting the GUI, and the SLURM jobs should run correctly.
+
+Multiple streams can now be imported at once, to create one "result".  This is
+useful when you want to merge a set of streams inside the GUI, for example if
+you've previously used "turbo-index" to split the indexing and integration work
+into multiple batches.
+
+When exporting data to MTZ format, the crystal and project fields of the MTZ
+file will now be filled in.  The crystal name will be the name of the indexing
+run, the project name will be the name of the folder from which the GUI was
+started.  As before, the dataset name will be the name of the merging run.
+See the section 'MTZ and XDS export' below for information about other changes
+to reflection data export.
+
+A new checkbox "Rescan streams when changing frame" has been added to the
+"Tools" menu.  In certain circumstances, such as when an indexing job is
+running, un-ticking this box will make it faster to switch between frames in
+the GUI.  However, you will need to occasionally run "Rescan streams" to make
+sure that the GUI's display is up-to-date.
+
+
+Build system
+------------
+
+When building with Meson, several dependencies will now automatically be
+downloaded and compiled.  In this version, this applies to the following
+dependencies: FastDiffractionImageProcessing (FDIP), LibCCP4, PinkIndexer,
+XGandalf, ZLib and Eigen (Eigen is a dependency of XGandalf and PinkIndexer).
+You can disable this feature if you don't want it - see INSTALL.md or the Meson
+manual for details.  It's also possible to use this feature for a restricted
+selection of dependencies.
+
+Bugs in the detection of certain dependencies (OpenCL and SLURM) have been
+fixed.  The dependency on ZLib has been made optional, and CrystFEL no longer
+depends on 'NCurses' at all.
+
+
+Geometry files
+--------------
+
+A new field 'mask_edge_pixels' was added to the geometry files.  As the name
+suggests, you can use this to easily mask the edge pixels of panels.  See "man
+crystfel_geometry" for details.
+
+CrystFEL has become more stringent about the contents of geometry files.  The
+wavelength (or photon energy etc) must be specified, as well as the experiment-
+space directions of the fs/ss panel vectors.  There is no longer any default
+for these.
+
+The 'detector rail direction' feature, which was broken in version 0.10.0 and
+not used at all in practice, has been removed completely.
+
+
+Indexing and integration
+------------------------
+
+A new option '--data-format' has been added to indexamajig.  This is to be used
+when receiving data from an online source, such as ZeroMQ.  The currently
+available values are 'msgpack' and 'hdf5'.  Previously, the only possibility
+was for online data to be in MessagePack format, but now it can be delivered in
+HDF5 format as well.  Other formats will be added in future.
+
+The 'peak pairing' procedure, an important part of the 'prediction refinement'
+and resolution estimation calculations, has been changed to use a reciprocal-
+space distance cutoff of one third of the smallest inter-Bragg distance.
+Previously, a hardcoded distance of ten pixels was used.  This change makes the
+results almost independent of image binning.
+
+
+MTZ and XDS export
+------------------
+
+There are now two modes for MTZ export, whether via the GUI or using get_hkl.
+
+Plain MTZ mode (get_hkl --output-format=mtz, or 'MTZ' in the GUI export
+dialog) simply places the reflections into CCP4's definition of the asymmetric
+unit and writes the intensity and error estimates into columns I and SIGI
+respectively.  This mode is the easiest to use for straightforward structure
+refinement.
+
+Bijvoet MTZ mode (get_hkl --output-format=mtz-bij, or 'MTZ, Bijvoet pairs
+together' in the GUI export dialog) puts Bijvoet pairs in the same row, under
+the columns I(+), SIGI(+), I(-) and SIGI(-).  This mode is obviously more
+appropriate for experiments involving the anomalous signal.
+
+Finally, a bug has been fixed in the export to XDS format, where the unit cell
+parameters were previously labelled as '!UNIT_CELL_CONSTANT' instead of
+'!UNIT_CELL_CONSTANTS'.
+
+
+API changes
+-----------
+
+There was only one change to the libcrystfel API:
+
+write_to_mtz: Addition of 'crystal_name' and 'project_name' parameters