This is CrystFEL 0.11.00.11.0

5 files changed, 178 insertions, 167 deletions

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 50cab3b5..349a9e50 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -13,8 +13,8 @@ The easiest way to ensure the future of CrystFEL is to use it and cite it in you
 
 You can also cite CrystFEL directly using one of the following DOIs:
 
-* All versions: [10.5281/zenodo.8183384](https://zenodo.org/record/8183384).
-* Version 0.10.2: [10.5281/zenodo.8183385](https://zenodo.org/record/8183385).
+* All versions: [10.5281/zenodo.8183384](https://doi.org/10.5281/zenodo.8183384).
+* This version (0.11.0): [10.5281/zenodo.10984700](https://10.5281/zenodo.10984700).
 
 
 Getting started
diff --git a/ChangeLog b/ChangeLog
index 7ea1073b..5d24622f 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,5 +1,5 @@
-Changes in this development version
------------------------------------
+CrystFEL version 0.11.0, 17 April 2024
+--------------------------------------
 
 - New detector geometry refinement system, using Millepede-II
 - New Homebrew formula and better build/test support for Mac OS
diff --git a/meson.build b/meson.build
index 1d6a0130..4717bb2a 100644
--- a/meson.build
+++ b/meson.build
@@ -1,11 +1,11 @@
 # Meson file for CrystFEL
 project('crystfel', 'c',
-        version: '0.10.2',
+        version: '0.11.0',
         license: 'GPL3+',
         meson_version: '>=0.55.0',
         default_options: ['buildtype=debugoptimized'])
 
-libcrystfel_api_version = 16
+libcrystfel_api_version = 17
 crystfel_rpath = '$ORIGIN/../lib64/:$ORIGIN/../lib:$ORIGIN/../../asapo-libraries/lib:$ORIGIN/../../asapo-libraries/lib64'
 
 add_project_arguments('-DHAVE_CONFIG_H', language: 'c')
diff --git a/relnotes-0.10.2.md b/relnotes-0.10.2.md
deleted file mode 100644
index 00bbad03..00000000
--- a/relnotes-0.10.2.md
+++ /dev/null
@@ -1,161 +0,0 @@
-Release notes for CrystFEL version 0.10.2
-=========================================
-
-Copyright © 2012-2023 Deutsches Elektronen-Synchrotron DESY,
-                      a research centre of the Helmholtz Association.
-
-See AUTHORS as well as the individual source code files for full contributor details.
-
-CrystFEL is free software: you can redistribute it and/or modify it under the
-terms of the GNU General Public License as published by the Free Software
-Foundation, either version 3 of the License, or (at your option) any later
-version.
-
-CrystFEL is distributed in the hope that it will be useful, but WITHOUT ANY
-WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
-PARTICULAR PURPOSE.  See the GNU General Public License for more details.
-
-You should have received a copy of the GNU General Public License along with
-CrystFEL.  If not, see <http://www.gnu.org/licenses/>.
-
-
-Overview
---------
-
-This release is primarily a bug-fixing update to CrystFEL, and there have been
-improvements around the entire suite.  The most important changes are detailed
-below.  See the ChangeLog or the Git history for a comprehensive list of all
-changes.
-
-
-Graphical user interface (GUI)
-------------------------------
-
-This version adds a colour scale widget to the GUI, which can be used to
-manually adjust the colours used to display the image.  The colour scale is
-found directly to the right of the image.  Click and drag to move
-the histogram of image values relative to the colour scale, and use the scroll
-wheel to compress and expand the range.
-
-Resolution rings can now be added to the image.  Enable them in the "View" menu.
-There are also menu items for running `detector-shift` and `peakogram`, found
-under the "Tools" menu.
-
-The figure of merit calculation tool has been improved, and now displays the
-results as a graph.  The graph values can be exported as a CSV file for further
-processing.
-
-This release also adds specially-designed icons to the GUI, which fixes
-problems with "broken" icons on some systems.
-
-
-Performance
------------
-
-Speed has been improved in many areas of CrystFEL.  The profiling option
-(`--profile`) of `indexamajig` has been re-implemented and can help you to find
-bottlenecks in processing on your data.  In addition, some new options were
-added to `indexamajig`:
-
-New option `--peakfinder8-fast` tells peakfinder8 to calculate some values in
-advance, approximately halving the time taken by the peak search.
-
-Another new option `--asdf-fast` slightly alters some thresholds within asdf,
-resulting in a large speedup.  Since this changes the behaviour from before, we
-decided to make it optional.
-
-Finally, `--cell-parameters-only` completely circumvents the spot prediction and
-integration calculations, which can make a very large speedup as long as you're
-not interested in spot positions.  Note that this isn't the same as the
-existing option ``--integration=none``, where the spot positions are predicted
-but the intensity integration is skipped.  For very large unit cells, which are
-often produced by unconstrained indexing (e.g. for online monitoring), the spot
-prediction can take signficant time.
-
-For advice about making CrystFEL faster, see this page:  
-<https://gitlab.desy.de/thomas.white/crystfel/-/blob/master/doc/articles/speed.rst>
-
-
-Documentation improvements
---------------------------
-
-Two long-form documentation articles have been added, on top of the other two
-mentioned in other sections of this page.
-
-A completely new tutorial has been added, describing how to process data using
-the graphical user interface:  
-<https://gitlab.desy.de/thomas.white/crystfel/-/blob/master/doc/articles/tutorial.rst>
-
-The following new article has been added, to help with the common question of
-choosing the correct point group for merging data:  
-<https://gitlab.desy.de/thomas.white/crystfel/-/blob/master/doc/articles/pointgroup.rst>
-
-
-Streaming interfaces
---------------------
-
-ZeroMQ and ASAP::O interfaces have been added, to allow data to be streamed
-directly to CrystFEL without any intermediate file storage.  For some
-information on how to get started, read the following document:  
-<https://gitlab.desy.de/thomas.white/crystfel/-/blob/master/doc/articles/online.rst>
-
-
-Refuses to overwrite stream
----------------------------
-
-Starting from this version, `indexamajig` and other CrystFEL tools will refuse to
-create a new stream if a file already exists with the same name.  This is meant
-to avoid the possibility of losing valuable results by (e.g.) re-running a
-processing script.  This becomes particularly important in a streaming data
-situation, where the original data might no longer be available.  If you really
-want to overwrite the results, you'll need to move or delete the old file
-yourself.
-
-
-Installation
-------------
-
-CrystFEL has become easier to install because of several developments.  First,
-the Slurm API headers are no longer needed for the GUI to submit Slurm jobs.
-In this version, it uses the Slurm commands directly
-(`sbatch/scontrol/scancel`).
-
-For environments based totally on CBFs or other types of files, it's now
-possible to compile CrystFEL without any reference to the HDF5 library,
-avoiding a lot of possible complications.
-
-There is a new script (`scripts/install-indexers`) which helps with installing
-Mosflm, DirAx and XDS.  After downloading CrystFEL, run the script with
-`--help` to get started.
-
-Finally, Docker images are now available.  These can be used with various
-container tools (not just Docker).  For example, using Apptainer (the new name
-for Singularity):
-```
-$ apptainer pull docker://gitlab.desy.de:5555/thomas.white/crystfel/crystfel:latest
-$ apptainer run -B /path/to/data crystfel_latest.sif
-```
-
-More details can be found on the installation page:
-<https://www.desy.de/~twhite/crystfel/install.html>
-
-
-API changes
------------
-
-Added routines:
-* `data_template_slabby_file_to_panel_coords`
-* `data_template_get_2d_detgeom_if_possible`
-* `data_template_get_clen_if_possible`
-* `image_create_dp_bad_sat`
-* `profile_{init,start,end,print_and_reset}`
-
-Removed routines:
-* `image_set_zero_mask`
-
-Changed routine prototypes:
-* `default_method_options` - addition of `asdf_opts`
-* `peakfinder8` - addition of `fast_mode` and `private_data`
-
-Changed enumerations:
-* `DataSourceType` - addition of `DATA_SOURCE_TYPE_SEEDEE`
diff --git a/relnotes-0.11.0.md b/relnotes-0.11.0.md
new file mode 100644
index 00000000..2691695b
--- /dev/null
+++ b/relnotes-0.11.0.md
@@ -0,0 +1,172 @@
+Release notes for CrystFEL version 0.11.0
+=========================================
+
+Copyright © 2012-2024 Deutsches Elektronen-Synchrotron DESY,
+                      a research centre of the Helmholtz Association.
+
+See AUTHORS as well as the individual source code files for full contributor details.
+
+CrystFEL is free software: you can redistribute it and/or modify it under the
+terms of the GNU General Public License as published by the Free Software
+Foundation, either version 3 of the License, or (at your option) any later
+version.
+
+CrystFEL is distributed in the hope that it will be useful, but WITHOUT ANY
+WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
+PARTICULAR PURPOSE.  See the GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License along with
+CrystFEL.  If not, see <http://www.gnu.org/licenses/>.
+
+
+Overview
+--------
+
+This CrystFEL release adds a totally new system for detector geometry
+refinement, as well as better support for compiling on Mac OS.  The old CMake
+build system has been removed in favour of Meson.  The simulation tools
+`pattern_sim` and `partial_sim`, which were not well maintained nor widely used,
+have been removed.
+
+These changes are detailed below.  In addition, there were many smaller fixes
+and improvements.  See the ChangeLog or the Git history for a comprehensive
+list of all changes.
+
+
+New system for detector geometry refinement
+-------------------------------------------
+
+The new detector geometry system is based on the Millepede-II program, the same
+software used for high energy physics experiments including the CMS.  It
+allows us to do a full refinement including all of the inter-dependencies
+between crystal orientations, cell parameters and the detector geometry.
+Previously, the refined updates to the detector positions would be biased
+because they were based on biased crystal parameters, which were in turn biased
+because of the incorrect detector geometry.  To break this circular dependency,
+we have to refine the individual parameters for each crystal at the same time
+as we refine the geometry.  Normally, doing this would result in an enormous
+and computationally prohibitive least squares calculation, even with
+specialised sparse matrix solvers.  However, the Millepede algorithm reduces
+the calculation to one with a sized based only on the number of geometrical
+parameters for the detector.
+
+To get started with the new geometry refinement, read the manual for the new
+`align_detector` program by running `man align_detector` or visiting
+<https://gitlab.desy.de/thomas.white/crystfel/-/blob/master/doc/man/align_detector.1.md>.
+An update of the geometry file will be necessary to add information about the
+hierarchy of detector panels, but example files are available.  The refinement
+can also be performed using the CrystFEL GUI.
+
+A side-effect of this change is that the beam center is no longer refined
+after indexing for each pattern individually.  This should make the prediction
+refinement more stable.
+
+Future CrystFEL versions will look at improving the stability and precision of
+this method, in particular for three-dimensional and fine-grained refinement
+tasks.
+
+
+Homebrew formula and improved Mac OS support
+--------------------------------------------
+
+This release adds a Homebrew formula for CrystFEL.  If you're running on a Mac,
+you can now install CrystFEL with a single `brew install` command.  See
+`INSTALL.md` for more details.  Eventually we hope to submit this formula to
+the main Homebrew repository to make it even easier to install.
+
+In addition, we are now regularly testing CrystFEL on Mac OS as part of our
+continuous integration pipeline.
+
+
+Removal of simulation tools and CMake build system
+--------------------------------------------------
+
+Due mostly to a insufficient development resources, the simulation tools
+`pattern_sim` and `partial_sim` have been removed from the CrystFEL suite
+starting from this version.  For more detailed discussion of the rationale for
+this, see https://gitlab.desy.de/thomas.white/crystfel/-/issues/81
+
+If you need quantitative image simulations, there are several better options
+including [nanoBragg]<https://bl831.als.lbl.gov/~jamesh/nanoBragg/>.  For
+relative-scale geometry-only simulations of the kind done by `partial_sim`,
+a better option is expected in a future CrystFEL version - watch this space!
+
+The CMake-based build system, which was barely maintained and lacked many
+features compared to the Meson-based system, has now been removed altogether.
+
+
+API changes
+-----------
+
+Removed routines:
+* `data_template_get_rigid_groups`
+* `r_gradient`
+* `x_gradient`
+* `y_gradient`
+* `image_hdf5_write`
+
+Added routines:
+* `crystfel_mille_new`
+* `crystfel_mille_free`
+* `crystfel_mille_delete_last_record`
+* `crystfel_mille_write_record`
+* `mille_label`
+* `mille_unlabel`
+* `write_mille`
+* `data_template_show_hierarchy`
+* `data_template_translate_group_px`
+* `data_template_translate_group_m`
+* `data_template_rotate_group`
+* `data_template_write_to_fh`
+* `data_template_write_to_file`
+* `data_template_group_info`
+* `detgeom_find_panel`
+* `detgeom_show_hierarchy`
+* `detgeom_translate_detector_m`
+* `detgeom_group_center`
+* `make_panel_minvs`
+* `is_cbf_file`
+* `is_cbfgz_file`
+* `is_hdf5_file`
+* `image_data_arrays_new`
+* `image_data_arrays_free`
+* `image_create_dp_bad`
+* `image_set_zero_data`
+* `index_pattern_4`
+* `r_dev`
+* `fs_dev`
+* `ss_dev`
+* `r_gradient`
+* `fs_ss_gradient`
+* `read_reflections_3`
+* `stream_write_data_template`
+* `show_vector`
+* `matrix_mult2`
+* `matrix_mult3`
+* `matrix_invert`
+* `sq`
+* `rotate2d`
+* `is_dir`
+
+Added type definitions:
+* `Mille`
+* `struct dg_group_info`
+* `struct detgeom_panel_group`
+* `ImageDataArrays`
+* `struct peak_params`
+* `struct reflpeak`
+
+Removed type definitions:
+* `struct rigid_group`
+* `struct rg_collection`
+
+Changed structure definitions:
+* `detgeom_panel` - addition of `group`
+* `detgeom` - addition of `top_group`
+* `image` - addition of `ida`
+* `enum gparam` - changed values, moved from geometry.h to predict-refine.h
+
+Changed routine prototypes:
+* `image_read` - addition of `ida`
+* `image_read_data_block` - addition of `ida`
+* `refine_prediction` - addition of `mille` and `max_mille_level`