aboutsummaryrefslogtreecommitdiff
path: root/libcrystfel/src/stream.c
AgeCommit message (Collapse)Author
2023-04-20Revert "Refuse to overwrite an existing stream"Thomas White
This was intended to help users avoid accidents. However, it seems to trip people up a lot, and creates problems such as breaking Slurm job pre-emption (re-scheduled job fails because output already exists). This reverts commit 6db9619ba5172b9e416a5939d375f5119aa652ff.
2022-11-05Stream: increase maximum geometry file size to 1 megabyteThomas White
The old maximum was 64 kilobytes, and a 68k geometry file has now been seen in the wild. The limit is only needed to prevent runaway memory allocation in the case of a corrupted stream. So it's safe to increase it quite a lot.
2022-09-22Implement data_template_get_2d_detgeom_if_possibleThomas White
It is horrible. But it's contained inside the DataTemplate module.
2022-09-21create_detgeom: Return detgeom structure rather than altering image argumentThomas White
2022-09-21Move create_detgeom to DataTemplate moduleThomas White
It seems to make more sense here, because it's all about interpreting the contents of the DataTemplate structure.
2022-09-08Stream: don't open stream if headers aren't understoodThomas White
This avoids e.g. crashing later if the geometry is bad. This commit also removes a problematic stanza from the geometry in test.stream, since the stream reading functions are now more fussy about this.
2022-08-18Restructure image data array creationThomas White
This adds a central procedure (image_create_dp_bad_sat) to create all the arrays. Then it's up to the loading procedure to put the values into the arrays. This also makes the loading procedures responsible for marking NaN/inf pixels in the bad pixel map. This avoids an additional sweep through the image data, and makes it possible to skip the NaN/inf check altogether if the image data comes, as it often does, in format which can't represent NaN/inf anyway. Finally, it removes quite a lot of duplicated code.
2022-06-02Refuse to overwrite an existing streamThomas White
With online processing, we might end up with a very long stream. It should not be so easily deleted if indexamajig is restarted!
2022-03-08read_peaks: Simplify logicThomas White
2022-03-08Fix slabbiness assumptionsThomas White
This adds a new routine, data_template_slabby_file_to_panel_coords, to be used (only!) in places where external forces require assumptions of slabbiness: pixel maps and MsgPack/HDF5 peak lists (including CXI-style). This also fixes the prototype of data_template_file_to_panel_coords to make the panel number strictly an input parameter. This was an oversight in the implementation of DataTemplate, and caused problems when reading non-slabby streams. Fixes: https://gitlab.desy.de/thomas.white/crystfel/-/issues/66
2022-03-08stream_open_for_write: Fix documentationThomas White
2022-03-08Drop compatibility with old stream format versionsThomas White
CrystFEL hasn't written any version other than 2.3 since before 2015. Supporting the old versions is getting difficult because of slabbiness assumptions, so it's time to get rid of it.
2021-07-23Add field widths to all sscanf("%s")Thomas White
2021-07-23stream_make_index: Remove unused assignmentThomas White
2021-07-23read_crystal: Remove unused varibalesThomas White
2021-07-22Clean up shadowed variablesThomas White
2021-07-21Reduce variable scope where possibleThomas White
2021-07-21Add missing cleanup on error pathsThomas White
2021-07-16Rename DST_* to DATA_SOURCE_TYPE_*Thomas White
On some systems, this conflicts with DST_NONE in sys/time.h.
2021-07-08Revert "Bump stream version to 2.4"Thomas White
Rationale: Although the format has technically changed, CrystFEL 0.9.1 and older do not read the "hdf5/" lines anyway. The new "header" lines are just noise, as far as they are concerned. However, increasing the version number causes older CrystFEL versions to reject streams written by newer CrystFEL versions. This is inconvenient, but will be a particular problem since CrystFEL 0.10.0 will most likely be released without a working "geoptimiser". We therefore need to be able to use older CrystFEL versions for refining geometry. Older versions are perfectly capable of reading newer streams. In general, the semantics of stream version numbers are a bit messed up, and we need to work out what they really mean. This reverts commit fdce01adf9fc5950904b644ea755760cb32ec5b2.
2021-07-07Bump stream version to 2.4Thomas White
The format for storing headers has changed from hdf5/location to header/type/location.
2021-07-07Interpret old-style stream header records as floating pointThomas White
The old way of recording metadata in the stream unfortunately doesn't include any information about the data type. Most of the time, floating point is what's intended. Older CrystFEL doesn't know about integer headers at all, but there might be string values if --copy-hdf5-field was used. By interpreting "hdf5/something = val" as floating-point, most things will work, e.g. camera length from HDF5 header enabling stream to be loaded without access to the image data. Anything that can't be interpreted as floating-point will be noisily ignored when loading the stream.
2021-06-16stream_read_chunk: Bail out if detgeom can't be createdThomas White
2021-06-16Add DST_NONE for 'no data source' (e.g. image came from stream)Thomas White
2021-05-17Resolve FIXMEs and TODOsThomas White
Prompted by the article linked below, for each FIXME/TODO I've either referenced an issue in the tracker, or removed it if it's not worth fixing. https://schleiss.io/plotting-source-code-todos-for-open-source-projects
2021-05-12Rework header caching layerThomas White
This clears up multiple layering violations which were starting to get in the way. It enables "string" headers to be stored directly, and it will make it much simpler to add new header types in the future. Along the way, this changes all the floating point header stuff to use double precision. This is needed for EuXFEL event IDs. Closes: https://gitlab.desy.de/thomas.white/crystfel/-/issues/34
2021-04-27Rename libcrystfel's config.h to libcrystfel-config.hThomas White
This removes the big potential for confusion, which has happened several times (see e.g. 095cbebaf6). It also fixes in-tree builds with CMake (but seriously, always use out-of-tree builds). Fixes #2.
2021-03-30Add header caching layerThomas White
This simultaneously implements --copy-header/--copy-hdf5-field and gives access to metadata (e.g. detector position or wavelength) when loading from a stream with header-specified values.
2021-03-11detgeom_transform_coords: Take an additional detector shiftThomas White
This allows the refined detector position to be used in a lot of places.
2021-03-05Mass update of copyright dates in source code commentsThomas White
2021-01-28Stream: Use separate DataTemplate for reading and writingThomas White
This solves a tricky conceptual issue: Stream owns the DataTemplate when it's reading (because it came from the stream headers), but not when writing (when it's provided to us).
2021-01-27stream_close: Free the DataTemplateThomas White
2021-01-14Remove old debuggingThomas White
2021-01-14Stream: Remove in_chunkThomas White
It's not used any more - nowhere is it set to 1.
2021-01-14stream_open_for_read: Avoid using fseek()Thomas White
2021-01-11UnitCell: Store all representations once they're calculatedThomas White
Previously, the "getter" functions would re-calculate the requested representation every time they were called. This could mean doing a matrix inversion in the middle of a tight loop, wasting loads of time. Now, it stores the calculated values and returns them directly next time. Setting the parameters invalidates the values for all representations other than the one given. The cost of doing this is that the cell can no longer be "const" in the getter functions. This tracked through some other code, but nothing too severe.
2020-10-27Add a new StreamFlag for generating the detgeom structureThomas White
This is a hack, and should be replaced by something which gets the metadata from the stream rather than hitting the read image file every time.
2020-09-25stream_make_index: s/Indexing/Scanning/Thomas White
Because it sounds strange and confusing to index the indexing results.
2020-09-08Remove "Command line:" in streamThomas White
This wasn't present in the old version, and it was missing a space anyway.
2020-09-04stream_read_chunk: Set image->spectrumThomas White
2020-09-04stream_make_index: Don't require an event ID for every chunkThomas White
Event ID is not written if not needed
2020-09-04make_key: Handle NULL eventThomas White
2020-09-04New API for random access to streamThomas White
2020-09-02Get rid of STREAM_IMAGE_DATAThomas White
No longer needed, and didn't fit very well in the first place.
2020-08-19Update old commentsThomas White
2020-08-19Remove STREAM_UNITCELL and STREAM_CRYSTALSThomas White
These flags weren't orthogonal at all, which was turning some routines into spaghetti-code. It's still possible to avoid read/write of peak lists, reflection data or image data. But that's it.
2020-08-19Const-cleaningThomas White
2020-08-19Make Stream responsible for its own DataTemplateThomas White
It makes no sense for a different DataTemplate to be used for every frame in a stream. And this way makes everything much easier for users ot the API.
2020-07-29New way of injecting version informationThomas White
Also, this fixes some incorrect headers and adds --version flags to programs which didn't yet have them.
2020-07-29stream_read_chunk: Add option to load image dataThomas White
generated by cgit v1.2.3 (git 2.25.1) at 2024-11-17 17:39:30 +0000