Data Management¶

To open or save raw experimental data one can use a special module. To call functions from the module one should create a corresponding class instance.

import atomize.general_modules.csv_opener_saver as openfile
file_handler = openfile.Saver_Opener()

Alternatively, it is possible to use the CSV Exporter embedded into Pyqtgraph for saving 1D data and a special option in Liveplot (right click → Save Data Action) for saving 2D data as comma separated two dimensional numpy array.

Every function below works with comma separated files and with HDF5 files; the format is chosen by the extension of the path, .h5 meaning HDF5 and anything else meaning CSV. See HDF5 files for the layout and for when it is worth using.

Functions¶

open_1d(file_path, header=0)¶

open_1d(file_path, header=0)    # -> (data_header, numpy.array)

Simple function to open a specified file with comma separated values. An .h5 file is read through the same function and returns the stored array transposed in exactly the same way, so a file saved from np.c_[x_axis, data_x, data_y] comes back as three rows whichever format it was written in.

Argument	Description
`file_path`	Path to file
`header`	Integer specifying the number of columns in the file header

open_2d(file_path, header=0)¶

open_2d(file_path, header=0)    # -> (data_header, numpy.array)

Simple function to open a specified file with 2D array of comma separated values. An .h5 file that holds both quadratures returns them stacked as a single (2, npoints, nsamples) array instead of the two separate matrices the CSV path keeps in two files; an .h5 file with one matrix returns that matrix, exactly as CSV does.

Argument	Description
`file_path`	Path to file
`header`	Integer specifying the number of columns in the file header

open_2d_appended(file_path, header=0, chunk_size=1)¶

open_2d_appended(file_path, header=0, chunk_size=1)    # -> (data_header, numpy.array)

This function opens a file with a single column array of values from 2D array. For an .h5 file written with per-scan snapshots the list of slices of its scans dataset is returned and chunk_size is ignored, since the file already knows where one scan ends and the next begins.

Argument	Description
`file_path`	Path to file
`header`	Integer specifying the number of columns in the file header
`chunk_size`	Y axis size of the initial 2D array

open_h5_axes(file_path)¶

open_h5_axes(file_path)    # -> { 't': numpy.array, 'sweep': numpy.array }

This function returns the axis datasets of an .h5 file written with the axes argument of save_data. Only the axes actually present appear in the dictionary, so a file saved without them yields an empty one. The data itself is read separately, through open_1d or open_2d.

Argument	Description
`file_path`	Path to file

open_h5_axis_units(file_path)¶

open_h5_axis_units(file_path)    # -> { 't': str, 'sweep': str }

This function returns the units stored beside those axes by the axes_units argument of save_data, keyed the same way. A file saved without them, or written by an earlier version, yields an empty dictionary; an axis vector holds only numbers, so a reader that wants to label it needs this.

data = file_handler.open_2d(file_path)[1]
axes = file_handler.open_h5_axes(file_path)
units = file_handler.open_h5_axis_units(file_path)
# axes['sweep'] in units.get('sweep', 'arb. u.')

Argument	Description
`file_path`	Path to file

open_h5_params(file_path)¶

open_h5_params(file_path)    # -> { 'Field': 3450.5, 'Experiment': 'Pulsed EPR AWG Experiment', … }

This function returns the header parameters of an .h5 file as a dictionary keyed by the name in front of the colon. Every Name: value unit line of the header above its first ---- separator is stored once more as a typed attribute when the file is written, so a value that is a number comes back as a float and anything else, a date or an experiment name, as text. Nothing has to be parsed: the field of a hundred files is a loop over this function. A file written before these attributes existed yields an empty dictionary, and the text header, read through open_1d or open_2d, is unchanged either way.

params = file_handler.open_h5_params(file_path)
units = file_handler.open_h5_param_units(file_path)
# params['Field'] in units.get('Field', '')      -> 3450.5 G
# params['Temperature']                           -> 80.05

Argument	Description
`file_path`	Path to file

open_h5_param_units(file_path)¶

open_h5_param_units(file_path)    # -> { 'Field': 'G', 'Temperature': 'K', … }

This function returns the units stored beside those parameters, keyed the same way; a parameter whose header line carried no unit does not appear. The unit is the text that followed the number on the header line, so Record Length: 500 Points gives 'Points'.

Argument	Description
`file_path`	Path to file

open_file_dialog(directory='')¶

open_file_dialog(directory='')    # -> path to file.csv

This function returns the path to the file selected in the dialog box that opens.

Argument	Description
`directory`	Path to preopened directory in the dialog window

create_file_dialog(directory='', fmt='csv')¶

create_file_dialog(directory='', fmt='csv')    # -> path to file.csv

This function returns the path to the file specified in the dialog box that opens. It can be used to manually save your data inside the experimental script to specified file.

Argument	Description
`directory`	Path to preopened directory in the dialog window
`fmt`	Extension offered by the dialog and appended to a name typed without one; `'h5'` to save HDF5

create_file_parameters(add_name, directory='')¶

# returns two paths: file with add_name extension and file.csv
create_file_parameters('.param')

This function has the full functionality of the create_file_dialog() function, but also returns a second file for saving parameters / header.

Argument	Description
`add_name`	String that will be added to the second file instead of `'.csv'` extension. Example: `create_file_parameters('.param')` will create a second file with `.param` extension
`directory`	Path to preopened directory in the dialog window

save_header(file_path, header='', mode='w')¶

save_header(file_path, header='', mode='w')

This function saves the string given by argument header to the file with the path file_path. Argument mode allows choosing whether the file will be rewritten (mode='w') or the data will be appended to the end of the file (mode='a'). For an .h5 file the header becomes the file attribute described in HDF5 files and no datasets are created yet, so a run that crashes before saving still leaves its header behind — that file reads back through open_2d as an empty array with its header intact. mode='a' updates the header of an existing .h5 without touching its data.

save_data(file_path, data, header='', mode='w', axes=None, fmt='%.6e', dtype=None, axes_units=None)¶

save_data(file_path, data, header='', mode='w', axes=None, fmt='%.6e', dtype=None, axes_units=None)

This function saves the numpy array given by the argument data and the string given by argument header to the file with the path file_path. Argument mode allows choosing whether the file will be rewritten (mode='w') or the data will be appended to the end of the file (mode='a'). Appending to an .h5 file raises a ValueError rather than quietly rewriting it, since appending rows of text and growing a dataset are not the same operation.

This function works for 1D, 2D, and 3D data. In case of 3D (an array of 2D arrays) data, a separate file will be created for each 2D array with the additional _i string in the file_path; an .h5 file keeps them as the I and Q datasets of one file instead (a third plane and beyond become D2, D3…, and open_2d stacks whatever it finds). The standard combination of function to save the experimental data together with a header is the following:

file_data, file_param = file_handler.create_file_parameters('.param')
header = 'Test Header'
file_handler.save_header(file_param, header=header, mode='w')
# Acquiring experimental data
file_handler.save_data(file_data, data, header=header, mode='w')

Argument	Description
`axes`	`(t, sweep)` pair of 1D arrays written as the axis datasets of an `.h5` file; ignored for CSV
`axes_units`	`(t, sweep)` pair of unit strings, e.g. `('s', 'G')`, stored beside those datasets; ignored for CSV. An axis vector carries only numbers, so without this the unit of a stored axis is lost
`fmt`	Number format of the CSV columns; it also picks the HDF5 precision unless `dtype` is given
`dtype`	HDF5 data type; `None` derives it from `fmt`, so `'%.6e'` (7 significant digits) gives `float32` and anything wider gives `float64`. Pass `'float64'` to store the array exactly whatever the CSV format is

HDF5 files¶

A path ending in .h5 is written as a single HDF5 file instead of comma separated text. It is worth doing for the full 2D arrays of an experiment: such a file is about three times smaller than the same data as text, is written in a fraction of the time, keeps both quadratures and the axes together, and is read by Origin, MATLAB and h5dump without any Atomize code. Small 1D result files gain nothing from it and are kept as CSV by every control center window, but the functions accept them at any rank, so an experimental script may use them freely.

example_2d.h5
├── attrs
│   ├── header          str   exact header text as passed to save_data (no '# ')
│   ├── format_version  int   2
│   ├── source          str   'atomize'
│   ├── t_unit          str   unit of the t axis, only when axes_units is passed
│   └── sweep_unit      str   unit of the sweep axis, only when axes_units is passed
├── params attrs                          every 'Name: value unit' header line, typed
│   ├── Field           float 3450.5      (see open_h5_params)
│   ├── Field_unit      str   'G'
│   └── …
├── I      float32  (npoints, nsamples)   same orientation as the CSV rows
├── Q      float32  (npoints, nsamples)   only when the source has a quadrature
├── t      float64  (nsamples,)           within-trace axis
└── sweep  float64  (npoints,)            tau / field / amplitude axis

The array is stored exactly as np.savetxt() would lay it out, so a 1D file is the same layout with one axis fewer and no separate concept: save_data() never has to guess what the array means, and open_1d() / open_2d() differ for HDF5 exactly as they differ for CSV. The t and sweep datasets are written only when axes is passed, and their t_unit / sweep_unit attributes only when axes_units is passed as well. The params group repeats the header as typed attributes, one per Name: value unit line above the first ---- separator, so an external script or h5dump sees the field, the frequency or the temperature as a number with its unit beside it rather than inside one text block; open_h5_params reads them back. A file written with per-scan snapshots carries one more dataset, scans, whose first axis is the scan number and whose slice j - 1 is the cumulative average after scan j.

An axis stored this way keeps its true origin, which a header cannot: a header line gives a step only, so a reader has to start the axis at zero. A field sweep written as an axis dataset comes back running from its real start field rather than from 0 G. This is why the control center windows that save 2D data pass both axes and axes_units.

Note

The default float32 is not a loss against CSV: the default '%.6e' format writes 7 significant digits, which is the same precision band. Save with dtype='float64' (or a wider fmt) if a particular array needs more.

Standard numpy savetxt() function¶

np.savetxt(path_to_file, data_to_save, fmt='%.4e', delimiter=' ',
           newline='n', header='field: %d' % i, footer='',
           comments='#', encoding=None)

For saving inside the script by create_file_dialog() a standard numpy function should be used.