SPEC filewriter scan numbering when "new" data file exists

[prjemian]

In SpecWriterCallback.newfile(), when kwarg scan_id=None (the default) and the chosen file already exists, the next scan_id should be determined from the existing file. One of these (perhaps the max of all):

• increment from the last scan number
• increment from the highest scan number in the file
• increment from the total number of scans

[prjemian]

Example:

In [8]: fl = !grep "^#S " 05_02_test.dat                                                                                                                               

In [9]: fl                                                                                                                                                             
Out[9]: 
['#S 1  tune_mr()',
 '#S 1  tune_mr()',
 '#S 2  tune_mr()',
 '#S 3  tune_mr()',
 '#S 4  tune_mr()',
 '#S 5  tune_mr()',
 '#S 1  tune_mr()',
 '#S 2  tune_ar()',
 '#S 3  tune_mr()',
 '#S 1  tune_ar()',
 '#S 1  tune_ar()',
 '#S 2  tune_ar()',
 '#S 3  tune_ar()',
 '#S 101  tune_mr()',
 '#S 102  tune_m2rp()',
 '#S 103  tune_ar()',
 '#S 104  tune_a2rp()',
 "#S 105  measure_USAXS_Transmission(detectors=['scaler0'], num=1)",
 "#S 1  measure_USAXS_Transmission(detectors=['scaler0'], num=1)",
 "#S 1  measure_USAXS_Transmission(detectors=['scaler0_channels_chan02', 'scaler0_channels_chan05'], num=1)",
 "#S 1  measure_USAXS_Transmission(detectors=['scaler0_channels_chan02', 'scaler0_channels_chan05'], num=1)",
 "#S 1  measure_USAXS_Transmission(detectors=['scaler0_channels_chan02', 'scaler0_channels_chan05'], num=1)",
 "#S 1  measure_USAXS_Transmission(detectors=['scaler0_channels_chan02', 'scaler0_channels_chan05'], num=1)",
 "#S 1  measure_USAXS_Transmission(detectors=['scaler0_channels_chan02', 'scaler0_channels_chan05'], num=1)",
 "#S 2  measure_USAXS_Transmission(detectors=['scaler0_channels_chan02', 'scaler0_channels_chan05'], num=1)",
 "#S 1  measure_USAXS_Transmission(detectors=['scaler0_channels_chan02', 'scaler0_channels_chan05'], num=1)",
 "#S 2  count(detectors=['scaler0'], num=1)",
 "#S 1  measure_USAXS_Transmission(detectors=['scaler0'], num=1)",
 "#S 1  measure_USAXS_Transmission(detectors=['scaler0'], num=1)",
 "#S 1  measure_USAXS_Transmission(detectors=['scaler0'], num=1)",
 "#S 1  measure_USAXS_Transmission(detectors=['scaler0'], num=1)",
 "#S 1  measure_USAXS_Transmission(detectors=['scaler0'], num=1)",
 "#S 1  measure_USAXS_Transmission(detectors=['scaler0'], num=1)",
 '#S 1  tune_mr()',
 '#S 1  tune_mr()',
 '#S 1  tune_mr()',
 '#S 108  tune_mr()',
 "#S 109  measure_USAXS_Transmission(detectors=['scaler0'], num=1)",
 '#S 110  Flyscan(pos_X=60, pos_Y=160, thickness=0, scan_title=blank)']

In [20]: sl = np.array(list(map(int, [f.split()[1] for f in fl])))                                                                                                     

In [21]: sl                                                                                                                                                            
Out[21]: 
array([  1,   1,   2,   3,   4,   5,   1,   2,   3,   1,   1,   2,   3,
       101, 102, 103, 104, 105,   1,   1,   1,   1,   1,   1,   2,   1,
         2,   1,   1,   1,   1,   1,   1,   1,   1,   1, 108, 109, 110])

In [22]: sl.max()                                                                                                                                                      
Out[22]: 110

In [23]: sl[-1]                                                                                                                                                        
Out[23]: 110

In [24]: len(sl)                                                                                                                                                       
Out[24]: 39

In [25]: max(sl[-1], sl.max(), len(sl))                                                                                                                                
Out[25]: 110

Yes, sl[-1] is redundant with sl.max()

[prjemian]

Should use a package to parse existing SPEC file. Possibilities are silx (ESRF) or spec2nexus (mine)

Here is the relevant silx documentation: http://www.silx.org/doc/silx/latest/modules/io/specfile.html?highlight=spec#module-silx.io.specfile

[prjemian]

This has bogged down trying to get a databroker document stream into a temporary db backend for testing. We've got the JSON and can read it but the db.insert fails on the first event document.

[prjemian]

Skipping the databroker for now, we have document streams (from scans with real hardware) ready to use in the unit tests.