can't plot SPEC file with MatLab tools

[prjemian]

Opened issue in the apstools package. Also, could not plot the data with SPEC's splot. Will resolve in apstools project and close this issue once we verify the problem is solved.

[prjemian]

It could be that the MatLab tools might also need some changes. The splot program is from SPEC so apstools will be changed to produce a file that can be plotted with splot. Will test with MatLab tools and decide from that point what to do next.

[sureshnaps]

Pete,
I agree and we are very open to modifying matlab specreader code to read this file. The motor params would be needed and the rest we can change. We do not need to chase every bit of it to make the file look exactly like spec.

[prjemian]

Perhaps related to BCDA-APS/apstools#203 (#N should be the number of data columns, not the expected number of data rows)

[prjemian]

@sureshnaps Let's take another scan now that BCDA-APS/apstools#203 is fixed. If MatLab still cannot read the file, let's get the details from MatLab about what it finds wrong.

[prjemian]

If it's the #O and #P control lines, that might take some work for the spec writer to handle (it does not know the names of all the motors that will be used until they are used).

[prjemian]

I created a new SPEC file with scans from last Monday. /home/beams10/8IDIUSER/20190802-164346.dat

Here are the steps (using the default SPEC data file name):

snow% bluesky8IDI
...
# look at the list of recent scans
In [2]: APS_utils.list_recent_scans(show_command=True)
========= ========================== ======= ========================================
short_uid date/time                  scan_id command
========= ========================== ======= ========================================
8e0739f   2019-07-29 14:00:40.396961 26      grid_scan(detectors=['scaler1'], arg ...
b0279ec   2019-07-29 13:35:26.517215 25      rel_scan(detectors=['scaler1', 'Atte ...
42b758a   2019-07-29 13:34:00.489015 24      scan(detectors=['scaler1', 'Atten1'] ...
f3a368f   2019-07-29 13:31:57.348687 23      rel_scan(detectors=['scaler1', 'Atte ...
05acf4e   2019-07-29 13:29:26.281539 22      rel_scan(detectors=['scaler1', 'Atte ...
92930e5   2019-07-29 13:29:02.416113 21      rel_scan(detectors=['scaler1', 'Atte ...
4d295d2   2019-07-29 13:26:26.603173 20      scan(detectors=['scaler1', 'Atten1'] ...
8731828   2019-07-29 13:21:49.175562 19      scan(detectors=['scaler1'], num=21,  ...
032a08e   2019-07-29 13:09:38.330084 18      count(detectors=['scaler1'], num=1)
a2abc76   2019-07-29 13:08:42.159739 17      count(detectors=['scaler1'], num=1)
e484a6a   2019-07-26 15:53:46.317911 16      scan(detectors=['noisy_det'], num=25 ...
cd6057e   2019-07-26 15:53:45.954193 15      scan(detectors=['noisy_det'], num=51 ...
d9c5bd6   2019-07-26 15:53:45.618354 14      scan(detectors=['noisy_det'], num=5. ...
d27599b   2019-07-26 15:53:45.304559 13      scan(detectors=['noisy_det'], num=11 ...
9fd190f   2019-07-26 15:51:40.737271 12      scan(detectors=['noisy_det'], num=12 ...
3fe1556   2019-07-26 15:51:40.339529 11      scan(detectors=['noisy_det'], num=25 ...
1eee259   2019-07-26 15:50:58.251555 10      scan(detectors=['noisy_det'], num=25 ...
638133e   2019-07-26 15:18:04.553335 9       scan(detectors=['noisy_det'], num=11 ...
280a478   2019-07-26 15:07:32.622573 8       count(detectors=['noisy_det'], num=5)
0c996a1   2019-07-26 15:06:56.376353 7       count(detectors=['noisy_det'], num=5)
========= ========================== ======= ========================================

Out[2]: <pyRestTable.rest_table.Table at 0x7f3b0c1de3c8>

In [3]: specwriter.spec_filename
Out[3]: '/home/beams10/8IDIUSER/20190802-164346.dat'

In [4]: APS_utils.replay(db[-10:], specwriter.receiver)

In [5]: exit

[sureshnaps]

This spec file complains as well about a missing #P. My feeling is we will ask Zhang to take a look at it and see where it can be fixed.
I am following a lot of things you are doing, this week is a relatively better time to try a few bluesky things even in parallel.

Index exceeds the number of array elements (0).
Error in readspecscan (line 22)
while ~strcmp(scanline(1:2),'#P')
Error in selectscan (line 39)
scan = readspecscan(file,scan,selection);
Error while evaluating UIControl Callback.

[sureshnaps]

specreader shows a list of the motors in each scan and it so needs the O fields in the header and P fields above each scan. These are quite useful features. This seems like a lot of undertaking in bluesky given that it has not been done so far. Maybe we have to stick to a python based spec plot that can be made to work. We can discuss this.

[sureshnaps]

One way to do would be:
make a spec file with just 1 or 2 scans (i can always crop this file) and then we can read that in matlab and then manually insert the O and P fields and then make sure that we can read it. May be it is a rabbit hole and will keep growing. Then we can make a judicious decision. Does this sound ok? I can do this on Monday.

[sureshnaps]

also noticed that these spec files grow upward, native spec files grow downward.

[prjemian]

Installed (on snow) the updated apstools 1.1.12 (released yesterday). Re-wrote the SPEC file:

In [1]: APS_utils.replay(db[-10:], specwriter.receiver)

In [2]: specwriter.spec_filename
Out[2]: '/home/beams10/8IDIUSER/20190806-074453.dat'

Give this a try. BUT, it still does not have any #O or #P lines. We can extract a scan or two from this and add #O0 / #P0 lines manually to see if that satisfies the Matlab reader.

[prjemian]

currently on snow:

RunEngine metadata dictionary:
=========== ==============================
key         value
=========== ==============================
beamline_id APS TRR 8-ID-I
login_id    8idiuser@snow.xray.aps.anl.gov
pid         26431
proposal_id None
scan_id     26
versions    =========== ========
            key         value
            =========== ========
            apstools    1.1.12
            bluesky     1.5.3
            databroker  0.13.0
            epics       3.3.3
            matplotlib  3.1.0
            numpy       1.16.4
            ophyd       1.3.3
            pyRestTable 2020.0.2
            spec2nexus  2021.1.1
            =========== ========
=========== ==============================