Added PPG raw decoding for Verity Sense

[HufflyCodes]

As I was working on the library in windows I could not modify core.py directly as it wouldn't point to that but instead the installed library so all of the changes I made were to bleakheart.py. That being said with those changes being added to core.py the package should work as intended for decoding the ppg values by running the corresponding ppg_parse.py. I am unsure of proper conventions and typical guidelines for doing pull requests so let me know if anything needs to be changed

[fsmeraldi]

I guess the easiest way when developing would be to check out the repo so that you have bleakheart in your PYTHONPATH or your working directory, without installing the package - but never mind, this is great, I'll merge it with core.py - no problem at all.

I have a couple of questions:

• what would be a good set of default setting values for PPG that one can add to default_settings? I guess it would be better if they do not require the SDK mode so that they won't cause errors if one just calls start_streaming
• related to this, can RESOLUTION and CHANNELS actually change on the Verity? If so, we should probably store the values passed to start_streaming as object attributes so that the decode function can have the right values.
• Related to this I realised I implicitly assumed fixed values for RESOLUTION in ECG and ACC decoders since on the H10 they do not change, is this still ok to do that?
• In fact, are the default settings I listed for the other measurements on H10 compatible with the Verity? I guess ECG does not apply, but what about ACC?

[HufflyCodes]

That's a great point, for the default settings for the verity sense I just iterated through all of the available sensors with the help of list measurements and got this:
Measurements supported by Polar device: ['PPG', 'ACC', 'PPI', 'GYRO', 'MAG']
Available PPG settings:
error_code: 0
error_msg: SUCCESS
SAMPLE_RATE: [55]
RESOLUTION: [22]
CHANNELS: [4]
Available ACC settings:
error_code: 0
error_msg: SUCCESS
SAMPLE_RATE: [52]
RESOLUTION: [16]
RANGE: [8]
CHANNELS: [3]
Available PPI settings:
error_code: 0
error_msg: SUCCESS
Available GYRO settings:
error_code: 0
error_msg: SUCCESS
SAMPLE_RATE: [52]
RESOLUTION: [16]
RANGE: [2000]
CHANNELS: [3]
Available MAG settings:
error_code: 0
error_msg: SUCCESS
SAMPLE_RATE: [10, 20, 50, 100]
RESOLUTION: [16]
RANGE: [50]
CHANNELS: [3]

So sadly they do seem to change and as I am sure you are aware this is what the H10 returns
Measurements supported by Polar device: ['ECG', 'ACC']
Available ECG settings:
error_code: 0
error_msg: SUCCESS
SAMPLE_RATE: [130]
RESOLUTION: [14]
Available ACC settings:
error_code: 0
error_msg: SUCCESS
SAMPLE_RATE: [25, 50, 100, 200]
RESOLUTION: [16]
RANGE: [2, 4, 8]

But it seems like resolution and range only change for the H10's accelerometer so those being fixed with a dictionary should be sufficient. I can check and see if adding the SDK mode changes anything if you would like more information regarding that also. I am not sure why the H10's accelerometer is giving different results from your hard coded range and sample rate, I wonder if they changed that in a newer hardware version. I know in the verity threads they made a fix for some sampling rate issue, could be similar to that and if it is the devices can be updated with their app.

Do the settings listed answer all of your questions?

One of my own so I understand it a bit better, any reason why the main file was renamed to core.py instead of it's old bleakheart.py? Is that so it can work better as an installed library?

[fsmeraldi]

Thank you, very interesting!

• For PPG: as far as I can see RESOLUTION is always 22 and CHANNELS always 4, both in your SDK example and without SDK, so it would seem we are fine - could you please check the other SDK settings for PPG just in case? I will add the PPG parameters you listed above as the default choices.
• For ACC on the H10: the decoder works with all the options - the only thing that is assumed constant is the resolution. I do get the same listed options as you do, the defaults are just a pick that seemed sensible to me. My understanding is that if you map a 8g range over 16 bits, you get lower resolution that if you have a 2g range over the same bits, so it's a trade off - at least, that should be the logic unless they're doing something really strange. The user can in any case override the defaults by passing their own settings.
• If we ever implement Verity support for ACC it will become more complicated as I see the same defaults won't work, but we'll worry about it once we get there. In fact, maybe these defaults should be eliminated altogether; I just thought it was a good way to give the user some guidance.

Concerning filenames: I decided to organise the library as a package, and then I did not want to have the same name to avoid doubts about what "import" was doing. My intention was to implement another module with accessory functions for reading offline measurements, but I had no luck getting documentation from Polar on the offline mode for the H10, and so far have not got very far with figuring that out.

[HufflyCodes]

• With SDK mode on the following is returned:
  Connected: True
  Measurements supported by Polar device: ['PPG', 'ACC', 'PPI', 'GYRO', 'MAG']

Available PPG settings:
error_code: 0
error_msg: SUCCESS
SAMPLE_RATE: [28, 44, 55, 135, 176]
RESOLUTION: [22]
CHANNELS: [4]

Available ACC settings:
error_code: 0
error_msg: SUCCESS
SAMPLE_RATE: [26, 52, 104, 208, 416]
RESOLUTION: [16]
RANGE: [2, 4, 8, 16]
CHANNELS: [3]

Available PPI settings:
error_code: 0
error_msg: SUCCESS

Available GYRO settings:
error_code: 0
error_msg: SUCCESS
SAMPLE_RATE: [26, 52, 104, 208, 416]
RESOLUTION: [16]
RANGE: [250, 500, 1000, 2000]
CHANNELS: [3]

Available MAG settings:
error_code: 0
error_msg: SUCCESS
SAMPLE_RATE: [10, 20, 50, 100]
RESOLUTION: [16]
RANGE: [50]
CHANNELS: [3]
So it seems only ACC and GYRO are the complicated ones with lots of variability.

• That makes sense to me!
• I think leaving the defaults is a good place to start and ensuring ease of use to me comes before all of the modularity but that's just preference.

Is there anything you'd like me to work on to get this more integrated?

[fsmeraldi]

Thank you @HufflyCodes, this is very interesting! It seems that on the verity RESOLUTION=16 for everything except for PPG where it is 22, and likewise CHANNELS=3 for everything (as expected) except PPG (where I gather it is 3 LEDs plus a background reading).

Looking at the example you quoted, are the frame type listed there for ACC (129) and the one you decoded for PPG (128) essentially the same? If so, it would seem that if one

• adds to the _pmd_data_handler if-else ladder a case for 'ACC' and frame_type==129 and
• explicitly passes the correct values for CHANNELS and RESOLUTION to your _decode_ppg_data method,
  then this should work also for ACC (and possibly for the other measurements too, if the frame type is essentially the same).

However, I may be over-simplifying. For one thing, start_streaming for ACC on the Verity should also return a "factor" parameter that I am returning but not decoding. I wonder if that would depend directly on RANGE and account for the scaling or if it is something else. What are your thoughts?

Incidentally, I am not sure I understand what is PPI - I read it as peak-to-peak intervals, but on the H10 this is bundled with the heart rate - maybe this is separate on the Verity because the heart rate standard requires proper RR intervals, and those returned by the Verity are not?

Finally: I had gotten the PPG frame type wrong in _pmd_data_handler (I thought I'd add a safety check but I'd gotten it wrong), the last commit should have fixed it.

[HufflyCodes]

I will look into the ACC frames here and see what I can find, from the reading I have previously done I do remember there being a scaling factor but only for certain frame types? We will see if any more reading clears things up or makes it more convoluted haha, I know for example I read (directly from their issues forum and it wasn't corrected anywhere...) the three channels were IR, Red, Green thinking I could do some SPO2 analysis with this and NOPE they are all green so who knows what the forums will bring.

I think the code for decoding should be the same, I did my best to make it 'modular' with allowing inputs for frame and header length and such so that should work well there, assuming it doesn't break with new changes, again pending the conversion factor.
One worry I have with working on this end is double checking my work, I am uncertain how I could verify the values other than trying to setup a rig to drop it and see if I can get ~1g - 0g accurately as I don't have another accelerometer to test it against minus the one on my phone but who knows how accurate that is. As for how the scaling factor relates to the range or not I am sure, honestly I haven't even looked into what the range is in regards to, better said, I cannot find a specific definition for each different sensor. The comments list multiples of G for ACC but what about the other sensor units? Regardless I didn't notice that is where the scaling factor is generated, I figured it would be part of each packet but I wonder if it is static. Can you give me any details on what you know regarding the PMD sending the scaling factor over? All that being said I will mess around with it to see what I can figure out just playing with it.

They mention in their description of it that the PPI is: PP interval (milliseconds) representing cardiac pulse-to-pulse interval extracted from PPG signal. So same as the ECG one it seems, just different data source? I can try it out but also from that page they mention this

Warning

Polar Verity Sense PPI algorithm is a separate algorithm than the HR one used when PPI data is not being requested. When PPI recording is enabled, HR is only updated every 5 seconds. Also it takes around 25 seconds for the first sample batch to be sent to the offline recording file or over BLE for streaming. As PPI recording is incompatible with the notion of training, enabling PPI recording will abort any ongoing training (internal training or swimming).

If movement is detected, the heart rate is fixed to the last reliable value.

Also, attempting to set TRIGER_EXERCISE_START with PPI measurement type when using the offline recording triggered settings will return ERROR_NOT_SUPPORTED

[fsmeraldi]

Hello @HufflyCodes , apologies for the very late reply, I've been swamped by work

I will look into the ACC frames here and see what I can find, from the reading I have previously done I do remember there being a scaling factor but only for certain frame types? We will see if any more reading clears things up or makes it more convoluted haha, I know for example I read (directly from their issues forum and it wasn't corrected anywhere...) the three channels were IR, Red, Green thinking I could do some SPO2 analysis with this and NOPE they are all green so who knows what the forums will bring.

aah, that would have been nice... too bad it isn't possible!

I think the code for decoding should be the same, I did my best to make it 'modular' with allowing inputs for frame and header length and such so that should work well there, assuming it doesn't break with new changes, again pending the conversion factor. One worry I have with working on this end is double checking my work, I am uncertain how I could verify the values other than trying to setup a rig to drop it and see if I can get ~1g - 0g accurately as I don't have another accelerometer to test it against minus the one on my phone but who knows how accurate that is.

Well the scaling factor can be checked by making sure that the norm of the acceleration vector when the sensor is at rest is g. As for the rest, I would not know - fancy ways I have seen for testing accelerometers (in fact, with mobile phones) included putting them on the turntable of a vinyl player or attaching them to a string or a spring to form a pendulum... which may still result in dropping them, but only as an unintended consequence! ;)

As for how the scaling factor relates to the range or not I am sure, honestly I haven't even looked into what the range is in regards to, better said, I cannot find a specific definition for each different sensor. The comments list multiples of G for ACC but what about the other sensor units?

I'd agree the PPG sensor is probably in arbitrary units... ECG is in microvolt

Regardless I didn't notice that is where the scaling factor is generated, I figured it would be part of each packet but I wonder if it is static. Can you give me any details on what you know regarding the PMD sending the scaling factor over? All that being said I will mess around with it to see what I can figure out just playing with it.

From a note I had made at the end of start_streaming, it seems that it is indeed static and it is decided when you specify the parameters for ACC acquisition. I think I had gotten this idea from page 10 of this old specification. In that example, it is indeed for delta frames. I guess they do a range-dependent scaling so that they always encode only the significant digits, which is more efficient; if this is correct, a static value makes sense.

They mention in their description of it that the PPI is: PP interval (milliseconds) representing cardiac pulse-to-pulse interval extracted from PPG signal. So same as the ECG one it seems, just different data source? I can try it out but also from that page they mention this

Warning

Polar Verity Sense PPI algorithm is a separate algorithm than the HR one used when PPI data is not being requested. When PPI recording is enabled, HR is only updated every 5 seconds. Also it takes around 25 seconds for the first sample batch to be sent to the offline recording file or over BLE for streaming. As PPI recording is incompatible with the notion of training, enabling PPI recording will abort any ongoing training (internal training or swimming).
If movement is detected, the heart rate is fixed to the last reliable value.
Also, attempting to set TRIGER_EXERCISE_START with PPI measurement type when using the offline recording triggered settings will return ERROR_NOT_SUPPORTED

Thank you, that clarifies it!

[HufflyCodes]

No worries work has been the same for me, this is my first week back on a computer in over two weeks.

Well the scaling factor can be checked by making sure that the norm of the acceleration vector when the sensor is at rest is g. As for the rest, I would not know - fancy ways I have seen for testing accelerometers (in fact, with mobile phones) included putting them on the turntable of a vinyl player or attaching them to a string or a spring to form a pendulum... which may still result in dropping them, but only as an unintended consequence! ;)

Welp that is what thinking before doing will get you! That is much easier than the other methods and such. Once I get it spooled up I will look into that and try to derive the scaling factor.

From a note I had made at the end of start_streaming, it seems that it is indeed static and it is decided when you specify the parameters for ACC acquisition. I think I had gotten this idea from page 10 of this old specification. In that example, it is indeed for delta frames. I guess they do a range-dependent scaling so that they always encode only the significant digits, which is more efficient; if this is correct, a static value makes sense.

Speaking of the scaling factor I will look at the notes you left at the end of the start_streaming function but upon reading that specification, I am only left with more confusion as they somehow convert 964680256 to 0.000244 using the IEEE-754 Floating point converter which is the first I have ever encountered that so I will look into implementing that if you haven't already. Doesn't seem too hard upon initial googling, I am just unsure how I would modify reading it from the start_streaming function so once I have that figured out I will probably just add another helper function to convert the scaling factor and store it for decoding the values. Or should the scaling factor be simply printed out along with the rest of the data?

[fsmeraldi]

Hello Wesley, start_streaming as it stands returns (err_code, err_msg, response) where response is the raw control point response, that should be a bytearray. err_code is in fact response[3]. So as I understand it, from the Verity you should have:

response[4]==0 # no other frames
response[5]==5 # data type: factor
response[6]==1 # one value only
response[7:] # the actual factor (4 bytes)

I tried decoding the example given in the specification, this way seems to work:

import struct
factor_array=bytes.fromhex("40DA7F39")
struct.unpack('f', factor_array)
0.00024399999529123306

so basically struct.unpack(response[7:]) should do the trick. I think this should then be stored in some attribute of the PMD object and used by the decoding method to perform the correct scaling. Once we are convinced this is correct, the attribute could directly be set by start_streaming; I will need to check out what happens with the H10, maybe the response simply does not include that field at all.

[HufflyCodes]

Awesome thank you for the snippet! I will check it out to confirm the response returns what is expected, and I had no idea struct.unpack was so powerful, python is awesome!

For the H10 I would assume that is does not contain it at all as the documents do not list any conversion factors but that could be old so good to check.

So for storing the conversion factor that should be a part of core.py where it can then be used to call the conversion helper function? If that's the case I can work on getting that implemented after some initial testing to make sure that everything we have discussed checks out.

[fsmeraldi]

I just checked for the H10 there are no conversion factors - the control point response is 6 bytes only and seems to end in 0x01 for ACC and 0x00 for ECG. So I took it that 0x05 indicates the Verity and the factor, and in that case I added a couple of lines to decode the factor and save it in self._verity_acc_factor. This is initialised to 1.0 in the constructor; after you call start_streaming, it should be ready to use for decoding. I also added a printout of the raw control point response in examples accel_callback and ecg_queue, for ease of debugging.

By the way, I had some time today so I published a new release with the PPG decoding (thanks again!) and merged some minor changes (including renaming core.py to _core.py) into verity_dev. Let me know if you find any issues!