[Bug]: Project-file/code-component content is truncated when containing (some) emojis

[megholm]

PsychoPy Version

2024.2.1post4

What OS are your PsychoPy running on?

Windows 10

Bug Description

I discovered a problem with the code-component ('s (de)serializer).

Take a simple project with just a code-component and the following (xml project) content:

<Param val="kButtonRatingText1 = '☺'&amp;#10;kButtonRatingText2 = '🙂'&amp;#10;kButtonRatingText3 = '😐'&amp;#10;kButtonRatingText4 = '🙁'&amp;#10;kButtonRatingText5 = '☹'&amp;#10;kText = '123'" valType="extendedCode" updates="constant" name="Before Experiment"/>

or:

kButtonRatingText1 = '☺'

kButtonRatingText2 = '🙂'

kButtonRatingText3 = '😐'

kButtonRatingText4 = '🙁'

kButtonRatingText5 = '☹'

kText = '123'

if you want.

Opening the code-block again the text gets truncated to:

<Param val="kButtonRatingText1 = '☺'&amp;#10;kButtonRatingText2 = '🙂'&amp;#10;kButtonRatingText3 = '😐'&amp;#10;kButtonRatingText4 = '🙁'&amp;#10;kButtonRatingText5 = '☹'&amp;#10;kText = '1" valType="extendedCode" updates="constant" name="Before Experiment"/>

Expected Behaviour

No truncation.

Steps to Reproduce

...

Additional context

No response

[TEParsons]

If I run the following code to create an experiment, add a Code Component with this text, save it and load it again, the loaded Code Component is identical:

from psychopy.experiment import Experiment
from psychopy.experiment.routines import Routine
from psychopy.experiment.components.code import CodeComponent

# code content to test
code = (
    "kButtonRatingText1 = '☺'\n"
    "kButtonRatingText2 = '🙂'\n"
    "kButtonRatingText3 = '😐'\n"
    "kButtonRatingText4 = '🙁'\n"
    "kButtonRatingText5 = '☹'\n"
    "kText = '123'\n"
)
# make experiment, routine and component
exp = Experiment()
rt = Routine(
    name="test",
    exp=exp
)
comp = CodeComponent(
    name="testCode",
    parentName="test",
    exp=exp
)
# set component content
comp.params['Before Experiment'].val = code
# arrange them into a simple experiment flow
exp.addRoutine("test", rt)
exp.flow.addRoutine(rt, 0)
rt.addComponent(comp)
# save experiment
exp.saveToXML("test.psyexp")
# load experiment
loadedExp = Experiment.fromFile("test.psyexp")
# check content is unchanged
assert loadedExp.routines['test'][1].params['Before Experiment'].val == code

I can also save and reload an experiment in PsychoPy Studio with that code in a Code Component without this issue, the truncation seems to happen specifically when the code is passed through the text control in PsychoPy Standalone. So I think this is something wxPython does specifically, most likely because some of these emojis are multiple characters long but need to be treated as one character, so it messes up the indices and cuts characters off the end.

I'll transfer this issue to psychopy-app as that's where the wx app code lives now, but for a fix I would recommend switching to PsychoPy Studio.