Skip to content

bpo-40280: Add build target for Emscripten/node.js #30495

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 11 commits into from
Closed

bpo-40280: Add build target for Emscripten/node.js #30495

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
717c09b
bpo-40280: Add build target for Emscripten/node.js
emmatyping Jan 9, 2022
5492cf8
Merge branch 'main' of github.com:python/cpython into nodejs
emmatyping Jan 9, 2022
a862bce
Update Makefile.pre.in
emmatyping Jan 10, 2022
6d88ce6
Change target name to python-rawfs.js
emmatyping Jan 10, 2022
3ee5687
Remove special casing for test files
emmatyping Jan 10, 2022
aadbd05
Merge branch 'nodejs' of github.com:ethanhs/cpython into nodejs
emmatyping Jan 10, 2022
ee99490
Fix threading for nodejs
emmatyping Jan 11, 2022
3a0dd1c
Update Makefile.pre.in
emmatyping Jan 11, 2022
faf52a4
Update Makefile.pre.in
emmatyping Jan 11, 2022
bdcaa48
Update Makefile.pre.in
emmatyping Jan 11, 2022
5284265
Update Modules/socketmodule.c
emmatyping Jan 11, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 12 additions & 2 deletions Makefile.pre.in
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ CC= @CC@
CXX= @CXX@
MAINCC= @MAINCC@
LINKCC= @LINKCC@
LINKCC_BUILDPYTHON = @LINKCC_BUILDPYTHON@
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the least intrusive way I could think of to customize the link step only for building the python.wasm executable...

If you know of a better way, please do tell :)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Meh! The ugly hacks are piling up. Maybe it's easier to not use ac_cv_pthread and to add -pthread to CFLAGS or CFLAGS_NODIST after all?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately then it would be passed to the $(BUILDPYTHON) target, and standalone wasm doesn't support -pthread

AR= @AR@
READELF= @READELF@
SOABI= @SOABI@
Expand Down Expand Up @@ -717,7 +718,7 @@ clinic: check-clean-src $(srcdir)/Modules/_blake2/blake2s_impl.c

# Build the interpreter
$(BUILDPYTHON): Programs/python.o $(LIBRARY_DEPS)
$(LINKCC) $(PY_CORE_LDFLAGS) $(LINKFORSHARED) -o $@ Programs/python.o $(BLDLIBRARY) $(LIBS) $(MODLIBS) $(SYSLIBS)
$(LINKCC_BUILDPYTHON) $(PY_CORE_LDFLAGS) $(LINKFORSHARED) -o $@ Programs/python.o $(BLDLIBRARY) $(LIBS) $(MODLIBS) $(SYSLIBS)

platform: $(BUILDPYTHON) pybuilddir.txt
$(RUNSHARED) $(PYTHON_FOR_BUILD) -c 'import sys ; from sysconfig import get_platform ; print("%s-%d.%d" % (get_platform(), *sys.version_info[:2]))' >platform
Expand Down Expand Up @@ -833,6 +834,7 @@ $(DLLLIBRARY) libpython$(LDVERSION).dll.a: $(LIBRARY_OBJS)
# wasm32-emscripten build
# wasm assets directory is relative to current build dir, e.g. "./usr/local".
# --preload-file turns a relative asset path into an absolute path.
# With node.js, we use the rawfs which is much easier for tests
WASM_ASSETS_DIR=".$(prefix)"
WASM_STDLIB="$(WASM_ASSETS_DIR)/local/lib/python$(VERSION)/os.py"

Expand All @@ -844,7 +846,15 @@ $(WASM_STDLIB): $(srcdir)/Lib/*.py $(srcdir)/Lib/*/*.py \
python.html: Programs/python.o $(LIBRARY_DEPS) $(WASM_STDLIB)
$(LINKCC) $(PY_CORE_LDFLAGS) $(LINKFORSHARED) -o $@ Programs/python.o \
$(BLDLIBRARY) $(LIBS) $(MODLIBS) $(SYSLIBS) \
-s ASSERTIONS=1 --preload-file $(WASM_ASSETS_DIR)
-s ASSERTIONS=1 -s ALLOW_MEMORY_GROWTH=1 -s USE_PTHREADS \
-s PROXY_TO_PTHREAD --preload-file $(WASM_ASSETS_DIR)

python-rawfs.js: Programs/python.o $(LIBRARY_DEPS) pybuilddir.txt
$(LINKCC) $(PY_CORE_LDFLAGS) $(LINKFORSHARED) -o $@ Programs/python.o \
$(BLDLIBRARY) $(LIBS) $(MODLIBS) $(SYSLIBS) \
-s ASSERTIONS=1 -s ALLOW_MEMORY_GROWTH=1 \
-s NODERAWFS=1 -s EXIT_RUNTIME=1 -s USE_PTHREADS \
-s PROXY_TO_PTHREAD

##########################################################################
# Build static libmpdec.a
Expand Down
2 changes: 1 addition & 1 deletion Modules/socketmodule.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7926,7 +7926,7 @@ PyInit__socket(void)
#ifdef IPPROTO_VRRP
PyModule_AddIntMacro(m, IPPROTO_VRRP);
#endif
#ifdef IPPROTO_SCTP
#if defined(IPPROTO_SCTP) && !defined(__EMSCRIPTEN__)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found a better way to deal with the issue. I guess you did the same mistake as I and added the skipif call after the check for AIX? That doesn't work because @requireSocket("AF_INET", "SOCK_STREAM", "IPPROTO_SCTP") tries to create an IPPROTO_SCTP socket and decorators are executed in reverse order!

This works:

--- a/Lib/test/test_socket.py
+++ b/Lib/test/test_socket.py
@@ -43,6 +43,7 @@
 
 VSOCKPORT = 1234
 AIX = platform.system() == "AIX"
+EMSCRIPTEN = sys.platform == "Emscripten"
 
 try:
     import _socket
...

 @requireAttrs(socket.socket, "sendmsg")
[email protected](AIX, "IPPROTO_SCTP: [Errno 62] Protocol not supported on AIX")
 @requireSocket("AF_INET", "SOCK_STREAM", "IPPROTO_SCTP")
[email protected](AIX, "IPPROTO_SCTP: [Errno 62] Protocol not supported on AIX")
[email protected](EMSCRIPTEN, "IPPROTO_SCTP: aborts on Emscripten")
 class SendmsgSCTPStreamTest(SendmsgStreamTests, SendrecvmsgSCTPStreamTestBase):

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see you've opened #30538 so I'll remove this change.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#30538 does not work ... it's still running the decorator and crashing node.js

PyModule_AddIntMacro(m, IPPROTO_SCTP);
#endif
#ifdef IPPROTO_BIP
Expand Down
7 changes: 7 additions & 0 deletions Tools/wasm/config.site-wasm32-emscripten
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,13 @@
# Partly based on pyodide's pyconfig.undefs.h file.
#

ac_exeext=.wasm

# Emscripten will build code that includes pthreads without the flag, but fail to run...
ac_cv_pthread_is_default=no
ac_cv_kthread=no
ac_cv_pthread=yes

# cannot be detected in cross builds
ac_cv_buggy_getaddrinfo=no

Expand Down
40 changes: 32 additions & 8 deletions configure
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -878,6 +878,7 @@ AR
GNULD
EXPORTSFROM
EXPORTSYMS
LINKCC_BUILDPYTHON
LINKCC
LDVERSION
RUNSHARED
Expand Down Expand Up @@ -6326,6 +6327,25 @@ fi
{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $LINKCC" >&5
$as_echo "$LINKCC" >&6; }


{ $as_echo "$as_me:${as_lineno-$LINENO}: checking LINKCC_BUILDPYTHON" >&5
$as_echo_n "checking LINKCC_BUILDPYTHON... " >&6; }
if test -z "$LINKCC_BUILDPYTHON"
then
# the .wasm executable doesn't support building with pthreads,
# so we hack around the symbol errors as they will be provided by
# Emscripten
case $ac_sys_system in #(
Emscripten) :
LINKCC_BUILDPYTHON="emcc -s ERROR_ON_UNDEFINED_SYMBOLS=0" ;; #(
*) :
LINKCC_BUILDPYTHON='$(LINKCC)'
;;
esac
fi
{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $LINKCC_BUILDPYTHON" >&5
$as_echo "$LINKCC_BUILDPYTHON" >&6; }

# EXPORTSYMS holds the list of exported symbols for AIX.
# EXPORTSFROM holds the module name exporting symbols on AIX.
EXPORTSYMS=
Expand Down Expand Up @@ -8276,7 +8296,6 @@ fi
{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_pthread_is_default" >&5
$as_echo "$ac_cv_pthread_is_default" >&6; }


if test $ac_cv_pthread_is_default = yes
then
ac_cv_kpthread=no
Expand Down Expand Up @@ -8451,16 +8470,21 @@ fi

if test $ac_cv_cxx_thread = yes
then
echo 'void foo();int main(){foo();}void foo(){}' > conftest.$ac_ext
$CXX -c conftest.$ac_ext 2>&5
if $CXX -o conftest$ac_exeext conftest.$ac_objext 2>&5 \
&& test -s conftest$ac_exeext && ./conftest$ac_exeext
then
if test "$cross_compiling" = yes; then
# assume that it works
ac_cv_cxx_thread=yes
else
ac_cv_cxx_thread=no
echo 'void foo();int main(){foo();}void foo(){}' > conftest.$ac_ext
$CXX -c conftest.$ac_ext 2>&5
if $CXX -o conftest$ac_exeext conftest.$ac_objext 2>&5 \
&& test -s conftest$ac_exeext && ./conftest$ac_exeext
then
ac_cv_cxx_thread=yes
else
ac_cv_cxx_thread=no
fi
rm -fr conftest*
fi
rm -fr conftest*
fi
{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_cxx_thread" >&5
$as_echo "$ac_cv_cxx_thread" >&6; }
Expand Down
34 changes: 26 additions & 8 deletions configure.ac
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1162,6 +1162,20 @@ then
fi
AC_MSG_RESULT($LINKCC)

AC_SUBST(LINKCC_BUILDPYTHON)
AC_MSG_CHECKING(LINKCC_BUILDPYTHON)
if test -z "$LINKCC_BUILDPYTHON"
then
# the .wasm executable doesn't support building with pthreads,
# so we hack around the symbol errors as they will be provided by
# Emscripten
AS_CASE([$ac_sys_system],
[Emscripten], [LINKCC_BUILDPYTHON="emcc -s ERROR_ON_UNDEFINED_SYMBOLS=0"],
[LINKCC_BUILDPYTHON='$(LINKCC)']
)
fi
AC_MSG_RESULT($LINKCC_BUILDPYTHON)

# EXPORTSYMS holds the list of exported symbols for AIX.
# EXPORTSFROM holds the module name exporting symbols on AIX.
EXPORTSYMS=
Expand Down Expand Up @@ -2125,7 +2139,6 @@ int main(){
],[ac_cv_pthread_is_default=no],[ac_cv_pthread_is_default=no])
])


if test $ac_cv_pthread_is_default = yes
then
ac_cv_kpthread=no
Expand Down Expand Up @@ -2233,16 +2246,21 @@ fi

if test $ac_cv_cxx_thread = yes
then
echo 'void foo();int main(){foo();}void foo(){}' > conftest.$ac_ext
$CXX -c conftest.$ac_ext 2>&5
if $CXX -o conftest$ac_exeext conftest.$ac_objext 2>&5 \
&& test -s conftest$ac_exeext && ./conftest$ac_exeext
then
if test "$cross_compiling" = yes; then
# assume that it works
ac_cv_cxx_thread=yes
else
ac_cv_cxx_thread=no
echo 'void foo();int main(){foo();}void foo(){}' > conftest.$ac_ext
$CXX -c conftest.$ac_ext 2>&5
if $CXX -o conftest$ac_exeext conftest.$ac_objext 2>&5 \
&& test -s conftest$ac_exeext && ./conftest$ac_exeext
then
ac_cv_cxx_thread=yes
else
ac_cv_cxx_thread=no
fi
rm -fr conftest*
fi
rm -fr conftest*
fi
AC_MSG_RESULT($ac_cv_cxx_thread)
fi
Expand Down