diff --git a/Doc/c-api/code.rst b/Doc/c-api/code.rst index 6ae6bfe4aa6ab4..6eae24b38fae48 100644 --- a/Doc/c-api/code.rst +++ b/Doc/c-api/code.rst @@ -32,11 +32,13 @@ bound into a function. .. c:function:: Py_ssize_t PyCode_GetNumFree(PyCodeObject *co) - Return the number of free variables in a code object. + Return the number of :term:`free (closure) variables ` + in a code object. .. c:function:: int PyUnstable_Code_GetFirstFree(PyCodeObject *co) - Return the position of the first free variable in a code object. + Return the position of the first :term:`free (closure) variable ` + in a code object. .. versionchanged:: 3.13 @@ -144,7 +146,8 @@ bound into a function. Equivalent to the Python code ``getattr(co, 'co_freevars')``. Returns a new reference to a :c:type:`PyTupleObject` containing the names of - the free variables. On error, ``NULL`` is returned and an exception is raised. + the :term:`free (closure) variables `. On error, ``NULL`` is returned + and an exception is raised. .. versionadded:: 3.11 diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index 6d16e04ac3d864..8e0cf7bb0fc088 100644 --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -7,7 +7,8 @@ Initialization, Finalization, and Threads ***************************************** -See also the :ref:`Python Initialization Configuration `. +See :ref:`Python Initialization Configuration ` for details +on how to configure the interpreter prior to initialization. .. _pre-init-safe: @@ -21,6 +22,15 @@ a few functions and the :ref:`global configuration variables The following functions can be safely called before Python is initialized: +* Functions that initialize the interpreter: + + * :c:func:`Py_Initialize` + * :c:func:`Py_InitializeEx` + * :c:func:`Py_InitializeFromConfig` + * :c:func:`Py_BytesMain` + * :c:func:`Py_Main` + * the runtime pre-initialization functions covered in :ref:`init-config` + * Configuration functions: * :c:func:`PyImport_AppendInittab` @@ -32,6 +42,7 @@ The following functions can be safely called before Python is initialized: * :c:func:`Py_SetProgramName` * :c:func:`Py_SetPythonHome` * :c:func:`PySys_ResetWarnOptions` + * the configuration functions covered in :ref:`init-config` * Informative functions: @@ -43,10 +54,12 @@ The following functions can be safely called before Python is initialized: * :c:func:`Py_GetCopyright` * :c:func:`Py_GetPlatform` * :c:func:`Py_GetVersion` + * :c:func:`Py_IsInitialized` * Utilities: * :c:func:`Py_DecodeLocale` + * the status reporting and utility functions covered in :ref:`init-config` * Memory allocators: @@ -62,11 +75,13 @@ The following functions can be safely called before Python is initialized: .. note:: - The following functions **should not be called** before - :c:func:`Py_Initialize`: :c:func:`Py_EncodeLocale`, :c:func:`Py_GetPath`, + Despite their apparent similarity to some of the functions listed above, + the following functions **should not be called** before the interpreter has + been initialized: :c:func:`Py_EncodeLocale`, :c:func:`Py_GetPath`, :c:func:`Py_GetPrefix`, :c:func:`Py_GetExecPrefix`, :c:func:`Py_GetProgramFullPath`, :c:func:`Py_GetPythonHome`, - :c:func:`Py_GetProgramName` and :c:func:`PyEval_InitThreads`. + :c:func:`Py_GetProgramName`, :c:func:`PyEval_InitThreads`, and + :c:func:`Py_RunMain`. .. _global-conf-vars: @@ -346,34 +361,42 @@ Initializing and finalizing the interpreter this should be called before using any other Python/C API functions; see :ref:`Before Python Initialization ` for the few exceptions. - This initializes - the table of loaded modules (``sys.modules``), and creates the fundamental - modules :mod:`builtins`, :mod:`__main__` and :mod:`sys`. It also initializes - the module search path (``sys.path``). It does not set ``sys.argv``; use - the new :c:type:`PyConfig` API of the :ref:`Python Initialization - Configuration ` for that. This is a no-op when called for a - second time - (without calling :c:func:`Py_FinalizeEx` first). There is no return value; it is a - fatal error if the initialization fails. - - Use the :c:func:`Py_InitializeFromConfig` function to customize the + This initializes the table of loaded modules (``sys.modules``), and creates + the fundamental modules :mod:`builtins`, :mod:`__main__` and :mod:`sys`. + It also initializes the module search path (``sys.path``). It does not set + ``sys.argv``; use the :ref:`Python Initialization Configuration ` + API for that. This is a no-op when called for a second time (without calling + :c:func:`Py_FinalizeEx` first). There is no return value; it is a fatal + error if the initialization fails. + + Use :c:func:`Py_InitializeFromConfig` to customize the :ref:`Python Initialization Configuration `. .. note:: - On Windows, changes the console mode from ``O_TEXT`` to ``O_BINARY``, which will - also affect non-Python uses of the console using the C Runtime. + On Windows, changes the console mode from ``O_TEXT`` to ``O_BINARY``, + which will also affect non-Python uses of the console using the C Runtime. .. c:function:: void Py_InitializeEx(int initsigs) This function works like :c:func:`Py_Initialize` if *initsigs* is ``1``. If - *initsigs* is ``0``, it skips initialization registration of signal handlers, which - might be useful when Python is embedded. + *initsigs* is ``0``, it skips initialization registration of signal handlers, + which may be useful when CPython is embedded as part of a larger application. - Use the :c:func:`Py_InitializeFromConfig` function to customize the + Use :c:func:`Py_InitializeFromConfig` to customize the :ref:`Python Initialization Configuration `. +.. c:function:: PyStatus Py_InitializeFromConfig(const PyConfig *config) + + Initialize Python from *config* configuration, as described in + :ref:`init-from-config`. + + See the :ref:`init-config` section for details on pre-initializing the + interpreter, populating the runtime configuration structure, and querying + the returned status structure. + + .. c:function:: int Py_IsInitialized() Return true (nonzero) when the Python interpreter has been initialized, false @@ -440,12 +463,111 @@ Initializing and finalizing the interpreter .. versionadded:: 3.6 + .. c:function:: void Py_Finalize() This is a backwards-compatible version of :c:func:`Py_FinalizeEx` that disregards the return value. +.. c:function:: int Py_BytesMain(int argc, char **argv) + + Similar to :c:func:`Py_Main` but *argv* is an array of bytes strings, + allowing the calling application to delegate the text decoding step to + the CPython runtime. + + .. versionadded:: 3.8 + + +.. c:function:: int Py_Main(int argc, wchar_t **argv) + + The main program for the standard interpreter, encapsulating a full + initialization/finalization cycle, as well as additional + behaviour to implement reading configurations settings from the environment + and command line, and then executing ``__main__`` in accordance with + :ref:`using-on-cmdline`. + + This is made available for programs which wish to support the full CPython + command line interface, rather than just embedding a Python runtime in a + larger application. + + The *argc* and *argv* parameters are similar to those which are passed to a + C program's :c:func:`main` function, except that the *argv* entries are first + converted to ``wchar_t`` using :c:func:`Py_DecodeLocale`. It is also + important to note that the argument list entries may be modified to point to + strings other than those passed in (however, the contents of the strings + pointed to by the argument list are not modified). + + The return value will be ``0`` if the interpreter exits normally (i.e., + without an exception), ``1`` if the interpreter exits due to an exception, + or ``2`` if the argument list does not represent a valid Python command + line. + + Note that if an otherwise unhandled :exc:`SystemExit` is raised, this + function will not return ``1``, but exit the process, as long as + ``Py_InspectFlag`` is not set. If ``Py_InspectFlag`` is set, execution will + drop into the interactive Python prompt, at which point a second otherwise + unhandled :exc:`SystemExit` will still exit the process, while any other + means of exiting will set the return value as described above. + + In terms of the CPython runtime configuration APIs documented in the + :ref:`runtime configuration ` section (and without accounting + for error handling), ``Py_Main`` is approximately equivalent to:: + + PyConfig config; + PyConfig_InitPythonConfig(&config); + PyConfig_SetArgv(&config, argc, argv); + Py_InitializeFromConfig(&config); + PyConfig_Clear(&config); + + Py_RunMain(); + + In normal usage, an embedding application will call this function + *instead* of calling :c:func:`Py_Initialize`, :c:func:`Py_InitializeEx` or + :c:func:`Py_InitializeFromConfig` directly, and all settings will be applied + as described elsewhere in this documentation. If this function is instead + called *after* a preceding runtime initialization API call, then exactly + which environmental and command line configuration settings will be updated + is version dependent (as it depends on which settings correctly support + being modified after they have already been set once when the runtime was + first initialized). + + +.. c:function:: int Py_RunMain(void) + + Executes the main module in a fully configured CPython runtime. + + Executes the command (:c:member:`PyConfig.run_command`), the script + (:c:member:`PyConfig.run_filename`) or the module + (:c:member:`PyConfig.run_module`) specified on the command line or in the + configuration. If none of these values are set, runs the interactive Python + prompt (REPL) using the ``__main__`` module's global namespace. + + If :c:member:`PyConfig.inspect` is not set (the default), the return value + will be ``0`` if the interpreter exits normally (that is, without raising + an exception), or ``1`` if the interpreter exits due to an exception. If an + otherwise unhandled :exc:`SystemExit` is raised, the function will immediately + exit the process instead of returning ``1``. + + If :c:member:`PyConfig.inspect` is set (such as when the :option:`-i` option + is used), rather than returning when the interpreter exits, execution will + instead resume in an interactive Python prompt (REPL) using the ``__main__`` + module's global namespace. If the interpreter exited with an exception, it + is immediately raised in the REPL session. The function return value is + then determined by the way the *REPL session* terminates: returning ``0`` + if the session terminates without raising an unhandled exception, exiting + immediately for an unhandled :exc:`SystemExit`, and returning ``1`` for + any other unhandled exception. + + This function always finalizes the Python interpreter regardless of whether + it returns a value or immediately exits the process due to an unhandled + :exc:`SystemExit` exception. + + See :ref:`Python Configuration ` for an example of a + customized Python that always runs in isolated mode using + :c:func:`Py_RunMain`. + + Process-wide parameters ======================= diff --git a/Doc/c-api/init_config.rst b/Doc/c-api/init_config.rst index 9dc9ba61e7a60f..6f8962afc7af0d 100644 --- a/Doc/c-api/init_config.rst +++ b/Doc/c-api/init_config.rst @@ -1356,14 +1356,13 @@ the :option:`-X` command line option. The ``show_alloc_count`` field has been removed. +.. _init-from-config: + Initialization with PyConfig ---------------------------- -Function to initialize Python: - -.. c:function:: PyStatus Py_InitializeFromConfig(const PyConfig *config) - - Initialize Python from *config* configuration. +Initializing the interpreter from a populated configuration struct is handled +by calling :c:func:`Py_InitializeFromConfig`. The caller is responsible to handle exceptions (error or exit) using :c:func:`PyStatus_Exception` and :c:func:`Py_ExitStatusException`. @@ -1835,26 +1834,6 @@ return ``-1`` on error: } -Py_RunMain() -============ - -.. c:function:: int Py_RunMain(void) - - Execute the command (:c:member:`PyConfig.run_command`), the script - (:c:member:`PyConfig.run_filename`) or the module - (:c:member:`PyConfig.run_module`) specified on the command line or in the - configuration. - - By default and when if :option:`-i` option is used, run the REPL. - - Finally, finalizes Python and returns an exit status that can be passed to - the ``exit()`` function. - -See :ref:`Python Configuration ` for an example of -customized Python always running in isolated mode using -:c:func:`Py_RunMain`. - - Runtime Python configuration API ================================ diff --git a/Doc/c-api/unicode.rst b/Doc/c-api/unicode.rst index b2ac0c903c2bd7..f5704cffa199a5 100644 --- a/Doc/c-api/unicode.rst +++ b/Doc/c-api/unicode.rst @@ -1438,6 +1438,31 @@ They all return ``NULL`` or ``-1`` if an exception occurs. This function returns ``-1`` upon failure, so one should call :c:func:`PyErr_Occurred` to check for errors. + .. seealso:: + + The :c:func:`PyUnicode_Equal` function. + + +.. c:function:: int PyUnicode_Equal(PyObject *a, PyObject *b) + + Test if two strings are equal: + + * Return ``1`` if *a* is equal to *b*. + * Return ``0`` if *a* is not equal to *b*. + * Set a :exc:`TypeError` exception and return ``-1`` if *a* or *b* is not a + :class:`str` object. + + The function always succeeds if *a* and *b* are :class:`str` objects. + + The function works for :class:`str` subclasses, but does not honor custom + ``__eq__()`` method. + + .. seealso:: + + The :c:func:`PyUnicode_Compare` function. + + .. versionadded:: 3.14 + .. c:function:: int PyUnicode_EqualToUTF8AndSize(PyObject *unicode, const char *string, Py_ssize_t size) diff --git a/Doc/c-api/veryhigh.rst b/Doc/c-api/veryhigh.rst index 67167444d0a685..9f02bdb5896563 100644 --- a/Doc/c-api/veryhigh.rst +++ b/Doc/c-api/veryhigh.rst @@ -25,30 +25,6 @@ are only passed to these functions if it is certain that they were created by the same library that the Python runtime is using. -.. c:function:: int Py_Main(int argc, wchar_t **argv) - - The main program for the standard interpreter. This is made available for - programs which embed Python. The *argc* and *argv* parameters should be - prepared exactly as those which are passed to a C program's :c:func:`main` - function (converted to wchar_t according to the user's locale). It is - important to note that the argument list may be modified (but the contents of - the strings pointed to by the argument list are not). The return value will - be ``0`` if the interpreter exits normally (i.e., without an exception), - ``1`` if the interpreter exits due to an exception, or ``2`` if the parameter - list does not represent a valid Python command line. - - Note that if an otherwise unhandled :exc:`SystemExit` is raised, this - function will not return ``1``, but exit the process, as long as - :c:member:`PyConfig.inspect` is zero. - - -.. c:function:: int Py_BytesMain(int argc, char **argv) - - Similar to :c:func:`Py_Main` but *argv* is an array of bytes strings. - - .. versionadded:: 3.8 - - .. c:function:: int PyRun_AnyFile(FILE *fp, const char *filename) This is a simplified interface to :c:func:`PyRun_AnyFileExFlags` below, leaving diff --git a/Doc/data/stable_abi.dat b/Doc/data/stable_abi.dat index 19dc71a345b474..9314facd2ad873 100644 --- a/Doc/data/stable_abi.dat +++ b/Doc/data/stable_abi.dat @@ -783,6 +783,7 @@ func,PyUnicode_DecodeUnicodeEscape,3.2,, func,PyUnicode_EncodeCodePage,3.7,on Windows, func,PyUnicode_EncodeFSDefault,3.2,, func,PyUnicode_EncodeLocale,3.7,, +func,PyUnicode_Equal,3.14,, func,PyUnicode_EqualToUTF8,3.13,, func,PyUnicode_EqualToUTF8AndSize,3.13,, func,PyUnicode_FSConverter,3.2,, diff --git a/Doc/glossary.rst b/Doc/glossary.rst index e72a8d002d507d..933fb0319452a6 100644 --- a/Doc/glossary.rst +++ b/Doc/glossary.rst @@ -231,6 +231,28 @@ Glossary A variable defined in a class and intended to be modified only at class level (i.e., not in an instance of the class). + closure variable + A :term:`free variable` referenced from a :term:`nested scope` that is defined in an outer + scope rather than being resolved at runtime from the globals or builtin namespaces. + May be explicitly defined with the :keyword:`nonlocal` keyword to allow write access, + or implicitly defined if the variable is only being read. + + For example, in the ``inner`` function in the following code, both ``x`` and ``print`` are + :term:`free variables `, but only ``x`` is a *closure variable*:: + + def outer(): + x = 0 + def inner(): + nonlocal x + x += 1 + print(x) + return inner + + Due to the :attr:`codeobject.co_freevars` attribute (which, despite its name, only + includes the names of closure variables rather than listing all referenced free + variables), the more general :term:`free variable` term is sometimes used even + when the intended meaning is to refer specifically to closure variables. + complex number An extension of the familiar real number system in which all numbers are expressed as a sum of a real part and an imaginary part. Imaginary @@ -454,6 +476,13 @@ Glossary the :term:`global interpreter lock` which allows only one thread to execute Python bytecode at a time. See :pep:`703`. + free variable + Formally, as defined in the :ref:`language execution model `, a free + variable is any variable used in a namespace which is not a local variable in that + namespace. See :term:`closure variable` for an example. + Pragmatically, due to the name of the :attr:`codeobject.co_freevars` attribute, + the term is also sometimes used as a synonym for :term:`closure variable`. + function A series of statements which returns some value to a caller. It can also be passed zero or more :term:`arguments ` which may be used in diff --git a/Doc/howto/argparse-optparse.rst b/Doc/howto/argparse-optparse.rst new file mode 100644 index 00000000000000..cef2d893b28a62 --- /dev/null +++ b/Doc/howto/argparse-optparse.rst @@ -0,0 +1,55 @@ +.. currentmodule:: argparse + +.. _upgrading-optparse-code: + +========================== +Upgrading optparse code +========================== + +Originally, the :mod:`argparse` module had attempted to maintain compatibility +with :mod:`optparse`. However, :mod:`optparse` was difficult to extend +transparently, particularly with the changes required to support +``nargs=`` specifiers and better usage messages. When most everything in +:mod:`optparse` had either been copy-pasted over or monkey-patched, it no +longer seemed practical to try to maintain the backwards compatibility. + +The :mod:`argparse` module improves on the :mod:`optparse` +module in a number of ways including: + +* Handling positional arguments. +* Supporting subcommands. +* Allowing alternative option prefixes like ``+`` and ``/``. +* Handling zero-or-more and one-or-more style arguments. +* Producing more informative usage messages. +* Providing a much simpler interface for custom ``type`` and ``action``. + +A partial upgrade path from :mod:`optparse` to :mod:`argparse`: + +* Replace all :meth:`optparse.OptionParser.add_option` calls with + :meth:`ArgumentParser.add_argument` calls. + +* Replace ``(options, args) = parser.parse_args()`` with ``args = + parser.parse_args()`` and add additional :meth:`ArgumentParser.add_argument` + calls for the positional arguments. Keep in mind that what was previously + called ``options``, now in the :mod:`argparse` context is called ``args``. + +* Replace :meth:`optparse.OptionParser.disable_interspersed_args` + by using :meth:`~ArgumentParser.parse_intermixed_args` instead of + :meth:`~ArgumentParser.parse_args`. + +* Replace callback actions and the ``callback_*`` keyword arguments with + ``type`` or ``action`` arguments. + +* Replace string names for ``type`` keyword arguments with the corresponding + type objects (e.g. int, float, complex, etc). + +* Replace :class:`optparse.Values` with :class:`Namespace` and + :exc:`optparse.OptionError` and :exc:`optparse.OptionValueError` with + :exc:`ArgumentError`. + +* Replace strings with implicit arguments such as ``%default`` or ``%prog`` with + the standard Python syntax to use dictionaries to format strings, that is, + ``%(default)s`` and ``%(prog)s``. + +* Replace the OptionParser constructor ``version`` argument with a call to + ``parser.add_argument('--version', action='version', version='')``. diff --git a/Doc/howto/sockets.rst b/Doc/howto/sockets.rst index 0bbf97da39768d..cbc49d15a0771b 100644 --- a/Doc/howto/sockets.rst +++ b/Doc/howto/sockets.rst @@ -100,8 +100,8 @@ mainloop of the web server:: (clientsocket, address) = serversocket.accept() # now do something with the clientsocket # in this case, we'll pretend this is a threaded server - ct = client_thread(clientsocket) - ct.run() + ct = make_client_thread(clientsocket) + ct.start() There's actually 3 general ways in which this loop could work - dispatching a thread to handle ``clientsocket``, create a new process to handle diff --git a/Doc/library/argparse.rst b/Doc/library/argparse.rst index 83d0a9ed7b1d0a..e9a08984f77c3a 100644 --- a/Doc/library/argparse.rst +++ b/Doc/library/argparse.rst @@ -1,4 +1,4 @@ -:mod:`!argparse` --- Parser for command-line options, arguments and sub-commands +:mod:`!argparse` --- Parser for command-line options, arguments and subcommands ================================================================================ .. module:: argparse @@ -19,36 +19,13 @@ introduction to Python command-line parsing, have a look at the :ref:`argparse tutorial `. -The :mod:`argparse` module makes it easy to write user-friendly command-line -interfaces. The program defines what arguments it requires, and :mod:`argparse` -will figure out how to parse those out of :data:`sys.argv`. The :mod:`argparse` +The :mod:`!argparse` module makes it easy to write user-friendly command-line +interfaces. The program defines what arguments it requires, and :mod:`!argparse` +will figure out how to parse those out of :data:`sys.argv`. The :mod:`!argparse` module also automatically generates help and usage messages. The module will also issue errors when users give the program invalid arguments. -Quick Links for ArgumentParser ---------------------------------------- -========================= =========================================================================================================== ================================================================================== -Name Description Values -========================= =========================================================================================================== ================================================================================== -prog_ The name of the program -usage_ The string describing the program usage -description_ A brief description of what the program does -epilog_ Additional description of the program after the argument help -parents_ A list of :class:`ArgumentParser` objects whose arguments should also be included -formatter_class_ A class for customizing the help output ``argparse.HelpFormatter`` -prefix_chars_ The set of characters that prefix optional arguments Defaults to ``'-'`` -fromfile_prefix_chars_ The set of characters that prefix files to read additional arguments from Defaults to ``None`` (meaning arguments will never be treated as file references) -argument_default_ The global default value for arguments -allow_abbrev_ Allows long options to be abbreviated if the abbreviation is unambiguous ``True`` or ``False`` (default: ``True``) -conflict_handler_ The strategy for resolving conflicting optionals -add_help_ Add a ``-h/--help`` option to the parser ``True`` or ``False`` (default: ``True``) -exit_on_error_ Determines whether or not to exit with error info when an error occurs ``True`` or ``False`` (default: ``True``) -========================= =========================================================================================================== ================================================================================== - -Core Functionality ------------------- - -The :mod:`argparse` module's support for command-line interfaces is built +The :mod:`!argparse` module's support for command-line interfaces is built around an instance of :class:`argparse.ArgumentParser`. It is a container for argument specifications and has options that apply to the parser as whole:: @@ -72,133 +49,9 @@ the extracted data in a :class:`argparse.Namespace` object:: args = parser.parse_args() print(args.filename, args.count, args.verbose) - -Quick Links for add_argument() ------------------------------- - -============================ =========================================================== ========================================================================================================================== -Name Description Values -============================ =========================================================== ========================================================================================================================== -action_ Specify how an argument should be handled ``'store'``, ``'store_const'``, ``'store_true'``, ``'append'``, ``'append_const'``, ``'count'``, ``'help'``, ``'version'`` -choices_ Limit values to a specific set of choices ``['foo', 'bar']``, ``range(1, 10)``, or :class:`~collections.abc.Container` instance -const_ Store a constant value -default_ Default value used when an argument is not provided Defaults to ``None`` -dest_ Specify the attribute name used in the result namespace -help_ Help message for an argument -metavar_ Alternate display name for the argument as shown in help -nargs_ Number of times the argument can be used :class:`int`, ``'?'``, ``'*'``, or ``'+'`` -required_ Indicate whether an argument is required or optional ``True`` or ``False`` -:ref:`type ` Automatically convert an argument to the given type :class:`int`, :class:`float`, ``argparse.FileType('w')``, or callable function -============================ =========================================================== ========================================================================================================================== - - -Example -------- - -The following code is a Python program that takes a list of integers and -produces either the sum or the max:: - - import argparse - - parser = argparse.ArgumentParser(description='Process some integers.') - parser.add_argument('integers', metavar='N', type=int, nargs='+', - help='an integer for the accumulator') - parser.add_argument('--sum', dest='accumulate', action='store_const', - const=sum, default=max, - help='sum the integers (default: find the max)') - - args = parser.parse_args() - print(args.accumulate(args.integers)) - -Assuming the above Python code is saved into a file called ``prog.py``, it can -be run at the command line and it provides useful help messages: - -.. code-block:: shell-session - - $ python prog.py -h - usage: prog.py [-h] [--sum] N [N ...] - - Process some integers. - - positional arguments: - N an integer for the accumulator - - options: - -h, --help show this help message and exit - --sum sum the integers (default: find the max) - -When run with the appropriate arguments, it prints either the sum or the max of -the command-line integers: - -.. code-block:: shell-session - - $ python prog.py 1 2 3 4 - 4 - - $ python prog.py 1 2 3 4 --sum - 10 - -If invalid arguments are passed in, an error will be displayed: - -.. code-block:: shell-session - - $ python prog.py a b c - usage: prog.py [-h] [--sum] N [N ...] - prog.py: error: argument N: invalid int value: 'a' - -The following sections walk you through this example. - - -Creating a parser -^^^^^^^^^^^^^^^^^ - -The first step in using the :mod:`argparse` is creating an -:class:`ArgumentParser` object:: - - >>> parser = argparse.ArgumentParser(description='Process some integers.') - -The :class:`ArgumentParser` object will hold all the information necessary to -parse the command line into Python data types. - - -Adding arguments -^^^^^^^^^^^^^^^^ - -Filling an :class:`ArgumentParser` with information about program arguments is -done by making calls to the :meth:`~ArgumentParser.add_argument` method. -Generally, these calls tell the :class:`ArgumentParser` how to take the strings -on the command line and turn them into objects. This information is stored and -used when :meth:`~ArgumentParser.parse_args` is called. For example:: - - >>> parser.add_argument('integers', metavar='N', type=int, nargs='+', - ... help='an integer for the accumulator') - >>> parser.add_argument('--sum', dest='accumulate', action='store_const', - ... const=sum, default=max, - ... help='sum the integers (default: find the max)') - -Later, calling :meth:`~ArgumentParser.parse_args` will return an object with -two attributes, ``integers`` and ``accumulate``. The ``integers`` attribute -will be a list of one or more integers, and the ``accumulate`` attribute will be -either the :func:`sum` function, if ``--sum`` was specified at the command line, -or the :func:`max` function if it was not. - - -Parsing arguments -^^^^^^^^^^^^^^^^^ - -:class:`ArgumentParser` parses arguments through the -:meth:`~ArgumentParser.parse_args` method. This will inspect the command line, -convert each argument to the appropriate type and then invoke the appropriate action. -In most cases, this means a simple :class:`Namespace` object will be built up from -attributes parsed out of the command line:: - - >>> parser.parse_args(['--sum', '7', '-1', '42']) - Namespace(accumulate=, integers=[7, -1, 42]) - -In a script, :meth:`~ArgumentParser.parse_args` will typically be called with no -arguments, and the :class:`ArgumentParser` will automatically determine the -command-line arguments from :data:`sys.argv`. - +.. note:: + If you're looking a guide about how to upgrade optparse code + to argparse, see :ref:`Upgrading Optparse Code `. ArgumentParser objects ---------------------- @@ -268,8 +121,9 @@ The following sections describe how each of these are used. prog ^^^^ + By default, :class:`ArgumentParser` calculates the name of the program -to display in help messages depending on the way the Python inerpreter was run: +to display in help messages depending on the way the Python interpreter was run: * The :func:`base name ` of ``sys.argv[0]`` if a file was passed as argument. @@ -278,48 +132,10 @@ to display in help messages depending on the way the Python inerpreter was run: * The Python interpreter name followed by ``-m`` followed by the module or package name if the :option:`-m` option was used. -This default is almost -always desirable because it will make the help messages match the string that was -used to invoke the program on the command line. For example, consider a file -named ``myprogram.py`` with the following code:: - - import argparse - parser = argparse.ArgumentParser() - parser.add_argument('--foo', help='foo help') - args = parser.parse_args() - -The help for this program will display ``myprogram.py`` as the program name -(regardless of where the program was invoked from) if it is run as a script: - -.. code-block:: shell-session - - $ python myprogram.py --help - usage: myprogram.py [-h] [--foo FOO] - - options: - -h, --help show this help message and exit - --foo FOO foo help - $ cd .. - $ python subdir/myprogram.py --help - usage: myprogram.py [-h] [--foo FOO] - - options: - -h, --help show this help message and exit - --foo FOO foo help - -If it is executed via the :option:`-m` option, the help will display a corresponding command line: - -.. code-block:: shell-session - - $ /usr/bin/python3 -m subdir.myprogram --help - usage: python3 -m subdir.myprogram [-h] [--foo FOO] - - options: - -h, --help show this help message and exit - --foo FOO foo help - -To change this default behavior, another value can be supplied using the -``prog=`` argument to :class:`ArgumentParser`:: +This default is almost always desirable because it will make the help messages +match the string that was used to invoke the program on the command line. +However, to change this default behavior, another value can be supplied using +the ``prog=`` argument to :class:`ArgumentParser`:: >>> parser = argparse.ArgumentParser(prog='myprogram') >>> parser.print_help() @@ -352,22 +168,8 @@ usage ^^^^^ By default, :class:`ArgumentParser` calculates the usage message from the -arguments it contains:: - - >>> parser = argparse.ArgumentParser(prog='PROG') - >>> parser.add_argument('--foo', nargs='?', help='foo help') - >>> parser.add_argument('bar', nargs='+', help='bar help') - >>> parser.print_help() - usage: PROG [-h] [--foo [FOO]] bar [bar ...] - - positional arguments: - bar bar help - - options: - -h, --help show this help message and exit - --foo [FOO] foo help - -The default message can be overridden with the ``usage=`` keyword argument:: +arguments it contains. The default message can be overridden with the +``usage=`` keyword argument:: >>> parser = argparse.ArgumentParser(prog='PROG', usage='%(prog)s [options]') >>> parser.add_argument('--foo', nargs='?', help='foo help') @@ -395,16 +197,7 @@ Most calls to the :class:`ArgumentParser` constructor will use the ``description=`` keyword argument. This argument gives a brief description of what the program does and how it works. In help messages, the description is displayed between the command-line usage string and the help messages for the -various arguments:: - - >>> parser = argparse.ArgumentParser(description='A foo that bars') - >>> parser.print_help() - usage: argparse.py [-h] - - A foo that bars - - options: - -h, --help show this help message and exit +various arguments. By default, the description will be line-wrapped so that it fits within the given space. To change this behavior, see the formatter_class_ argument. @@ -534,7 +327,7 @@ should not be line-wrapped:: -h, --help show this help message and exit :class:`RawTextHelpFormatter` maintains whitespace for all sorts of help text, -including argument descriptions. However, multiple new lines are replaced with +including argument descriptions. However, multiple newlines are replaced with one. If you wish to preserve multiple blank lines, add spaces between the newlines. @@ -628,8 +421,8 @@ arguments will never be treated as file references. .. versionchanged:: 3.12 :class:`ArgumentParser` changed encoding and errors to read arguments files - from default (e.g. :func:`locale.getpreferredencoding(False) ` and - ``"strict"``) to :term:`filesystem encoding and error handler`. + from default (e.g. :func:`locale.getpreferredencoding(False) ` + and ``"strict"``) to the :term:`filesystem encoding and error handler`. Arguments file should be encoded in UTF-8 instead of ANSI Codepage on Windows. @@ -715,25 +508,8 @@ add_help ^^^^^^^^ By default, ArgumentParser objects add an option which simply displays -the parser's help message. For example, consider a file named -``myprogram.py`` containing the following code:: - - import argparse - parser = argparse.ArgumentParser() - parser.add_argument('--foo', help='foo help') - args = parser.parse_args() - -If ``-h`` or ``--help`` is supplied at the command line, the ArgumentParser -help will be printed: - -.. code-block:: shell-session - - $ python myprogram.py --help - usage: myprogram.py [-h] [--foo FOO] - - options: - -h, --help show this help message and exit - --foo FOO foo help +the parser's help message. If ``-h`` or ``--help`` is supplied at the command +line, the ArgumentParser help will be printed. Occasionally, it may be useful to disable the addition of this help option. This can be achieved by passing ``False`` as the ``add_help=`` argument to @@ -872,12 +648,7 @@ them, though most actions simply add an attribute to the object returned by how the command-line arguments should be handled. The supplied actions are: * ``'store'`` - This just stores the argument's value. This is the default - action. For example:: - - >>> parser = argparse.ArgumentParser() - >>> parser.add_argument('--foo') - >>> parser.parse_args('--foo 1'.split()) - Namespace(foo='1') + action. * ``'store_const'`` - This stores the value specified by the const_ keyword argument; note that the const_ keyword argument defaults to ``None``. The @@ -892,7 +663,7 @@ how the command-line arguments should be handled. The supplied actions are: * ``'store_true'`` and ``'store_false'`` - These are special cases of ``'store_const'`` used for storing the values ``True`` and ``False`` respectively. In addition, they create default values of ``False`` and - ``True`` respectively. For example:: + ``True`` respectively:: >>> parser = argparse.ArgumentParser() >>> parser.add_argument('--foo', action='store_true') @@ -1137,7 +908,7 @@ was not present at the command line:: Namespace(foo=42) If the target namespace already has an attribute set, the action *default* -will not over write it:: +will not overwrite it:: >>> parser = argparse.ArgumentParser() >>> parser.add_argument('--foo', default=42) @@ -1211,7 +982,6 @@ Common built-in types and functions can be used as type converters: parser.add_argument('distance', type=float) parser.add_argument('street', type=ascii) parser.add_argument('code_point', type=ord) - parser.add_argument('source_file', type=open) parser.add_argument('dest_file', type=argparse.FileType('w', encoding='latin-1')) parser.add_argument('datapath', type=pathlib.Path) @@ -1242,10 +1012,11 @@ better reporting than can be given by the ``type`` keyword. A :exc:`FileNotFoundError` exception would not be handled at all. Even :class:`~argparse.FileType` has its limitations for use with the ``type`` -keyword. If one argument uses *FileType* and then a subsequent argument fails, -an error is reported but the file is not automatically closed. In this case, it -would be better to wait until after the parser has run and then use the -:keyword:`with`-statement to manage the files. +keyword. If one argument uses :class:`~argparse.FileType` and then a +subsequent argument fails, an error is reported but the file is not +automatically closed. In this case, it would be better to wait until after +the parser has run and then use the :keyword:`with`-statement to manage the +files. For type checkers that simply check against a fixed set of values, consider using the choices_ keyword instead. @@ -1273,15 +1044,7 @@ if the argument was not one of the acceptable values:: Note that inclusion in the *choices* sequence is checked after any type_ conversions have been performed, so the type of the objects in the *choices* -sequence should match the type_ specified:: - - >>> parser = argparse.ArgumentParser(prog='doors.py') - >>> parser.add_argument('door', type=int, choices=range(1, 4)) - >>> print(parser.parse_args(['3'])) - Namespace(door=3) - >>> parser.parse_args(['4']) - usage: doors.py [-h] {1,2,3} - doors.py: error: argument door: invalid choice: 4 (choose from 1, 2, 3) +sequence should match the type_ specified. Any sequence can be passed as the *choices* value, so :class:`list` objects, :class:`tuple` objects, and custom sequences are all supported. @@ -1331,22 +1094,7 @@ help The ``help`` value is a string containing a brief description of the argument. When a user requests help (usually by using ``-h`` or ``--help`` at the command line), these ``help`` descriptions will be displayed with each -argument:: - - >>> parser = argparse.ArgumentParser(prog='frobble') - >>> parser.add_argument('--foo', action='store_true', - ... help='foo the bars before frobbling') - >>> parser.add_argument('bar', nargs='+', - ... help='one of the bars to be frobbled') - >>> parser.parse_args(['-h']) - usage: frobble [-h] [--foo] bar [bar ...] - - positional arguments: - bar one of the bars to be frobbled - - options: - -h, --help show this help message and exit - --foo foo the bars before frobbling +argument. The ``help`` strings can include various format specifiers to avoid repetition of things like the program name or the argument default_. The available @@ -1527,40 +1275,41 @@ this API may be passed as the ``action`` parameter to type=None, choices=None, required=False, help=None, \ metavar=None) -Action objects are used by an ArgumentParser to represent the information -needed to parse a single argument from one or more strings from the -command line. The Action class must accept the two positional arguments -plus any keyword arguments passed to :meth:`ArgumentParser.add_argument` -except for the ``action`` itself. + Action objects are used by an ArgumentParser to represent the information + needed to parse a single argument from one or more strings from the + command line. The Action class must accept the two positional arguments + plus any keyword arguments passed to :meth:`ArgumentParser.add_argument` + except for the ``action`` itself. -Instances of Action (or return value of any callable to the ``action`` -parameter) should have attributes "dest", "option_strings", "default", "type", -"required", "help", etc. defined. The easiest way to ensure these attributes -are defined is to call ``Action.__init__``. + Instances of Action (or return value of any callable to the ``action`` + parameter) should have attributes "dest", "option_strings", "default", "type", + "required", "help", etc. defined. The easiest way to ensure these attributes + are defined is to call ``Action.__init__``. -Action instances should be callable, so subclasses must override the -``__call__`` method, which should accept four parameters: + Action instances should be callable, so subclasses must override the + ``__call__`` method, which should accept four parameters: -* ``parser`` - The ArgumentParser object which contains this action. + * *parser* - The ArgumentParser object which contains this action. -* ``namespace`` - The :class:`Namespace` object that will be returned by - :meth:`~ArgumentParser.parse_args`. Most actions add an attribute to this - object using :func:`setattr`. + * *namespace* - The :class:`Namespace` object that will be returned by + :meth:`~ArgumentParser.parse_args`. Most actions add an attribute to this + object using :func:`setattr`. -* ``values`` - The associated command-line arguments, with any type conversions - applied. Type conversions are specified with the type_ keyword argument to - :meth:`~ArgumentParser.add_argument`. + * *values* - The associated command-line arguments, with any type conversions + applied. Type conversions are specified with the type_ keyword argument to + :meth:`~ArgumentParser.add_argument`. -* ``option_string`` - The option string that was used to invoke this action. - The ``option_string`` argument is optional, and will be absent if the action - is associated with a positional argument. + * *option_string* - The option string that was used to invoke this action. + The ``option_string`` argument is optional, and will be absent if the action + is associated with a positional argument. -The ``__call__`` method may perform arbitrary actions, but will typically set -attributes on the ``namespace`` based on ``dest`` and ``values``. + The ``__call__`` method may perform arbitrary actions, but will typically set + attributes on the ``namespace`` based on ``dest`` and ``values``. + + Action subclasses can define a ``format_usage`` method that takes no argument + and return a string which will be used when printing the usage of the program. + If such method is not provided, a sensible default will be used. -Action subclasses can define a ``format_usage`` method that takes no argument -and return a string which will be used when printing the usage of the program. -If such method is not provided, a sensible default will be used. The parse_args() method ----------------------- @@ -1755,29 +1504,29 @@ The Namespace object Simple class used by default by :meth:`~ArgumentParser.parse_args` to create an object holding attributes and return it. -This class is deliberately simple, just an :class:`object` subclass with a -readable string representation. If you prefer to have dict-like view of the -attributes, you can use the standard Python idiom, :func:`vars`:: - - >>> parser = argparse.ArgumentParser() - >>> parser.add_argument('--foo') - >>> args = parser.parse_args(['--foo', 'BAR']) - >>> vars(args) - {'foo': 'BAR'} - -It may also be useful to have an :class:`ArgumentParser` assign attributes to an -already existing object, rather than a new :class:`Namespace` object. This can -be achieved by specifying the ``namespace=`` keyword argument:: + This class is deliberately simple, just an :class:`object` subclass with a + readable string representation. If you prefer to have dict-like view of the + attributes, you can use the standard Python idiom, :func:`vars`:: - >>> class C: - ... pass - ... - >>> c = C() - >>> parser = argparse.ArgumentParser() - >>> parser.add_argument('--foo') - >>> parser.parse_args(args=['--foo', 'BAR'], namespace=c) - >>> c.foo - 'BAR' + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo') + >>> args = parser.parse_args(['--foo', 'BAR']) + >>> vars(args) + {'foo': 'BAR'} + + It may also be useful to have an :class:`ArgumentParser` assign attributes to an + already existing object, rather than a new :class:`Namespace` object. This can + be achieved by specifying the ``namespace=`` keyword argument:: + + >>> class C: + ... pass + ... + >>> c = C() + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo') + >>> parser.parse_args(args=['--foo', 'BAR'], namespace=c) + >>> c.foo + 'BAR' Other utilities @@ -1791,12 +1540,12 @@ Sub-commands [option_strings], [dest], [required], \ [help], [metavar]) - Many programs split up their functionality into a number of sub-commands, - for example, the ``svn`` program can invoke sub-commands like ``svn + Many programs split up their functionality into a number of subcommands, + for example, the ``svn`` program can invoke subcommands like ``svn checkout``, ``svn update``, and ``svn commit``. Splitting up functionality this way can be a particularly good idea when a program performs several different functions which require different kinds of command-line arguments. - :class:`ArgumentParser` supports the creation of such sub-commands with the + :class:`ArgumentParser` supports the creation of such subcommands with the :meth:`add_subparsers` method. The :meth:`add_subparsers` method is normally called with no arguments and returns a special action object. This object has a single method, :meth:`~_SubParsersAction.add_parser`, which takes a @@ -1805,18 +1554,18 @@ Sub-commands Description of parameters: - * title - title for the sub-parser group in help output; by default + * *title* - title for the sub-parser group in help output; by default "subcommands" if description is provided, otherwise uses title for positional arguments - * description - description for the sub-parser group in help output, by + * *description* - description for the sub-parser group in help output, by default ``None`` - * prog - usage information that will be displayed with sub-command help, + * *prog* - usage information that will be displayed with sub-command help, by default the name of the program and any positional arguments before the subparser argument - * parser_class - class which will be used to create sub-parser instances, by + * *parser_class* - class which will be used to create sub-parser instances, by default the class of the current parser (e.g. ArgumentParser) * action_ - the basic type of action to be taken when this argument is @@ -1830,15 +1579,15 @@ Sub-commands * help_ - help for sub-parser group in help output, by default ``None`` - * metavar_ - string presenting available sub-commands in help; by default it - is ``None`` and presents sub-commands in form {cmd1, cmd2, ..} + * metavar_ - string presenting available subcommands in help; by default it + is ``None`` and presents subcommands in form {cmd1, cmd2, ..} Some example usage:: >>> # create the top-level parser >>> parser = argparse.ArgumentParser(prog='PROG') >>> parser.add_argument('--foo', action='store_true', help='foo help') - >>> subparsers = parser.add_subparsers(help='sub-command help') + >>> subparsers = parser.add_subparsers(help='subcommand help') >>> >>> # create the parser for the "a" command >>> parser_a = subparsers.add_parser('a', help='a help') @@ -1873,7 +1622,7 @@ Sub-commands usage: PROG [-h] [--foo] {a,b} ... positional arguments: - {a,b} sub-command help + {a,b} subcommand help a a help b b help @@ -1944,12 +1693,12 @@ Sub-commands .. versionadded:: 3.13 - One particularly effective way of handling sub-commands is to combine the use + One particularly effective way of handling subcommands is to combine the use of the :meth:`add_subparsers` method with calls to :meth:`set_defaults` so that each subparser knows which Python function it should execute. For example:: - >>> # sub-command functions + >>> # subcommand functions >>> def foo(args): ... print(args.x * args.y) ... @@ -2231,20 +1980,20 @@ Partial parsing .. method:: ArgumentParser.parse_known_args(args=None, namespace=None) -Sometimes a script may only parse a few of the command-line arguments, passing -the remaining arguments on to another script or program. In these cases, the -:meth:`~ArgumentParser.parse_known_args` method can be useful. It works much like -:meth:`~ArgumentParser.parse_args` except that it does not produce an error when -extra arguments are present. Instead, it returns a two item tuple containing -the populated namespace and the list of remaining argument strings. + Sometimes a script may only parse a few of the command-line arguments, passing + the remaining arguments on to another script or program. In these cases, the + :meth:`~ArgumentParser.parse_known_args` method can be useful. It works much like + :meth:`~ArgumentParser.parse_args` except that it does not produce an error when + extra arguments are present. Instead, it returns a two item tuple containing + the populated namespace and the list of remaining argument strings. -:: + :: - >>> parser = argparse.ArgumentParser() - >>> parser.add_argument('--foo', action='store_true') - >>> parser.add_argument('bar') - >>> parser.parse_known_args(['--foo', '--badger', 'BAR', 'spam']) - (Namespace(bar='BAR', foo=True), ['--badger', 'spam']) + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo', action='store_true') + >>> parser.add_argument('bar') + >>> parser.parse_known_args(['--foo', '--badger', 'BAR', 'spam']) + (Namespace(bar='BAR', foo=True), ['--badger', 'spam']) .. warning:: :ref:`Prefix matching ` rules apply to @@ -2302,90 +2051,38 @@ Intermixed parsing .. method:: ArgumentParser.parse_intermixed_args(args=None, namespace=None) .. method:: ArgumentParser.parse_known_intermixed_args(args=None, namespace=None) -A number of Unix commands allow the user to intermix optional arguments with -positional arguments. The :meth:`~ArgumentParser.parse_intermixed_args` -and :meth:`~ArgumentParser.parse_known_intermixed_args` methods -support this parsing style. - -These parsers do not support all the argparse features, and will raise -exceptions if unsupported features are used. In particular, subparsers, -and mutually exclusive groups that include both -optionals and positionals are not supported. - -The following example shows the difference between -:meth:`~ArgumentParser.parse_known_args` and -:meth:`~ArgumentParser.parse_intermixed_args`: the former returns ``['2', -'3']`` as unparsed arguments, while the latter collects all the positionals -into ``rest``. :: - - >>> parser = argparse.ArgumentParser() - >>> parser.add_argument('--foo') - >>> parser.add_argument('cmd') - >>> parser.add_argument('rest', nargs='*', type=int) - >>> parser.parse_known_args('doit 1 --foo bar 2 3'.split()) - (Namespace(cmd='doit', foo='bar', rest=[1]), ['2', '3']) - >>> parser.parse_intermixed_args('doit 1 --foo bar 2 3'.split()) - Namespace(cmd='doit', foo='bar', rest=[1, 2, 3]) - -:meth:`~ArgumentParser.parse_known_intermixed_args` returns a two item tuple -containing the populated namespace and the list of remaining argument strings. -:meth:`~ArgumentParser.parse_intermixed_args` raises an error if there are any -remaining unparsed argument strings. - -.. versionadded:: 3.7 - -.. _upgrading-optparse-code: - -Upgrading optparse code ------------------------ - -Originally, the :mod:`argparse` module had attempted to maintain compatibility -with :mod:`optparse`. However, :mod:`optparse` was difficult to extend -transparently, particularly with the changes required to support the new -``nargs=`` specifiers and better usage messages. When most everything in -:mod:`optparse` had either been copy-pasted over or monkey-patched, it no -longer seemed practical to try to maintain the backwards compatibility. - -The :mod:`argparse` module improves on the standard library :mod:`optparse` -module in a number of ways including: + A number of Unix commands allow the user to intermix optional arguments with + positional arguments. The :meth:`~ArgumentParser.parse_intermixed_args` + and :meth:`~ArgumentParser.parse_known_intermixed_args` methods + support this parsing style. -* Handling positional arguments. -* Supporting sub-commands. -* Allowing alternative option prefixes like ``+`` and ``/``. -* Handling zero-or-more and one-or-more style arguments. -* Producing more informative usage messages. -* Providing a much simpler interface for custom ``type`` and ``action``. + These parsers do not support all the argparse features, and will raise + exceptions if unsupported features are used. In particular, subparsers, + and mutually exclusive groups that include both + optionals and positionals are not supported. -A partial upgrade path from :mod:`optparse` to :mod:`argparse`: + The following example shows the difference between + :meth:`~ArgumentParser.parse_known_args` and + :meth:`~ArgumentParser.parse_intermixed_args`: the former returns ``['2', + '3']`` as unparsed arguments, while the latter collects all the positionals + into ``rest``. :: -* Replace all :meth:`optparse.OptionParser.add_option` calls with - :meth:`ArgumentParser.add_argument` calls. - -* Replace ``(options, args) = parser.parse_args()`` with ``args = - parser.parse_args()`` and add additional :meth:`ArgumentParser.add_argument` - calls for the positional arguments. Keep in mind that what was previously - called ``options``, now in the :mod:`argparse` context is called ``args``. - -* Replace :meth:`optparse.OptionParser.disable_interspersed_args` - by using :meth:`~ArgumentParser.parse_intermixed_args` instead of - :meth:`~ArgumentParser.parse_args`. - -* Replace callback actions and the ``callback_*`` keyword arguments with - ``type`` or ``action`` arguments. - -* Replace string names for ``type`` keyword arguments with the corresponding - type objects (e.g. int, float, complex, etc). + >>> parser = argparse.ArgumentParser() + >>> parser.add_argument('--foo') + >>> parser.add_argument('cmd') + >>> parser.add_argument('rest', nargs='*', type=int) + >>> parser.parse_known_args('doit 1 --foo bar 2 3'.split()) + (Namespace(cmd='doit', foo='bar', rest=[1]), ['2', '3']) + >>> parser.parse_intermixed_args('doit 1 --foo bar 2 3'.split()) + Namespace(cmd='doit', foo='bar', rest=[1, 2, 3]) -* Replace :class:`optparse.Values` with :class:`Namespace` and - :exc:`optparse.OptionError` and :exc:`optparse.OptionValueError` with - :exc:`ArgumentError`. + :meth:`~ArgumentParser.parse_known_intermixed_args` returns a two item tuple + containing the populated namespace and the list of remaining argument strings. + :meth:`~ArgumentParser.parse_intermixed_args` raises an error if there are any + remaining unparsed argument strings. -* Replace strings with implicit arguments such as ``%default`` or ``%prog`` with - the standard Python syntax to use dictionaries to format strings, that is, - ``%(default)s`` and ``%(prog)s``. + .. versionadded:: 3.7 -* Replace the OptionParser constructor ``version`` argument with a call to - ``parser.add_argument('--version', action='version', version='')``. Exceptions ---------- @@ -2400,3 +2097,12 @@ Exceptions .. exception:: ArgumentTypeError Raised when something goes wrong converting a command line string to a type. + + +.. rubric:: Guides and Tutorials + +.. toctree:: + :maxdepth: 1 + + ../howto/argparse.rst + ../howto/argparse-optparse.rst diff --git a/Doc/library/ctypes.rst b/Doc/library/ctypes.rst index 535c5173be50de..d76b8d4809c078 100644 --- a/Doc/library/ctypes.rst +++ b/Doc/library/ctypes.rst @@ -2589,6 +2589,8 @@ fields, or any other data types containing pointer type fields. the structure when being packed or unpacked to/from memory. Setting this attribute to 0 is the same as not setting it at all. + .. versionadded:: 3.13 + .. attribute:: _layout_ An optional string naming the struct/union layout. It can currently diff --git a/Doc/library/datetime.rst b/Doc/library/datetime.rst index 64510a77c67c11..f0b465bc9ce39c 100644 --- a/Doc/library/datetime.rst +++ b/Doc/library/datetime.rst @@ -948,6 +948,10 @@ Other constructors, all class methods: This function is preferred over :meth:`today` and :meth:`utcnow`. + .. note:: + + Subsequent calls to :meth:`!datetime.now` may return the same + instant depending on the precision of the underlying clock. .. classmethod:: datetime.utcnow() diff --git a/Doc/library/dis.rst b/Doc/library/dis.rst index e3919c2ffad84c..1d084a8bf38d98 100644 --- a/Doc/library/dis.rst +++ b/Doc/library/dis.rst @@ -969,7 +969,8 @@ iterations of the loop. .. opcode:: GET_LEN - Perform ``STACK.append(len(STACK[-1]))``. + Perform ``STACK.append(len(STACK[-1]))``. Used in :keyword:`match` statements where + comparison with structure of pattern is needed. .. versionadded:: 3.10 @@ -1434,7 +1435,7 @@ iterations of the loop. slot ``i`` of the "fast locals" storage in this mapping. If the name is not found there, loads it from the cell contained in slot ``i``, similar to :opcode:`LOAD_DEREF`. This is used for loading - free variables in class bodies (which previously used + :term:`closure variables ` in class bodies (which previously used :opcode:`!LOAD_CLASSDEREF`) and in :ref:`annotation scopes ` within class bodies. @@ -1463,8 +1464,8 @@ iterations of the loop. .. opcode:: COPY_FREE_VARS (n) - Copies the ``n`` free variables from the closure into the frame. - Removes the need for special code on the caller's side when calling + Copies the ``n`` :term:`free (closure) variables ` from the closure + into the frame. Removes the need for special code on the caller's side when calling closures. .. versionadded:: 3.11 @@ -1937,10 +1938,10 @@ instructions: .. data:: hasfree - Sequence of bytecodes that access a free variable. 'free' in this - context refers to names in the current scope that are referenced by inner - scopes or names in outer scopes that are referenced from this scope. It does - *not* include references to global or builtin scopes. + Sequence of bytecodes that access a :term:`free (closure) variable `. + 'free' in this context refers to names in the current scope that are + referenced by inner scopes or names in outer scopes that are referenced + from this scope. It does *not* include references to global or builtin scopes. .. data:: hasname diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst index a96f69e6170f00..7f8df704a33327 100644 --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -684,9 +684,10 @@ are always available. They are listed here in alphabetical order. ``__builtins__`` dictionary into *globals* before passing it to :func:`exec`. The *closure* argument specifies a closure--a tuple of cellvars. - It's only valid when the *object* is a code object containing free variables. - The length of the tuple must exactly match the number of free variables - referenced by the code object. + It's only valid when the *object* is a code object containing + :term:`free (closure) variables `. + The length of the tuple must exactly match the length of the code object'S + :attr:`~codeobject.co_freevars` attribute. .. audit-event:: exec code_object exec diff --git a/Doc/library/inspect.rst b/Doc/library/inspect.rst index 853671856b2a14..1eaf1cc5d9a68e 100644 --- a/Doc/library/inspect.rst +++ b/Doc/library/inspect.rst @@ -694,7 +694,7 @@ and its return annotation. To retrieve a :class:`!Signature` object, use the :func:`!signature` function. -.. function:: signature(callable, *, follow_wrapped=True, globals=None, locals=None, eval_str=False) +.. function:: signature(callable, *, follow_wrapped=True, globals=None, locals=None, eval_str=False, annotation_format=Format.VALUE) Return a :class:`Signature` object for the given *callable*: @@ -725,7 +725,12 @@ function. *globals*, *locals*, and *eval_str* parameters are passed into :func:`!annotationlib.get_annotations` when resolving the annotations; see the documentation for :func:`!annotationlib.get_annotations` - for instructions on how to use these parameters. + for instructions on how to use these parameters. A member of the + :class:`annotationlib.Format` enum can be passed to the + *annotation_format* parameter to control the format of the returned + annotations. For example, use + ``annotation_format=annotationlib.Format.STRING`` to return annotations in string + format. Raises :exc:`ValueError` if no signature can be provided, and :exc:`TypeError` if that type of object is not supported. Also, @@ -733,7 +738,7 @@ function. the ``eval()`` call(s) to un-stringize the annotations in :func:`annotationlib.get_annotations` could potentially raise any kind of exception. - A slash(/) in the signature of a function denotes that the parameters prior + A slash (/) in the signature of a function denotes that the parameters prior to it are positional-only. For more info, see :ref:`the FAQ entry on positional-only parameters `. @@ -746,6 +751,9 @@ function. .. versionchanged:: 3.10 The *globals*, *locals*, and *eval_str* parameters were added. + .. versionchanged:: 3.14 + The *annotation_format* parameter was added. + .. note:: Some callables may not be introspectable in certain implementations of @@ -838,7 +846,7 @@ function. :class:`Signature` objects are also supported by the generic function :func:`copy.replace`. - .. method:: format(*, max_width=None) + .. method:: format(*, max_width=None, quote_annotation_strings=True) Create a string representation of the :class:`Signature` object. @@ -847,8 +855,17 @@ function. If the signature is longer than *max_width*, all parameters will be on separate lines. + If *quote_annotation_strings* is False, :term:`annotations ` + in the signature are displayed without opening and closing quotation + marks if they are strings. This is useful if the signature was created with the + :attr:`~annotationlib.Format.STRING` format or if + ``from __future__ import annotations`` was used. + .. versionadded:: 3.13 + .. versionchanged:: 3.14 + The *unquote_annotations* parameter was added. + .. classmethod:: Signature.from_callable(obj, *, follow_wrapped=True, globals=None, locals=None, eval_str=False) Return a :class:`Signature` (or its subclass) object for a given callable diff --git a/Doc/library/itertools.rst b/Doc/library/itertools.rst index 9a62249816c9bf..c138e903fa5a0f 100644 --- a/Doc/library/itertools.rst +++ b/Doc/library/itertools.rst @@ -134,7 +134,7 @@ loops that truncate the stream. To compute a running minimum, set *function* to :func:`min`. For a running maximum, set *function* to :func:`max`. Or for a running product, set *function* to :func:`operator.mul`. - To build an `Amortization table + To build an `amortization table `_, accumulate the interest and apply payments: @@ -736,6 +736,26 @@ loops that truncate the stream. allows nested :func:`tee` calls to share the same underlying data chain and to have a single update step rather than a chain of calls. + The flattening property makes tee iterators efficiently peekable: + + .. testcode:: + + def lookahead(tee_iterator): + "Return the next value without moving the input forward" + [forked_iterator] = tee(tee_iterator, 1) + return next(forked_iterator) + + .. doctest:: + + >>> iterator = iter('abcdef') + >>> [iterator] = tee(iterator, 1) # Make the input peekable + >>> next(iterator) # Move the iterator forward + 'a' + >>> lookahead(iterator) # Check next value + 'b' + >>> next(iterator) # Continue moving forward + 'b' + ``tee`` iterators are not threadsafe. A :exc:`RuntimeError` may be raised when simultaneously using iterators returned by the same :func:`tee` call, even if the original *iterable* is threadsafe. @@ -952,15 +972,6 @@ and :term:`generators ` which incur interpreter overhead. iterators = cycle(islice(iterators, num_active)) yield from map(next, iterators) - def partition(predicate, iterable): - """Partition entries into false entries and true entries. - - If *predicate* is slow, consider wrapping it with functools.lru_cache(). - """ - # partition(is_odd, range(10)) → 0 2 4 6 8 and 1 3 5 7 9 - t1, t2 = tee(iterable) - return filterfalse(predicate, t1), filter(predicate, t2) - def subslices(seq): "Return all contiguous non-empty subslices of a sequence." # subslices('ABCD') → A AB ABC ABCD B BC BCD C CD D @@ -1178,15 +1189,19 @@ The following recipes have a more mathematical flavor: >>> list(it) ['d', 'e', 'f'] + >>> list(prepend(1, [2, 3, 4])) [1, 2, 3, 4] + >>> list(enumerate('abc')) [(0, 'a'), (1, 'b'), (2, 'c')] + >>> list(islice(tabulate(lambda x: 2*x), 4)) [0, 2, 4, 6] + >>> list(tail(3, 'ABCDEFG')) ['E', 'F', 'G'] >>> # Verify the input is consumed greedily @@ -1195,6 +1210,7 @@ The following recipes have a more mathematical flavor: >>> list(input_iterator) [] + >>> it = iter(range(10)) >>> consume(it, 3) >>> # Verify the input is consumed lazily @@ -1205,6 +1221,7 @@ The following recipes have a more mathematical flavor: >>> next(it, 'Done') 'Done' + >>> nth('abcde', 3) 'd' >>> nth('abcde', 9) is None @@ -1216,6 +1233,7 @@ The following recipes have a more mathematical flavor: >>> list(it) ['d', 'e'] + >>> [all_equal(s) for s in ('', 'A', 'AAAA', 'AAAB', 'AAABA')] [True, True, True, False, False] >>> [all_equal(s, key=str.casefold) for s in ('', 'A', 'AaAa', 'AAAB', 'AAABA')] @@ -1229,24 +1247,19 @@ The following recipes have a more mathematical flavor: >>> ''.join(it) 'bbccc' + >>> quantify(range(99), lambda x: x%2==0) 50 - >>> quantify([True, False, False, True, True]) 3 - >>> quantify(range(12), predicate=lambda x: x%2==1) 6 + >>> a = [[1, 2, 3], [4, 5, 6]] >>> list(flatten(a)) [1, 2, 3, 4, 5, 6] - >>> list(repeatfunc(pow, 5, 2, 3)) - [8, 8, 8, 8, 8] - - >>> take(5, map(int, repeatfunc(random.random))) - [0, 0, 0, 0, 0] >>> list(ncycles('abc', 3)) ['a', 'b', 'c', 'a', 'b', 'c', 'a', 'b', 'c'] @@ -1256,9 +1269,11 @@ The following recipes have a more mathematical flavor: >>> list(input_iterator) [] + >>> sum_of_squares([10, 20, 30]) 1400 + >>> list(reshape([(0, 1), (2, 3), (4, 5)], 3)) [(0, 1, 2), (3, 4, 5)] >>> M = [(0, 1, 2, 3), (4, 5, 6, 7), (8, 9, 10, 11)] @@ -1279,6 +1294,7 @@ The following recipes have a more mathematical flavor: >>> list(reshape(M, 12)) [(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11)] + >>> list(transpose([(1, 2, 3), (11, 22, 33)])) [(1, 11), (2, 22), (3, 33)] >>> # Verify that the inputs are consumed lazily @@ -1290,11 +1306,13 @@ The following recipes have a more mathematical flavor: >>> list(zip(input1, input2)) [(2, 22), (3, 33)] + >>> list(matmul([(7, 5), (3, 5)], [[2, 5], [7, 9]])) [(49, 80), (41, 60)] >>> list(matmul([[2, 5], [7, 9], [3, 4]], [[7, 11, 5, 4, 9], [3, 5, 2, 6, 3]])) [(29, 47, 20, 38, 33), (76, 122, 53, 82, 90), (33, 53, 23, 36, 39)] + >>> list(convolve([1, -1, -20], [1, -3])) == [1, -4, -17, 60] True >>> data = [20, 40, 24, 32, 20, 28, 16] @@ -1317,6 +1335,7 @@ The following recipes have a more mathematical flavor: >>> list(signal_iterator) [30, 40, 50] + >>> from fractions import Fraction >>> from decimal import Decimal >>> polynomial_eval([1, -4, -17, 60], x=5) @@ -1348,6 +1367,7 @@ The following recipes have a more mathematical flavor: >>> polynomial_eval([11, 2], 7) == 11 * 7 + 2 True + >>> polynomial_from_roots([5, -4, 3]) [1, -4, -17, 60] >>> factored = lambda x: (x - 5) * (x + 4) * (x - 3) @@ -1355,9 +1375,11 @@ The following recipes have a more mathematical flavor: >>> all(factored(x) == expanded(x) for x in range(-10, 11)) True + >>> polynomial_derivative([1, -4, -17, 60]) [3, -8, -17] + >>> list(iter_index('AABCADEAF', 'A')) [0, 1, 4, 7] >>> list(iter_index('AABCADEAF', 'B')) @@ -1415,12 +1437,14 @@ The following recipes have a more mathematical flavor: >>> ''.join(input_iterator) 'DEAF' + >>> # Verify that the target value can be a sequence. >>> seq = [[10, 20], [30, 40], 30, 40, [30, 40], 50] >>> target = [30, 40] >>> list(iter_index(seq, target)) [1, 4] + >>> # Verify faithfulness to type specific index() method behaviors. >>> # For example, bytes and str perform continuous-subsequence searches >>> # that do not match the general behavior specified @@ -1450,6 +1474,7 @@ The following recipes have a more mathematical flavor: >>> set(sieve(10_000)).isdisjoint(carmichael) True + >>> list(factor(99)) # Code example 1 [3, 3, 11] >>> list(factor(1_000_000_000_000_007)) # Code example 2 @@ -1495,6 +1520,7 @@ The following recipes have a more mathematical flavor: >>> all(list(factor(n)) == sorted(factor(n)) for n in range(2_000)) True + >>> totient(0) # https://www.wolframalpha.com/input?i=totient+0 0 >>> first_totients = [1, 1, 2, 2, 4, 2, 6, 4, 6, 4, 10, 4, 12, 6, 8, 8, 16, 6, @@ -1514,9 +1540,15 @@ The following recipes have a more mathematical flavor: >>> totient(6 ** 20) == 1 * 2**19 * 2 * 3**19 # repeated primes True + >>> list(flatten([('a', 'b'), (), ('c', 'd', 'e'), ('f',), ('g', 'h', 'i')])) ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i'] + + >>> list(repeatfunc(pow, 5, 2, 3)) + [8, 8, 8, 8, 8] + >>> take(5, map(int, repeatfunc(random.random))) + [0, 0, 0, 0, 0] >>> random.seed(85753098575309) >>> list(repeatfunc(random.random, 3)) [0.16370491282496968, 0.45889608687313455, 0.3747076837820118] @@ -1525,9 +1557,11 @@ The following recipes have a more mathematical flavor: >>> list(repeatfunc(pow, 3, 2, 5)) [32, 32, 32] + >>> list(grouper('abcdefg', 3, fillvalue='x')) [('a', 'b', 'c'), ('d', 'e', 'f'), ('g', 'x', 'x')] + >>> it = grouper('abcdefg', 3, incomplete='strict') >>> next(it) ('a', 'b', 'c') @@ -1541,6 +1575,7 @@ The following recipes have a more mathematical flavor: >>> list(grouper('abcdefg', n=3, incomplete='ignore')) [('a', 'b', 'c'), ('d', 'e', 'f')] + >>> list(sliding_window('ABCDEFG', 1)) [('A',), ('B',), ('C',), ('D',), ('E',), ('F',), ('G',)] >>> list(sliding_window('ABCDEFG', 2)) @@ -1570,6 +1605,7 @@ The following recipes have a more mathematical flavor: ... 'zero or negative n not supported' + >>> list(roundrobin('abc', 'd', 'ef')) ['a', 'd', 'e', 'b', 'f', 'c'] >>> ranges = [range(5, 1000), range(4, 3000), range(0), range(3, 2000), range(2, 5000), range(1, 3500)] @@ -1583,38 +1619,19 @@ The following recipes have a more mathematical flavor: >>> ''.join(chain(*input_iterators)) 'dijkopqr' - >>> def is_odd(x): - ... return x % 2 == 1 - - >>> evens, odds = partition(is_odd, range(10)) - >>> list(evens) - [0, 2, 4, 6, 8] - >>> list(odds) - [1, 3, 5, 7, 9] - >>> # Verify that the input is consumed lazily - >>> input_iterator = iter(range(10)) - >>> evens, odds = partition(is_odd, input_iterator) - >>> next(odds) - 1 - >>> next(odds) - 3 - >>> next(evens) - 0 - >>> list(input_iterator) - [4, 5, 6, 7, 8, 9] >>> list(subslices('ABCD')) ['A', 'AB', 'ABC', 'ABCD', 'B', 'BC', 'BCD', 'C', 'CD', 'D'] + >>> list(powerset([1,2,3])) [(), (1,), (2,), (3,), (1, 2), (1, 3), (2, 3), (1, 2, 3)] - >>> all(len(list(powerset(range(n)))) == 2**n for n in range(18)) True - >>> list(powerset('abcde')) == sorted(sorted(set(powerset('abcde'))), key=len) True + >>> list(unique_everseen('AAAABBBCCDAABBB')) ['A', 'B', 'C', 'D'] >>> list(unique_everseen('ABBCcAD', str.casefold)) @@ -1629,6 +1646,7 @@ The following recipes have a more mathematical flavor: >>> ''.join(input_iterator) 'AAABBBCCDAABBB' + >>> list(unique_justseen('AAAABBBCCDAABBB')) ['A', 'B', 'C', 'D', 'A', 'B'] >>> list(unique_justseen('ABBCcAD', str.casefold)) @@ -1643,6 +1661,7 @@ The following recipes have a more mathematical flavor: >>> ''.join(input_iterator) 'AAABBBCCDAABBB' + >>> list(unique([[1, 2], [3, 4], [1, 2]])) [[1, 2], [3, 4]] >>> list(unique('ABBcCAD', str.casefold)) @@ -1650,6 +1669,7 @@ The following recipes have a more mathematical flavor: >>> list(unique('ABBcCAD', str.casefold, reverse=True)) ['D', 'c', 'B', 'A'] + >>> d = dict(a=1, b=2, c=3) >>> it = iter_except(d.popitem, KeyError) >>> d['d'] = 4 @@ -1667,6 +1687,7 @@ The following recipes have a more mathematical flavor: >>> next(it, 'empty') 'empty' + >>> first_true('ABC0DEF1', '9', str.isdigit) '0' >>> # Verify that inputs are consumed lazily @@ -1747,21 +1768,36 @@ The following recipes have a more mathematical flavor: return true_iterator(), chain(transition, it) + def partition(predicate, iterable): + """Partition entries into false entries and true entries. + + If *predicate* is slow, consider wrapping it with functools.lru_cache(). + """ + # partition(is_odd, range(10)) → 0 2 4 6 8 and 1 3 5 7 9 + t1, t2 = tee(iterable) + return filterfalse(predicate, t1), filter(predicate, t2) + + + .. doctest:: :hide: >>> dotproduct([1,2,3], [4,5,6]) 32 + >>> sumprod([1,2,3], [4,5,6]) 32 + >>> list(islice(pad_none('abc'), 0, 6)) ['a', 'b', 'c', None, None, None] + >>> list(triplewise('ABCDEFG')) [('A', 'B', 'C'), ('B', 'C', 'D'), ('C', 'D', 'E'), ('D', 'E', 'F'), ('E', 'F', 'G')] + >>> population = 'ABCDEFGH' >>> for r in range(len(population) + 1): ... seq = list(combinations(population, r)) @@ -1769,16 +1805,38 @@ The following recipes have a more mathematical flavor: ... assert nth_combination(population, r, i) == seq[i] ... for i in range(-len(seq), 0): ... assert nth_combination(population, r, i) == seq[i] - + ... >>> iterable = 'abcde' >>> r = 3 >>> combos = list(combinations(iterable, r)) >>> all(nth_combination(iterable, r, i) == comb for i, comb in enumerate(combos)) True + >>> it = iter('ABCdEfGhI') >>> all_upper, remainder = before_and_after(str.isupper, it) >>> ''.join(all_upper) 'ABC' >>> ''.join(remainder) 'dEfGhI' + + + >>> def is_odd(x): + ... return x % 2 == 1 + ... + >>> evens, odds = partition(is_odd, range(10)) + >>> list(evens) + [0, 2, 4, 6, 8] + >>> list(odds) + [1, 3, 5, 7, 9] + >>> # Verify that the input is consumed lazily + >>> input_iterator = iter(range(10)) + >>> evens, odds = partition(is_odd, input_iterator) + >>> next(odds) + 1 + >>> next(odds) + 3 + >>> next(evens) + 0 + >>> list(input_iterator) + [4, 5, 6, 7, 8, 9] diff --git a/Doc/library/locale.rst b/Doc/library/locale.rst index 0246f99157024a..5f3c4840b5cc70 100644 --- a/Doc/library/locale.rst +++ b/Doc/library/locale.rst @@ -158,7 +158,8 @@ The :mod:`locale` module defines the following exception and functions: .. function:: nl_langinfo(option) - Return some locale-specific information as a string. This function is not + Return some locale-specific information as a string (or a tuple for + ``ALT_DIGITS``). This function is not available on all systems, and the set of possible options might also vary across platforms. The possible argument values are numbers, for which symbolic constants are available in the locale module. @@ -311,8 +312,16 @@ The :mod:`locale` module defines the following exception and functions: .. data:: ALT_DIGITS - Get a representation of up to 100 values used to represent the values - 0 to 99. + Get a tuple of up to 100 strings used to represent the values 0 to 99. + + The function temporarily sets the ``LC_CTYPE`` locale to the locale + of the category that determines the requested value (``LC_TIME``, + ``LC_NUMERIC``, ``LC_MONETARY`` or ``LC_MESSAGES``) if locales are + different and the resulting string is non-ASCII. + This temporary change affects other threads. + + .. versionchanged:: 3.14 + The function now temporarily sets the ``LC_CTYPE`` locale in some cases. .. function:: getdefaultlocale([envvars]) diff --git a/Doc/library/symtable.rst b/Doc/library/symtable.rst index 15e0b23aa12bf0..56cd6b8afaa73e 100644 --- a/Doc/library/symtable.rst +++ b/Doc/library/symtable.rst @@ -167,11 +167,12 @@ Examining Symbol Tables .. method:: get_nonlocals() - Return a tuple containing names of nonlocals in this function. + Return a tuple containing names of explicitly declared nonlocals in this function. .. method:: get_frees() - Return a tuple containing names of free variables in this function. + Return a tuple containing names of :term:`free (closure) variables ` + in this function. .. class:: Class diff --git a/Doc/library/types.rst b/Doc/library/types.rst index 3c3c760c206ff2..84b80ec6efd59f 100644 --- a/Doc/library/types.rst +++ b/Doc/library/types.rst @@ -199,7 +199,7 @@ Standard names are defined for the following types: .. data:: CellType The type for cell objects: such objects are used as containers for - a function's free variables. + a function's :term:`closure variables `. .. versionadded:: 3.8 diff --git a/Doc/library/venv.rst b/Doc/library/venv.rst index e2c77963ff3040..5205c6c211d9bf 100644 --- a/Doc/library/venv.rst +++ b/Doc/library/venv.rst @@ -215,7 +215,7 @@ containing the virtual environment): | +------------+--------------------------------------------------+ | | csh/tcsh | :samp:`$ source {}/bin/activate.csh` | | +------------+--------------------------------------------------+ -| | PowerShell | :samp:`$ {}/bin/Activate.ps1` | +| | pwsh | :samp:`$ {}/bin/Activate.ps1` | +-------------+------------+--------------------------------------------------+ | Windows | cmd.exe | :samp:`C:\\> {}\\Scripts\\activate.bat` | | +------------+--------------------------------------------------+ diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst index 513199d21456bf..d059a660548c7e 100644 --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -564,8 +564,9 @@ Special read-only attributes in which the function was defined. * - .. attribute:: function.__closure__ - - ``None`` or a :class:`tuple` of cells that contain bindings for the - function's free variables. + - ``None`` or a :class:`tuple` of cells that contain bindings for the names specified + in the :attr:`~codeobject.co_freevars` attribute of the function's + :attr:`code object `. A cell object has the attribute ``cell_contents``. This can be used to get the value of the cell, as well as set the value. @@ -1285,10 +1286,14 @@ Special read-only attributes * - .. attribute:: codeobject.co_cellvars - A :class:`tuple` containing the names of :ref:`local variables ` - that are referenced by nested functions inside the function + that are referenced from at least one :term:`nested scope` inside the function * - .. attribute:: codeobject.co_freevars - - A :class:`tuple` containing the names of free variables in the function + - A :class:`tuple` containing the names of + :term:`free (closure) variables ` that a :term:`nested scope` + references in an outer scope. See also :attr:`function.__closure__`. + + Note: references to global and builtin names are *not* included. * - .. attribute:: codeobject.co_code - A string representing the sequence of :term:`bytecode` instructions in diff --git a/Doc/reference/executionmodel.rst b/Doc/reference/executionmodel.rst index 99cb09d09331d8..cb6c524dd97a30 100644 --- a/Doc/reference/executionmodel.rst +++ b/Doc/reference/executionmodel.rst @@ -90,7 +90,7 @@ If a name is bound in a block, it is a local variable of that block, unless declared as :keyword:`nonlocal` or :keyword:`global`. If a name is bound at the module level, it is a global variable. (The variables of the module code block are local and global.) If a variable is used in a code block but not -defined there, it is a :dfn:`free variable`. +defined there, it is a :term:`free variable`. Each occurrence of a name in the program text refers to the :dfn:`binding` of that name established by the following name resolution rules. @@ -337,6 +337,9 @@ enclosing namespace, but in the global namespace. [#]_ The :func:`exec` and :func:`eval` functions have optional arguments to override the global and local namespace. If only one namespace is specified, it is used for both. +.. XXX(ncoghlan) above is only accurate for string execution. When executing code objects, + closure cells may now be passed explicitly to resolve co_freevars references. + Docs issue: https://github.com/python/cpython/issues/122826 .. _exceptions: diff --git a/Doc/reference/expressions.rst b/Doc/reference/expressions.rst index ab72ad49d041e1..decde0d297cf59 100644 --- a/Doc/reference/expressions.rst +++ b/Doc/reference/expressions.rst @@ -1809,6 +1809,8 @@ returns a boolean value regardless of the type of its argument single: named expression pair: assignment; expression +.. _assignment-expressions: + Assignment expressions ====================== diff --git a/Doc/tools/extensions/pyspecific.py b/Doc/tools/extensions/pyspecific.py index c89b1693343b4e..b6623a2b8e01f1 100644 --- a/Doc/tools/extensions/pyspecific.py +++ b/Doc/tools/extensions/pyspecific.py @@ -353,8 +353,8 @@ def run(self): # Support for building "topic help" for pydoc pydoc_topic_labels = [ - 'assert', 'assignment', 'async', 'atom-identifiers', 'atom-literals', - 'attribute-access', 'attribute-references', 'augassign', 'await', + 'assert', 'assignment', 'assignment-expressions', 'async', 'atom-identifiers', + 'atom-literals', 'attribute-access', 'attribute-references', 'augassign', 'await', 'binary', 'bitwise', 'bltin-code-objects', 'bltin-ellipsis-object', 'bltin-null-object', 'bltin-type-objects', 'booleans', 'break', 'callable-types', 'calls', 'class', 'comparisons', 'compound', diff --git a/Doc/tutorial/introduction.rst b/Doc/tutorial/introduction.rst index 054bac59c955d5..65e3b1938bca9c 100644 --- a/Doc/tutorial/introduction.rst +++ b/Doc/tutorial/introduction.rst @@ -197,21 +197,19 @@ and workarounds. String literals can span multiple lines. One way is using triple-quotes: ``"""..."""`` or ``'''...'''``. End of lines are automatically included in the string, but it's possible to prevent this by adding a ``\`` at -the end of the line. The following example:: - - print("""\ +the end of the line. In the following example, the initial newline is not +included:: + + >>> print("""\ + ... Usage: thingy [OPTIONS] + ... -h Display this usage message + ... -H hostname Hostname to connect to + ... """) Usage: thingy [OPTIONS] -h Display this usage message -H hostname Hostname to connect to - """) - -produces the following output (note that the initial newline is not included): -.. code-block:: text - - Usage: thingy [OPTIONS] - -h Display this usage message - -H hostname Hostname to connect to + >>> Strings can be concatenated (glued together) with the ``+`` operator, and repeated with ``*``:: diff --git a/Doc/whatsnew/3.13.rst b/Doc/whatsnew/3.13.rst index a47d5e077a357b..565f74149725d5 100644 --- a/Doc/whatsnew/3.13.rst +++ b/Doc/whatsnew/3.13.rst @@ -814,6 +814,10 @@ ctypes See :gh:`124520` for discussion and links to changes in some affected projects. +* :class:`ctypes.Structure` objects have a new :attr:`~ctypes.Structure._align_` + attribute which allows the alignment of the structure being packed to/from + memory to be specified explicitly. + (Contributed by Matt Sanderson in :gh:`112433`) dbm --- diff --git a/Doc/whatsnew/3.14.rst b/Doc/whatsnew/3.14.rst index 67d8d389b58082..c62a3ca5872eef 100644 --- a/Doc/whatsnew/3.14.rst +++ b/Doc/whatsnew/3.14.rst @@ -281,6 +281,18 @@ http (Contributed by Yorik Hansen in :gh:`123430`.) +inspect +------- + +* :func:`inspect.signature` takes a new argument *annotation_format* to control + the :class:`annotationlib.Format` used for representing annotations. + (Contributed by Jelle Zijlstra in :gh:`101552`.) + +* :meth:`inspect.Signature.format` takes a new argument *unquote_annotations*. + If true, string :term:`annotations ` are displayed without surrounding quotes. + (Contributed by Jelle Zijlstra in :gh:`101552`.) + + json ---- @@ -356,6 +368,14 @@ pickle of the error. (Contributed by Serhiy Storchaka in :gh:`122213`.) +pydoc +----- + +* :term:`Annotations ` in help output are now usually + displayed in a format closer to that in the original source. + (Contributed by Jelle Zijlstra in :gh:`101552`.) + + symtable -------- @@ -587,6 +607,12 @@ Changes in the Python API Wrap it in :func:`staticmethod` if you want to preserve the old behavior. (Contributed by Serhiy Storchaka and Dominykas Grigonis in :gh:`121027`.) +* The :func:`locale.nl_langinfo` function now sets temporarily the ``LC_CTYPE`` + locale in some cases. + This temporary change affects other threads. + (Contributed by Serhiy Storchaka in :gh:`69998`.) + + Build Changes ============= @@ -687,6 +713,10 @@ New Features `__ mentioned in :pep:`630` (:gh:`124153`). +* Add :c:func:`PyUnicode_Equal` function to the limited C API: + test if two strings are equal. + (Contributed by Victor Stinner in :gh:`124502`.) + Porting to Python 3.14 ---------------------- diff --git a/Include/internal/pycore_function.h b/Include/internal/pycore_function.h index 6d44e933e8a8cb..c45d281125febb 100644 --- a/Include/internal/pycore_function.h +++ b/Include/internal/pycore_function.h @@ -18,6 +18,10 @@ extern PyObject* _PyFunction_Vectorcall( #define FUNC_MAX_WATCHERS 8 +#define FUNC_VERSION_UNSET 0 +#define FUNC_VERSION_CLEARED 1 +#define FUNC_VERSION_FIRST_VALID 2 + #define FUNC_VERSION_CACHE_SIZE (1<<12) /* Must be a power of 2 */ struct _func_version_cache_item { @@ -41,6 +45,12 @@ struct _py_func_state { extern PyFunctionObject* _PyFunction_FromConstructor(PyFrameConstructor *constr); +static inline int +_PyFunction_IsVersionValid(uint32_t version) +{ + return version >= FUNC_VERSION_FIRST_VALID; +} + extern uint32_t _PyFunction_GetVersionForCurrentState(PyFunctionObject *func); PyAPI_FUNC(void) _PyFunction_SetVersion(PyFunctionObject *func, uint32_t version); void _PyFunction_ClearCodeByVersion(uint32_t version); diff --git a/Include/internal/pycore_global_objects_fini_generated.h b/Include/internal/pycore_global_objects_fini_generated.h index 28a76c36801b4b..3140a75a47c5ee 100644 --- a/Include/internal/pycore_global_objects_fini_generated.h +++ b/Include/internal/pycore_global_objects_fini_generated.h @@ -757,6 +757,7 @@ _PyStaticObjects_CheckRefcnt(PyInterpreterState *interp) { _PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(_initializing)); _PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(_io)); _PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(_is_text_encoding)); + _PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(_isatty_open_only)); _PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(_length_)); _PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(_limbo)); _PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(_lock_unlock_module)); diff --git a/Include/internal/pycore_global_strings.h b/Include/internal/pycore_global_strings.h index ac789b06fb8a61..1591cb0a3f114f 100644 --- a/Include/internal/pycore_global_strings.h +++ b/Include/internal/pycore_global_strings.h @@ -246,6 +246,7 @@ struct _Py_global_strings { STRUCT_FOR_ID(_initializing) STRUCT_FOR_ID(_io) STRUCT_FOR_ID(_is_text_encoding) + STRUCT_FOR_ID(_isatty_open_only) STRUCT_FOR_ID(_length_) STRUCT_FOR_ID(_limbo) STRUCT_FOR_ID(_lock_unlock_module) diff --git a/Include/internal/pycore_magic_number.h b/Include/internal/pycore_magic_number.h index 2414d25d41bfbf..a88ff2deeba941 100644 --- a/Include/internal/pycore_magic_number.h +++ b/Include/internal/pycore_magic_number.h @@ -259,6 +259,7 @@ Known values: Python 3.14a1 3605 (Move ENTER_EXECUTOR to opcode 255) Python 3.14a1 3606 (Specialize CALL_KW) Python 3.14a1 3607 (Add pseudo instructions JUMP_IF_TRUE/FALSE) + Python 3.14a1 3608 (Add support for slices) Python 3.15 will start with 3650 @@ -271,7 +272,7 @@ PC/launcher.c must also be updated. */ -#define PYC_MAGIC_NUMBER 3607 +#define PYC_MAGIC_NUMBER 3608 /* This is equivalent to converting PYC_MAGIC_NUMBER to 2 bytes (little-endian) and then appending b'\r\n'. */ #define PYC_MAGIC_NUMBER_TOKEN \ diff --git a/Include/internal/pycore_runtime_init.h b/Include/internal/pycore_runtime_init.h index e6adb98eb19130..a17ba46966daa1 100644 --- a/Include/internal/pycore_runtime_init.h +++ b/Include/internal/pycore_runtime_init.h @@ -11,6 +11,7 @@ extern "C" { #include "pycore_ceval_state.h" // _PyEval_RUNTIME_PERF_INIT #include "pycore_faulthandler.h" // _faulthandler_runtime_state_INIT #include "pycore_floatobject.h" // _py_float_format_unknown +#include "pycore_function.h" #include "pycore_object.h" // _PyObject_HEAD_INIT #include "pycore_obmalloc_init.h" // _obmalloc_global_state_INIT #include "pycore_parser.h" // _parser_runtime_state_INIT @@ -243,7 +244,7 @@ extern PyTypeObject _PyExc_MemoryError; .dict_state = _dict_state_INIT, \ .mem_free_queue = _Py_mem_free_queue_INIT(INTERP.mem_free_queue), \ .func_state = { \ - .next_version = 1, \ + .next_version = FUNC_VERSION_FIRST_VALID, \ }, \ .types = { \ .next_version_tag = _Py_TYPE_BASE_VERSION_TAG, \ diff --git a/Include/internal/pycore_runtime_init_generated.h b/Include/internal/pycore_runtime_init_generated.h index 7847a5c63ebf3f..c9d20d0b5aacdb 100644 --- a/Include/internal/pycore_runtime_init_generated.h +++ b/Include/internal/pycore_runtime_init_generated.h @@ -755,6 +755,7 @@ extern "C" { INIT_ID(_initializing), \ INIT_ID(_io), \ INIT_ID(_is_text_encoding), \ + INIT_ID(_isatty_open_only), \ INIT_ID(_length_), \ INIT_ID(_limbo), \ INIT_ID(_lock_unlock_module), \ diff --git a/Include/internal/pycore_unicodeobject.h b/Include/internal/pycore_unicodeobject.h index 20497ee93016d0..a60372f58295a9 100644 --- a/Include/internal/pycore_unicodeobject.h +++ b/Include/internal/pycore_unicodeobject.h @@ -252,11 +252,7 @@ extern Py_ssize_t _PyUnicode_InsertThousandsGrouping( extern PyObject* _PyUnicode_FormatLong(PyObject *, int, int, int); -/* Fast equality check when the inputs are known to be exact unicode types - and where the hash values are equal (i.e. a very probable match) */ -extern int _PyUnicode_EQ(PyObject *, PyObject *); - -// Equality check. +// Fast equality check when the inputs are known to be exact unicode types. // Export for '_pickle' shared extension. PyAPI_FUNC(int) _PyUnicode_Equal(PyObject *, PyObject *); diff --git a/Include/internal/pycore_unicodeobject_generated.h b/Include/internal/pycore_unicodeobject_generated.h index a688f70a2ba36f..d335373e88ee74 100644 --- a/Include/internal/pycore_unicodeobject_generated.h +++ b/Include/internal/pycore_unicodeobject_generated.h @@ -784,6 +784,10 @@ _PyUnicode_InitStaticStrings(PyInterpreterState *interp) { _PyUnicode_InternStatic(interp, &string); assert(_PyUnicode_CheckConsistency(string, 1)); assert(PyUnicode_GET_LENGTH(string) != 1); + string = &_Py_ID(_isatty_open_only); + _PyUnicode_InternStatic(interp, &string); + assert(_PyUnicode_CheckConsistency(string, 1)); + assert(PyUnicode_GET_LENGTH(string) != 1); string = &_Py_ID(_length_); _PyUnicode_InternStatic(interp, &string); assert(_PyUnicode_CheckConsistency(string, 1)); diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h index dee00715b3c51d..2ce3a008b7129e 100644 --- a/Include/unicodeobject.h +++ b/Include/unicodeobject.h @@ -966,6 +966,10 @@ PyAPI_FUNC(int) PyUnicode_EqualToUTF8(PyObject *, const char *); PyAPI_FUNC(int) PyUnicode_EqualToUTF8AndSize(PyObject *, const char *, Py_ssize_t); #endif +#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x030e0000 +PyAPI_FUNC(int) PyUnicode_Equal(PyObject *str1, PyObject *str2); +#endif + /* Rich compare two strings and return one of the following: - NULL in case an exception was raised diff --git a/Lib/_pyio.py b/Lib/_pyio.py index 18849b309b8605..7b6d10c008d3cb 100644 --- a/Lib/_pyio.py +++ b/Lib/_pyio.py @@ -238,7 +238,7 @@ def open(file, mode="r", buffering=-1, encoding=None, errors=None, result = raw try: line_buffering = False - if buffering == 1 or buffering < 0 and raw.isatty(): + if buffering == 1 or buffering < 0 and raw._isatty_open_only(): buffering = -1 line_buffering = True if buffering < 0: @@ -1794,6 +1794,21 @@ def isatty(self): self._checkClosed() return os.isatty(self._fd) + def _isatty_open_only(self): + """Checks whether the file is a TTY using an open-only optimization. + + TTYs are always character devices. If the interpreter knows a file is + not a character device when it would call ``isatty``, can skip that + call. Inside ``open()`` there is a fresh stat result that contains that + information. Use the stat result to skip a system call. Outside of that + context TOCTOU issues (the fd could be arbitrarily modified by + surrounding code). + """ + if (self._stat_atopen is not None + and not stat.S_ISCHR(self._stat_atopen.st_mode)): + return False + return os.isatty(self._fd) + @property def closefd(self): """True if the file descriptor will be closed by close().""" diff --git a/Lib/enum.py b/Lib/enum.py index 9d53eb86bc2116..17d72738792982 100644 --- a/Lib/enum.py +++ b/Lib/enum.py @@ -1092,6 +1092,21 @@ def _add_member_(cls, name, member): # now add to _member_map_ (even aliases) cls._member_map_[name] = member + @property + def __signature__(cls): + from inspect import Parameter, Signature + if cls._member_names_: + return Signature([Parameter('values', Parameter.VAR_POSITIONAL)]) + else: + return Signature([Parameter('new_class_name', Parameter.POSITIONAL_ONLY), + Parameter('names', Parameter.POSITIONAL_OR_KEYWORD), + Parameter('module', Parameter.KEYWORD_ONLY, default=None), + Parameter('qualname', Parameter.KEYWORD_ONLY, default=None), + Parameter('type', Parameter.KEYWORD_ONLY, default=None), + Parameter('start', Parameter.KEYWORD_ONLY, default=1), + Parameter('boundary', Parameter.KEYWORD_ONLY, default=None)]) + + EnumMeta = EnumType # keep EnumMeta name for backwards compatibility @@ -1135,13 +1150,6 @@ class Enum(metaclass=EnumType): attributes -- see the documentation for details. """ - @classmethod - def __signature__(cls): - if cls._member_names_: - return '(*values)' - else: - return '(new_class_name, /, names, *, module=None, qualname=None, type=None, start=1, boundary=None)' - def __new__(cls, value): # all enum instances are actually created during class construction # without calling this method; this method is called by the metaclass' diff --git a/Lib/inspect.py b/Lib/inspect.py index 17314564f35397..0c33c6cc995a03 100644 --- a/Lib/inspect.py +++ b/Lib/inspect.py @@ -140,6 +140,7 @@ import abc +from annotationlib import Format from annotationlib import get_annotations # re-exported import ast import dis @@ -1319,7 +1320,9 @@ def getargvalues(frame): args, varargs, varkw = getargs(frame.f_code) return ArgInfo(args, varargs, varkw, frame.f_locals) -def formatannotation(annotation, base_module=None): +def formatannotation(annotation, base_module=None, *, quote_annotation_strings=True): + if not quote_annotation_strings and isinstance(annotation, str): + return annotation if getattr(annotation, '__module__', None) == 'typing': def repl(match): text = match.group() @@ -2270,7 +2273,8 @@ def _signature_from_builtin(cls, func, skip_bound_arg=True): def _signature_from_function(cls, func, skip_bound_arg=True, - globals=None, locals=None, eval_str=False): + globals=None, locals=None, eval_str=False, + *, annotation_format=Format.VALUE): """Private helper: constructs Signature for the given python function.""" is_duck_function = False @@ -2296,7 +2300,8 @@ def _signature_from_function(cls, func, skip_bound_arg=True, positional = arg_names[:pos_count] keyword_only_count = func_code.co_kwonlyargcount keyword_only = arg_names[pos_count:pos_count + keyword_only_count] - annotations = get_annotations(func, globals=globals, locals=locals, eval_str=eval_str) + annotations = get_annotations(func, globals=globals, locals=locals, eval_str=eval_str, + format=annotation_format) defaults = func.__defaults__ kwdefaults = func.__kwdefaults__ @@ -2379,7 +2384,8 @@ def _signature_from_callable(obj, *, globals=None, locals=None, eval_str=False, - sigcls): + sigcls, + annotation_format=Format.VALUE): """Private helper function to get signature for arbitrary callable objects. @@ -2391,7 +2397,8 @@ def _signature_from_callable(obj, *, globals=globals, locals=locals, sigcls=sigcls, - eval_str=eval_str) + eval_str=eval_str, + annotation_format=annotation_format) if not callable(obj): raise TypeError('{!r} is not a callable object'.format(obj)) @@ -2424,18 +2431,10 @@ def _signature_from_callable(obj, *, pass else: if sig is not None: - # since __text_signature__ is not writable on classes, __signature__ - # may contain text (or be a callable that returns text); - # if so, convert it - o_sig = sig - if not isinstance(sig, (Signature, str)) and callable(sig): - sig = sig() - if isinstance(sig, str): - sig = _signature_fromstr(sigcls, obj, sig) if not isinstance(sig, Signature): raise TypeError( 'unexpected object {!r} in __signature__ ' - 'attribute'.format(o_sig)) + 'attribute'.format(sig)) return sig try: @@ -2480,7 +2479,8 @@ def _signature_from_callable(obj, *, # of a Python function (Cython functions, for instance), then: return _signature_from_function(sigcls, obj, skip_bound_arg=skip_bound_arg, - globals=globals, locals=locals, eval_str=eval_str) + globals=globals, locals=locals, eval_str=eval_str, + annotation_format=annotation_format) if _signature_is_builtin(obj): return _signature_from_builtin(sigcls, obj, @@ -2715,13 +2715,17 @@ def replace(self, *, name=_void, kind=_void, return type(self)(name, kind, default=default, annotation=annotation) def __str__(self): + return self._format() + + def _format(self, *, quote_annotation_strings=True): kind = self.kind formatted = self._name # Add annotation and default value if self._annotation is not _empty: - formatted = '{}: {}'.format(formatted, - formatannotation(self._annotation)) + annotation = formatannotation(self._annotation, + quote_annotation_strings=quote_annotation_strings) + formatted = '{}: {}'.format(formatted, annotation) if self._default is not _empty: if self._annotation is not _empty: @@ -2969,11 +2973,13 @@ def __init__(self, parameters=None, *, return_annotation=_empty, @classmethod def from_callable(cls, obj, *, - follow_wrapped=True, globals=None, locals=None, eval_str=False): + follow_wrapped=True, globals=None, locals=None, eval_str=False, + annotation_format=Format.VALUE): """Constructs Signature for the given callable object.""" return _signature_from_callable(obj, sigcls=cls, follow_wrapper_chains=follow_wrapped, - globals=globals, locals=locals, eval_str=eval_str) + globals=globals, locals=locals, eval_str=eval_str, + annotation_format=annotation_format) @property def parameters(self): @@ -3188,19 +3194,24 @@ def __repr__(self): def __str__(self): return self.format() - def format(self, *, max_width=None): + def format(self, *, max_width=None, quote_annotation_strings=True): """Create a string representation of the Signature object. If *max_width* integer is passed, signature will try to fit into the *max_width*. If signature is longer than *max_width*, all parameters will be on separate lines. + + If *quote_annotation_strings* is False, annotations + in the signature are displayed without opening and closing quotation + marks. This is useful when the signature was created with the + STRING format or when ``from __future__ import annotations`` was used. """ result = [] render_pos_only_separator = False render_kw_only_separator = True for param in self.parameters.values(): - formatted = str(param) + formatted = param._format(quote_annotation_strings=quote_annotation_strings) kind = param.kind @@ -3237,16 +3248,19 @@ def format(self, *, max_width=None): rendered = '(\n {}\n)'.format(',\n '.join(result)) if self.return_annotation is not _empty: - anno = formatannotation(self.return_annotation) + anno = formatannotation(self.return_annotation, + quote_annotation_strings=quote_annotation_strings) rendered += ' -> {}'.format(anno) return rendered -def signature(obj, *, follow_wrapped=True, globals=None, locals=None, eval_str=False): +def signature(obj, *, follow_wrapped=True, globals=None, locals=None, eval_str=False, + annotation_format=Format.VALUE): """Get a signature object for the passed callable.""" return Signature.from_callable(obj, follow_wrapped=follow_wrapped, - globals=globals, locals=locals, eval_str=eval_str) + globals=globals, locals=locals, eval_str=eval_str, + annotation_format=annotation_format) class BufferFlags(enum.IntFlag): diff --git a/Lib/pydoc.py b/Lib/pydoc.py index eec7b0770f56ca..c863794ea14ef9 100644 --- a/Lib/pydoc.py +++ b/Lib/pydoc.py @@ -71,6 +71,7 @@ class or function within a module or module in a package. If the import tokenize import urllib.parse import warnings +from annotationlib import Format from collections import deque from reprlib import Repr from traceback import format_exception_only @@ -212,12 +213,12 @@ def splitdoc(doc): def _getargspec(object): try: - signature = inspect.signature(object) + signature = inspect.signature(object, annotation_format=Format.STRING) if signature: name = getattr(object, '__name__', '') # function are always single-line and should not be formatted max_width = (80 - len(name)) if name != '' else None - return signature.format(max_width=max_width) + return signature.format(max_width=max_width, quote_annotation_strings=False) except (ValueError, TypeError): argspec = getattr(object, '__text_signature__', None) if argspec: diff --git a/Lib/pydoc_data/topics.py b/Lib/pydoc_data/topics.py index 97bb4eb52f4386..ae56c136608472 100644 --- a/Lib/pydoc_data/topics.py +++ b/Lib/pydoc_data/topics.py @@ -417,33 +417,43 @@ 'caused a\n' 'syntax error.\n', 'assignment-expressions': 'Assignment expressions\n' - '**********************\n' - '\n' - 'An assignment expression (sometimes also called a “named expression”' - '\nor “walrus”) assigns an expression to an identifier, while also\n' - 'returning the value of the expression.\n' - '\n' - 'One common use case is when handling matched regular expressions:\n' - '\n' - ' if matching := pattern.search(data):\n' - ' do_something(matching)\n' - '\n' - 'Or, when processing a file stream in chunks:\n' - '\n' - ' while chunk := file.read(9000):\n' - ' process(chunk)\n' - '\n' - 'Assignment expressions must be surrounded by parentheses when used as\n' - 'expression statements and when used as sub-expressions in slicing,\n' - 'conditional, lambda, keyword-argument, and comprehension-if\n' - 'expressions and in assert, with, and assignment statements. In all\n' - 'other places where they can be used, parentheses are not required,\n' - 'including in if and while statements.\n' - '\n' - 'Added in version 3.8.\n' - 'See also:\n' - '\n' - ' **PEP 572** - Assignment Expressions\n', + '**********************\n' + '\n' + ' assignment_expression ::= [identifier ":="] ' + 'expression\n' + '\n' + 'An assignment expression (sometimes also called a ' + '“named expression”\n' + 'or “walrus”) assigns an "expression" to an ' + '"identifier", while also\n' + 'returning the value of the "expression".\n' + '\n' + 'One common use case is when handling matched ' + 'regular expressions:\n' + '\n' + ' if matching := pattern.search(data):\n' + ' do_something(matching)\n' + '\n' + 'Or, when processing a file stream in chunks:\n' + '\n' + ' while chunk := file.read(9000):\n' + ' process(chunk)\n' + '\n' + 'Assignment expressions must be surrounded by ' + 'parentheses when used as\n' + 'expression statements and when used as ' + 'sub-expressions in slicing,\n' + 'conditional, lambda, keyword-argument, and ' + 'comprehension-if\n' + 'expressions and in "assert", "with", and ' + '"assignment" statements. In\n' + 'all other places where they can be used, ' + 'parentheses are not required,\n' + 'including in "if" and "while" statements.\n' + '\n' + 'Added in version 3.8: See **PEP 572** for more ' + 'details about\n' + 'assignment expressions.\n', 'async': 'Coroutines\n' '**********\n' '\n' diff --git a/Lib/site.py b/Lib/site.py index cafd3ab70b2cac..b3194d79fb5ab8 100644 --- a/Lib/site.py +++ b/Lib/site.py @@ -491,12 +491,21 @@ def register_readline(): This can be overridden in the sitecustomize or usercustomize module, or in a PYTHONSTARTUP file. """ + if not sys.flags.ignore_environment: + PYTHON_BASIC_REPL = os.getenv("PYTHON_BASIC_REPL") + else: + PYTHON_BASIC_REPL = False + import atexit try: import readline import rlcompleter # noqa: F401 - import _pyrepl.readline - import _pyrepl.unix_console + if PYTHON_BASIC_REPL: + CAN_USE_PYREPL = False + else: + import _pyrepl.readline + import _pyrepl.unix_console + from _pyrepl.main import CAN_USE_PYREPL except ImportError: return @@ -517,7 +526,6 @@ def register_readline(): pass if readline.get_current_history_length() == 0: - from _pyrepl.main import CAN_USE_PYREPL # If no history was loaded, default to .python_history, # or PYTHON_HISTORY. # The guard is necessary to avoid doubling history size at @@ -525,13 +533,17 @@ def register_readline(): # through a PYTHONSTARTUP hook, see: # http://bugs.python.org/issue5845#msg198636 history = gethistoryfile() - if os.getenv("PYTHON_BASIC_REPL") or not CAN_USE_PYREPL: - readline_module = readline - else: + + if CAN_USE_PYREPL: readline_module = _pyrepl.readline + exceptions = (OSError, *_pyrepl.unix_console._error) + else: + readline_module = readline + exceptions = OSError + try: readline_module.read_history_file(history) - except (OSError,* _pyrepl.unix_console._error): + except exceptions: pass def write_history(): diff --git a/Lib/test/certdata/make_ssl_certs.py b/Lib/test/certdata/make_ssl_certs.py index 198c64035c5044..18e614496385fd 100644 --- a/Lib/test/certdata/make_ssl_certs.py +++ b/Lib/test/certdata/make_ssl_certs.py @@ -9,8 +9,8 @@ from subprocess import * startdate = "20180829142316Z" -enddate_default = "20371028142316Z" -days_default = "7000" +enddate_default = "25251028142316Z" +days_default = "140000" req_template = """ [ default ] diff --git a/Lib/test/pickletester.py b/Lib/test/pickletester.py index 1722cc8612ca6b..8ef8fae44f1d25 100644 --- a/Lib/test/pickletester.py +++ b/Lib/test/pickletester.py @@ -26,7 +26,7 @@ from test import support from test.support import os_helper from test.support import ( - TestFailed, run_with_locale, no_tracing, + TestFailed, run_with_locales, no_tracing, _2G, _4G, bigmemtest ) from test.support.import_helper import forget @@ -2895,7 +2895,7 @@ def test_float(self): got = self.loads(pickle) self.assert_is_copy(value, got) - @run_with_locale('LC_ALL', 'de_DE', 'fr_FR') + @run_with_locales('LC_ALL', 'de_DE', 'fr_FR', '') def test_float_format(self): # make sure that floats are formatted locale independent with proto 0 self.assertEqual(self.dumps(1.2, 0)[0:3], b'F1.') diff --git a/Lib/test/support/__init__.py b/Lib/test/support/__init__.py index 1a44cc638b5714..72ce5dacd1be4c 100644 --- a/Lib/test/support/__init__.py +++ b/Lib/test/support/__init__.py @@ -930,8 +930,8 @@ def check_sizeof(test, o, size): test.assertEqual(result, size, msg) #======================================================================= -# Decorator for running a function in a different locale, correctly resetting -# it afterwards. +# Decorator/context manager for running a code in a different locale, +# correctly resetting it afterwards. @contextlib.contextmanager def run_with_locale(catstr, *locales): @@ -942,16 +942,21 @@ def run_with_locale(catstr, *locales): except AttributeError: # if the test author gives us an invalid category string raise - except: + except Exception: # cannot retrieve original locale, so do nothing locale = orig_locale = None + if '' not in locales: + raise unittest.SkipTest('no locales') else: for loc in locales: try: locale.setlocale(category, loc) break - except: + except locale.Error: pass + else: + if '' not in locales: + raise unittest.SkipTest(f'no locales {locales}') try: yield @@ -959,6 +964,46 @@ def run_with_locale(catstr, *locales): if locale and orig_locale: locale.setlocale(category, orig_locale) +#======================================================================= +# Decorator for running a function in multiple locales (if they are +# availasble) and resetting the original locale afterwards. + +def run_with_locales(catstr, *locales): + def deco(func): + @functools.wraps(func) + def wrapper(self, /, *args, **kwargs): + dry_run = '' in locales + try: + import locale + category = getattr(locale, catstr) + orig_locale = locale.setlocale(category) + except AttributeError: + # if the test author gives us an invalid category string + raise + except Exception: + # cannot retrieve original locale, so do nothing + pass + else: + try: + for loc in locales: + with self.subTest(locale=loc): + try: + locale.setlocale(category, loc) + except locale.Error: + self.skipTest(f'no locale {loc!r}') + else: + dry_run = False + func(self, *args, **kwargs) + finally: + locale.setlocale(category, orig_locale) + if dry_run: + # no locales available, so just run the test + # with the current locale + with self.subTest(locale=None): + func(self, *args, **kwargs) + return wrapper + return deco + #======================================================================= # Decorator for running a function in a specific timezone, correctly # resetting it afterwards. diff --git a/Lib/test/test__locale.py b/Lib/test/test__locale.py index 0947464bb8c04e..02d2acc6d1c417 100644 --- a/Lib/test/test__locale.py +++ b/Lib/test/test__locale.py @@ -1,4 +1,4 @@ -from _locale import (setlocale, LC_ALL, LC_CTYPE, LC_NUMERIC, localeconv, Error) +from _locale import (setlocale, LC_ALL, LC_CTYPE, LC_NUMERIC, LC_TIME, localeconv, Error) try: from _locale import (RADIXCHAR, THOUSEP, nl_langinfo) except ImportError: @@ -74,6 +74,17 @@ def accept(loc): 'ps_AF': ('\u066b', '\u066c'), } +known_alt_digits = { + 'C': (0, {}), + 'en_US': (0, {}), + 'fa_IR': (100, {0: '\u06f0\u06f0', 10: '\u06f1\u06f0', 99: '\u06f9\u06f9'}), + 'ja_JP': (100, {0: '\u3007', 10: '\u5341', 99: '\u4e5d\u5341\u4e5d'}), + 'lzh_TW': (32, {0: '\u3007', 10: '\u5341', 31: '\u5345\u4e00'}), + 'my_MM': (100, {0: '\u1040\u1040', 10: '\u1041\u1040', 99: '\u1049\u1049'}), + 'or_IN': (100, {0: '\u0b66', 10: '\u0b67\u0b66', 99: '\u0b6f\u0b6f'}), + 'shn_MM': (100, {0: '\u1090\u1090', 10: '\u1091\u1090', 99: '\u1099\u1099'}), +} + if sys.platform == 'win32': # ps_AF doesn't work on Windows: see bpo-38324 (msg361830) del known_numerics['ps_AF'] @@ -115,16 +126,17 @@ def numeric_tester(self, calc_type, calc_value, data_type, used_locale): def test_lc_numeric_nl_langinfo(self): # Test nl_langinfo against known values tested = False + oldloc = setlocale(LC_CTYPE) for loc in candidate_locales: try: setlocale(LC_NUMERIC, loc) - setlocale(LC_CTYPE, loc) except Error: continue for li, lc in ((RADIXCHAR, "decimal_point"), (THOUSEP, "thousands_sep")): if self.numeric_tester('nl_langinfo', nl_langinfo(li), lc, loc): tested = True + self.assertEqual(setlocale(LC_CTYPE), oldloc) if not tested: self.skipTest('no suitable locales') @@ -135,10 +147,10 @@ def test_lc_numeric_nl_langinfo(self): def test_lc_numeric_localeconv(self): # Test localeconv against known values tested = False + oldloc = setlocale(LC_CTYPE) for loc in candidate_locales: try: setlocale(LC_NUMERIC, loc) - setlocale(LC_CTYPE, loc) except Error: continue formatting = localeconv() @@ -146,6 +158,7 @@ def test_lc_numeric_localeconv(self): "thousands_sep"): if self.numeric_tester('localeconv', formatting[lc], lc, loc): tested = True + self.assertEqual(setlocale(LC_CTYPE), oldloc) if not tested: self.skipTest('no suitable locales') @@ -153,10 +166,10 @@ def test_lc_numeric_localeconv(self): def test_lc_numeric_basic(self): # Test nl_langinfo against localeconv tested = False + oldloc = setlocale(LC_CTYPE) for loc in candidate_locales: try: setlocale(LC_NUMERIC, loc) - setlocale(LC_CTYPE, loc) except Error: continue for li, lc in ((RADIXCHAR, "decimal_point"), @@ -173,6 +186,35 @@ def test_lc_numeric_basic(self): nl_radixchar, li_radixchar, loc, set_locale)) tested = True + self.assertEqual(setlocale(LC_CTYPE), oldloc) + if not tested: + self.skipTest('no suitable locales') + + @unittest.skipUnless(nl_langinfo, "nl_langinfo is not available") + @unittest.skipUnless(hasattr(locale, 'ALT_DIGITS'), "requires locale.ALT_DIGITS") + @unittest.skipIf( + support.is_emscripten or support.is_wasi, + "musl libc issue on Emscripten, bpo-46390" + ) + def test_alt_digits_nl_langinfo(self): + # Test nl_langinfo(ALT_DIGITS) + tested = False + for loc, (count, samples) in known_alt_digits.items(): + with self.subTest(locale=loc): + try: + setlocale(LC_TIME, loc) + except Error: + self.skipTest(f'no locale {loc!r}') + continue + with self.subTest(locale=loc): + alt_digits = nl_langinfo(locale.ALT_DIGITS) + self.assertIsInstance(alt_digits, tuple) + if count and not alt_digits and sys.platform == 'darwin': + self.skipTest(f'ALT_DIGITS is not set for locale {loc!r} on macOS') + self.assertEqual(len(alt_digits), count) + for i in samples: + self.assertEqual(alt_digits[i], samples[i]) + tested = True if not tested: self.skipTest('no suitable locales') @@ -180,10 +222,10 @@ def test_float_parsing(self): # Bug #1391872: Test whether float parsing is okay on European # locales. tested = False + oldloc = setlocale(LC_CTYPE) for loc in candidate_locales: try: setlocale(LC_NUMERIC, loc) - setlocale(LC_CTYPE, loc) except Error: continue @@ -199,6 +241,7 @@ def test_float_parsing(self): self.assertRaises(ValueError, float, localeconv()['decimal_point'].join(['1', '23'])) tested = True + self.assertEqual(setlocale(LC_CTYPE), oldloc) if not tested: self.skipTest('no suitable locales') diff --git a/Lib/test/test_capi/test_unicode.py b/Lib/test/test_capi/test_unicode.py index e6f85427214958..65d8242ad3fc60 100644 --- a/Lib/test/test_capi/test_unicode.py +++ b/Lib/test/test_capi/test_unicode.py @@ -1903,6 +1903,39 @@ def test_recover_error(self): self.assertEqual(writer.finish(), 'Hello World.') + def test_unicode_equal(self): + unicode_equal = _testlimitedcapi.unicode_equal + + def copy(text): + return text.encode().decode() + + self.assertTrue(unicode_equal("", "")) + self.assertTrue(unicode_equal("abc", "abc")) + self.assertTrue(unicode_equal("abc", copy("abc"))) + self.assertTrue(unicode_equal("\u20ac", copy("\u20ac"))) + self.assertTrue(unicode_equal("\U0010ffff", copy("\U0010ffff"))) + + self.assertFalse(unicode_equal("abc", "abcd")) + self.assertFalse(unicode_equal("\u20ac", "\u20ad")) + self.assertFalse(unicode_equal("\U0010ffff", "\U0010fffe")) + + # str subclass + self.assertTrue(unicode_equal("abc", Str("abc"))) + self.assertTrue(unicode_equal(Str("abc"), "abc")) + self.assertFalse(unicode_equal("abc", Str("abcd"))) + self.assertFalse(unicode_equal(Str("abc"), "abcd")) + + # invalid type + for invalid_type in (b'bytes', 123, ("tuple",)): + with self.subTest(invalid_type=invalid_type): + with self.assertRaises(TypeError): + unicode_equal("abc", invalid_type) + with self.assertRaises(TypeError): + unicode_equal(invalid_type, "abc") + + # CRASHES unicode_equal("abc", NULL) + # CRASHES unicode_equal(NULL, "abc") + if __name__ == "__main__": unittest.main() diff --git a/Lib/test/test_codecs.py b/Lib/test/test_codecs.py index 428036e1765b8f..290656f070503a 100644 --- a/Lib/test/test_codecs.py +++ b/Lib/test/test_codecs.py @@ -2,7 +2,6 @@ import contextlib import copy import io -import locale import pickle import sys import unittest @@ -1812,16 +1811,10 @@ def test_getwriter(self): self.assertRaises(TypeError, codecs.getwriter) self.assertRaises(LookupError, codecs.getwriter, "__spam__") + @support.run_with_locale('LC_CTYPE', 'tr_TR') def test_lookup_issue1813(self): # Issue #1813: under Turkish locales, lookup of some codecs failed # because 'I' is lowercased as "ı" (dotless i) - oldlocale = locale.setlocale(locale.LC_CTYPE) - self.addCleanup(locale.setlocale, locale.LC_CTYPE, oldlocale) - try: - locale.setlocale(locale.LC_CTYPE, 'tr_TR') - except locale.Error: - # Unsupported locale on this system - self.skipTest('test needs Turkish locale') c = codecs.lookup('ASCII') self.assertEqual(c.name, 'ascii') diff --git a/Lib/test/test_compile.py b/Lib/test/test_compile.py index e6dc7a53189e12..6f838da6018741 100644 --- a/Lib/test/test_compile.py +++ b/Lib/test/test_compile.py @@ -1386,6 +1386,14 @@ def check_op_count(func, op, expected): actual += 1 self.assertEqual(actual, expected) + def check_consts(func, typ, expected): + slice_consts = 0 + consts = func.__code__.co_consts + for instr in dis.Bytecode(func): + if instr.opname == "LOAD_CONST" and isinstance(consts[instr.oparg], typ): + slice_consts += 1 + self.assertEqual(slice_consts, expected) + def load(): return x[a:b] + x [a:] + x[:b] + x[:] @@ -1401,15 +1409,30 @@ def long_slice(): def aug(): x[a:b] += y - check_op_count(load, "BINARY_SLICE", 4) + def aug_const(): + x[1:2] += y + + def compound_const_slice(): + x[1:2:3, 4:5:6] = y + + check_op_count(load, "BINARY_SLICE", 3) check_op_count(load, "BUILD_SLICE", 0) - check_op_count(store, "STORE_SLICE", 4) + check_consts(load, slice, 1) + check_op_count(store, "STORE_SLICE", 3) check_op_count(store, "BUILD_SLICE", 0) + check_consts(store, slice, 1) check_op_count(long_slice, "BUILD_SLICE", 1) check_op_count(long_slice, "BINARY_SLICE", 0) check_op_count(aug, "BINARY_SLICE", 1) check_op_count(aug, "STORE_SLICE", 1) check_op_count(aug, "BUILD_SLICE", 0) + check_op_count(aug_const, "BINARY_SLICE", 0) + check_op_count(aug_const, "STORE_SLICE", 0) + check_consts(aug_const, slice, 1) + check_op_count(compound_const_slice, "BINARY_SLICE", 0) + check_op_count(compound_const_slice, "BUILD_SLICE", 0) + check_consts(compound_const_slice, slice, 0) + check_consts(compound_const_slice, tuple, 1) def test_compare_positions(self): for opname_prefix, op in [ diff --git a/Lib/test/test_decimal.py b/Lib/test/test_decimal.py index c591fd54430b18..d1e7e69e7e951b 100644 --- a/Lib/test/test_decimal.py +++ b/Lib/test/test_decimal.py @@ -1253,7 +1253,7 @@ def test_deprecated_N_format(self): self.assertRaises(ValueError, format, h, '10Nf') self.assertRaises(ValueError, format, h, 'Nx') - @run_with_locale('LC_ALL', 'ps_AF') + @run_with_locale('LC_ALL', 'ps_AF', '') def test_wide_char_separator_decimal_point(self): # locale with wide char separator and decimal point Decimal = self.decimal.Decimal diff --git a/Lib/test/test_embed.py b/Lib/test/test_embed.py index 3edc19d8254754..035d4418b15c51 100644 --- a/Lib/test/test_embed.py +++ b/Lib/test/test_embed.py @@ -168,7 +168,8 @@ def run_repeated_init_and_subinterpreters(self): # Parse the line from the loop. The first line is the main # interpreter and the 3 afterward are subinterpreters. interp = Interp(*match.groups()) - if support.verbose > 1: + if support.verbose > 2: + # 5 lines per pass is super-spammy, so limit that to -vvv print(interp) self.assertTrue(interp.interp) self.assertTrue(interp.tstate) @@ -279,6 +280,10 @@ def test_pre_initialization_api(self): """ env = dict(os.environ, PYTHONPATH=os.pathsep.join(sys.path)) out, err = self.run_embedded_interpreter("test_pre_initialization_api", env=env) + if support.verbose > 1: + print() + print(out) + print(err) if MS_WINDOWS: expected_path = self.test_exe else: @@ -296,6 +301,10 @@ def test_pre_initialization_sys_options(self): env['PYTHONPATH'] = os.pathsep.join(sys.path) out, err = self.run_embedded_interpreter( "test_pre_initialization_sys_options", env=env) + if support.verbose > 1: + print() + print(out) + print(err) expected_output = ( "sys.warnoptions: ['once', 'module', 'default']\n" "sys._xoptions: {'not_an_option': '1', 'also_not_an_option': '2'}\n" diff --git a/Lib/test/test_exceptions.py b/Lib/test/test_exceptions.py index ba858c49400911..b3c21cd4f3d585 100644 --- a/Lib/test/test_exceptions.py +++ b/Lib/test/test_exceptions.py @@ -8,6 +8,7 @@ import weakref import errno from codecs import BOM_UTF8 +from itertools import product from textwrap import dedent from test.support import (captured_stderr, check_impl_detail, @@ -1336,6 +1337,29 @@ def test_unicode_errors_no_object(self): for klass in klasses: self.assertEqual(str(klass.__new__(klass)), "") + def test_unicode_error_str_does_not_crash(self): + # Test that str(UnicodeError(...)) does not crash. + # See https://github.com/python/cpython/issues/123378. + + for start, end, objlen in product( + range(-5, 5), + range(-5, 5), + range(7), + ): + obj = 'a' * objlen + with self.subTest('encode', objlen=objlen, start=start, end=end): + exc = UnicodeEncodeError('utf-8', obj, start, end, '') + self.assertIsInstance(str(exc), str) + + with self.subTest('translate', objlen=objlen, start=start, end=end): + exc = UnicodeTranslateError(obj, start, end, '') + self.assertIsInstance(str(exc), str) + + encoded = obj.encode() + with self.subTest('decode', objlen=objlen, start=start, end=end): + exc = UnicodeDecodeError('utf-8', encoded, start, end, '') + self.assertIsInstance(str(exc), str) + @no_tracing def test_badisinstance(self): # Bug #2542: if issubclass(e, MyException) raises an exception, diff --git a/Lib/test/test_float.py b/Lib/test/test_float.py index 048bb14509064b..f588e16b70123a 100644 --- a/Lib/test/test_float.py +++ b/Lib/test/test_float.py @@ -175,7 +175,7 @@ def check(s): # non-UTF-8 byte string check(b'123\xa0') - @support.run_with_locale('LC_NUMERIC', 'fr_FR', 'de_DE') + @support.run_with_locale('LC_NUMERIC', 'fr_FR', 'de_DE', '') def test_float_with_comma(self): # set locale to something that doesn't use '.' for the decimal point # float must not accept the locale specific decimal point but diff --git a/Lib/test/test_frame.py b/Lib/test/test_frame.py index 32de8ed9a13f80..80c5379b69ca4b 100644 --- a/Lib/test/test_frame.py +++ b/Lib/test/test_frame.py @@ -515,6 +515,61 @@ def make_frame(): with self.assertRaises(TypeError): FrameLocalsProxy(frame=sys._getframe()) # no keyword arguments + def test_generator_f_locals(self): + def get_generator_genexpr(new_value): + g = (x for x in range(10)) + g.gi_frame.f_locals['.0'] = new_value + return g + + def get_generator_fn_call(new_value): + def gen(it): + for x in it: + yield x + + g = gen(range(10)) + g.gi_frame.f_locals['it'] = new_value + return g + + err_msg_pattern_genexpr = "'for' requires an object with __iter__ method, got %s" + err_msg_pattern_fn_call = "'%s' object is not iterable" + + sequences = [ + range(0), + range(20), + [1, 2, 3], + (2,), + set((13, 48, 211)), + frozenset((15, 8, 6)), + dict([(1, 2), (3, 4)]), + ] + + for seq in sequences: + err_msg_genexpr = err_msg_pattern_genexpr % type(seq).__name__ + with self.assertRaisesRegex(TypeError, err_msg_genexpr): + list(get_generator_genexpr(seq)) + self.assertListEqual(list(get_generator_genexpr(iter(seq))), + list(seq)) + + self.assertListEqual(list(get_generator_fn_call(seq)), + list(seq)) + self.assertListEqual(list(get_generator_fn_call(iter(seq))), + list(seq)) + + non_sequences = [ + None, + 42, + 3.0, + 2j, + ] + + for obj in non_sequences: + err_msg_genexpr = err_msg_pattern_genexpr % type(obj).__name__ + with self.assertRaisesRegex(TypeError, err_msg_genexpr): + list(get_generator_genexpr(obj)) + err_msg_fn_call = err_msg_pattern_fn_call % type(obj).__name__ + with self.assertRaisesRegex(TypeError, err_msg_fn_call): + list(get_generator_fn_call(obj)) + class FrameLocalsProxyMappingTests(mapping_tests.TestHashMappingProtocol): """Test that FrameLocalsProxy behaves like a Mapping (with exceptions)""" diff --git a/Lib/test/test_imaplib.py b/Lib/test/test_imaplib.py index 1fd75d0a3f4c7b..a6509fc3ba0eae 100644 --- a/Lib/test/test_imaplib.py +++ b/Lib/test/test_imaplib.py @@ -57,7 +57,7 @@ def timevalues(self): timezone(timedelta(0, 2 * 60 * 60))), '"18-May-2033 05:33:20 +0200"'] - @run_with_locale('LC_ALL', 'de_DE', 'fr_FR') + @run_with_locale('LC_ALL', 'de_DE', 'fr_FR', '') # DST rules included to work around quirk where the Gnu C library may not # otherwise restore the previous time zone @run_with_tz('STD-1DST,M3.2.0,M11.1.0') diff --git a/Lib/test/test_inspect/inspect_deferred_annotations.py b/Lib/test/test_inspect/inspect_deferred_annotations.py new file mode 100644 index 00000000000000..bb59ef1035b3c1 --- /dev/null +++ b/Lib/test/test_inspect/inspect_deferred_annotations.py @@ -0,0 +1,2 @@ +def f(x: undefined): + pass diff --git a/Lib/test/test_inspect/test_inspect.py b/Lib/test/test_inspect/test_inspect.py index d2dc9e147d29c2..9fa6d23d15f06a 100644 --- a/Lib/test/test_inspect/test_inspect.py +++ b/Lib/test/test_inspect/test_inspect.py @@ -1,3 +1,4 @@ +from annotationlib import Format, ForwardRef import asyncio import builtins import collections @@ -22,7 +23,6 @@ import types import tempfile import textwrap -from typing import Unpack import unicodedata import unittest import unittest.mock @@ -46,6 +46,7 @@ from test.test_inspect import inspect_fodder as mod from test.test_inspect import inspect_fodder2 as mod2 from test.test_inspect import inspect_stringized_annotations +from test.test_inspect import inspect_deferred_annotations # Functions tested in this suite: @@ -4622,6 +4623,18 @@ def func( expected_multiline, ) + def test_signature_format_unquote(self): + def func(x: 'int') -> 'str': ... + + self.assertEqual( + inspect.signature(func).format(), + "(x: 'int') -> 'str'" + ) + self.assertEqual( + inspect.signature(func).format(quote_annotation_strings=False), + "(x: int) -> str" + ) + def test_signature_replace_parameters(self): def test(a, b) -> 42: pass @@ -4854,6 +4867,26 @@ def test_signature_eval_str(self): par('b', PORK, annotation=tuple), ))) + def test_signature_annotation_format(self): + ida = inspect_deferred_annotations + sig = inspect.Signature + par = inspect.Parameter + PORK = inspect.Parameter.POSITIONAL_OR_KEYWORD + for signature_func in (inspect.signature, inspect.Signature.from_callable): + with self.subTest(signature_func=signature_func): + self.assertEqual( + signature_func(ida.f, annotation_format=Format.STRING), + sig([par("x", PORK, annotation="undefined")]) + ) + self.assertEqual( + signature_func(ida.f, annotation_format=Format.FORWARDREF), + sig([par("x", PORK, annotation=ForwardRef("undefined"))]) + ) + with self.assertRaisesRegex(NameError, "undefined"): + signature_func(ida.f, annotation_format=Format.VALUE) + with self.assertRaisesRegex(NameError, "undefined"): + signature_func(ida.f) + def test_signature_none_annotation(self): class funclike: # Has to be callable, and have correct @@ -4879,38 +4912,6 @@ def foo(): pass self.assertEqual(signature_func(foo), inspect.Signature()) self.assertEqual(inspect.get_annotations(foo), {}) - def test_signature_as_str(self): - self.maxDiff = None - class S: - __signature__ = '(a, b=2)' - - self.assertEqual(self.signature(S), - ((('a', ..., ..., 'positional_or_keyword'), - ('b', 2, ..., 'positional_or_keyword')), - ...)) - - def test_signature_as_callable(self): - # __signature__ should be either a staticmethod or a bound classmethod - class S: - @classmethod - def __signature__(cls): - return '(a, b=2)' - - self.assertEqual(self.signature(S), - ((('a', ..., ..., 'positional_or_keyword'), - ('b', 2, ..., 'positional_or_keyword')), - ...)) - - class S: - @staticmethod - def __signature__(): - return '(a, b=2)' - - self.assertEqual(self.signature(S), - ((('a', ..., ..., 'positional_or_keyword'), - ('b', 2, ..., 'positional_or_keyword')), - ...)) - def test_signature_on_derived_classes(self): # gh-105080: Make sure that signatures are consistent on derived classes diff --git a/Lib/test/test_math.py b/Lib/test/test_math.py index a3eebc97ada23b..e2e2a419c7778c 100644 --- a/Lib/test/test_math.py +++ b/Lib/test/test_math.py @@ -2722,7 +2722,7 @@ def test_fma_infinities(self): # gh-73468: On some platforms, libc fma() doesn't implement IEE 754-2008 # properly: it doesn't use the right sign when the result is zero. @unittest.skipIf( - sys.platform.startswith(("freebsd", "wasi")) + sys.platform.startswith(("freebsd", "wasi", "netbsd")) or (sys.platform == "android" and platform.machine() == "x86_64"), f"this platform doesn't implement IEE 754-2008 properly") def test_fma_zero_result(self): diff --git a/Lib/test/test_pydoc/test_pydoc.py b/Lib/test/test_pydoc/test_pydoc.py index 776e02f41a1cec..2a4d3ab73db608 100644 --- a/Lib/test/test_pydoc/test_pydoc.py +++ b/Lib/test/test_pydoc/test_pydoc.py @@ -1073,7 +1073,7 @@ def __init__(self, class A(builtins.object) | A( - | arg1: collections.abc.Callable[[int, int, int], str], + | arg1: Callable[[int, int, int], str], | arg2: Literal['some value', 'other value'], | arg3: Annotated[int, 'some docs about this type'] | ) -> None @@ -1082,7 +1082,7 @@ class A(builtins.object) | | __init__( | self, - | arg1: collections.abc.Callable[[int, int, int], str], + | arg1: Callable[[int, int, int], str], | arg2: Literal['some value', 'other value'], | arg3: Annotated[int, 'some docs about this type'] | ) -> None @@ -1109,7 +1109,7 @@ def func( self.assertEqual(doc, '''Python Library Documentation: function func in module %s func( - arg1: collections.abc.Callable[[typing.Annotated[int, 'Some doc']], str], + arg1: Callable[[Annotated[int, 'Some doc']], str], arg2: Literal[1, 2, 3, 4, 5, 6, 7, 8] ) -> Annotated[int, 'Some other'] ''' % __name__) @@ -1394,8 +1394,8 @@ def foo(data: typing.List[typing.Any], T = typing.TypeVar('T') class C(typing.Generic[T], typing.Mapping[int, str]): ... self.assertEqual(pydoc.render_doc(foo).splitlines()[-1], - 'f\x08fo\x08oo\x08o(data: List[Any], x: int)' - ' -> Iterator[Tuple[int, Any]]') + 'f\x08fo\x08oo\x08o(data: typing.List[typing.Any], x: int)' + ' -> typing.Iterator[typing.Tuple[int, typing.Any]]') self.assertEqual(pydoc.render_doc(C).splitlines()[2], 'class C\x08C(collections.abc.Mapping, typing.Generic)') diff --git a/Lib/test/test_pyrepl/test_pyrepl.py b/Lib/test/test_pyrepl/test_pyrepl.py index 36f940eaea4eac..1a76832386bf1d 100644 --- a/Lib/test/test_pyrepl/test_pyrepl.py +++ b/Lib/test/test_pyrepl/test_pyrepl.py @@ -1204,6 +1204,18 @@ def test_python_basic_repl(self): self.assertNotIn("Exception", output) self.assertNotIn("Traceback", output) + # The site module must not load _pyrepl if PYTHON_BASIC_REPL is set + commands = ("import sys\n" + "print('_pyrepl' in sys.modules)\n" + "exit()\n") + env["PYTHON_BASIC_REPL"] = "1" + output, exit_code = self.run_repl(commands, env=env) + self.assertEqual(exit_code, 0) + self.assertIn("False", output) + self.assertNotIn("True", output) + self.assertNotIn("Exception", output) + self.assertNotIn("Traceback", output) + @force_not_colorized def test_bad_sys_excepthook_doesnt_crash_pyrepl(self): env = os.environ.copy() diff --git a/Lib/test/test_sqlite3/test_dump.py b/Lib/test/test_sqlite3/test_dump.py index d508f238f84fb5..550cea41976441 100644 --- a/Lib/test/test_sqlite3/test_dump.py +++ b/Lib/test/test_sqlite3/test_dump.py @@ -10,6 +10,7 @@ class DumpTests(MemoryDatabaseMixin, unittest.TestCase): def test_table_dump(self): expected_sqls = [ + "PRAGMA foreign_keys=OFF;", """CREATE TABLE "index"("index" blob);""" , """INSERT INTO "index" VALUES(X'01');""" @@ -48,7 +49,7 @@ def test_table_dump(self): expected_sqls = [ "PRAGMA foreign_keys=OFF;", "BEGIN TRANSACTION;", - *expected_sqls, + *expected_sqls[1:], "COMMIT;", ] [self.assertEqual(expected_sqls[i], actual_sqls[i]) diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py index 216aa84a8c147b..b93fa0ed99f8ce 100644 --- a/Lib/test/test_ssl.py +++ b/Lib/test/test_ssl.py @@ -383,6 +383,7 @@ def test_random(self): ssl.RAND_add(bytearray(b"this is a random bytearray object"), 75.0) def test_parse_cert(self): + self.maxDiff = None # note that this uses an 'unofficial' function in _ssl.c, # provided solely for this test, to exercise the certificate # parsing code diff --git a/Lib/test/test_stable_abi_ctypes.py b/Lib/test/test_stable_abi_ctypes.py index d16ad7ef5d4328..b14d500a9c6e97 100644 --- a/Lib/test/test_stable_abi_ctypes.py +++ b/Lib/test/test_stable_abi_ctypes.py @@ -805,6 +805,7 @@ def test_windows_feature_macros(self): "PyUnicode_DecodeUnicodeEscape", "PyUnicode_EncodeFSDefault", "PyUnicode_EncodeLocale", + "PyUnicode_Equal", "PyUnicode_EqualToUTF8", "PyUnicode_EqualToUTF8AndSize", "PyUnicode_FSConverter", diff --git a/Lib/test/test_str.py b/Lib/test/test_str.py index 6600dcf9157971..4de6c1cba152bd 100644 --- a/Lib/test/test_str.py +++ b/Lib/test/test_str.py @@ -1665,7 +1665,7 @@ def test_startswith_endswith_errors(self): self.assertIn('str', exc) self.assertIn('tuple', exc) - @support.run_with_locale('LC_ALL', 'de_DE', 'fr_FR') + @support.run_with_locale('LC_ALL', 'de_DE', 'fr_FR', '') def test_format_float(self): # should not format with a comma, but always with C locale self.assertEqual('1.0', '%.1f' % 1.0) diff --git a/Lib/test/test_strptime.py b/Lib/test/test_strptime.py index 038746e26c24ad..37f6b08db28b3e 100644 --- a/Lib/test/test_strptime.py +++ b/Lib/test/test_strptime.py @@ -7,7 +7,8 @@ import os import sys from test import support -from test.support import skip_if_buggy_ucrt_strfptime, warnings_helper +from test.support import warnings_helper +from test.support import skip_if_buggy_ucrt_strfptime, run_with_locales from datetime import date as datetime_date import _strptime @@ -207,8 +208,8 @@ class StrptimeTests(unittest.TestCase): """Tests for _strptime.strptime.""" def setUp(self): - """Create testing time tuple.""" - self.time_tuple = time.gmtime() + """Create testing time tuples.""" + self.time_tuple = time.localtime() def test_ValueError(self): # Make sure ValueError is raised when match fails or format is bad @@ -289,54 +290,67 @@ def test_unconverteddata(self): # Check ValueError is raised when there is unconverted data self.assertRaises(ValueError, _strptime._strptime_time, "10 12", "%m") - def helper(self, directive, position): + def roundtrip(self, fmt, position, time_tuple=None): """Helper fxn in testing.""" - fmt = "%d %Y" if directive == 'd' else "%" + directive - strf_output = time.strftime(fmt, self.time_tuple) + if time_tuple is None: + time_tuple = self.time_tuple + strf_output = time.strftime(fmt, time_tuple) strp_output = _strptime._strptime_time(strf_output, fmt) - self.assertTrue(strp_output[position] == self.time_tuple[position], - "testing of '%s' directive failed; '%s' -> %s != %s" % - (directive, strf_output, strp_output[position], - self.time_tuple[position])) + self.assertEqual(strp_output[position], time_tuple[position], + "testing of %r format failed; %r -> %r != %r" % + (fmt, strf_output, strp_output[position], + time_tuple[position])) + if support.verbose >= 3: + print("testing of %r format: %r -> %r" % + (fmt, strf_output, strp_output[position])) def test_year(self): # Test that the year is handled properly - for directive in ('y', 'Y'): - self.helper(directive, 0) + self.roundtrip('%Y', 0) + self.roundtrip('%y', 0) + self.roundtrip('%Y', 0, (1900, 1, 1, 0, 0, 0, 0, 1, 0)) + # Must also make sure %y values are correct for bounds set by Open Group - for century, bounds in ((1900, ('69', '99')), (2000, ('00', '68'))): - for bound in bounds: - strp_output = _strptime._strptime_time(bound, '%y') - expected_result = century + int(bound) - self.assertTrue(strp_output[0] == expected_result, - "'y' test failed; passed in '%s' " - "and returned '%s'" % (bound, strp_output[0])) + strptime = _strptime._strptime_time + self.assertEqual(strptime('00', '%y')[0], 2000) + self.assertEqual(strptime('68', '%y')[0], 2068) + self.assertEqual(strptime('69', '%y')[0], 1969) + self.assertEqual(strptime('99', '%y')[0], 1999) def test_month(self): # Test for month directives - for directive in ('B', 'b', 'm'): - self.helper(directive, 1) + self.roundtrip('%m', 1) + + @run_with_locales('LC_TIME', 'C', 'en_US', 'fr_FR', 'de_DE', 'ja_JP', 'he_IL', '') + def test_month_locale(self): + # Test for month directives + self.roundtrip('%B', 1) + self.roundtrip('%b', 1) + for m in range(1, 13): + self.roundtrip('%B', 1, (1900, m, 1, 0, 0, 0, 0, 1, 0)) + self.roundtrip('%b', 1, (1900, m, 1, 0, 0, 0, 0, 1, 0)) def test_day(self): # Test for day directives - self.helper('d', 2) + self.roundtrip('%d %Y', 2) def test_hour(self): # Test hour directives - self.helper('H', 3) - strf_output = time.strftime("%I %p", self.time_tuple) - strp_output = _strptime._strptime_time(strf_output, "%I %p") - self.assertTrue(strp_output[3] == self.time_tuple[3], - "testing of '%%I %%p' directive failed; '%s' -> %s != %s" % - (strf_output, strp_output[3], self.time_tuple[3])) + self.roundtrip('%H', 3) + + # NB: Only works on locales with AM/PM + @run_with_locales('LC_TIME', 'C', 'en_US', 'ja_JP') + def test_hour_locale(self): + # Test hour directives + self.roundtrip('%I %p', 3) def test_minute(self): # Test minute directives - self.helper('M', 4) + self.roundtrip('%M', 4) def test_second(self): # Test second directives - self.helper('S', 5) + self.roundtrip('%S', 5) def test_fraction(self): # Test microseconds @@ -347,12 +361,18 @@ def test_fraction(self): def test_weekday(self): # Test weekday directives - for directive in ('A', 'a', 'w', 'u'): - self.helper(directive,6) + self.roundtrip('%w', 6) + self.roundtrip('%u', 6) + + @run_with_locales('LC_TIME', 'C', 'en_US', 'fr_FR', 'de_DE', 'ja_JP', '') + def test_weekday_locale(self): + # Test weekday directives + self.roundtrip('%A', 6) + self.roundtrip('%a', 6) def test_julian(self): # Test julian directives - self.helper('j', 7) + self.roundtrip('%j', 7) def test_offset(self): one_hour = 60 * 60 @@ -449,20 +469,96 @@ def test_bad_timezone(self): "time.daylight set to %s and passing in %s" % (time.tzname, tz_value, time.daylight, tz_name)) - def test_date_time(self): + # NB: Does not roundtrip in some locales due to the ambiguity of + # the date and time representation (bugs in locales?): + # * Seconds are not included: bem_ZM, bokmal, ff_SN, nb_NO, nn_NO, + # no_NO, norwegian, nynorsk. + # * Hours are in 12-hour notation without AM/PM indication: hy_AM, + # id_ID, ms_MY. + # * Year is not included: ha_NG. + # * Use non-Gregorian calendar: lo_LA, thai, th_TH. + # + # BUG: Generates invalid regexp for br_FR, csb_PL, Arabic. + # BUG: Generates regexp that does not match the current date and time + # for fa_IR, gez_ER, gez_ET, lzh_TW, my_MM, or_IN, shn_MM, yo_NG. + # BUG: Generates regexp that does not match the current date and time + # for fa_IR, gez_ER, gez_ET, lzh_TW, my_MM, or_IN, shn_MM, yo_NG, + # fr_FR, ja_JP, he_IL, ko_KR, zh_CN, etc. + @run_with_locales('LC_TIME', 'C', 'en_US', 'de_DE', + 'eu_ES', 'mfe_MU') + def test_date_time_locale(self): # Test %c directive - for position in range(6): - self.helper('c', position) - - def test_date(self): + now = time.time() + self.roundtrip('%c', slice(0, 6), time.localtime(now)) + # 1 hour 20 minutes 30 seconds ago + self.roundtrip('%c', slice(0, 6), time.localtime(now - 4830)) + # 12 hours ago + self.roundtrip('%c', slice(0, 6), time.localtime(now - 12*3600)) + # different days of the week + for i in range(1, 7): + self.roundtrip('%c', slice(0, 6), time.localtime(now - i*24*3600)) + # different months + for i in range(1, 12): + self.roundtrip('%c', slice(0, 6), time.localtime(now - i*30*24*3600)) + # different year + self.roundtrip('%c', slice(0, 6), time.localtime(now - 366*24*3600)) + + # NB: Dates before 1969 do not roundtrip on some locales: + # bo_CN, bo_IN, dz_BT, eu_ES, eu_FR. + @run_with_locales('LC_TIME', 'C', 'en_US', 'de_DE', 'ja_JP') + def test_date_time_locale2(self): + # Test %c directive + self.roundtrip('%c', slice(0, 6), (1900, 1, 1, 0, 0, 0, 0, 1, 0)) + + # NB: Does not roundtrip because use non-Gregorian calendar: + # lo_LA, thai, th_TH. + # BUG: Generates regexp that does not match the current date + # for az_IR, fa_IR, lzh_TW, my_MM, or_IN, shn_MM, + # Arabic, ja_JP, ko_KR, zh_CN, etc. + @run_with_locales('LC_TIME', 'C', 'en_US', 'fr_FR', 'de_DE', + 'he_IL', 'eu_ES') + def test_date_locale(self): # Test %x directive - for position in range(0,3): - self.helper('x', position) - - def test_time(self): + now = time.time() + self.roundtrip('%x', slice(0, 3), time.localtime(now)) + # different days of the week + for i in range(1, 7): + self.roundtrip('%x', slice(0, 3), time.localtime(now - i*24*3600)) + # different months + for i in range(1, 12): + self.roundtrip('%x', slice(0, 3), time.localtime(now - i*30*24*3600)) + # different year + self.roundtrip('%x', slice(0, 3), time.localtime(now - 366*24*3600)) + + # NB: Dates before 1969 do not roundtrip on many locales, including C. + @unittest.skipIf( + support.is_emscripten or support.is_wasi, + "musl libc issue on Emscripten, bpo-46390" + ) + @run_with_locales('LC_TIME', 'en_US', 'fr_FR', 'de_DE', 'ja_JP') + def test_date_locale2(self): + # Test %x directive + self.roundtrip('%x', slice(0, 3), (1900, 1, 1, 0, 0, 0, 0, 1, 0)) + + # NB: Does not roundtrip in some locales due to the ambiguity of + # the time representation (bugs in locales?): + # * Seconds are not included: bokmal, ff_SN, nb_NO, nn_NO, no_NO, + # norwegian, nynorsk. + # * Hours are in 12-hour notation without AM/PM indication: hy_AM, + # ms_MY, sm_WS. + # BUG: Generates regexp that does not match the current time for + # aa_DJ, aa_ER, aa_ET, am_ET, az_IR, byn_ER, fa_IR, gez_ER, gez_ET, + # lzh_TW, my_MM, om_ET, om_KE, or_IN, shn_MM, sid_ET, so_DJ, so_ET, + # so_SO, ti_ER, ti_ET, tig_ER, wal_ET. + @run_with_locales('LC_TIME', 'C', 'en_US', 'fr_FR', 'de_DE', 'ja_JP') + def test_time_locale(self): # Test %X directive - for position in range(3,6): - self.helper('X', position) + now = time.time() + self.roundtrip('%X', slice(3, 6), time.localtime(now)) + # 1 hour 20 minutes 30 seconds ago + self.roundtrip('%X', slice(3, 6), time.localtime(now - 4830)) + # 12 hours ago + self.roundtrip('%X', slice(3, 6), time.localtime(now - 12*3600)) def test_percent(self): # Make sure % signs are handled properly @@ -714,12 +810,7 @@ def test_new_localetime(self): def test_TimeRE_recreation_locale(self): # The TimeRE instance should be recreated upon changing the locale. - locale_info = locale.getlocale(locale.LC_TIME) - try: - locale.setlocale(locale.LC_TIME, ('en_US', 'UTF8')) - except locale.Error: - self.skipTest('test needs en_US.UTF8 locale') - try: + with support.run_with_locale('LC_TIME', 'en_US.UTF8'): _strptime._strptime_time('10 2004', '%d %Y') # Get id of current cache object. first_time_re = _strptime._TimeRE_cache @@ -736,10 +827,6 @@ def test_TimeRE_recreation_locale(self): # to the resetting to the original locale. except locale.Error: self.skipTest('test needs de_DE.UTF8 locale') - # Make sure we don't trample on the locale setting once we leave the - # test. - finally: - locale.setlocale(locale.LC_TIME, locale_info) @support.run_with_tz('STD-1DST,M4.1.0,M10.1.0') def test_TimeRE_recreation_timezone(self): diff --git a/Lib/test/test_time.py b/Lib/test/test_time.py index 530c317a852e77..5b5779231f06ce 100644 --- a/Lib/test/test_time.py +++ b/Lib/test/test_time.py @@ -2,7 +2,6 @@ from test.support import warnings_helper import decimal import enum -import locale import math import platform import sys @@ -600,17 +599,8 @@ def test_get_clock_info(self): class TestLocale(unittest.TestCase): - def setUp(self): - self.oldloc = locale.setlocale(locale.LC_ALL) - - def tearDown(self): - locale.setlocale(locale.LC_ALL, self.oldloc) - + @support.run_with_locale('LC_ALL', 'fr_FR', '') def test_bug_3061(self): - try: - tmp = locale.setlocale(locale.LC_ALL, "fr_FR") - except locale.Error: - self.skipTest('could not set locale.LC_ALL to fr_FR') # This should not cause an exception time.strftime("%B", (2009,2,1,0,0,0,0,0,0)) diff --git a/Lib/test/test_types.py b/Lib/test/test_types.py index 3c9e33e3c9dbfc..d1161719d98040 100644 --- a/Lib/test/test_types.py +++ b/Lib/test/test_types.py @@ -398,7 +398,7 @@ def test(i, format_spec, result): test(123456, "1=20", '11111111111111123456') test(123456, "*=20", '**************123456') - @run_with_locale('LC_NUMERIC', 'en_US.UTF8') + @run_with_locale('LC_NUMERIC', 'en_US.UTF8', '') def test_float__format__locale(self): # test locale support for __format__ code 'n' @@ -407,7 +407,7 @@ def test_float__format__locale(self): self.assertEqual(locale.format_string('%g', x, grouping=True), format(x, 'n')) self.assertEqual(locale.format_string('%.10g', x, grouping=True), format(x, '.10n')) - @run_with_locale('LC_NUMERIC', 'en_US.UTF8') + @run_with_locale('LC_NUMERIC', 'en_US.UTF8', '') def test_int__format__locale(self): # test locale support for __format__ code 'n' for integers diff --git a/Misc/NEWS.d/next/C API/2018-06-30-21-48-16.bpo-34008.2Wjtm0.rst b/Misc/NEWS.d/next/C API/2018-06-30-21-48-16.bpo-34008.2Wjtm0.rst new file mode 100644 index 00000000000000..1a01dafc758004 --- /dev/null +++ b/Misc/NEWS.d/next/C API/2018-06-30-21-48-16.bpo-34008.2Wjtm0.rst @@ -0,0 +1,2 @@ +Added ``Py_IsInitialized`` to the list of APIs that are safe to call before +the interpreter is initialized, and updated the embedding tests to cover it. diff --git a/Misc/NEWS.d/next/C_API/2024-09-25-11-44-02.gh-issue-124502.qWuDjT.rst b/Misc/NEWS.d/next/C_API/2024-09-25-11-44-02.gh-issue-124502.qWuDjT.rst new file mode 100644 index 00000000000000..f515619328b359 --- /dev/null +++ b/Misc/NEWS.d/next/C_API/2024-09-25-11-44-02.gh-issue-124502.qWuDjT.rst @@ -0,0 +1,2 @@ +Add :c:func:`PyUnicode_Equal` function to the limited C API: test if two +strings are equal. Patch by Victor Stinner. diff --git a/Misc/NEWS.d/next/Core_and_Builtins/2024-10-03-14-39-41.gh-issue-123378.dCxANf.rst b/Misc/NEWS.d/next/Core_and_Builtins/2024-10-03-14-39-41.gh-issue-123378.dCxANf.rst new file mode 100644 index 00000000000000..5cd34535d674d3 --- /dev/null +++ b/Misc/NEWS.d/next/Core_and_Builtins/2024-10-03-14-39-41.gh-issue-123378.dCxANf.rst @@ -0,0 +1,3 @@ +Fix a crash in the :meth:`~object.__str__` method of :exc:`UnicodeError` +objects when the :attr:`UnicodeError.start` and :attr:`UnicodeError.end` +values are invalid or out-of-range. Patch by Bénédikt Tran. diff --git a/Misc/NEWS.d/next/Core_and_Builtins/2024-10-07-19-26-50.gh-issue-125038.ffSLCz.rst b/Misc/NEWS.d/next/Core_and_Builtins/2024-10-07-19-26-50.gh-issue-125038.ffSLCz.rst new file mode 100644 index 00000000000000..1cdeac708d17e8 --- /dev/null +++ b/Misc/NEWS.d/next/Core_and_Builtins/2024-10-07-19-26-50.gh-issue-125038.ffSLCz.rst @@ -0,0 +1,2 @@ +Fix crash when iterating over a generator expression after direct changes on ``gi_frame.f_locals``. Patch by +Mikhail Efimov. diff --git a/Misc/NEWS.d/next/Documentation/2018-07-04-20-35-25.bpo-34008.bqecIb.rst b/Misc/NEWS.d/next/Documentation/2018-07-04-20-35-25.bpo-34008.bqecIb.rst new file mode 100644 index 00000000000000..a89086af35bfc1 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2018-07-04-20-35-25.bpo-34008.bqecIb.rst @@ -0,0 +1,8 @@ +The :c:func:`Py_Main` documentation moved from the "Very High Level API" section to the +"Initialization and Finalization" section. + +Also make it explicit that we expect ``Py_Main`` to typically be called instead +of ``Py_Initialize`` rather than after it (since ``Py_Main`` makes its own +call to ``Py_Initialize``). Document that calling both is +supported but is version dependent on which settings +will be applied correctly. diff --git a/Misc/NEWS.d/next/Documentation/2024-08-01-17-18-21.gh-issue-70870.fZnBM9.rst b/Misc/NEWS.d/next/Documentation/2024-08-01-17-18-21.gh-issue-70870.fZnBM9.rst new file mode 100644 index 00000000000000..ba607bf7a42cd9 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2024-08-01-17-18-21.gh-issue-70870.fZnBM9.rst @@ -0,0 +1,3 @@ +Clarified the dual usage of the term "free variable" (both the formal +meaning of any reference to names defined outside the local scope, and the +narrower pragmatic meaning of nonlocal variables named in ``co_freevars``). diff --git a/Misc/NEWS.d/next/Library/2024-02-27-10-22-15.gh-issue-115937.0cVNur.rst b/Misc/NEWS.d/next/Library/2024-02-27-10-22-15.gh-issue-115937.0cVNur.rst new file mode 100644 index 00000000000000..f9dae0c9b8e2b2 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2024-02-27-10-22-15.gh-issue-115937.0cVNur.rst @@ -0,0 +1,3 @@ +Removed extra preprocessing for the ``__signature__`` attribute: the code +just check if it's a :class:`inspect.Signature` instance. Patch by Sergey B +Kirpichev. diff --git a/Misc/NEWS.d/next/Library/2024-09-27-06-39-32.gh-issue-101552.xYkzag.rst b/Misc/NEWS.d/next/Library/2024-09-27-06-39-32.gh-issue-101552.xYkzag.rst new file mode 100644 index 00000000000000..913a84de5fe6a3 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2024-09-27-06-39-32.gh-issue-101552.xYkzag.rst @@ -0,0 +1,4 @@ +Add an *annoation_format* parameter to :func:`inspect.signature`. Add an +*quote_annotation_strings* parameter to :meth:`inspect.Signature.format`. Use the +new functionality to improve the display of annotations in signatures in +:mod:`pydoc`. Patch by Jelle Zijlstra. diff --git a/Misc/NEWS.d/next/Library/2024-10-02-22-53-48.gh-issue-90102.4qX52R.rst b/Misc/NEWS.d/next/Library/2024-10-02-22-53-48.gh-issue-90102.4qX52R.rst new file mode 100644 index 00000000000000..0e708ed11d21b5 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2024-10-02-22-53-48.gh-issue-90102.4qX52R.rst @@ -0,0 +1,3 @@ +Skip the ``isatty`` system call during open() when the file is known to not +be a character device. This provides a slight performance improvement when +reading whole files. diff --git a/Misc/NEWS.d/next/Library/2024-10-03-19-16-38.gh-issue-123961.ik1Dgs.rst b/Misc/NEWS.d/next/Library/2024-10-03-19-16-38.gh-issue-123961.ik1Dgs.rst new file mode 100644 index 00000000000000..b637b895d0b803 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2024-10-03-19-16-38.gh-issue-123961.ik1Dgs.rst @@ -0,0 +1,2 @@ +Convert :mod:`curses` to multi-phase initialization (:pep:`489`), thereby +fixing reference leaks at interpreter shutdown. Patch by Bénédikt Tran. diff --git a/Misc/NEWS.d/next/Library/2024-10-04-12-43-03.gh-issue-69998.DVqOXX.rst b/Misc/NEWS.d/next/Library/2024-10-04-12-43-03.gh-issue-69998.DVqOXX.rst new file mode 100644 index 00000000000000..65388e0b4e7ee4 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2024-10-04-12-43-03.gh-issue-69998.DVqOXX.rst @@ -0,0 +1,3 @@ +Fix :func:`locale.nl_langinfo` in case when different categories have +different locales. The function now sets temporarily the ``LC_CTYPE`` locale +in some cases. This temporary change affects other threads. diff --git a/Misc/NEWS.d/next/Library/2024-10-08-12-09-09.gh-issue-124969._VBQLq.rst b/Misc/NEWS.d/next/Library/2024-10-08-12-09-09.gh-issue-124969._VBQLq.rst new file mode 100644 index 00000000000000..b5082b90721d42 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2024-10-08-12-09-09.gh-issue-124969._VBQLq.rst @@ -0,0 +1,3 @@ +Fix ``locale.nl_langinfo(locale.ALT_DIGITS)``. Now it returns a tuple of up +to 100 strings (an empty tuple on most locales). Previously it returned the +first item of that tuple or an empty string. diff --git a/Misc/NEWS.d/next/Library/2024-10-08-13-28-22.gh-issue-125096.Vz0W5g.rst b/Misc/NEWS.d/next/Library/2024-10-08-13-28-22.gh-issue-125096.Vz0W5g.rst new file mode 100644 index 00000000000000..c582a2dfe7243c --- /dev/null +++ b/Misc/NEWS.d/next/Library/2024-10-08-13-28-22.gh-issue-125096.Vz0W5g.rst @@ -0,0 +1,5 @@ +If the :envvar:`PYTHON_BASIC_REPL` environment variable is set, the +:mod:`site` module no longer imports the :mod:`!_pyrepl` module. Moreover, +the :mod:`site` module now respects :option:`-E` and :option:`-I` command +line options: ignore :envvar:`PYTHON_BASIC_REPL` in this case. Patch by +Victor Stinner. diff --git a/Misc/NEWS.d/next/Tests/2023-08-03-17-26-55.gh-issue-107562.ZnbscS.rst b/Misc/NEWS.d/next/Tests/2023-08-03-17-26-55.gh-issue-107562.ZnbscS.rst new file mode 100644 index 00000000000000..cc368d4bdffbd2 --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2023-08-03-17-26-55.gh-issue-107562.ZnbscS.rst @@ -0,0 +1,3 @@ +Test certificates have been updated to expire far in the future. This allows +testing Y2038 with system time set to after that, so that actual Y2038 +issues can be exposed, and not masked by expired certificate errors. diff --git a/Misc/stable_abi.toml b/Misc/stable_abi.toml index fe0a5e44f8fb15..62978261745d79 100644 --- a/Misc/stable_abi.toml +++ b/Misc/stable_abi.toml @@ -2536,3 +2536,5 @@ added = '3.14' [const.Py_TP_USE_SPEC] added = '3.14' +[function.PyUnicode_Equal] + added = '3.14' diff --git a/Modules/_cursesmodule.c b/Modules/_cursesmodule.c index 61b65675375547..27d5df08de933e 100644 --- a/Modules/_cursesmodule.c +++ b/Modules/_cursesmodule.c @@ -160,30 +160,31 @@ typedef chtype attr_t; /* No attr_t type is available */ #define _CURSES_PAIR_CONTENT_FUNC pair_content #endif /* _NCURSES_EXTENDED_COLOR_FUNCS */ -typedef struct _cursesmodule_state { - PyObject *error; // PyCursesError - PyTypeObject *window_type; // PyCursesWindow_Type -} _cursesmodule_state; +typedef struct { + PyObject *error; // curses exception type + PyTypeObject *window_type; // exposed by PyCursesWindow_Type +} cursesmodule_state; -// For now, we keep a global state variable to prepare for PEP 489. -static _cursesmodule_state curses_global_state; - -static inline _cursesmodule_state * -get_cursesmodule_state(PyObject *Py_UNUSED(module)) +static inline cursesmodule_state * +get_cursesmodule_state(PyObject *module) { - return &curses_global_state; + void *state = PyModule_GetState(module); + assert(state != NULL); + return (cursesmodule_state *)state; } -static inline _cursesmodule_state * -get_cursesmodule_state_by_cls(PyTypeObject *Py_UNUSED(cls)) +static inline cursesmodule_state * +get_cursesmodule_state_by_cls(PyTypeObject *cls) { - return &curses_global_state; + void *state = PyType_GetModuleState(cls); + assert(state != NULL); + return (cursesmodule_state *)state; } -static inline _cursesmodule_state * -get_cursesmodule_state_by_win(PyCursesWindowObject *Py_UNUSED(win)) +static inline cursesmodule_state * +get_cursesmodule_state_by_win(PyCursesWindowObject *win) { - return &curses_global_state; + return get_cursesmodule_state_by_cls(Py_TYPE(win)); } /*[clinic input] @@ -192,6 +193,9 @@ class _curses.window "PyCursesWindowObject *" "clinic_state()->window_type" [clinic start generated code]*/ /*[clinic end generated code: output=da39a3ee5e6b4b0d input=ae6cb623018f2cbc]*/ +/* Indicate whether the module has already been loaded or not. */ +static int curses_module_loaded = 0; + /* Tells whether setupterm() has been called to initialise terminfo. */ static int curses_setupterm_called = FALSE; @@ -211,8 +215,8 @@ static const char *curses_screen_encoding = NULL; * set and this returns 0. Otherwise, this returns 1. * * Since this function can be called in functions that do not - * have a direct access to the module's state, the exception - * type is directly taken from the global state for now. + * have a direct access to the module's state, '_curses.error' + * is imported on demand. */ static inline int _PyCursesCheckFunction(int called, const char *funcname) @@ -220,7 +224,12 @@ _PyCursesCheckFunction(int called, const char *funcname) if (called == TRUE) { return 1; } - PyErr_Format(curses_global_state.error, "must call %s() first", funcname); + PyObject *exc = _PyImport_GetModuleAttrString("_curses", "error"); + if (exc != NULL) { + PyErr_Format(exc, "must call %s() first", funcname); + Py_DECREF(exc); + } + assert(PyErr_Occurred()); return 0; } @@ -237,7 +246,7 @@ _PyCursesStatefulCheckFunction(PyObject *module, int called, const char *funcnam if (called == TRUE) { return 1; } - _cursesmodule_state *state = get_cursesmodule_state(module); + cursesmodule_state *state = get_cursesmodule_state(module); PyErr_Format(state->error, "must call %s() first", funcname); return 0; } @@ -275,7 +284,7 @@ _PyCursesStatefulCheckFunction(PyObject *module, int called, const char *funcnam /* Utility Functions */ static inline void -_PyCursesSetError(_cursesmodule_state *state, const char *funcname) +_PyCursesSetError(cursesmodule_state *state, const char *funcname) { if (funcname == NULL) { PyErr_SetString(state->error, catchall_ERR); @@ -296,7 +305,7 @@ PyCursesCheckERR(PyObject *module, int code, const char *fname) if (code != ERR) { Py_RETURN_NONE; } else { - _cursesmodule_state *state = get_cursesmodule_state(module); + cursesmodule_state *state = get_cursesmodule_state(module); _PyCursesSetError(state, fname); return NULL; } @@ -308,7 +317,7 @@ PyCursesCheckERR_ForWin(PyCursesWindowObject *win, int code, const char *fname) if (code != ERR) { Py_RETURN_NONE; } else { - _cursesmodule_state *state = get_cursesmodule_state_by_win(win); + cursesmodule_state *state = get_cursesmodule_state_by_win(win); _PyCursesSetError(state, fname); return NULL; } @@ -746,7 +755,7 @@ Window_TwoArgNoReturnFunction(wresize, int, "ii;lines,columns") /* Allocation and deallocation of Window Objects */ static PyObject * -PyCursesWindow_New(_cursesmodule_state *state, +PyCursesWindow_New(cursesmodule_state *state, WINDOW *win, const char *encoding) { if (encoding == NULL) { @@ -1405,12 +1414,12 @@ _curses_window_derwin_impl(PyCursesWindowObject *self, int group_left_1, win = derwin(self->win,nlines,ncols,begin_y,begin_x); if (win == NULL) { - _cursesmodule_state *state = get_cursesmodule_state_by_win(self); + cursesmodule_state *state = get_cursesmodule_state_by_win(self); PyErr_SetString(state->error, catchall_NULL); return NULL; } - _cursesmodule_state *state = get_cursesmodule_state_by_win(self); + cursesmodule_state *state = get_cursesmodule_state_by_win(self); return PyCursesWindow_New(state, win, NULL); } @@ -1559,7 +1568,7 @@ _curses_window_getkey_impl(PyCursesWindowObject *self, int group_right_1, /* getch() returns ERR in nodelay mode */ PyErr_CheckSignals(); if (!PyErr_Occurred()) { - _cursesmodule_state *state = get_cursesmodule_state_by_win(self); + cursesmodule_state *state = get_cursesmodule_state_by_win(self); PyErr_SetString(state->error, "no input"); } return NULL; @@ -1619,7 +1628,7 @@ _curses_window_get_wch_impl(PyCursesWindowObject *self, int group_right_1, return NULL; /* get_wch() returns ERR in nodelay mode */ - _cursesmodule_state *state = get_cursesmodule_state_by_win(self); + cursesmodule_state *state = get_cursesmodule_state_by_win(self); PyErr_SetString(state->error, "no input"); return NULL; } @@ -2133,7 +2142,7 @@ _curses_window_noutrefresh_impl(PyCursesWindowObject *self) #ifdef py_is_pad if (py_is_pad(self->win)) { if (!group_right_1) { - _cursesmodule_state *state = get_cursesmodule_state_by_win(self); + cursesmodule_state *state = get_cursesmodule_state_by_win(self); PyErr_SetString(state->error, "noutrefresh() called for a pad " "requires 6 arguments"); @@ -2358,7 +2367,7 @@ _curses_window_refresh_impl(PyCursesWindowObject *self, int group_right_1, #ifdef py_is_pad if (py_is_pad(self->win)) { if (!group_right_1) { - _cursesmodule_state *state = get_cursesmodule_state_by_win(self); + cursesmodule_state *state = get_cursesmodule_state_by_win(self); PyErr_SetString(state->error, "refresh() for a pad requires 6 arguments"); return NULL; @@ -2441,12 +2450,12 @@ _curses_window_subwin_impl(PyCursesWindowObject *self, int group_left_1, win = subwin(self->win, nlines, ncols, begin_y, begin_x); if (win == NULL) { - _cursesmodule_state *state = get_cursesmodule_state_by_win(self); + cursesmodule_state *state = get_cursesmodule_state_by_win(self); PyErr_SetString(state->error, catchall_NULL); return NULL; } - _cursesmodule_state *state = get_cursesmodule_state_by_win(self); + cursesmodule_state *state = get_cursesmodule_state_by_win(self); return PyCursesWindow_New(state, win, self->encoding); } @@ -2846,7 +2855,7 @@ _curses_color_content_impl(PyObject *module, int color_number) PyCursesStatefulInitialisedColor(module); if (_COLOR_CONTENT_FUNC(color_number, &r, &g, &b) == ERR) { - _cursesmodule_state *state = get_cursesmodule_state(module); + cursesmodule_state *state = get_cursesmodule_state(module); PyErr_Format(state->error, "%s() returned ERR", Py_STRINGIFY(_COLOR_CONTENT_FUNC)); return NULL; @@ -3086,7 +3095,7 @@ _curses_getmouse_impl(PyObject *module) rtn = getmouse( &event ); if (rtn == ERR) { - _cursesmodule_state *state = get_cursesmodule_state(module); + cursesmodule_state *state = get_cursesmodule_state(module); PyErr_SetString(state->error, "getmouse() returned ERR"); return NULL; } @@ -3181,11 +3190,11 @@ _curses_getwin(PyObject *module, PyObject *file) fseek(fp, 0, 0); win = getwin(fp); if (win == NULL) { - _cursesmodule_state *state = get_cursesmodule_state(module); + cursesmodule_state *state = get_cursesmodule_state(module); PyErr_SetString(state->error, catchall_NULL); goto error; } - _cursesmodule_state *state = get_cursesmodule_state(module); + cursesmodule_state *state = get_cursesmodule_state(module); res = PyCursesWindow_New(state, win, NULL); error: @@ -3332,7 +3341,7 @@ _curses_init_pair_impl(PyObject *module, int pair_number, int fg, int bg) COLOR_PAIRS - 1); } else { - _cursesmodule_state *state = get_cursesmodule_state(module); + cursesmodule_state *state = get_cursesmodule_state(module); PyErr_Format(state->error, "%s() returned ERR", Py_STRINGIFY(_CURSES_INIT_PAIR_FUNC)); } @@ -3358,14 +3367,14 @@ _curses_initscr_impl(PyObject *module) if (curses_initscr_called) { wrefresh(stdscr); - _cursesmodule_state *state = get_cursesmodule_state(module); + cursesmodule_state *state = get_cursesmodule_state(module); return PyCursesWindow_New(state, stdscr, NULL); } win = initscr(); if (win == NULL) { - _cursesmodule_state *state = get_cursesmodule_state(module); + cursesmodule_state *state = get_cursesmodule_state(module); PyErr_SetString(state->error, catchall_NULL); return NULL; } @@ -3462,7 +3471,7 @@ _curses_initscr_impl(PyObject *module) SetDictInt("COLS", COLS); #undef SetDictInt - _cursesmodule_state *state = get_cursesmodule_state(module); + cursesmodule_state *state = get_cursesmodule_state(module); PyObject *winobj = PyCursesWindow_New(state, win, NULL); if (winobj == NULL) { return NULL; @@ -3496,7 +3505,7 @@ _curses_setupterm_impl(PyObject *module, const char *term, int fd) sys_stdout = PySys_GetObject("stdout"); if (sys_stdout == NULL || sys_stdout == Py_None) { - _cursesmodule_state *state = get_cursesmodule_state(module); + cursesmodule_state *state = get_cursesmodule_state(module); PyErr_SetString(state->error, "lost sys.stdout"); return NULL; } @@ -3517,7 +3526,7 @@ _curses_setupterm_impl(PyObject *module, const char *term, int fd) s = "setupterm: could not find terminfo database"; } - _cursesmodule_state *state = get_cursesmodule_state(module); + cursesmodule_state *state = get_cursesmodule_state(module); PyErr_SetString(state->error, s); return NULL; } @@ -3835,12 +3844,12 @@ _curses_newpad_impl(PyObject *module, int nlines, int ncols) win = newpad(nlines, ncols); if (win == NULL) { - _cursesmodule_state *state = get_cursesmodule_state(module); + cursesmodule_state *state = get_cursesmodule_state(module); PyErr_SetString(state->error, catchall_NULL); return NULL; } - _cursesmodule_state *state = get_cursesmodule_state(module); + cursesmodule_state *state = get_cursesmodule_state(module); return PyCursesWindow_New(state, win, NULL); } @@ -3876,12 +3885,12 @@ _curses_newwin_impl(PyObject *module, int nlines, int ncols, win = newwin(nlines,ncols,begin_y,begin_x); if (win == NULL) { - _cursesmodule_state *state = get_cursesmodule_state(module); + cursesmodule_state *state = get_cursesmodule_state(module); PyErr_SetString(state->error, catchall_NULL); return NULL; } - _cursesmodule_state *state = get_cursesmodule_state(module); + cursesmodule_state *state = get_cursesmodule_state(module); return PyCursesWindow_New(state, win, NULL); } @@ -3996,7 +4005,7 @@ _curses_pair_content_impl(PyObject *module, int pair_number) COLOR_PAIRS - 1); } else { - _cursesmodule_state *state = get_cursesmodule_state(module); + cursesmodule_state *state = get_cursesmodule_state(module); PyErr_Format(state->error, "%s() returned ERR", Py_STRINGIFY(_CURSES_PAIR_CONTENT_FUNC)); } @@ -4327,7 +4336,7 @@ _curses_start_color_impl(PyObject *module) PyCursesStatefulInitialised(module); if (start_color() == ERR) { - _cursesmodule_state *state = get_cursesmodule_state(module); + cursesmodule_state *state = get_cursesmodule_state(module); PyErr_SetString(state->error, "start_color() returned ERR"); return NULL; } @@ -4480,7 +4489,7 @@ _curses_tparm_impl(PyObject *module, const char *str, int i1, int i2, int i3, result = tparm((char *)str,i1,i2,i3,i4,i5,i6,i7,i8,i9); if (!result) { - _cursesmodule_state *state = get_cursesmodule_state(module); + cursesmodule_state *state = get_cursesmodule_state(module); PyErr_SetString(state->error, "tparm() returned NULL"); return NULL; } @@ -4682,7 +4691,7 @@ _curses_use_default_colors_impl(PyObject *module) if (code != ERR) { Py_RETURN_NONE; } else { - _cursesmodule_state *state = get_cursesmodule_state(module); + cursesmodule_state *state = get_cursesmodule_state(module); PyErr_SetString(state->error, "use_default_colors() returned ERR"); return NULL; } @@ -4763,7 +4772,7 @@ _curses_has_extended_color_support_impl(PyObject *module) /* List of functions defined in the module */ -static PyMethodDef PyCurses_methods[] = { +static PyMethodDef cursesmodule_methods[] = { _CURSES_BAUDRATE_METHODDEF _CURSES_BEEP_METHODDEF _CURSES_CAN_CHANGE_COLOR_METHODDEF @@ -4872,7 +4881,7 @@ curses_capi_start_color_called(void) } static void * -curses_capi_new(_cursesmodule_state *state) +curses_capi_new(cursesmodule_state *state) { assert(state->window_type != NULL); void **capi = (void **)PyMem_Calloc(PyCurses_API_pointers, sizeof(void *)); @@ -4892,8 +4901,9 @@ curses_capi_free(void *capi) { assert(capi != NULL); void **capi_ptr = (void **)capi; - assert(capi_ptr[0] != NULL); - Py_DECREF(capi_ptr[0]); // decref curses window type + // In free-threaded builds, capi_ptr[0] may have been already cleared + // by curses_capi_capsule_destructor(), hence the use of Py_XDECREF(). + Py_XDECREF(capi_ptr[0]); // decref curses window type PyMem_Free(capi_ptr); } @@ -4942,12 +4952,44 @@ curses_capi_capsule_new(void *capi) return capsule; } -/* Module initialization */ +/* Module initialization and cleanup functions */ + +static int +cursesmodule_traverse(PyObject *mod, visitproc visit, void *arg) +{ + cursesmodule_state *state = get_cursesmodule_state(mod); + Py_VISIT(state->error); + Py_VISIT(state->window_type); + return 0; +} + +static int +cursesmodule_clear(PyObject *mod) +{ + cursesmodule_state *state = get_cursesmodule_state(mod); + Py_CLEAR(state->error); + Py_CLEAR(state->window_type); + return 0; +} + +static void +cursesmodule_free(void *mod) +{ + (void)cursesmodule_clear((PyObject *)mod); + curses_module_loaded = 0; // allow reloading once garbage-collected +} static int cursesmodule_exec(PyObject *module) { - _cursesmodule_state *state = get_cursesmodule_state(module); + if (curses_module_loaded) { + PyErr_SetString(PyExc_ImportError, + "module 'curses' can only be loaded once per process"); + return -1; + } + curses_module_loaded = 1; + + cursesmodule_state *state = get_cursesmodule_state(module); /* Initialize object type */ state->window_type = (PyTypeObject *)PyType_FromModuleAndSpec( module, &PyCursesWindow_Type_spec, NULL); @@ -4987,7 +5029,6 @@ cursesmodule_exec(PyObject *module) return -1; } rc = PyDict_SetItemString(module_dict, "error", state->error); - Py_DECREF(state->error); if (rc < 0) { return -1; } @@ -5181,33 +5222,26 @@ cursesmodule_exec(PyObject *module) /* Initialization function for the module */ -static struct PyModuleDef _cursesmodule = { +static PyModuleDef_Slot cursesmodule_slots[] = { + {Py_mod_exec, cursesmodule_exec}, + {Py_mod_multiple_interpreters, Py_MOD_MULTIPLE_INTERPRETERS_NOT_SUPPORTED}, + {Py_mod_gil, Py_MOD_GIL_NOT_USED}, + {0, NULL} +}; + +static struct PyModuleDef cursesmodule = { PyModuleDef_HEAD_INIT, .m_name = "_curses", - .m_size = -1, - .m_methods = PyCurses_methods, + .m_size = sizeof(cursesmodule_state), + .m_methods = cursesmodule_methods, + .m_slots = cursesmodule_slots, + .m_traverse = cursesmodule_traverse, + .m_clear = cursesmodule_clear, + .m_free = cursesmodule_free }; PyMODINIT_FUNC PyInit__curses(void) { - // create the module - PyObject *mod = PyModule_Create(&_cursesmodule); - if (mod == NULL) { - goto error; - } -#ifdef Py_GIL_DISABLED - if (PyUnstable_Module_SetGIL(mod, Py_MOD_GIL_NOT_USED) < 0) { - goto error; - } -#endif - // populate the module - if (cursesmodule_exec(mod) < 0) { - goto error; - } - return mod; - -error: - Py_XDECREF(mod); - return NULL; + return PyModuleDef_Init(&cursesmodule); } diff --git a/Modules/_io/_iomodule.c b/Modules/_io/_iomodule.c index 1238e6074246d0..6622f2cabb908b 100644 --- a/Modules/_io/_iomodule.c +++ b/Modules/_io/_iomodule.c @@ -346,7 +346,7 @@ _io_open_impl(PyObject *module, PyObject *file, const char *mode, /* buffering */ if (buffering < 0) { - PyObject *res = PyObject_CallMethodNoArgs(raw, &_Py_ID(isatty)); + PyObject *res = PyObject_CallMethodNoArgs(raw, &_Py_ID(_isatty_open_only)); if (res == NULL) goto error; isatty = PyObject_IsTrue(res); diff --git a/Modules/_io/fileio.c b/Modules/_io/fileio.c index d9597f860b9429..f374592eb95967 100644 --- a/Modules/_io/fileio.c +++ b/Modules/_io/fileio.c @@ -12,9 +12,6 @@ #ifdef HAVE_SYS_TYPES_H # include #endif -#ifdef HAVE_SYS_STAT_H -# include -#endif #ifdef HAVE_IO_H # include #endif @@ -1218,6 +1215,24 @@ _io_FileIO_isatty_impl(fileio *self) return PyBool_FromLong(res); } +/* Checks whether the file is a TTY using an open-only optimization. + + TTYs are always character devices. If the interpreter knows a file is + not a character device when it would call ``isatty``, can skip that + call. Inside ``open()`` there is a fresh stat result that contains that + information. Use the stat result to skip a system call. Outside of that + context TOCTOU issues (the fd could be arbitrarily modified by + surrounding code). */ +static PyObject * +_io_FileIO_isatty_open_only(PyObject *op, PyObject *Py_UNUSED(ignored)) +{ + fileio *self = _PyFileIO_CAST(op); + if (self->stat_atopen != NULL && !S_ISCHR(self->stat_atopen->st_mode)) { + Py_RETURN_FALSE; + } + return _io_FileIO_isatty_impl(self); +} + #include "clinic/fileio.c.h" static PyMethodDef fileio_methods[] = { @@ -1234,6 +1249,7 @@ static PyMethodDef fileio_methods[] = { _IO_FILEIO_WRITABLE_METHODDEF _IO_FILEIO_FILENO_METHODDEF _IO_FILEIO_ISATTY_METHODDEF + {"_isatty_open_only", _io_FileIO_isatty_open_only, METH_NOARGS}, {"_dealloc_warn", fileio_dealloc_warn, METH_O, NULL}, {"__reduce__", _PyIOBase_cannot_pickle, METH_NOARGS}, {"__reduce_ex__", _PyIOBase_cannot_pickle, METH_O}, diff --git a/Modules/_io/winconsoleio.c b/Modules/_io/winconsoleio.c index ec5c298066a587..d7cb5abfdc0abd 100644 --- a/Modules/_io/winconsoleio.c +++ b/Modules/_io/winconsoleio.c @@ -1128,6 +1128,7 @@ static PyMethodDef winconsoleio_methods[] = { _IO__WINDOWSCONSOLEIO_WRITABLE_METHODDEF _IO__WINDOWSCONSOLEIO_FILENO_METHODDEF _IO__WINDOWSCONSOLEIO_ISATTY_METHODDEF + {"_isatty_open_only", (PyCFunction)_io__WindowsConsoleIO_isatty, METH_NOARGS}, {NULL, NULL} /* sentinel */ }; diff --git a/Modules/_localemodule.c b/Modules/_localemodule.c index 9452df492bb23b..0daec646605775 100644 --- a/Modules/_localemodule.c +++ b/Modules/_localemodule.c @@ -144,6 +144,17 @@ locale_is_ascii(const char *str) return (strlen(str) == 1 && ((unsigned char)str[0]) <= 127); } +static int +is_all_ascii(const char *str) +{ + for (; *str; str++) { + if ((unsigned char)*str > 127) { + return 0; + } + } + return 1; +} + static int locale_decode_monetary(PyObject *dict, struct lconv *lc) { @@ -478,113 +489,153 @@ _locale__getdefaultlocale_impl(PyObject *module) #endif #ifdef HAVE_LANGINFO_H -#define LANGINFO(X) {#X, X} +#define LANGINFO(X, Y) {#X, X, Y} static struct langinfo_constant{ - char* name; + const char *name; int value; + int category; } langinfo_constants[] = { /* These constants should exist on any langinfo implementation */ - LANGINFO(DAY_1), - LANGINFO(DAY_2), - LANGINFO(DAY_3), - LANGINFO(DAY_4), - LANGINFO(DAY_5), - LANGINFO(DAY_6), - LANGINFO(DAY_7), - - LANGINFO(ABDAY_1), - LANGINFO(ABDAY_2), - LANGINFO(ABDAY_3), - LANGINFO(ABDAY_4), - LANGINFO(ABDAY_5), - LANGINFO(ABDAY_6), - LANGINFO(ABDAY_7), - - LANGINFO(MON_1), - LANGINFO(MON_2), - LANGINFO(MON_3), - LANGINFO(MON_4), - LANGINFO(MON_5), - LANGINFO(MON_6), - LANGINFO(MON_7), - LANGINFO(MON_8), - LANGINFO(MON_9), - LANGINFO(MON_10), - LANGINFO(MON_11), - LANGINFO(MON_12), - - LANGINFO(ABMON_1), - LANGINFO(ABMON_2), - LANGINFO(ABMON_3), - LANGINFO(ABMON_4), - LANGINFO(ABMON_5), - LANGINFO(ABMON_6), - LANGINFO(ABMON_7), - LANGINFO(ABMON_8), - LANGINFO(ABMON_9), - LANGINFO(ABMON_10), - LANGINFO(ABMON_11), - LANGINFO(ABMON_12), + LANGINFO(DAY_1, LC_TIME), + LANGINFO(DAY_2, LC_TIME), + LANGINFO(DAY_3, LC_TIME), + LANGINFO(DAY_4, LC_TIME), + LANGINFO(DAY_5, LC_TIME), + LANGINFO(DAY_6, LC_TIME), + LANGINFO(DAY_7, LC_TIME), + + LANGINFO(ABDAY_1, LC_TIME), + LANGINFO(ABDAY_2, LC_TIME), + LANGINFO(ABDAY_3, LC_TIME), + LANGINFO(ABDAY_4, LC_TIME), + LANGINFO(ABDAY_5, LC_TIME), + LANGINFO(ABDAY_6, LC_TIME), + LANGINFO(ABDAY_7, LC_TIME), + + LANGINFO(MON_1, LC_TIME), + LANGINFO(MON_2, LC_TIME), + LANGINFO(MON_3, LC_TIME), + LANGINFO(MON_4, LC_TIME), + LANGINFO(MON_5, LC_TIME), + LANGINFO(MON_6, LC_TIME), + LANGINFO(MON_7, LC_TIME), + LANGINFO(MON_8, LC_TIME), + LANGINFO(MON_9, LC_TIME), + LANGINFO(MON_10, LC_TIME), + LANGINFO(MON_11, LC_TIME), + LANGINFO(MON_12, LC_TIME), + + LANGINFO(ABMON_1, LC_TIME), + LANGINFO(ABMON_2, LC_TIME), + LANGINFO(ABMON_3, LC_TIME), + LANGINFO(ABMON_4, LC_TIME), + LANGINFO(ABMON_5, LC_TIME), + LANGINFO(ABMON_6, LC_TIME), + LANGINFO(ABMON_7, LC_TIME), + LANGINFO(ABMON_8, LC_TIME), + LANGINFO(ABMON_9, LC_TIME), + LANGINFO(ABMON_10, LC_TIME), + LANGINFO(ABMON_11, LC_TIME), + LANGINFO(ABMON_12, LC_TIME), #ifdef RADIXCHAR /* The following are not available with glibc 2.0 */ - LANGINFO(RADIXCHAR), - LANGINFO(THOUSEP), + LANGINFO(RADIXCHAR, LC_NUMERIC), + LANGINFO(THOUSEP, LC_NUMERIC), /* YESSTR and NOSTR are deprecated in glibc, since they are a special case of message translation, which should be rather done using gettext. So we don't expose it to Python in the first place. - LANGINFO(YESSTR), - LANGINFO(NOSTR), + LANGINFO(YESSTR, LC_MESSAGES), + LANGINFO(NOSTR, LC_MESSAGES), */ - LANGINFO(CRNCYSTR), + LANGINFO(CRNCYSTR, LC_MONETARY), #endif - LANGINFO(D_T_FMT), - LANGINFO(D_FMT), - LANGINFO(T_FMT), - LANGINFO(AM_STR), - LANGINFO(PM_STR), + LANGINFO(D_T_FMT, LC_TIME), + LANGINFO(D_FMT, LC_TIME), + LANGINFO(T_FMT, LC_TIME), + LANGINFO(AM_STR, LC_TIME), + LANGINFO(PM_STR, LC_TIME), /* The following constants are available only with XPG4, but... OpenBSD doesn't have CODESET but has T_FMT_AMPM, and doesn't have a few of the others. Solution: ifdef-test them all. */ #ifdef CODESET - LANGINFO(CODESET), + LANGINFO(CODESET, LC_CTYPE), #endif #ifdef T_FMT_AMPM - LANGINFO(T_FMT_AMPM), + LANGINFO(T_FMT_AMPM, LC_TIME), #endif #ifdef ERA - LANGINFO(ERA), + LANGINFO(ERA, LC_TIME), #endif #ifdef ERA_D_FMT - LANGINFO(ERA_D_FMT), + LANGINFO(ERA_D_FMT, LC_TIME), #endif #ifdef ERA_D_T_FMT - LANGINFO(ERA_D_T_FMT), + LANGINFO(ERA_D_T_FMT, LC_TIME), #endif #ifdef ERA_T_FMT - LANGINFO(ERA_T_FMT), + LANGINFO(ERA_T_FMT, LC_TIME), #endif #ifdef ALT_DIGITS - LANGINFO(ALT_DIGITS), + LANGINFO(ALT_DIGITS, LC_TIME), #endif #ifdef YESEXPR - LANGINFO(YESEXPR), + LANGINFO(YESEXPR, LC_MESSAGES), #endif #ifdef NOEXPR - LANGINFO(NOEXPR), + LANGINFO(NOEXPR, LC_MESSAGES), #endif #ifdef _DATE_FMT /* This is not available in all glibc versions that have CODESET. */ - LANGINFO(_DATE_FMT), + LANGINFO(_DATE_FMT, LC_TIME), #endif - {0, 0} + {0, 0, 0} }; +/* Temporary make the LC_CTYPE locale to be the same as + * the locale of the specified category. */ +static int +change_locale(int category, char **oldloc) +{ + /* Keep a copy of the LC_CTYPE locale */ + *oldloc = setlocale(LC_CTYPE, NULL); + if (!*oldloc) { + PyErr_SetString(PyExc_RuntimeError, "faild to get LC_CTYPE locale"); + return -1; + } + *oldloc = _PyMem_Strdup(*oldloc); + if (!*oldloc) { + PyErr_NoMemory(); + return -1; + } + + /* Set a new locale if it is different. */ + char *loc = setlocale(category, NULL); + if (loc == NULL || strcmp(loc, *oldloc) == 0) { + PyMem_Free(*oldloc); + *oldloc = NULL; + return 0; + } + + setlocale(LC_CTYPE, loc); + return 1; +} + +/* Restore the old LC_CTYPE locale. */ +static void +restore_locale(char *oldloc) +{ + if (oldloc != NULL) { + setlocale(LC_CTYPE, oldloc); + PyMem_Free(oldloc); + } +} + /*[clinic input] _locale.nl_langinfo @@ -602,14 +653,50 @@ _locale_nl_langinfo_impl(PyObject *module, int item) /* Check whether this is a supported constant. GNU libc sometimes returns numeric values in the char* return value, which would crash PyUnicode_FromString. */ - for (i = 0; langinfo_constants[i].name; i++) + for (i = 0; langinfo_constants[i].name; i++) { if (langinfo_constants[i].value == item) { /* Check NULL as a workaround for GNU libc's returning NULL instead of an empty string for nl_langinfo(ERA). */ const char *result = nl_langinfo(item); result = result != NULL ? result : ""; - return PyUnicode_DecodeLocale(result, NULL); + char *oldloc = NULL; + if (langinfo_constants[i].category != LC_CTYPE + && !is_all_ascii(result) + && change_locale(langinfo_constants[i].category, &oldloc) < 0) + { + return NULL; + } + PyObject *pyresult; +#ifdef ALT_DIGITS + if (item == ALT_DIGITS) { + /* The result is a sequence of up to 100 NUL-separated strings. */ + const char *s = result; + int count = 0; + for (; count < 100 && *s; count++) { + s += strlen(s) + 1; + } + pyresult = PyTuple_New(count); + if (pyresult != NULL) { + for (int i = 0; i < count; i++) { + PyObject *unicode = PyUnicode_DecodeLocale(result, NULL); + if (unicode == NULL) { + Py_CLEAR(pyresult); + break; + } + PyTuple_SET_ITEM(pyresult, i, unicode); + result += strlen(result) + 1; + } + } + } + else +#endif + { + pyresult = PyUnicode_DecodeLocale(result, NULL); + } + restore_locale(oldloc); + return pyresult; } + } PyErr_SetString(PyExc_ValueError, "unsupported langinfo constant"); return NULL; } diff --git a/Modules/_testlimitedcapi/unicode.c b/Modules/_testlimitedcapi/unicode.c index 2b70d09108a333..c7a23d5d1cbd71 100644 --- a/Modules/_testlimitedcapi/unicode.c +++ b/Modules/_testlimitedcapi/unicode.c @@ -1,7 +1,7 @@ #include "pyconfig.h" // Py_GIL_DISABLED #ifndef Py_GIL_DISABLED - // Need limited C API 3.13 to test PyUnicode_EqualToUTF8() -# define Py_LIMITED_API 0x030d0000 + // Need limited C API 3.14 to test PyUnicode_Equal() +# define Py_LIMITED_API 0x030e0000 #endif #include "parts.h" @@ -1837,6 +1837,23 @@ test_string_from_format(PyObject *self, PyObject *Py_UNUSED(ignored)) #undef CHECK_FORMAT_0 } + +/* Test PyUnicode_Equal() */ +static PyObject * +unicode_equal(PyObject *module, PyObject *args) +{ + PyObject *str1, *str2; + if (!PyArg_ParseTuple(args, "OO", &str1, &str2)) { + return NULL; + } + + NULLABLE(str1); + NULLABLE(str2); + RETURN_INT(PyUnicode_Equal(str1, str2)); +} + + + static PyMethodDef TestMethods[] = { {"codec_incrementalencoder", codec_incrementalencoder, METH_VARARGS}, {"codec_incrementaldecoder", codec_incrementaldecoder, METH_VARARGS}, @@ -1924,6 +1941,7 @@ static PyMethodDef TestMethods[] = { {"unicode_format", unicode_format, METH_VARARGS}, {"unicode_contains", unicode_contains, METH_VARARGS}, {"unicode_isidentifier", unicode_isidentifier, METH_O}, + {"unicode_equal", unicode_equal, METH_VARARGS}, {NULL}, }; diff --git a/Modules/_threadmodule.c b/Modules/_threadmodule.c index 153fe85597749d..9617f9cafe76ff 100644 --- a/Modules/_threadmodule.c +++ b/Modules/_threadmodule.c @@ -705,9 +705,7 @@ static PyType_Spec ThreadHandle_Type_spec = { typedef struct { PyObject_HEAD - PyThread_type_lock lock_lock; - PyObject *in_weakreflist; - char locked; /* for sanity checking */ + PyMutex lock; } lockobject; static int @@ -722,15 +720,7 @@ lock_dealloc(PyObject *op) { lockobject *self = (lockobject*)op; PyObject_GC_UnTrack(self); - if (self->in_weakreflist != NULL) { - PyObject_ClearWeakRefs((PyObject *) self); - } - if (self->lock_lock != NULL) { - /* Unlock the lock so it's safe to free it */ - if (self->locked) - PyThread_release_lock(self->lock_lock); - PyThread_free_lock(self->lock_lock); - } + PyObject_ClearWeakRefs((PyObject *) self); PyTypeObject *tp = Py_TYPE(self); tp->tp_free((PyObject*)self); Py_DECREF(tp); @@ -798,13 +788,12 @@ lock_PyThread_acquire_lock(PyObject *op, PyObject *args, PyObject *kwds) return NULL; } - PyLockStatus r = acquire_timed(self->lock_lock, timeout); + PyLockStatus r = _PyMutex_LockTimed(&self->lock, timeout, + _PY_LOCK_HANDLE_SIGNALS | _PY_LOCK_DETACH); if (r == PY_LOCK_INTR) { return NULL; } - if (r == PY_LOCK_ACQUIRED) - self->locked = 1; return PyBool_FromLong(r == PY_LOCK_ACQUIRED); } @@ -836,13 +825,11 @@ lock_PyThread_release_lock(PyObject *op, PyObject *Py_UNUSED(ignored)) { lockobject *self = (lockobject*)op; /* Sanity check: the lock must be locked */ - if (!self->locked) { + if (_PyMutex_TryUnlock(&self->lock) < 0) { PyErr_SetString(ThreadError, "release unlocked lock"); return NULL; } - self->locked = 0; - PyThread_release_lock(self->lock_lock); Py_RETURN_NONE; } @@ -870,7 +857,7 @@ static PyObject * lock_locked_lock(PyObject *op, PyObject *Py_UNUSED(ignored)) { lockobject *self = (lockobject*)op; - return PyBool_FromLong((long)self->locked); + return PyBool_FromLong(PyMutex_IsLocked(&self->lock)); } PyDoc_STRVAR(locked_doc, @@ -890,21 +877,15 @@ lock_repr(PyObject *op) { lockobject *self = (lockobject*)op; return PyUnicode_FromFormat("<%s %s object at %p>", - self->locked ? "locked" : "unlocked", Py_TYPE(self)->tp_name, self); + PyMutex_IsLocked(&self->lock) ? "locked" : "unlocked", Py_TYPE(self)->tp_name, self); } #ifdef HAVE_FORK static PyObject * lock__at_fork_reinit(PyObject *op, PyObject *Py_UNUSED(args)) { - lockobject *self = (lockobject*)op; - if (_PyThread_at_fork_reinit(&self->lock_lock) < 0) { - PyErr_SetString(ThreadError, "failed to reinitialize lock at fork"); - return NULL; - } - - self->locked = 0; - + lockobject *self = (lockobject *)op; + _PyMutex_at_fork_reinit(&self->lock); Py_RETURN_NONE; } #endif /* HAVE_FORK */ @@ -970,18 +951,12 @@ A lock is not owned by the thread that locked it; another thread may\n\ unlock it. A thread attempting to lock a lock that it has already locked\n\ will block until another thread unlocks it. Deadlocks may ensue."); -static PyMemberDef lock_type_members[] = { - {"__weaklistoffset__", Py_T_PYSSIZET, offsetof(lockobject, in_weakreflist), Py_READONLY}, - {NULL}, -}; - static PyType_Slot lock_type_slots[] = { {Py_tp_dealloc, lock_dealloc}, {Py_tp_repr, lock_repr}, {Py_tp_doc, (void *)lock_doc}, {Py_tp_methods, lock_methods}, {Py_tp_traverse, lock_traverse}, - {Py_tp_members, lock_type_members}, {Py_tp_new, lock_new}, {0, 0} }; @@ -990,7 +965,7 @@ static PyType_Spec lock_type_spec = { .name = "_thread.lock", .basicsize = sizeof(lockobject), .flags = (Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC | - Py_TPFLAGS_IMMUTABLETYPE), + Py_TPFLAGS_IMMUTABLETYPE | Py_TPFLAGS_MANAGED_WEAKREF), .slots = lock_type_slots, }; @@ -1340,16 +1315,7 @@ newlockobject(PyObject *module) if (self == NULL) { return NULL; } - - self->lock_lock = PyThread_allocate_lock(); - self->locked = 0; - self->in_weakreflist = NULL; - - if (self->lock_lock == NULL) { - Py_DECREF(self); - PyErr_SetString(ThreadError, "can't allocate lock"); - return NULL; - } + self->lock = (PyMutex){0}; return self; } diff --git a/Objects/codeobject.c b/Objects/codeobject.c index 6f0b3f8b9a3262..8a2f4d32b911d9 100644 --- a/Objects/codeobject.c +++ b/Objects/codeobject.c @@ -2336,6 +2336,7 @@ _PyCode_ConstantKey(PyObject *op) if (op == Py_None || op == Py_Ellipsis || PyLong_CheckExact(op) || PyUnicode_CheckExact(op) + || PySlice_Check(op) /* code_richcompare() uses _PyCode_ConstantKey() internally */ || PyCode_Check(op)) { diff --git a/Objects/exceptions.c b/Objects/exceptions.c index b3910855165494..c685481b13a93a 100644 --- a/Objects/exceptions.c +++ b/Objects/exceptions.c @@ -2994,46 +2994,55 @@ UnicodeEncodeError_init(PyObject *self, PyObject *args, PyObject *kwds) static PyObject * UnicodeEncodeError_str(PyObject *self) { - PyUnicodeErrorObject *uself = (PyUnicodeErrorObject *)self; + PyUnicodeErrorObject *exc = (PyUnicodeErrorObject *)self; PyObject *result = NULL; PyObject *reason_str = NULL; PyObject *encoding_str = NULL; - if (!uself->object) + if (exc->object == NULL) { /* Not properly initialized. */ return PyUnicode_FromString(""); + } /* Get reason and encoding as strings, which they might not be if they've been modified after we were constructed. */ - reason_str = PyObject_Str(uself->reason); - if (reason_str == NULL) + reason_str = PyObject_Str(exc->reason); + if (reason_str == NULL) { goto done; - encoding_str = PyObject_Str(uself->encoding); - if (encoding_str == NULL) + } + encoding_str = PyObject_Str(exc->encoding); + if (encoding_str == NULL) { goto done; + } + + Py_ssize_t len = PyUnicode_GET_LENGTH(exc->object); + Py_ssize_t start = exc->start, end = exc->end; - if (uself->start < PyUnicode_GET_LENGTH(uself->object) && uself->end == uself->start+1) { - Py_UCS4 badchar = PyUnicode_ReadChar(uself->object, uself->start); + if ((start >= 0 && start < len) && (end >= 0 && end <= len) && end == start + 1) { + Py_UCS4 badchar = PyUnicode_ReadChar(exc->object, start); const char *fmt; - if (badchar <= 0xff) + if (badchar <= 0xff) { fmt = "'%U' codec can't encode character '\\x%02x' in position %zd: %U"; - else if (badchar <= 0xffff) + } + else if (badchar <= 0xffff) { fmt = "'%U' codec can't encode character '\\u%04x' in position %zd: %U"; - else + } + else { fmt = "'%U' codec can't encode character '\\U%08x' in position %zd: %U"; + } result = PyUnicode_FromFormat( fmt, encoding_str, (int)badchar, - uself->start, + start, reason_str); } else { result = PyUnicode_FromFormat( "'%U' codec can't encode characters in position %zd-%zd: %U", encoding_str, - uself->start, - uself->end-1, + start, + end - 1, reason_str); } done: @@ -3107,41 +3116,46 @@ UnicodeDecodeError_init(PyObject *self, PyObject *args, PyObject *kwds) static PyObject * UnicodeDecodeError_str(PyObject *self) { - PyUnicodeErrorObject *uself = (PyUnicodeErrorObject *)self; + PyUnicodeErrorObject *exc = (PyUnicodeErrorObject *)self; PyObject *result = NULL; PyObject *reason_str = NULL; PyObject *encoding_str = NULL; - if (!uself->object) + if (exc->object == NULL) { /* Not properly initialized. */ return PyUnicode_FromString(""); + } /* Get reason and encoding as strings, which they might not be if they've been modified after we were constructed. */ - reason_str = PyObject_Str(uself->reason); - if (reason_str == NULL) + reason_str = PyObject_Str(exc->reason); + if (reason_str == NULL) { goto done; - encoding_str = PyObject_Str(uself->encoding); - if (encoding_str == NULL) + } + encoding_str = PyObject_Str(exc->encoding); + if (encoding_str == NULL) { goto done; + } + + Py_ssize_t len = PyBytes_GET_SIZE(exc->object); + Py_ssize_t start = exc->start, end = exc->end; - if (uself->start < PyBytes_GET_SIZE(uself->object) && uself->end == uself->start+1) { - int byte = (int)(PyBytes_AS_STRING(((PyUnicodeErrorObject *)self)->object)[uself->start]&0xff); + if ((start >= 0 && start < len) && (end >= 0 && end <= len) && end == start + 1) { + int badbyte = (int)(PyBytes_AS_STRING(exc->object)[start] & 0xff); result = PyUnicode_FromFormat( "'%U' codec can't decode byte 0x%02x in position %zd: %U", encoding_str, - byte, - uself->start, + badbyte, + start, reason_str); } else { result = PyUnicode_FromFormat( "'%U' codec can't decode bytes in position %zd-%zd: %U", encoding_str, - uself->start, - uself->end-1, - reason_str - ); + start, + end - 1, + reason_str); } done: Py_XDECREF(reason_str); @@ -3204,42 +3218,49 @@ UnicodeTranslateError_init(PyUnicodeErrorObject *self, PyObject *args, static PyObject * UnicodeTranslateError_str(PyObject *self) { - PyUnicodeErrorObject *uself = (PyUnicodeErrorObject *)self; + PyUnicodeErrorObject *exc = (PyUnicodeErrorObject *)self; PyObject *result = NULL; PyObject *reason_str = NULL; - if (!uself->object) + if (exc->object == NULL) { /* Not properly initialized. */ return PyUnicode_FromString(""); + } /* Get reason as a string, which it might not be if it's been modified after we were constructed. */ - reason_str = PyObject_Str(uself->reason); - if (reason_str == NULL) + reason_str = PyObject_Str(exc->reason); + if (reason_str == NULL) { goto done; + } + + Py_ssize_t len = PyUnicode_GET_LENGTH(exc->object); + Py_ssize_t start = exc->start, end = exc->end; - if (uself->start < PyUnicode_GET_LENGTH(uself->object) && uself->end == uself->start+1) { - Py_UCS4 badchar = PyUnicode_ReadChar(uself->object, uself->start); + if ((start >= 0 && start < len) && (end >= 0 && end <= len) && end == start + 1) { + Py_UCS4 badchar = PyUnicode_ReadChar(exc->object, start); const char *fmt; - if (badchar <= 0xff) + if (badchar <= 0xff) { fmt = "can't translate character '\\x%02x' in position %zd: %U"; - else if (badchar <= 0xffff) + } + else if (badchar <= 0xffff) { fmt = "can't translate character '\\u%04x' in position %zd: %U"; - else + } + else { fmt = "can't translate character '\\U%08x' in position %zd: %U"; + } result = PyUnicode_FromFormat( fmt, (int)badchar, - uself->start, - reason_str - ); - } else { + start, + reason_str); + } + else { result = PyUnicode_FromFormat( "can't translate characters in position %zd-%zd: %U", - uself->start, - uself->end-1, - reason_str - ); + start, + end - 1, + reason_str); } done: Py_XDECREF(reason_str); diff --git a/Objects/funcobject.c b/Objects/funcobject.c index 98e6766d0ca0d8..855d1a2eeca819 100644 --- a/Objects/funcobject.c +++ b/Objects/funcobject.c @@ -128,7 +128,7 @@ _PyFunction_FromConstructor(PyFrameConstructor *constr) op->func_annotate = NULL; op->func_typeparams = NULL; op->vectorcall = _PyFunction_Vectorcall; - op->func_version = 0; + op->func_version = FUNC_VERSION_UNSET; // NOTE: functions created via FrameConstructor do not use deferred // reference counting because they are typically not part of cycles // nor accessed by multiple threads. @@ -207,7 +207,7 @@ PyFunction_NewWithQualName(PyObject *code, PyObject *globals, PyObject *qualname op->func_annotate = NULL; op->func_typeparams = NULL; op->vectorcall = _PyFunction_Vectorcall; - op->func_version = 0; + op->func_version = FUNC_VERSION_UNSET; if ((code_obj->co_flags & CO_NESTED) == 0) { // Use deferred reference counting for top-level functions, but not // nested functions because they are more likely to capture variables, @@ -287,31 +287,59 @@ functions is running. */ +static inline struct _func_version_cache_item * +get_cache_item(PyInterpreterState *interp, uint32_t version) +{ + return interp->func_state.func_version_cache + + (version % FUNC_VERSION_CACHE_SIZE); +} + void _PyFunction_SetVersion(PyFunctionObject *func, uint32_t version) { + assert(func->func_version == FUNC_VERSION_UNSET); + assert(version >= FUNC_VERSION_FIRST_VALID); + // This should only be called from MAKE_FUNCTION. No code is specialized + // based on the version, so we do not need to stop the world to set it. + func->func_version = version; #ifndef Py_GIL_DISABLED PyInterpreterState *interp = _PyInterpreterState_GET(); - if (func->func_version != 0) { - struct _func_version_cache_item *slot = - interp->func_state.func_version_cache - + (func->func_version % FUNC_VERSION_CACHE_SIZE); - if (slot->func == func) { - slot->func = NULL; - // Leave slot->code alone, there may be use for it. - } - } + struct _func_version_cache_item *slot = get_cache_item(interp, version); + slot->func = func; + slot->code = func->func_code; #endif - func->func_version = version; +} + +static void +func_clear_version(PyInterpreterState *interp, PyFunctionObject *func) +{ + if (func->func_version < FUNC_VERSION_FIRST_VALID) { + // Version was never set or has already been cleared. + return; + } #ifndef Py_GIL_DISABLED - if (version != 0) { - struct _func_version_cache_item *slot = - interp->func_state.func_version_cache - + (version % FUNC_VERSION_CACHE_SIZE); - slot->func = func; - slot->code = func->func_code; + struct _func_version_cache_item *slot = + get_cache_item(interp, func->func_version); + if (slot->func == func) { + slot->func = NULL; + // Leave slot->code alone, there may be use for it. } #endif + func->func_version = FUNC_VERSION_CLEARED; +} + +// Called when any of the critical function attributes are changed +static void +_PyFunction_ClearVersion(PyFunctionObject *func) +{ + if (func->func_version < FUNC_VERSION_FIRST_VALID) { + // Version was never set or has already been cleared. + return; + } + PyInterpreterState *interp = _PyInterpreterState_GET(); + _PyEval_StopTheWorld(interp); + func_clear_version(interp, func); + _PyEval_StartTheWorld(interp); } void @@ -319,9 +347,7 @@ _PyFunction_ClearCodeByVersion(uint32_t version) { #ifndef Py_GIL_DISABLED PyInterpreterState *interp = _PyInterpreterState_GET(); - struct _func_version_cache_item *slot = - interp->func_state.func_version_cache - + (version % FUNC_VERSION_CACHE_SIZE); + struct _func_version_cache_item *slot = get_cache_item(interp, version); if (slot->code) { assert(PyCode_Check(slot->code)); PyCodeObject *code = (PyCodeObject *)slot->code; @@ -340,9 +366,7 @@ _PyFunction_LookupByVersion(uint32_t version, PyObject **p_code) return NULL; #else PyInterpreterState *interp = _PyInterpreterState_GET(); - struct _func_version_cache_item *slot = - interp->func_state.func_version_cache - + (version % FUNC_VERSION_CACHE_SIZE); + struct _func_version_cache_item *slot = get_cache_item(interp, version); if (slot->code) { assert(PyCode_Check(slot->code)); PyCodeObject *code = (PyCodeObject *)slot->code; @@ -431,7 +455,7 @@ PyFunction_SetDefaults(PyObject *op, PyObject *defaults) } handle_func_event(PyFunction_EVENT_MODIFY_DEFAULTS, (PyFunctionObject *) op, defaults); - _PyFunction_SetVersion((PyFunctionObject *)op, 0); + _PyFunction_ClearVersion((PyFunctionObject *)op); Py_XSETREF(((PyFunctionObject *)op)->func_defaults, defaults); return 0; } @@ -440,7 +464,7 @@ void PyFunction_SetVectorcall(PyFunctionObject *func, vectorcallfunc vectorcall) { assert(func != NULL); - _PyFunction_SetVersion(func, 0); + _PyFunction_ClearVersion(func); func->vectorcall = vectorcall; } @@ -473,7 +497,7 @@ PyFunction_SetKwDefaults(PyObject *op, PyObject *defaults) } handle_func_event(PyFunction_EVENT_MODIFY_KWDEFAULTS, (PyFunctionObject *) op, defaults); - _PyFunction_SetVersion((PyFunctionObject *)op, 0); + _PyFunction_ClearVersion((PyFunctionObject *)op); Py_XSETREF(((PyFunctionObject *)op)->func_kwdefaults, defaults); return 0; } @@ -506,7 +530,7 @@ PyFunction_SetClosure(PyObject *op, PyObject *closure) Py_TYPE(closure)->tp_name); return -1; } - _PyFunction_SetVersion((PyFunctionObject *)op, 0); + _PyFunction_ClearVersion((PyFunctionObject *)op); Py_XSETREF(((PyFunctionObject *)op)->func_closure, closure); return 0; } @@ -658,7 +682,7 @@ func_set_code(PyObject *self, PyObject *value, void *Py_UNUSED(ignored)) } handle_func_event(PyFunction_EVENT_MODIFY_CODE, op, value); - _PyFunction_SetVersion(op, 0); + _PyFunction_ClearVersion(op); Py_XSETREF(op->func_code, Py_NewRef(value)); return 0; } @@ -744,7 +768,7 @@ func_set_defaults(PyObject *self, PyObject *value, void *Py_UNUSED(ignored)) } handle_func_event(PyFunction_EVENT_MODIFY_DEFAULTS, op, value); - _PyFunction_SetVersion(op, 0); + _PyFunction_ClearVersion(op); Py_XSETREF(op->func_defaults, Py_XNewRef(value)); return 0; } @@ -787,7 +811,7 @@ func_set_kwdefaults(PyObject *self, PyObject *value, void *Py_UNUSED(ignored)) } handle_func_event(PyFunction_EVENT_MODIFY_KWDEFAULTS, op, value); - _PyFunction_SetVersion(op, 0); + _PyFunction_ClearVersion(op); Py_XSETREF(op->func_kwdefaults, Py_XNewRef(value)); return 0; } @@ -1030,7 +1054,7 @@ static int func_clear(PyObject *self) { PyFunctionObject *op = _PyFunction_CAST(self); - _PyFunction_SetVersion(op, 0); + func_clear_version(_PyInterpreterState_GET(), op); Py_CLEAR(op->func_globals); Py_CLEAR(op->func_builtins); Py_CLEAR(op->func_module); @@ -1068,7 +1092,6 @@ func_dealloc(PyObject *self) if (op->func_weakreflist != NULL) { PyObject_ClearWeakRefs((PyObject *) op); } - _PyFunction_SetVersion(op, 0); (void)func_clear((PyObject*)op); // These aren't cleared by func_clear(). Py_DECREF(op->func_code); diff --git a/Objects/setobject.c b/Objects/setobject.c index 8bff4d99f81b81..9f40e085f06fa6 100644 --- a/Objects/setobject.c +++ b/Objects/setobject.c @@ -40,6 +40,8 @@ #include "pycore_pyatomic_ft_wrappers.h" // FT_ATOMIC_LOAD_SSIZE_RELAXED() #include "pycore_pyerrors.h" // _PyErr_SetKeyError() #include "pycore_setobject.h" // _PySet_NextEntry() definition + +#include "stringlib/eq.h" // unicode_eq() #include // offsetof() #include "clinic/setobject.c.h" @@ -96,7 +98,7 @@ set_lookkey(PySetObject *so, PyObject *key, Py_hash_t hash) return entry; if (PyUnicode_CheckExact(startkey) && PyUnicode_CheckExact(key) - && _PyUnicode_EQ(startkey, key)) + && unicode_eq(startkey, key)) return entry; table = so->table; Py_INCREF(startkey); @@ -157,7 +159,7 @@ set_add_entry(PySetObject *so, PyObject *key, Py_hash_t hash) goto found_active; if (PyUnicode_CheckExact(startkey) && PyUnicode_CheckExact(key) - && _PyUnicode_EQ(startkey, key)) + && unicode_eq(startkey, key)) goto found_active; table = so->table; Py_INCREF(startkey); diff --git a/Objects/sliceobject.c b/Objects/sliceobject.c index 1b6d35998c2b69..4fef0af93fe095 100644 --- a/Objects/sliceobject.c +++ b/Objects/sliceobject.c @@ -343,7 +343,7 @@ Create a slice object. This is used for extended slicing (e.g. a[0:10:2])."); static void slice_dealloc(PySliceObject *r) { - _PyObject_GC_UNTRACK(r); + PyObject_GC_UnTrack(r); Py_DECREF(r->step); Py_DECREF(r->start); Py_DECREF(r->stop); diff --git a/Objects/stringlib/eq.h b/Objects/stringlib/eq.h index 2eac4baf5ca9ce..821b692f26b830 100644 --- a/Objects/stringlib/eq.h +++ b/Objects/stringlib/eq.h @@ -4,14 +4,19 @@ * unicode_eq() is called when the hash of two unicode objects is equal. */ Py_LOCAL_INLINE(int) -unicode_eq(PyObject *a, PyObject *b) +unicode_eq(PyObject *str1, PyObject *str2) { - if (PyUnicode_GET_LENGTH(a) != PyUnicode_GET_LENGTH(b)) + Py_ssize_t len = PyUnicode_GET_LENGTH(str1); + if (PyUnicode_GET_LENGTH(str2) != len) { return 0; - if (PyUnicode_GET_LENGTH(a) == 0) - return 1; - if (PyUnicode_KIND(a) != PyUnicode_KIND(b)) + } + + int kind = PyUnicode_KIND(str1); + if (PyUnicode_KIND(str2) != kind) { return 0; - return memcmp(PyUnicode_1BYTE_DATA(a), PyUnicode_1BYTE_DATA(b), - PyUnicode_GET_LENGTH(a) * PyUnicode_KIND(a)) == 0; + } + + const void *data1 = PyUnicode_DATA(str1); + const void *data2 = PyUnicode_DATA(str2); + return (memcmp(data1, data2, len * kind) == 0); } diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c index e9589cfe44f3bf..bd5bb5048fdacc 100644 --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -261,7 +261,6 @@ _PyUnicode_InternedSize_Immortal(void) } static Py_hash_t unicode_hash(PyObject *); -static int unicode_compare_eq(PyObject *, PyObject *); static Py_uhash_t hashtable_unicode_hash(const void *key) @@ -275,7 +274,7 @@ hashtable_unicode_compare(const void *key1, const void *key2) PyObject *obj1 = (PyObject *)key1; PyObject *obj2 = (PyObject *)key2; if (obj1 != NULL && obj2 != NULL) { - return unicode_compare_eq(obj1, obj2); + return unicode_eq(obj1, obj2); } else { return obj1 == obj2; @@ -10968,26 +10967,6 @@ unicode_compare(PyObject *str1, PyObject *str2) #undef COMPARE } -static int -unicode_compare_eq(PyObject *str1, PyObject *str2) -{ - int kind; - const void *data1, *data2; - Py_ssize_t len; - int cmp; - - len = PyUnicode_GET_LENGTH(str1); - if (PyUnicode_GET_LENGTH(str2) != len) - return 0; - kind = PyUnicode_KIND(str1); - if (PyUnicode_KIND(str2) != kind) - return 0; - data1 = PyUnicode_DATA(str1); - data2 = PyUnicode_DATA(str2); - - cmp = memcmp(data1, data2, len * kind); - return (cmp == 0); -} int _PyUnicode_Equal(PyObject *str1, PyObject *str2) @@ -10997,7 +10976,25 @@ _PyUnicode_Equal(PyObject *str1, PyObject *str2) if (str1 == str2) { return 1; } - return unicode_compare_eq(str1, str2); + return unicode_eq(str1, str2); +} + + +int +PyUnicode_Equal(PyObject *str1, PyObject *str2) +{ + if (!PyUnicode_Check(str1)) { + PyErr_Format(PyExc_TypeError, + "first argument must be str, not %T", str1); + return -1; + } + if (!PyUnicode_Check(str2)) { + PyErr_Format(PyExc_TypeError, + "second argument must be str, not %T", str2); + return -1; + } + + return _PyUnicode_Equal(str1, str2); } @@ -11195,7 +11192,7 @@ _PyUnicode_EqualToASCIIId(PyObject *left, _Py_Identifier *right) return 0; } - return unicode_compare_eq(left, right_uni); + return unicode_eq(left, right_uni); } PyObject * @@ -11223,7 +11220,7 @@ PyUnicode_RichCompare(PyObject *left, PyObject *right, int op) } } else if (op == Py_EQ || op == Py_NE) { - result = unicode_compare_eq(left, right); + result = unicode_eq(left, right); result ^= (op == Py_NE); return PyBool_FromLong(result); } @@ -11233,12 +11230,6 @@ PyUnicode_RichCompare(PyObject *left, PyObject *right, int op) } } -int -_PyUnicode_EQ(PyObject *aa, PyObject *bb) -{ - return unicode_eq(aa, bb); -} - int PyUnicode_Contains(PyObject *str, PyObject *substr) { diff --git a/PC/python3dll.c b/PC/python3dll.c index 6b8208ab90bd95..9296474617e115 100755 --- a/PC/python3dll.c +++ b/PC/python3dll.c @@ -717,6 +717,7 @@ EXPORT_FUNC(PyUnicode_DecodeUTF8Stateful) EXPORT_FUNC(PyUnicode_EncodeCodePage) EXPORT_FUNC(PyUnicode_EncodeFSDefault) EXPORT_FUNC(PyUnicode_EncodeLocale) +EXPORT_FUNC(PyUnicode_Equal) EXPORT_FUNC(PyUnicode_EqualToUTF8) EXPORT_FUNC(PyUnicode_EqualToUTF8AndSize) EXPORT_FUNC(PyUnicode_Find) diff --git a/Programs/_testembed.c b/Programs/_testembed.c index ab2b2d06cca15d..ab619e32429d63 100644 --- a/Programs/_testembed.c +++ b/Programs/_testembed.c @@ -311,14 +311,36 @@ static int test_pre_initialization_api(void) _Py_EMBED_PREINIT_CHECK("Checking Py_SetProgramName\n"); Py_SetProgramName(program); + _Py_EMBED_PREINIT_CHECK("Checking !Py_IsInitialized pre-initialization\n"); + if (Py_IsInitialized()) { + fprintf(stderr, "Fatal error: initialized before initialization!\n"); + return 1; + } + _Py_EMBED_PREINIT_CHECK("Initializing interpreter\n"); Py_Initialize(); + + _Py_EMBED_PREINIT_CHECK("Checking Py_IsInitialized post-initialization\n"); + if (!Py_IsInitialized()) { + fprintf(stderr, "Fatal error: not initialized after initialization!\n"); + return 1; + } + _Py_EMBED_PREINIT_CHECK("Check sys module contents\n"); - PyRun_SimpleString("import sys; " - "print('sys.executable:', sys.executable)"); + PyRun_SimpleString( + "import sys; " + "print('sys.executable:', sys.executable); " + "sys.stdout.flush(); " + ); _Py_EMBED_PREINIT_CHECK("Finalizing interpreter\n"); Py_Finalize(); + _Py_EMBED_PREINIT_CHECK("Checking !Py_IsInitialized post-finalization\n"); + if (Py_IsInitialized()) { + fprintf(stderr, "Fatal error: still initialized after finalization!\n"); + return 1; + } + _Py_EMBED_PREINIT_CHECK("Freeing memory allocated by Py_DecodeLocale\n"); PyMem_RawFree(program); return 0; @@ -364,12 +386,15 @@ static int test_pre_initialization_sys_options(void) _Py_EMBED_PREINIT_CHECK("Initializing interpreter\n"); _testembed_Py_InitializeFromConfig(); _Py_EMBED_PREINIT_CHECK("Check sys module contents\n"); - PyRun_SimpleString("import sys; " - "print('sys.warnoptions:', sys.warnoptions); " - "print('sys._xoptions:', sys._xoptions); " - "warnings = sys.modules['warnings']; " - "latest_filters = [f[0] for f in warnings.filters[:3]]; " - "print('warnings.filters[:3]:', latest_filters)"); + PyRun_SimpleString( + "import sys; " + "print('sys.warnoptions:', sys.warnoptions); " + "print('sys._xoptions:', sys._xoptions); " + "warnings = sys.modules['warnings']; " + "latest_filters = [f[0] for f in warnings.filters[:3]]; " + "print('warnings.filters[:3]:', latest_filters); " + "sys.stdout.flush(); " + ); _Py_EMBED_PREINIT_CHECK("Finalizing interpreter\n"); Py_Finalize(); diff --git a/Python/bytecodes.c b/Python/bytecodes.c index 228d82173e6126..92cb031f83e7e5 100644 --- a/Python/bytecodes.c +++ b/Python/bytecodes.c @@ -2806,7 +2806,16 @@ dummy_func( replaced op(_FOR_ITER, (iter -- iter, next)) { /* before: [iter]; after: [iter, iter()] *or* [] (and jump over END_FOR.) */ PyObject *iter_o = PyStackRef_AsPyObjectBorrow(iter); - PyObject *next_o = (*Py_TYPE(iter_o)->tp_iternext)(iter_o); + PyTypeObject *type = Py_TYPE(iter_o); + iternextfunc iternext = type->tp_iternext; + if (iternext == NULL) { + _PyErr_Format(tstate, PyExc_TypeError, + "'for' requires an object with " + "__iter__ method, got %.100s", + type->tp_name); + ERROR_NO_POP(); + } + PyObject *next_o = (*iternext)(iter_o); if (next_o == NULL) { if (_PyErr_Occurred(tstate)) { int matches = _PyErr_ExceptionMatches(tstate, PyExc_StopIteration); @@ -2832,7 +2841,16 @@ dummy_func( op(_FOR_ITER_TIER_TWO, (iter -- iter, next)) { /* before: [iter]; after: [iter, iter()] *or* [] (and jump over END_FOR.) */ PyObject *iter_o = PyStackRef_AsPyObjectBorrow(iter); - PyObject *next_o = (*Py_TYPE(iter_o)->tp_iternext)(iter_o); + PyTypeObject *type = Py_TYPE(iter_o); + iternextfunc iternext = type->tp_iternext; + if (iternext == NULL) { + _PyErr_Format(tstate, PyExc_TypeError, + "'for' requires an object with " + "__iter__ method, got %.100s", + type->tp_name); + ERROR_NO_POP(); + } + PyObject *next_o = (*iternext)(iter_o); if (next_o == NULL) { if (_PyErr_Occurred(tstate)) { int matches = _PyErr_ExceptionMatches(tstate, PyExc_StopIteration); @@ -2856,7 +2874,16 @@ dummy_func( _Py_CODEUNIT *target; _PyStackRef iter_stackref = TOP(); PyObject *iter = PyStackRef_AsPyObjectBorrow(iter_stackref); - PyObject *next = (*Py_TYPE(iter)->tp_iternext)(iter); + PyTypeObject *type = Py_TYPE(iter); + iternextfunc iternext = type->tp_iternext; + if (iternext == NULL) { + _PyErr_Format(tstate, PyExc_TypeError, + "'for' requires an object with " + "__iter__ method, got %.100s", + type->tp_name); + ERROR_NO_POP(); + } + PyObject *next = (*iternext)(iter); if (next != NULL) { PUSH(PyStackRef_FromPyObjectSteal(next)); target = next_instr; diff --git a/Python/codegen.c b/Python/codegen.c index 896c30cc14952a..689d2b5124e9d3 100644 --- a/Python/codegen.c +++ b/Python/codegen.c @@ -194,6 +194,7 @@ static int codegen_visit_expr(compiler *, expr_ty); static int codegen_augassign(compiler *, stmt_ty); static int codegen_annassign(compiler *, stmt_ty); static int codegen_subscript(compiler *, expr_ty); +static int codegen_slice_two_parts(compiler *, expr_ty); static int codegen_slice(compiler *, expr_ty); static bool are_all_items_const(asdl_expr_seq *, Py_ssize_t, Py_ssize_t); @@ -5005,12 +5006,8 @@ codegen_visit_expr(compiler *c, expr_ty e) } break; case Slice_kind: - { - int n = codegen_slice(c, e); - RETURN_IF_ERROR(n); - ADDOP_I(c, loc, BUILD_SLICE, n); + RETURN_IF_ERROR(codegen_slice(c, e)); break; - } case Name_kind: return codegen_nameop(c, loc, e->v.Name.id, e->v.Name.ctx); /* child nodes of List and Tuple will have expr_context set */ @@ -5023,9 +5020,22 @@ codegen_visit_expr(compiler *c, expr_ty e) } static bool -is_two_element_slice(expr_ty s) +is_constant_slice(expr_ty s) { return s->kind == Slice_kind && + (s->v.Slice.lower == NULL || + s->v.Slice.lower->kind == Constant_kind) && + (s->v.Slice.upper == NULL || + s->v.Slice.upper->kind == Constant_kind) && + (s->v.Slice.step == NULL || + s->v.Slice.step->kind == Constant_kind); +} + +static bool +should_apply_two_element_slice_optimization(expr_ty s) +{ + return !is_constant_slice(s) && + s->kind == Slice_kind && s->v.Slice.step == NULL; } @@ -5046,8 +5056,8 @@ codegen_augassign(compiler *c, stmt_ty s) break; case Subscript_kind: VISIT(c, expr, e->v.Subscript.value); - if (is_two_element_slice(e->v.Subscript.slice)) { - RETURN_IF_ERROR(codegen_slice(c, e->v.Subscript.slice)); + if (should_apply_two_element_slice_optimization(e->v.Subscript.slice)) { + RETURN_IF_ERROR(codegen_slice_two_parts(c, e->v.Subscript.slice)); ADDOP_I(c, loc, COPY, 3); ADDOP_I(c, loc, COPY, 3); ADDOP_I(c, loc, COPY, 3); @@ -5084,7 +5094,7 @@ codegen_augassign(compiler *c, stmt_ty s) ADDOP_NAME(c, loc, STORE_ATTR, e->v.Attribute.attr, names); break; case Subscript_kind: - if (is_two_element_slice(e->v.Subscript.slice)) { + if (should_apply_two_element_slice_optimization(e->v.Subscript.slice)) { ADDOP_I(c, loc, SWAP, 4); ADDOP_I(c, loc, SWAP, 3); ADDOP_I(c, loc, SWAP, 2); @@ -5231,8 +5241,10 @@ codegen_subscript(compiler *c, expr_ty e) } VISIT(c, expr, e->v.Subscript.value); - if (is_two_element_slice(e->v.Subscript.slice) && ctx != Del) { - RETURN_IF_ERROR(codegen_slice(c, e->v.Subscript.slice)); + if (should_apply_two_element_slice_optimization(e->v.Subscript.slice) && + ctx != Del + ) { + RETURN_IF_ERROR(codegen_slice_two_parts(c, e->v.Subscript.slice)); if (ctx == Load) { ADDOP(c, loc, BINARY_SLICE); } @@ -5254,15 +5266,9 @@ codegen_subscript(compiler *c, expr_ty e) return SUCCESS; } -/* Returns the number of the values emitted, - * thus are needed to build the slice, or -1 if there is an error. */ static int -codegen_slice(compiler *c, expr_ty s) +codegen_slice_two_parts(compiler *c, expr_ty s) { - int n = 2; - assert(s->kind == Slice_kind); - - /* only handles the cases where BUILD_SLICE is emitted */ if (s->v.Slice.lower) { VISIT(c, expr, s->v.Slice.lower); } @@ -5277,11 +5283,45 @@ codegen_slice(compiler *c, expr_ty s) ADDOP_LOAD_CONST(c, LOC(s), Py_None); } + return 0; +} + +static int +codegen_slice(compiler *c, expr_ty s) +{ + int n = 2; + assert(s->kind == Slice_kind); + + if (is_constant_slice(s)) { + PyObject *start = NULL; + if (s->v.Slice.lower) { + start = s->v.Slice.lower->v.Constant.value; + } + PyObject *stop = NULL; + if (s->v.Slice.upper) { + stop = s->v.Slice.upper->v.Constant.value; + } + PyObject *step = NULL; + if (s->v.Slice.step) { + step = s->v.Slice.step->v.Constant.value; + } + PyObject *slice = PySlice_New(start, stop, step); + if (slice == NULL) { + return ERROR; + } + ADDOP_LOAD_CONST_NEW(c, LOC(s), slice); + return SUCCESS; + } + + RETURN_IF_ERROR(codegen_slice_two_parts(c, s)); + if (s->v.Slice.step) { n++; VISIT(c, expr, s->v.Slice.step); } - return n; + + ADDOP_I(c, LOC(s), BUILD_SLICE, n); + return SUCCESS; } diff --git a/Python/compile.c b/Python/compile.c index 9826d3fbbde976..d463fcde204a05 100644 --- a/Python/compile.c +++ b/Python/compile.c @@ -1534,7 +1534,7 @@ _PyCompile_CodeGen(PyObject *ast, PyObject *filename, PyCompilerFlags *pflags, _PyCompile_CodeUnitMetadata *umd = &c->u->u_metadata; -#define SET_MATADATA_INT(key, value) do { \ +#define SET_METADATA_INT(key, value) do { \ PyObject *v = PyLong_FromLong((long)value); \ if (v == NULL) goto finally; \ int res = PyDict_SetItemString(metadata, key, v); \ @@ -1542,10 +1542,10 @@ _PyCompile_CodeGen(PyObject *ast, PyObject *filename, PyCompilerFlags *pflags, if (res < 0) goto finally; \ } while (0); - SET_MATADATA_INT("argcount", umd->u_argcount); - SET_MATADATA_INT("posonlyargcount", umd->u_posonlyargcount); - SET_MATADATA_INT("kwonlyargcount", umd->u_kwonlyargcount); -#undef SET_MATADATA_INT + SET_METADATA_INT("argcount", umd->u_argcount); + SET_METADATA_INT("posonlyargcount", umd->u_posonlyargcount); + SET_METADATA_INT("kwonlyargcount", umd->u_kwonlyargcount); +#undef SET_METADATA_INT int addNone = mod->kind != Expression_kind; if (_PyCodegen_AddReturnAtEnd(c, addNone) < 0) { diff --git a/Python/executor_cases.c.h b/Python/executor_cases.c.h index 4574e183921006..f260e59e7f6a63 100644 --- a/Python/executor_cases.c.h +++ b/Python/executor_cases.c.h @@ -3337,8 +3337,19 @@ iter = stack_pointer[-1]; /* before: [iter]; after: [iter, iter()] *or* [] (and jump over END_FOR.) */ PyObject *iter_o = PyStackRef_AsPyObjectBorrow(iter); + PyTypeObject *type = Py_TYPE(iter_o); + iternextfunc iternext = type->tp_iternext; + if (iternext == NULL) { + _PyFrame_SetStackPointer(frame, stack_pointer); + _PyErr_Format(tstate, PyExc_TypeError, + "'for' requires an object with " + "__iter__ method, got %.100s", + type->tp_name); + stack_pointer = _PyFrame_GetStackPointer(frame); + JUMP_TO_ERROR(); + } _PyFrame_SetStackPointer(frame, stack_pointer); - PyObject *next_o = (*Py_TYPE(iter_o)->tp_iternext)(iter_o); + PyObject *next_o = (*iternext)(iter_o); stack_pointer = _PyFrame_GetStackPointer(frame); if (next_o == NULL) { if (_PyErr_Occurred(tstate)) { diff --git a/Python/generated_cases.c.h b/Python/generated_cases.c.h index e1107caa197d7a..830e4f69c71c92 100644 --- a/Python/generated_cases.c.h +++ b/Python/generated_cases.c.h @@ -3921,8 +3921,19 @@ { /* before: [iter]; after: [iter, iter()] *or* [] (and jump over END_FOR.) */ PyObject *iter_o = PyStackRef_AsPyObjectBorrow(iter); + PyTypeObject *type = Py_TYPE(iter_o); + iternextfunc iternext = type->tp_iternext; + if (iternext == NULL) { + _PyFrame_SetStackPointer(frame, stack_pointer); + _PyErr_Format(tstate, PyExc_TypeError, + "'for' requires an object with " + "__iter__ method, got %.100s", + type->tp_name); + stack_pointer = _PyFrame_GetStackPointer(frame); + goto error; + } _PyFrame_SetStackPointer(frame, stack_pointer); - PyObject *next_o = (*Py_TYPE(iter_o)->tp_iternext)(iter_o); + PyObject *next_o = (*iternext)(iter_o); stack_pointer = _PyFrame_GetStackPointer(frame); if (next_o == NULL) { if (_PyErr_Occurred(tstate)) { @@ -4614,8 +4625,19 @@ _Py_CODEUNIT *target; _PyStackRef iter_stackref = TOP(); PyObject *iter = PyStackRef_AsPyObjectBorrow(iter_stackref); + PyTypeObject *type = Py_TYPE(iter); + iternextfunc iternext = type->tp_iternext; + if (iternext == NULL) { + _PyFrame_SetStackPointer(frame, stack_pointer); + _PyErr_Format(tstate, PyExc_TypeError, + "'for' requires an object with " + "__iter__ method, got %.100s", + type->tp_name); + stack_pointer = _PyFrame_GetStackPointer(frame); + goto error; + } _PyFrame_SetStackPointer(frame, stack_pointer); - PyObject *next = (*Py_TYPE(iter)->tp_iternext)(iter); + PyObject *next = (*iternext)(iter); stack_pointer = _PyFrame_GetStackPointer(frame); if (next != NULL) { PUSH(PyStackRef_FromPyObjectSteal(next)); diff --git a/Python/getargs.c b/Python/getargs.c index 1b9b39939ae94e..a764343ea9ee3f 100644 --- a/Python/getargs.c +++ b/Python/getargs.c @@ -2064,7 +2064,7 @@ find_keyword(PyObject *kwnames, PyObject *const *kwstack, PyObject *key) for (i = 0; i < nkwargs; i++) { PyObject *kwname = PyTuple_GET_ITEM(kwnames, i); assert(PyUnicode_Check(kwname)); - if (_PyUnicode_EQ(kwname, key)) { + if (_PyUnicode_Equal(kwname, key)) { return Py_NewRef(kwstack[i]); } } diff --git a/Python/marshal.c b/Python/marshal.c index b1708a7306f9e7..3d127b4e331d0d 100644 --- a/Python/marshal.c +++ b/Python/marshal.c @@ -76,6 +76,7 @@ module marshal #define TYPE_UNKNOWN '?' #define TYPE_SET '<' #define TYPE_FROZENSET '>' +#define TYPE_SLICE ':' #define FLAG_REF '\x80' /* with a type, add obj to index */ #define TYPE_ASCII 'a' @@ -613,6 +614,13 @@ w_complex_object(PyObject *v, char flag, WFILE *p) w_pstring(view.buf, view.len, p); PyBuffer_Release(&view); } + else if (PySlice_Check(v)) { + PySliceObject *slice = (PySliceObject *)v; + W_TYPE(TYPE_SLICE, p); + w_object(slice->start, p); + w_object(slice->stop, p); + w_object(slice->step, p); + } else { W_TYPE(TYPE_UNKNOWN, p); p->error = WFERR_UNMARSHALLABLE; @@ -1534,6 +1542,32 @@ r_object(RFILE *p) retval = Py_NewRef(v); break; + case TYPE_SLICE: + { + Py_ssize_t idx = r_ref_reserve(flag, p); + PyObject *stop = NULL; + PyObject *step = NULL; + PyObject *start = r_object(p); + if (start == NULL) { + goto cleanup; + } + stop = r_object(p); + if (stop == NULL) { + goto cleanup; + } + step = r_object(p); + if (step == NULL) { + goto cleanup; + } + retval = PySlice_New(start, stop, step); + r_ref_insert(retval, idx, flag, p); + cleanup: + Py_XDECREF(start); + Py_XDECREF(stop); + Py_XDECREF(step); + break; + } + default: /* Bogus data got written, which isn't ideal. This will let you keep working and recover. */ diff --git a/Python/specialize.c b/Python/specialize.c index d8bff39511cf12..4b33a468733d6b 100644 --- a/Python/specialize.c +++ b/Python/specialize.c @@ -1599,7 +1599,7 @@ function_get_version(PyObject *o, int opcode) assert(Py_IS_TYPE(o, &PyFunction_Type)); PyFunctionObject *func = (PyFunctionObject *)o; uint32_t version = _PyFunction_GetVersionForCurrentState(func); - if (version == 0) { + if (!_PyFunction_IsVersionValid(version)) { SPECIALIZATION_FAIL(opcode, SPEC_FAIL_OUT_OF_VERSIONS); return 0; } @@ -1692,7 +1692,7 @@ _Py_Specialize_BinarySubscr( goto fail; } uint32_t version = _PyFunction_GetVersionForCurrentState(func); - if (version == 0) { + if (!_PyFunction_IsVersionValid(version)) { SPECIALIZATION_FAIL(BINARY_SUBSCR, SPEC_FAIL_OUT_OF_VERSIONS); goto fail; } @@ -1977,7 +1977,7 @@ specialize_py_call(PyFunctionObject *func, _Py_CODEUNIT *instr, int nargs, argcount = code->co_argcount; } int version = _PyFunction_GetVersionForCurrentState(func); - if (version == 0) { + if (!_PyFunction_IsVersionValid(version)) { SPECIALIZATION_FAIL(CALL, SPEC_FAIL_OUT_OF_VERSIONS); return -1; } @@ -2009,7 +2009,7 @@ specialize_py_call_kw(PyFunctionObject *func, _Py_CODEUNIT *instr, int nargs, return -1; } int version = _PyFunction_GetVersionForCurrentState(func); - if (version == 0) { + if (!_PyFunction_IsVersionValid(version)) { SPECIALIZATION_FAIL(CALL, SPEC_FAIL_OUT_OF_VERSIONS); return -1; } diff --git a/Tools/build/regen-configure.sh b/Tools/build/regen-configure.sh index e34a36c1a573e5..efc80c8527885c 100755 --- a/Tools/build/regen-configure.sh +++ b/Tools/build/regen-configure.sh @@ -5,12 +5,10 @@ set -e -x # The check_generated_files job of .github/workflows/build.yml must kept in # sync with this script. Use the same container image than the job so the job # doesn't need to run autoreconf in a container. -IMAGE="ubuntu:22.04" -DEPENDENCIES="autotools-dev autoconf autoconf-archive pkg-config" +IMAGE="ghcr.io/python/autoconf:2024.10.06.11200919239" AUTORECONF="autoreconf -ivf -Werror" WORK_DIR="/src" -SHELL_CMD="apt-get update && apt-get -yq install $DEPENDENCIES && cd $WORK_DIR && $AUTORECONF" abs_srcdir=$(cd $(dirname $0)/../..; pwd) @@ -28,4 +26,4 @@ if command -v selinuxenabled >/dev/null && selinuxenabled; then PATH_OPT=":Z" fi -"$RUNTIME" run --rm -v "$abs_srcdir:$WORK_DIR$PATH_OPT" "$IMAGE" /usr/bin/bash -c "$SHELL_CMD" +"$RUNTIME" run --rm -v "$abs_srcdir:$WORK_DIR$PATH_OPT" "$IMAGE" diff --git a/Tools/c-analyzer/cpython/globals-to-fix.tsv b/Tools/c-analyzer/cpython/globals-to-fix.tsv index a0be2a0a203f8c..badd7b79102310 100644 --- a/Tools/c-analyzer/cpython/globals-to-fix.tsv +++ b/Tools/c-analyzer/cpython/globals-to-fix.tsv @@ -358,7 +358,6 @@ Modules/_testclinic.c - TestClass - ##----------------------- ## static types -Modules/_cursesmodule.c - PyCursesWindow_Type - Modules/_datetimemodule.c - PyDateTime_DateTimeType - Modules/_datetimemodule.c - PyDateTime_DateType - Modules/_datetimemodule.c - PyDateTime_DeltaType - @@ -383,7 +382,6 @@ Modules/_tkinter.c - Tktt_Type - Modules/xxlimited_35.c - Xxo_Type - ## exception types -Modules/_cursesmodule.c - PyCursesError - Modules/_tkinter.c - Tkinter_TclError - Modules/xxlimited_35.c - ErrorObject - Modules/xxmodule.c - ErrorObject - @@ -409,6 +407,7 @@ Modules/_tkinter.c - trbInCmd - Include/datetime.h - PyDateTimeAPI - Modules/_ctypes/cfield.c _ctypes_get_fielddesc initialized - Modules/_ctypes/malloc_closure.c - _pagesize - +Modules/_cursesmodule.c - curses_module_loaded - Modules/_cursesmodule.c - curses_initscr_called - Modules/_cursesmodule.c - curses_setupterm_called - Modules/_cursesmodule.c - curses_start_color_called - @@ -423,7 +422,6 @@ Modules/readline.c - libedit_history_start - Modules/_ctypes/cfield.c - formattable - Modules/_ctypes/malloc_closure.c - free_list - -Modules/_cursesmodule.c - curses_global_state - Modules/_curses_panel.c - lop - Modules/_ssl/debughelpers.c _PySSL_keylog_callback lock - Modules/_tkinter.c - quitMainLoop - diff --git a/Tools/cases_generator/generators_common.py b/Tools/cases_generator/generators_common.py index f32a20b304c354..0bfa1a3b56fbc2 100644 --- a/Tools/cases_generator/generators_common.py +++ b/Tools/cases_generator/generators_common.py @@ -43,13 +43,13 @@ def peek(self) -> Token | None: break return self.look_ahead -ROOT = Path(__file__).parent.parent.parent -DEFAULT_INPUT = (ROOT / "Python/bytecodes.c").absolute().as_posix() +ROOT = Path(__file__).parent.parent.parent.resolve() +DEFAULT_INPUT = (ROOT / "Python/bytecodes.c").as_posix() def root_relative_path(filename: str) -> str: try: - return Path(filename).absolute().relative_to(ROOT).as_posix() + return Path(filename).resolve().relative_to(ROOT).as_posix() except ValueError: # Not relative to root, just return original path. return filename