Merge pull request #144 from lmfit/with_lambda

add support for lambda expressions

Author: newville

README.rst

@@ -50,42 +50,62 @@ About Asteval
 --------------
 
 Asteval is a safe(ish) evaluator of Python expressions and statements,
-using Python's ast module. It provides a simple and robust restricted
-Python interpreter that can safely handle user input.  The emphasis
-here is on mathematical expressions so that many functions from
-``NumPy`` are imported and used if available.
-
-Asteval supports many Python language constructs by default. These
-include conditionals (if-elif-else blocks and if expressions), flow
-control (for loops, while loops, and try-except-finally blocks), list
-comprehension, slicing, subscripting, f-strings, and more.  All data
-are Python objects and built-in data structures (dictionaries, tuples,
-lists, strings, and ``Numpy`` nd-arrays) are fully supported by
-default.  It supports these language features by converting input into
-Python's own abstract syntax tree (AST) representation and walking
-through that tree.  This approach effectively guarantees that parsing
-of input will be identical to that of Python.
+using Python's ast module.  It emphasizes mathematical expressions so
+that many functions from ``NumPy`` are imported and used if available,
+but also provides a pretty complete subset of the Python language.
+Asteval provides a simple and robust restricted Python interpreter
+that can safely handle user input, and can be used as an embedded
+macro language within a large application.
+
+Asteval supports many Python language constructs by default, including
+conditionals (if-elif-else blocks and if expressions), flow control
+(for loops, while loops, with blocks, and try-except-finally blocks),
+list comprehension, slicing, subscripting, and f-strings.  All data
+are Python objects and the standard built-in data structures
+(dictionaries, tuples, lists, sets, strings, functions, and ``Numpy``
+nd-arrays) are well supported, but with limited to look "under the
+hood" and get private and unsafe methods.
 
 Many of the standard built-in Python functions are available, as are
-all mathematical functions from the ``math`` module.  If the ``NumPy``
-is installed, many of its functions will also be available.  Users can
-define and run their own functions within the confines of the
-limitations of Asteval.
+the functions from the ``math`` module.  Some of the built-in
+operators and functions, such as `getattr`, and `setattr` are not
+allowed, and some including `open` and `**` are replaced with versions
+intended to make them safer for user input.  If the ``NumPy`` is
+installed, many of its functions will also be available.  Programmers
+can add custom functions and data into each Asteval session.  Users
+can define and run their own functions within the confines of the
+limitations of the Asteval language.
+
+Asteval converts user input into Python's own abstract syntax tree
+(AST) representation and determines the result by walking through that
+tree.  This approach guarantees the parsing of input will be identical
+to that of Python, eliminating many lexing and parsing challenges and
+generating a result that is straightforward to interpret.  This makes
+"correctness" easy to test and verify with high confidence, so that
+the emphasis can be placed on balancing features with safety.
 
 There are several absences and differences with Python, and Asteval is
 by no means an attempt to reproduce Python with its own ``ast``
-module.  Some of the most important differences and absences are:
-
- 1. accessing many internal methods and classes of Python objects is
-    forbidden. This strengthens Asteval against malicious user code.
- 2. creating classes is not supported.
- 3. function decorators, `yield`, `async`, `lambda`, `exec`, and
-    `eval` are not supported.
- 4. importing modules is not supported by default (it can be enabled).
- 5. files will be opened in read-only mode by default.
-
-Even with these restrictions, Asteval provides a pretty full-features
-``mini-Python`` language that might be useful to expose to user input.
+module.  While, it does support a large subset of Python, the
+following features found in Python are not supported in Asteval:
+
+ 1. many internal methods and classes of Python objects,
+    especially ``__dunder__`` methods cannot be accessed.
+ 2. creating classes is not supported
+ 3. `eval`, `exec`, `yield`, `async`, `match/case`, function
+    decorators, generators, and type annotations are not supported.
+ 4. `f-strings` are supported, but `t-strings` are not supported.
+ 5. importing modules is not supported by default, though it can be
+    enabled.
+
+Most of these omissions and limitations are intentional, and aimed to
+strengthen Asteval against dangerous user code. Some of these
+omissions may simply be viewed as not particularly compelling for an
+embedded interpreter exposed to user input.
+
+Even with these
+restrictions,
+
 
 
 Matt Newville <[email protected]>

asteval/asteval.py

@@ -54,9 +54,9 @@
              'constant', 'continue', 'delete', 'dict', 'dictcomp',
              'excepthandler', 'expr', 'extslice', 'for',
              'functiondef', 'if', 'ifexp', 'import', 'importfrom',
-             'index', 'interrupt', 'list', 'listcomp', 'module',
-             'name', 'pass', 'raise', 'repr', 'return', 'set',
-             'setcomp', 'slice', 'subscript', 'try', 'tuple',
+             'index', 'interrupt', 'lambda', 'list', 'listcomp',
+             'module', 'name', 'pass', 'raise', 'repr', 'return',
+             'set', 'setcomp', 'slice', 'subscript', 'try', 'tuple',
              'unaryop', 'while', 'with', 'formattedvalue',
              'joinedstr']
 
@@ -65,8 +65,9 @@
 DEFAULT_CONFIG = {'import': False, 'importfrom': False}
 
 for _tnode in ('assert', 'augassign', 'delete', 'if', 'ifexp', 'for',
-             'formattedvalue', 'functiondef', 'print', 'raise', 'listcomp',
-             'dictcomp', 'setcomp', 'try', 'while', 'with'):
+             'formattedvalue', 'functiondef', 'print', 'raise',
+             'lambda', 'listcomp', 'dictcomp', 'setcomp', 'try',
+             'while', 'with'):
     MINIMAL_CONFIG[_tnode] = False
     DEFAULT_CONFIG[_tnode] = True
 
@@ -103,7 +104,7 @@ class Interpreter:
     -----
     1. setting `minimal=True` is equivalent to setting a config with the following
        nodes disabled: ('import', 'importfrom', 'if', 'for', 'while', 'try', 'with',
-       'functiondef', 'ifexp', 'listcomp', 'dictcomp', 'setcomp', 'augassign',
+       'functiondef', 'ifexp', 'lambda', 'listcomp', 'dictcomp', 'setcomp', 'augassign',
        'assert', 'delete', 'raise', 'print')
     2. by default 'import' and 'importfrom' are disabled, though they can be enabled.
     """
@@ -951,18 +952,24 @@ def on_arg(self, node):    # ('test', 'msg')
         """Arg for function definitions."""
         return node.arg
 
-    def on_functiondef(self, node):
+    def on_functiondef(self, node, is_lambda=False):
         """Define procedures."""
         # ('name', 'args', 'body', 'decorator_list')
-        if node.decorator_list:
-            raise Warning("decorated procedures not supported!")
-        kwargs = []
-
-        if (not valid_symbol_name(node.name) or
-                node.name in self.readonly_symbols):
-            errmsg = f"invalid function name (reserved word?) {node.name}"
-            self.raise_exception(node, exc=NameError, msg=errmsg)
+        if is_lambda:
+            name = 'lambda'
+            body = [node.body]
+        else:
+            name = node.name
+            body = node.body
+
+            if node.decorator_list:
+                raise Warning("decorated procedures not supported!")
+            if (not valid_symbol_name(name) or
+                    name in self.readonly_symbols):
+                errmsg = f"invalid function name (reserved word?) {name}"
+                self.raise_exception(node, exc=NameError, msg=errmsg)
 
+        kwargs = []
         offset = len(node.args.args) - len(node.args.defaults)
         for idef, defnode in enumerate(node.args.defaults):
             defval = self.run(defnode)
@@ -971,7 +978,7 @@ def on_functiondef(self, node):
 
         args = [tnode.arg for tnode in node.args.args[:offset]]
         doc = None
-        nb0 = node.body[0]
+        nb0 = body[0]
         if isinstance(nb0, ast.Expr) and isinstance(nb0.value, ast.Constant):
             doc = nb0.value
         varkws = node.args.kwarg
@@ -980,11 +987,19 @@ def on_functiondef(self, node):
             vararg = vararg.arg
         if isinstance(varkws, ast.arg):
             varkws = varkws.arg
-        self.symtable[node.name] = Procedure(node.name, self, doc=doc,
-                                             lineno=self.lineno,
-                                             body=node.body,
-                                             text=ast.unparse(node),
-                                             args=args, kwargs=kwargs,
-                                             vararg=vararg, varkws=varkws)
-        if node.name in self.no_deepcopy:
-            self.no_deepcopy.remove(node.name)
+
+        proc = Procedure(name, self, doc=doc, lineno=self.lineno,
+                         body=body, text=ast.unparse(node),
+                         args=args, kwargs=kwargs, vararg=vararg,
+                         varkws=varkws, is_lambda=is_lambda)
+
+        if is_lambda:
+            return proc
+        else:
+            self.symtable[name] = proc
+            if name in self.no_deepcopy:
+                self.no_deepcopy.remove(name)
+
+    def on_lambda(self, node):
+        """Lambda."""
+        return self.on_functiondef(node, is_lambda=True)

asteval/astutils.py

@@ -571,7 +571,7 @@ class Procedure:
 
     def __init__(self, name, interp, doc=None, lineno=None,
                  body=None, text=None, args=None, kwargs=None,
-                 vararg=None, varkws=None):
+                 vararg=None, varkws=None, is_lambda=False):
         """TODO: docstring in public method."""
         self.__ininit__ = True
         self.name = name
@@ -586,6 +586,10 @@ def __init__(self, name, interp, doc=None, lineno=None,
         self.__varkws__ = varkws
         self.lineno = lineno
         self.__text__ = text
+        self.__is_lambda__ = is_lambda
+        if is_lambda:
+            self.name = self.__name__ = 'lambda'
+
         if text is None:
             self.__text__ = f'{self.__signature__()}\n' + ast.unparse(self.__body__)
         self.__ininit__ = False
@@ -732,9 +736,12 @@ def __call__(self, *args, **kwargs):
         # evaluate script of function
         self.__asteval__.code_text.append(self.__text__)
         for node in self.__body__:
-            self.__asteval__.run(node, lineno=node.lineno)
+            out = self.__asteval__.run(node, lineno=node.lineno)
             if len(self.__asteval__.error) > 0:
                 break
+            if self.__is_lambda__:
+                retval = out
+                break
             if self.__asteval__.retval is not None:
                 retval = self.__asteval__.retval
                 self.__asteval__.retval = None

doc/api.rst

@@ -80,6 +80,8 @@ but will full support for Python data types and array slicing.
   +----------------+----------------------+-------------------+-------------------+
   | raise          | raise statements     |  True             | False             |
   +----------------+----------------------+-------------------+-------------------+
+  | lambda         | lambda expressions   |  True             | False             |
+  +----------------+----------------------+-------------------+-------------------+
   | listcomp       | list comprehension   |  True             | False             |
   +----------------+----------------------+-------------------+-------------------+
   | dictcomp       | dict comprehension   |  True             | False             |
@@ -105,6 +107,7 @@ The ``default`` configuration adds many language constructs, including
   *  with blocks
   *  augmented assignments:  ``x += 1``
   *  if-expressions:      ``x = a if TEST else b``
+  *  lambda expressions:  ``dist = lambda x,y: sqrt(x**2 + y**2)``
   *  list comprehension:  ``out = [sqrt(i) for i in values]``
   *  set and dict comprehension, too.
   *  print formatting with ``%``, ``str.format()``, or f-strings.
@@ -133,11 +136,13 @@ Passing, ``minimal=True`` will turn off all the nodes listed in Table
     >>>
     >>> aeval_min = Interpreter(minimal=True)
     >>> aeval_min.config
-    {'import': False, 'importfrom': False, 'assert': False, 'augassign': False,
-    'delete': False, 'if': False, 'ifexp': False, 'for': False,
-    'formattedvalue': False, 'functiondef': False, 'print': False,
-    'raise': False, 'listcomp': False, 'dictcomp': False, 'setcomp': False,
-    'try': False, 'while': False, 'with': False}
+    {'import': False, 'importfrom': False, 'assert': False,
+    'augassign': False, 'delete': False, 'if': False, 'ifexp': False,
+    'for': False, 'formattedvalue': False, 'functiondef': False,
+    'print': False, 'raise': False, 'lambda': False,
+    'listcomp': False, 'dictcomp': False, 'setcomp': False,
+    'try': False, 'while': False, 'with': False,
+    'nested_symtable': False}
 
 As shown above, importing Python modules with ``import module`` or ``from
 module import method`` can be enabled, but is disabled by default.  To enable
@@ -146,12 +151,13 @@ these, use ``with_import=True`` and ``with_importfrom=True``, as ::
     >>> from asteval import Interpreter
     >>> aeval_max = Interpreter(with_import=True, with_importfrom=True)
 
-or by setting the config dictionary as described above:
+or by setting the config dictionary passed to ``Interpreter`` as
+described above.
 
 Interpreter methods and attributes
 ====================================
 
-An Interpreter instance has many methods, but most of them are
+The Asteval Interpreter instance has many methods, but most of them are
 implementation details for how to handle particular AST nodes, and should
 not be considered as part of the usable API.  The methods described below,
 and the examples elsewhere in this documentation should be used as the
@@ -312,8 +318,6 @@ Utility Functions
 
 .. autofunction:: valid_symbol_name
 
-
-
 .. autofunction:: make_symbol_table
 
 

doc/basics.rst

@@ -155,7 +155,8 @@ User-defined functions can be written and executed, as in python with a
 
    >>> from asteval import Interpreter
    >>> aeval = Interpreter()
-   >>> code = """def func(a, b, norm=1.0):
+   >>> code = """
+   ... def func(a, b, norm=1.0):
    ...     return (a + b)/norm
    ... """
    >>> aeval(code)

doc/index.rst

@@ -41,7 +41,7 @@ These differences and absences include:
     access can also be used.
  2. creating classes is not allowed.
  3. importing modules is not allowed, unless specifically enabled.
- 4. decorators, generators, type hints, and ``lambda`` are not supported.
+ 4. decorators, generators, and type hints are not supported.
  5. ``yield``, ``await``,  and async programming are not supported.
  6. Many builtin functions (:py:func:`eval`, :py:func:`getattr`,
     :py:func:`hasattr`, :py:func:`setattr`, and :py:func:`delattr`) are not allowed.

doc/motivation.rst

@@ -20,7 +20,7 @@ makes user input difficult to trust.  Since asteval does not support the
 code cannot access the :py:mod:`os` and :py:mod:`sys` modules or any functions
 or classes outside those provided in the symbol table.
 
-Many of the other missing features (modules, classes, lambda, yield,
+Many of the other missing features (modules, classes, yield,
 generators) are similarly motivated by a desire for a safer version of
 :py:func:`eval`.  The idea for asteval is to make a simple procedural,
 mathematically-oriented language that can be embedded into larger applications.

tests/test_asteval.py

@@ -1149,6 +1149,19 @@ def c(x=10):
     isvalue(interp, 'o1', 3.5)
     isvalue(interp, 'o2', 1.5)
 
+@pytest.mark.parametrize("nested", [False, True])
+def test_lambda(nested):
+    """test using lambda definitions"""
+    interp = make_interpreter(nested_symtable=nested)
+
+    interp(textwrap.dedent("""
+    my_func = lambda x: 2 + 3*x
+    out = my_func(3)
+    """))
+    assert len(interp.error) == 0
+    isvalue(interp, 'out', 11.0)
+
+
 @pytest.mark.parametrize("nested", [False, True])
 def test_astdump(nested):
     """test ast parsing and dumping"""
@@ -1231,7 +1244,7 @@ def test_kaboom(nested):
     interp("""(lambda fc=(lambda n: [c for c in ().__class__.__bases__[0].__subclasses__() if c.__name__ == n][0]):
     fc("function")(fc("code")(0,0,0,0,"KABOOM",(),(),(),"","",0,""),{})()
 )()""")
-    check_error(interp, 'NotImplementedError')  # Safe, lambda is not supported
+    check_error(interp, 'AttributeError')  # Safe, unassigned lambda is not supported
 
     interp("""[print(c) for c in ().__class__.__bases__[0].__subclasses__()]""")  # Try a portion of the kaboom...