fix: add extra_plugin_dir support to FullStackTest (errbotio#1726)

* fix: add extra_plugin_dir support to FullStackTest

* docs: add example on using FullStackTest

* refactor: remove deprecated extra_test_file parameter

* docs: add info to CHANGES

* style: format file

* docs: remove travis-ci / coverall references and reordered pages

(cherry picked from commit 7c587d3)

Author: sijis

CHANGES.rst

@@ -25,6 +25,7 @@ fixes:
 - fix: type hints (#1698)
 - fix: update plugin config message (#1727)
 - docs: add example on how to use threaded replies (#1728)
+- fix: add extra_plugin_dir support to FullStackTest (#1726)
 
 
 v6.2.0 (2024-01-01)

docs/user_guide/plugin_development/index.rst

@@ -26,6 +26,7 @@ with sets of recipes on a range of topics describing how to handle more advanced
   scheduling
   webhooks
   testing
+  testing_plugins_fullstack
   logging
   exceptions
   plugin_compatibility_settings

docs/user_guide/plugin_development/testing.rst

@@ -251,37 +251,6 @@ You can now have a look at coverage statistics through :command:`coverage report
 
 It's also possible to generate an HTML report with :command:`coverage html` and opening the resulting `htmlcov/index.html`.
 
-Travis and Coveralls
---------------------
-
-Last but not least, you can run your tests on Travis-CI_ so when you update code or others submit pull requests the tests will automatically run confirming everything still works.
-
-In order to do that you'll need a `.travis.yml` similar to this:
-
-.. code-block:: yaml
-
-    language: python
-    python:
-      - 3.6
-      - 3.7
-    install:
-      - pip install -q errbot pytest pytest-pep8 --use-wheel
-      - pip install -q coverage coveralls --use-wheel
-    script:
-      - coverage run --source myplugin -m py.test --pep8
-    after_success:
-      - coveralls
-    notifications:
-      email: false
-
-Most of it is self-explanatory, except for perhaps the `after_success`. The author of this plugin uses Coveralls.io_ to keep track of code coverage so after a successful build we call out to coveralls and upload the statistics. It's for this reason that we `pip install [..] coveralls [..]` in the `.travis.yml`.
-
-The `-q` flag causes pip to be a lot more quiet and `--use-wheel` will cause pip to use wheels_ if available, speeding up your builds if you happen to depend on something that builds a C-extension.
-
-Both Travis-CI and Coveralls easily integrate with Github hosted code.
 
 .. _py.test: http://pytest.org
 .. _conftest.py: http://doc.pytest.org/en/latest/writing_plugins.html#conftest-py-local-per-directory-plugins
-.. _Coveralls.io: https://coveralls.io
-.. _Travis-CI: https://travis-ci.org
-.. _wheels: http://www.python.org/dev/peps/pep-0427/

docs/user_guide/plugin_development/testing_plugins_fullstack.rst

@@ -0,0 +1,167 @@
+Testing your plugins with unittest
+==================================
+
+This guide explains how to test your Errbot plugins using the built-in testing framework. Errbot provides a powerful testing backend called ``FullStackTest`` that allows you to write unit tests for your plugins in a familiar unittest style.
+
+Basic Test Setup
+--------------
+
+To test your plugin, create a test file (e.g., `test_myplugin.py`) in your plugin's directory. Here's a basic example:
+
+.. code-block:: python
+
+    import unittest
+    from pathlib import Path
+
+    from errbot.backends.test import FullStackTest
+
+    path = str(Path(__file__).resolve().parent)
+    extra_plugin_dir = path
+
+
+    class TestMyPlugin(FullStackTest):
+        def setUp(self):
+            super().setUp(extra_plugin_dir=extra_plugin_dir)
+
+        def test_my_command(self):
+            # Simulate a user sending a command
+            self.push_message('!hello')
+            self.assertIn('Hello!', self.pop_message())
+
+Running Tests
+------------
+
+You can run your tests using Python's unittest framework:
+
+.. code-block:: bash
+
+    python -m unittest test_myplugin.py
+
+Test Methods
+-----------
+
+FullStackTest provides several methods to help test your plugin's behavior:
+
+1. **Message Handling**:
+   - ``push_message(command)``: Simulate a user sending a command
+   - ``pop_message(timeout=5, block=True)``: Get the bot's response
+   - ``assertInCommand(command, response, timeout=5)``: Assert a command returns expected output
+   - ``assertCommandFound(command, timeout=5)``: Assert a command exists
+
+2. **Room Operations**:
+   - ``push_presence(presence)``: Simulate presence changes
+   - Test room joining/leaving
+   - Test room topic changes
+
+3. **Plugin Management**:
+   - ``inject_mocks(plugin_name, mock_dict)``: Inject mock objects into a plugin
+   - Test plugin configuration
+   - Test plugin dependencies
+
+Example Test Cases
+----------------
+
+Here are some example test cases showing different testing scenarios:
+
+1. **Basic Command Testing**:
+
+   .. code-block:: python
+
+       def test_basic_command(self):
+           self.push_message('!echo test')
+           self.assertIn('test', self.pop_message())
+
+2. **Command with Arguments**:
+
+   .. code-block:: python
+
+       def test_command_with_args(self):
+           self.push_message('!repeat test 3')
+           response = self.pop_message()
+           self.assertIn('testtesttest', response)
+
+3. **Error Handling**:
+
+   .. code-block:: python
+
+       def test_error_handling(self):
+           self.push_message('!nonexistent')
+           response = self.pop_message()
+           self.assertIn('Command not found', response)
+
+4. **Mocking Dependencies**:
+
+   .. code-block:: python
+
+       def test_with_mocks(self):
+           # Create mock objects
+           mock_dict = {
+               'external_api': MockExternalAPI()
+           }
+           self.inject_mocks('MyPlugin', mock_dict)
+
+           # Test plugin behavior with mocks
+           self.push_message('!api_test')
+           self.assertIn('Mock response', self.pop_message())
+
+Best Practices
+-------------
+
+1. **Test Isolation**: Each test should be independent and not rely on the state from other tests.
+
+2. **Setup and Teardown**: Use ``setUp()`` to initialize your test environment and ``tearDown()`` to clean up.
+
+3. **Timeout Handling**: Always specify appropriate timeouts for message operations to avoid hanging tests.
+
+4. **Error Cases**: Include tests for error conditions and edge cases.
+
+5. **Documentation**: Document your test cases to explain what they're testing and why.
+
+Complete Example
+--------------
+
+Here's a complete example of a test suite for a plugin:
+
+.. code-block:: python
+
+    import unittest
+    from pathlib import Path
+
+    from errbot.backends.test import FullStackTest
+
+    path = str(Path(__file__).resolve().parent)
+    extra_plugin_dir = path
+
+    class TestGreetingPlugin(FullStackTest):
+        def setUp(self):
+            super().setUp(extra_plugin_dir=extra_plugin_dir)
+
+        def test_basic_greeting(self):
+            """Test the basic greeting command."""
+            self.push_message('!greet Alice')
+            self.assertIn('Hello, Alice!', self.pop_message())
+
+        def test_greeting_with_options(self):
+            """Test greeting with different options."""
+            # Test with count
+            self.push_message('!greet Bob --count 2')
+            response = self.pop_message()
+            self.assertIn('Hello, Bob!Hello, Bob!', response)
+
+            # Test with shout
+            self.push_message('!greet Charlie --shout')
+            self.assertIn('HELLO, CHARLIE!', self.pop_message())
+
+        def test_error_handling(self):
+            """Test how the plugin handles errors."""
+            # Test missing name
+            self.push_message('!greet')
+            self.assertIn('Please provide a name', self.pop_message())
+
+            # Test invalid count
+            self.push_message('!greet Eve --count abc')
+            self.assertIn('must be an integer', self.pop_message())
+
+
+    if __name__ == '__main__':
+        unittest.main()

errbot/backends/test.py

@@ -1,9 +1,10 @@
 import importlib
 import logging
+import pathlib
 import sys
 import textwrap
 import unittest
-from os.path import abspath, sep
+from os.path import sep
 from queue import Empty, Queue
 from tempfile import mkdtemp
 from threading import Thread
@@ -604,25 +605,31 @@ def test_about(self):
                 self.assertIn('Err version', self.pop_message())
     """
 
+    def __init__(
+        self,
+        methodName,
+        extra_plugin_dir=None,
+        loglevel=logging.DEBUG,
+        extra_config=None,
+    ):
+        self.bot_thread = None
+        super().__init__(methodName)
+
     def setUp(
         self,
         extra_plugin_dir=None,
-        extra_test_file=None,
         loglevel=logging.DEBUG,
         extra_config=None,
     ) -> None:
         """
         :param extra_plugin_dir: Path to a directory from which additional
             plugins should be loaded.
-        :param extra_test_file: [Deprecated but kept for backward-compatibility,
-            use extra_plugin_dir instead]
-            Path to an additional plugin which should be loaded.
         :param loglevel: Logging verbosity. Expects one of the constants
             defined by the logging module.
         :param extra_config: Piece of extra bot config in a dict.
         """
-        if extra_plugin_dir is None and extra_test_file is not None:
-            extra_plugin_dir = sep.join(abspath(extra_test_file).split(sep)[:-2])
+        if extra_plugin_dir is None:
+            extra_plugin_dir = str(pathlib.Path(".").resolve().parent.parent.absolute())
 
         self.setup(
             extra_plugin_dir=extra_plugin_dir,