Added SQlite and Python dictionary options for backend #323
Conversation
…s is the interface) - then created a base class, abstract class, and independent subclasses to for each backend option (redis, sqlite, and python dictionary). The default option remains redis.
| logging.basicConfig(level=logging.WARNING) | ||
| logging.getLogger("bluesky_queueserver").setLevel("ERROR") | ||
|
|
||
| # Determine the backend type |
There was a problem hiding this comment.
This isqserver client application, it is not using the backend. I would suggest, that for prototyping we just use an environment variable, say QSERVER_PLAN_QUEUE_BACKEND and check it directly in the code. The parameter may be easily added later.
|
I would suggest putting the abstract class, the wrapper and each backend in a separate file so that the code is easier to manage. The tests that are common to all backends should probably be in a separate file and executed on all backends. The backend-specific tests, depending on the number of tests, could be in separate files or in one file. Also in naming classes and files I would put the common part first, so that when the names are sorted they remain together. For example, for class names it could be |
…or each backend type. Additionally, split out the abstract class into its own py file.
…d test to check interfaces contracts
2 participants
The following changes were made to plan_queue_ops.py to add support for the sqlite and dict backends:
Purpose: Implements queue operations using an SQLite database.
Key Features:
Uses aiosqlite for asynchronous SQLite operations.
Supports all required queue operations, including adding, retrieving, and clearing items in the queue.
Stores data in SQLite tables for persistence.
Reads the SQLite database path from the PLAN_QUEUE_SQLITE_PATH environment variable or defaults to plan_queue.db in the current working directory.
Implements methods for managing user group permissions, lock information, autostart mode, and stop-pending state.
Purpose: Implements queue operations using an in-memory Python dictionary with optional persistence to a JSON file.
Key Features:
Uses aiofiles for asynchronous file I/O to save and load data to/from a JSON file.
Supports all required queue operations, including adding, retrieving, and clearing items in the queue.
Reads the storage file path from the PLAN_QUEUE_DICT_PATH environment variable or defaults to plan_queue_storage.json in the current working directory.
Designed for lightweight or testing scenarios, not for production environments.
Purpose: Dynamically selects the appropriate backend (redis, sqlite, or dict) based on the PLAN_QUEUE_BACKEND environment variable.
Key Changes:
Added logic to initialize SQLitePlanQueueOperations if PLAN_QUEUE_BACKEND is set to sqlite.
Added logic to initialize DictPlanQueueOperations if PLAN_QUEUE_BACKEND is set to dict.
Retains support for the redis backend as the default option.
Purpose: Ensures all backends implement the required queue operations.
Key Changes:
Abstract methods such as add_item_to_queue, get_queue_full, clear_queue, set_plan_queue_mode, and others are implemented in both SQLitePlanQueueOperations and DictPlanQueueOperations.
Added methods for managing user group permissions, lock information, autostart mode, and stop-pending state.
SQLite Backend:
Uses SQLite tables to store queue, history, and metadata.
Implements methods for managing SQLite-specific operations, such as initializing tables and handling transactions.
Raises NotImplementedError for methods not applicable to SQLite (e.g., new_item_uid).
Dict Backend:
Uses an in-memory dictionary to store queue, history, and metadata.
Saves and loads data to/from a JSON file for persistence.
Implements all required methods for queue operations.
6. Environment Variable Support
PLAN_QUEUE_BACKEND:
Determines the backend to use (redis, sqlite, or dict).
PLAN_QUEUE_SQLITE_PATH:
Specifies the SQLite database file path for the sqlite backend.
PLAN_QUEUE_DICT_PATH:
Specifies the JSON file path for the dict backend.
Summary
These changes add support for two additional backends (sqlite and dict) to plan_queue_ops.py, making the queue operations more flexible and adaptable to different use cases. The redis backend remains the default, while the sqlite and dict backends provide lightweight and persistent alternatives.