add smi replay simulator #19
Conversation
| conn = sqlite3.connect(db_path) | ||
| cursor = conn.cursor() | ||
|
|
||
| query = "SELECT id, tiled_url FROM vectors ORDER BY id" |
There was a problem hiding this comment.
Now that we have multiple modules using the same sqlite db...I think we should extract all db operations from both modules and put them into their own module. This will make it a lot easier to deal with potential schema changes...and also do support postgres, which we might decide to do in the future.
There was a problem hiding this comment.
Sounds good.
Do you mean extracting all database operations (e.g., get_tiled_urls, insert_vector) into a dedicated module (e.g., db_utils.py), placing it in mlex_utils (for example), and then simply importing the functions wherever they are needed?
We plan to refactor our code and mlex_utils by November, introducing prefect_utils (Prefect 3 version with David), db_utils, mlflow_utils, and tiled_utils. We can go over the details later and have you review the changes.
| return [] | ||
|
|
||
|
|
||
| def read_image_from_tiled_url(tiled_url, api_key=None): |
There was a problem hiding this comment.
in the future, this will belong in a shared library that we maintain.
There was a problem hiding this comment.
I see. For the shared library, Are you referring to mlex_utils (https://github.com/mlexchange/mlex_utils)?
| socket.bind(address) | ||
|
|
||
| # Get URLs from database | ||
| urls = get_urls_from_db(db_path, limit=max_frames) |
There was a problem hiding this comment.
Calling a synchronous database option inside an async function, while probably not a real performance problem here, is not recommended. I recommend making get_urls_from_db asynch and use an asynch sqlite driver.
There was a problem hiding this comment.
Thanks for your suggestion. I make the changes in my last commit.
I added a simulator to read the tiled_url saved in
latent_vectors.db, which we collected during the SMI beamtime in July 2025, and send it to the listener.A companion PR for
LSEcan be found at mlexchange/mlex_latent_explorer#52.