Implement cached schema & variant materialized views for luminork #6938
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Dependency Review✅ No vulnerabilities or OpenSSF Scorecard issues found.Scanned FilesNone |
github-actions
bot
added
A-si-settings
|
1 similar comment
|
jhelwig
force-pushed
the
jhelwig/eng-3182-get-access-to-installed-uninstalled-schema-variants-in
branch
from
e230398 to
2d896dc
Compare
|
1 similar comment
|
jhelwig
force-pushed
the
jhelwig/eng-3182-get-access-to-installed-uninstalled-schema-variants-in
branch
from
918d1e5 to
8439563
Compare
|
jhelwig
force-pushed
the
jhelwig/eng-3182-get-access-to-installed-uninstalled-schema-variants-in
branch
from
8439563 to
c943635
Compare
|
jhelwig
force-pushed
the
jhelwig/eng-3182-get-access-to-installed-uninstalled-schema-variants-in
branch
from
c943635 to
93b793f
Compare
|
jhelwig
changed the title
jhelwig
force-pushed
the
jhelwig/eng-3182-get-access-to-installed-uninstalled-schema-variants-in
branch
from
93b793f to
a6eed41
Compare
|
…ions Add HasUniqueId trait providing unified function ID access across all function spec types. This eliminates repetitive implementation patterns while maintaining zero-allocation semantics through Option<&str> returns. The trait enables consistent function collection logic with shared error handling for invalid function IDs, stable materialized view output through deterministic ordering, and deduplication of collected function identifiers. Management functions always return Some() while other function types delegate to their unique_id fields. Both spec/ and pkg/ module implementations follow the same patterns for consistency. Testing covers all trait implementations with direct unit tests for each function type including edge cases for invalid UUIDs and error propagation paths.
jhelwig
force-pushed
the
jhelwig/eng-3182-get-access-to-installed-uninstalled-schema-variants-in
branch
from
a6eed41 to
692f01f
Compare
|
jhelwig
force-pushed
the
jhelwig/eng-3182-get-access-to-installed-uninstalled-schema-variants-in
branch
from
692f01f to
4b14d0a
Compare
|
jhelwig
force-pushed
the
jhelwig/eng-3182-get-access-to-installed-uninstalled-schema-variants-in
branch
from
4b14d0a to
6973e41
Compare
|
This change establishes the foundation for materialized views (MVs) that provide cached access to schema and variant data from uninstalled packages. The implementation uses the MV building pattern established by the change set MVs and supports graceful fallback chains in API endpoints. Key components: - CachedSchema and CachedDefaultVariant MV types for schema metadata access - Deployment MV discovery system with direct key lookup in content- addressable store vs O(nm) traversal through layers - JoinSet task scheduling for parallel MV building in edda-server - Luminork API endpoints updated with MV-first lookups and DAL fallbacks - Function collection system with error handling for variant data assembly The MV system enables efficient access to schema information without requiring full package installation, supporting API integrations and MCP servers.
jhelwig
force-pushed
the
jhelwig/eng-3182-get-access-to-installed-uninstalled-schema-variants-in
branch
from
6973e41 to
0cdc002
Compare
|
|
@jhelwig on the luminork side this is great! |
fnichol
approved these changes
Aug 20, 2025
jhelwig
deleted the
jhelwig/eng-3182-get-access-to-installed-uninstalled-schema-variants-in
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR implements materialized views (MVs) for cached access to schema and variant data from uninstalled packages, enabling efficient schema information retrieval without requiring full package installation.
Add trait-based function collection system - HasUniqueId trait provides unified function ID access across all function spec types, eliminating repetitive patterns while maintaining zero-allocation semantics through Option<&str> returns
Implement cached schema MV infrastructure - Uses the MV building pattern established by the change set MVs with deployment MV discovery system providing direct key lookup in content-addressable store vs O(nm) traversal through layers
Update Luminork API endpoints - Maintain current behavior of looking up from the graph first, but instead of 404ing if the schema/variant isn't found, check to see if there is an MV. If there is one, return it. If there isn't one, check to see if there should be one by consulting the cached_modules table. If there should be one, return a 202 and trigger a build. If not, return a 404.
Add parallel MV building - JoinSet task scheduling in edda-server with function collection system providing error handling for variant data assembly
Test plan