General query engine interface

[rubensworks]

After some internal discussions with @gsvarovsky and @jacoscaz, we identified the need to come up with a base query engine interface for RDF/JS (for declarative queries).

In essence, it should expose an interface that allows you to do operations
such as const resultStream = await engine.query('some query');

The goal of this issue is to collect input on what already exists, so we can identify what the requirements are for such an interface.

---

Projects I contribute to that would benefit from this interface:

• The Comunica query engine. Relevant types are defined here: https://github.com/comunica/comunica/blob/master/packages/types/index.ts
• rdf-test-suite.js, which can execute specification tests on query engines. Relevant interface is here: https://github.com/rubensworks/rdf-test-suite.js/blob/master/lib/testcase/sparql/IQueryEngine.ts
• SPARQLAlgebra.js, which defines typings for SPARQL algebra
• fetch-sparql-endpoint.js, exposes access to a SPARQL endpoint

Big open question for me is how close the relationship to SPARQL should be. (We could for example start off with defining it in terms of SPARQL, but leave room for other query languages)

[tpluscode]

https://github.com/zazuko/sparql-http-client (types)
https://github.com/Callidon/sparql-engine

[RubenVerborgh]

(We could for example start off with defining it in terms of SPARQL, but leave room for other query languages)

Optional second argument, defaulting to { language: 'SPARQL', version: '1.1', extensions: [] }?

[gsvarovsky]

I would want to implement this interface with m-ld's Javascript engine, so that it can operate in environments that use SPARQL queries directly.

I have implemented my own interface, which looks like this:
https://github.com/m-ld/m-ld-js/blob/edge/src/rdfjs-support.ts

Note the dependency on the sparqlalgebrajs types – I would prefer not to have to pass strings, so there may be a need to have an interface package extracted from sparqlalgebrajs.

I would also prefer that the interface allowed a store to be quite explicit about which queries it supports (e.g. Construct but not Describe).

[jacoscaz]

I maintain quadstore, a persistent RDF store with SPARQL capabilities via Comunica, and I would happily implement the proposed base query engine interface.

[tpluscode]

Does it have to be complicated much? I've been working with @bergos' sparql-http-client and I think's just about right. It comes in two forms, both of which have methods select/construct/ask/update but differ on the return types from select and construct

declare module 'sparql-http-client/StreamClient' {
  class StreamClient {
    query: {
      select(query: string): Promise<EventEmitter>
      construct(query: string): Promise<import('rdf-js').Stream>
      ask(query: string): Promise<boolean>
      update(query: string): Promise<void>
    }
  }
}

declare module 'sparql-http-client/ParsingClient' {
  class StreamClient {
    query: {
      select(query: string): Promise<Array<Record<string, import('rdf-js').Term>>>
      construct(query: string): Promise<import('rdf-js').Dataset>
      ask(query: string): Promise<boolean>
      update(query: string): Promise<void>
    }
  }
}

The only change that I would make above is for the StreamClient not to return promises

class StreamClient {
  query: {
-    select(query: string): Promise<EventEmitter>
-    construct(query: string): Promise<import('rdf-js').Stream>
+    select(query: string): EventEmitter
+    construct(query: string): import('rdf-js').Stream
  }
}

[jacoscaz]

both of which have methods select/construct/ask/update

I think I see the point you're making but doing this would imply leaking aspects of the query language into the query interface. It would be equivalent to having separate methods for SELECT, INSERT, DELETE and UPDATE queries in SQL-oriented database drivers and ORMs. IMHO, this would make it much harder to deal with use cases in which the nature of a query is not known ahead of time and also introduce too strong a coupling between SPARQL and the interface itself.

[tpluscode]

Database drivers (SDK?) do very much have similar distinction and it's nothing wrong. Think query/execute/executeScalar. I will not even comment on ORMs because that is the worst comparison ever. Might have a look at micro ORMs like dapper which are much "closer to metal" to mitigate the impedance mismatch issues plaguing ORMs

use cases in which the nature of a query is not known ahead of time

I'm curious about this statement. In what scenarios is it not known what is the desired kind of result? (tabular, graph or boolean).

and also introduce too strong a coupling between SPARQL and the interface itself.

What is the nature of this coupling? Arguably, the whole RDF stack is built on uniformity and standards. The RDF graph is the same graph in every software component. SPARQL, being similarly important core standard, can act the same. Otherwise I read you comment as an invitation to build (IMO unnecessary) abstractions

[rubensworks]

I've created a new issue to follow up on the discussion of query methods: #6
(So that we can keep this issue here focussed on collecting existing approaches)

[tpluscode]

I might actually mention also my lib @tpluscode/sparql-builder. Right now it does rely on sparql-http-client for execution but a standard interface would be nice, especially if we could get an in-memory query engine

[rubensworks]

Ah yes indeed, libs that depend on engines are also relevant to include here.

In that respect, the following libs may also be relevant:

• https://github.com/LDflex/LDflex (currently hardcoded on Comunica's interface)
• https://github.com/rubensworks/graphql-ld.js (currently hardcoded on Comunica's interface)

[ericprud]

I'd suggest starting with the SPARQL Algebra (but be willing to depart from it as use cases indicate). You can do a lot with it (like all of SPARQL) but is simpler than SPARQL. For instance, the idiosyncracies of a SolutionModifier's GROUP BY and HAVING reuse aggregation and filter. You may also opt to lop off large parts of it, but having a set of composable operations should be familiar to programmers.

[gsvarovsky]

@tpluscode #5 (comment)

not to return promises

👍

[rubensworks]

not to return promises

That would actually depend on the outcome of #6. Because if we only expose a single method there, then the return type would vary based on the query. Since query type and return type may be determined async, we may require promises.

[rubensworks]

I've had a look at all the suggested libraries, and I've tried to create an overview aspects that I feel may require some standardization.
If I missed any, please let me know!

Once we agree upon a list of aspect, we can branch of into separate issues to see how we want to tackle the specifics of each one.

1. Query method interface

How to pass a query to a library, and obtain results.

Discussion in #6.

Single method

All query forms are handled via a single method, possibly via method overloading or union types.

Example:

      query(query: string): Promise<SomeUnionType>

Implemented by:

• M-ld
• Quadstore
• Comunica
• rdf-test-suite.js

Form-based methods

Each query form has its own dedicated method.

Example:

      select(query: string): Promise<Array<Record<string, import('rdf-js').Term>>>
      construct(query: string): Promise<import('rdf-js').Dataset>
      ask(query: string): Promise<boolean>
      update(query: string): Promise<void>

Implemented by:

• sparql-http-client
• fetch-sparql-endpoint.js

Other

The following libraries follow another query interface, which seem to be use-case-specific, and may not benefit that much from standardization:

• sparql-engine
• LDflex
• GraphQL-LD
• @tpluscode/sparql-builder

2. Representing bindings

How to represent the results of tabular queries such as SELECT.

JSON-based

Example of a single bindings object:

{
  '?varA': namedNode('ex:a'),
  '?varB': namedNode('ex:b'),
}

Implemented by:

• M-ld
• rdf-test-suite.js
• sparql-http-client
• sparql-engine
• @tpluscode/sparql-builder
• fetch-sparql-endpoint.js

Object-based

A custom datastructure that exposes methods and allows bindings to be stored internally in a different manner.

Example of a single bindings object:

const bindings = ...
const term: RDF.Term = bindings.get('?a');

Implemented by:

• Quadstore
• Comunica

3. Exposing metadata

Both on query-level as on source-level, it may be beneficial to expose metadata such as cardinality (estimates).
Such information may be useful for query optimization.

Dedicated method for obtaining metadata

interface CountableSource {
  countQuads(): Promise<number> | number
}

Implemented by:

• Comunica
• M-ld

Generic object that provides metadata

const results = engine.query(...);
const metadata = await results.metadata();
console.log(metadata.cardinality);

Implemented by:

• Comunica

4. Serializing results

A method to serialize query results to a standard format, such as SPARQL JSON results.
Related to this, methods may be added that expose the available formats.

  resultToString: (queryResult: ..., format?: string) => Promise<Stream<string>>;

Implemented by:

• Comunica

5. Defining sources

Some engines allow query sources to vary per query execution,
and therefore enable passing it as an additional argument.

  query: (query: string, context: { sources: IDataSource[] }) => Promise<IQueryResult>;

export type IDataSource = string | RDF.Source | {
  type?: string;
  value: string | RDF.Source;
  context?: ActionContext;
};

Implemented by:

• Comunica
• rdf-test-suite.js

6. Passing query as algebra

Instead of passing a query string to an engine, a (pre-optimized?) algebra object may be passed.
Related to this, methods for parsing a query string to algebra may also be valuable to standardize.

Example:

export interface QueryableRdf<Q extends BaseQuad = Quad> {
  query(query: Algebra.Construct): Stream<Q>;
  query(query: Algebra.Describe): Stream<Q>;
  query(query: Algebra.Project): BaseStream<Binding>;
}

Implemented by:

• M-ld
• Quadstore
• Comunica

7. Defining query syntax format

If engines support different query syntaxes, they typically allow this to be customized via an optional argument.

Example:

  query: (query: string, context: { queryFormat: string }) => Promise<IQueryResult>;

Implemented by:

• Comunica