GH-2701: Fuseki Mod to list and abort running executions.

[Aklakan]

GitHub issue resolved #2701

Pull request Description: ARQ plugin + Fuseki Plugin to track and abort ongoing SPARQL executions.
I tried to design the changes to jena's core in such a way that execution tracking can be enabled without requiring any changes to existing code.

Summary of changes.

Interception of SPARQL Requests and Execution Tracking

Changes in jena-arq:

• Added SparqlDispatcherRegistry + infrastructure which allows to intercept SPARQL update and query statements (as objects or strings) against DatasetGraphs. Update-/QueryExecDataset now first delegates to the dispatcher chain.
• The usual Update-/QueryEngineFactory handling is now the last handler in this dispatcher chain.
• Execution tracking is the first element in the chain, initialized with the plugin InitExecTracking. If in the dispatcher chain there is a TaskListener is the context, then the Update-/QueryExec instances are wrapped with a tracking wrapping such that the listener is notified.
• The default TaskListener implementation is TaskEventBroker which supports de-/registering of further listeners.
• For keeping track of running tasks and a history of the last n executions, there is TaskEventHistory which is a subclass of TaskEventBroker.
• Added a parseCheck dataset context attribute. If false, then update-/query requests are forwarded via the dispatcher without parsing.

RDFLink-based Execution Tracking

This adds infrastructure to jena-rdfconnection in order to track executions against RDFLinks via the newly introduced class DatasetGraphOverRDFLink.

• Replaced the unfinished class GraphSPARQLService with DatasetGraphOverSparql. This is a base class in jena-arq that implements all methods by means of SPARQL requests. It is wired up with the tests in TS_SparqlCore. Caveat: As each update is a separate request and bulk updates may be split into multiple requests, blank nodes may not work as expected.
• Added DatasetGraphOverRDFLink as a subclass of DatasetGraphOverSparql which provides a newRDFLink() method and implements all DatasetGraph methods based on the RDFLink.
• Added dispatchers that intercept requests against DatasetGraphOverRDFLink and delegates them to the RDFLink.
• Added a DatasetAssemblerHTTP which configures a DatasetGraphOverRDFLink instance, which allows use of this system with Fuseki.
• Added ExampleDBpediaViaRemoteDataset which demonstrates a query against such a dataset making use of virtuoso specific features.

PREFIX fuseki:    <http://jena.apache.org/fuseki#>
PREFIX rdf:       <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
PREFIX ja:        <http://jena.hpl.hp.com/2005/11/Assembler#>
PREFIX remote:    <http://jena.apache.org/2025/http#>

<#service> rdf:type fuseki:Service ;
    fuseki:name "dbpedia-remote" ;
    fuseki:endpoint [
      fuseki:operation fuseki:query ;
    ] ;
    fuseki:endpoint [
      fuseki:operation fuseki:tracker ;
      fuseki:name "tracker" ;
    ] ;
    fuseki:dataset  <#baseDS>
   .

<#baseDS> rdf:type remote:DatasetHTTP ;
    remote:queryEndpoint "http://dbpedia.org/sparql" ;
    .

Fuseki Mod: Execution Tracker

Added a simple web page that will show a live view of ongoing and completed queries.

• Fuseki plugin will register a TaskEventHistory in the endpoint context and connect it to the dataset context's TaskEventBroker (broker will be created if absent).
• Web page will disconnect from event stream if there is too much activity and prompt for manual reload - prevents browser tab from freezing.
• Abort action can be disabled via context attribute:

<#service> rdf:type fuseki:Service ;
  fuseki:endpoint [
    fuseki:operation fuseki:tracker ;
    fuseki:name "tracker" ;
    ja:context [ ja:cxtName "allowAbort" ; ja:cxtValue false ] ; # Disable abort buttons in the Web page and the REST API
  ] ;
  fuseki:dataset <#baseDS> .

Jena-Query-Dashboard.webm

Misc changes

• QueryExecHTTP: Improved support to cancel HTTP requests. So far cancel would hang until the InputStream of the HTTP response could be obtained. Now the HTTP request can be cancelled immediately.
• HttpLib: Updated the methods to work on the async variant in order to make immediate abort on QueryExecHTTP work.
• Consolidated Task system of jena-geosparql with that of the execution tracking.
• Endpoint context is now never null so that Fuseki mods can always associate information with endpoints.

---

• Tests are included.
• Documentation change and updates are provided for the Apache Jena website
• Commits have been squashed to remove intermediate development commit messages.
• Key commit messages start with the issue number (GH-xxxx)

By submitting this pull request, I acknowledge that I am making a contribution to the Apache Software Foundation under the terms and conditions of the Contributor's Agreement.

---

See the Apache Jena "Contributing" guide.

[rvesse]

Really interesting piece of work, once did something much cruder (at least UI wise) in a previous $dayjob

Perhaps this could also serve as a base for discussion about further improvements and any necessary core changes for Jena 6. Ideally the ExecTracker mechanism would not require a wrapping with DatasetGraphWithExecTracker and instead this would be handled in the core machinery already. In cases where a specific DatasetGraph implementation is expected, the need for a wrapper to track executions may make things complex.

Yes I think this would be much cleaner if the tracking mechanism was integrated directly into the execution machinery without requiring extra wrapping as you do in this PR.

It would be nice if there were programmatic APIs for interacting with tracked queries/updates (there's some pieces towards that here but appears mostly focused on exposing stuff to the UI from my skim-reading of the code) so that applications that embed Jena could access and manage tracked queries/updates as desired.

Fuseki already has the concept of Tasks that's used for things like backups and compactions, would it make sense to integrate query/update tracking into that rather than creating a separate tracking mechanism. That might need generalising that mechanism, or pulling it more into Jena's core rather than Fuseki machinery, so might not be worth the effort, wdyt?

[Aklakan]

It would be nice if there were programmatic APIs for interacting with tracked queries/updates (there's some pieces towards that here but appears mostly focused on exposing stuff to the UI from my skim-reading of the code) so that applications that embed Jena could access and manage tracked queries/updates as desired.

The ARQ Plugin adds ExecTracker which so far would be the programmatic API. (Still a bit crude but intended as a starting point towards this goal.)
The DatasetGraphWithExecTracker wrapping is needed such that the query/update engines process queries by adding the tracking.
So by integrating the tracking closer to the core, the DatasetGraphWithExecTracker would only be needed for adding tracking to alternative query engines, but the main query/update engines could interact with the tracker directly.

An important question is, whether tracking executions within the DatasetGraph's context is the way to move forward.

Fuseki already has the concept of Tasks [...] pulling it more into Jena's core [...]

Yes, I yet need to look into how much effort it would be to disentangle Fuseki' task tracker from the Fuseki - but adding such a mechanism to core (and updating Fuseki for it) would be most likely the way to go.

[Aklakan]

Relevant code for intercepting query execution construction over a dataset graph using the newly proposed dispatcher system. Here, execution tracking is added.

[Aklakan]

The original code integrated exec tracking into the Update-/QueryEngineFactory system. However, this was sub-par because execution could only be intercepted on the QueryIterator level.

My updated proposal is to introduce a new layer Update-/QueryDispatcher on top of the Update-/QueryEngineFactory machinery. Now it is possible to intercept any query / update request to a dataset - even without having to parse the query.

Old design: QueryExecBuilderDataset -build-> QueryExecDataset -exec-> QueryEngineRegistry

New design: QueryExecBuilderDataset -build-> QueryDispatcherRegistry -customized build-> QueryExec

The last element of the dispatcher chain forwards the request to the usual Update-/QueryEngineFactory system.
The returned QueryExec may be of any type, such as a tracking wrapper - it is no longer necessarily a QueryExecDataset.

Consequences:

• Code that relies on QueryExecBuilderDataset to always return a QueryExecDataset may break when adding custom interceptors. This should be a low price to pay because conventional code relies on the QueryExec interface.
• Execution Tracking can be added without any change to the existing code.

Related Ongoing Work

As a demo for related work based on this infrastructure, we are using it to integrate third party triple stores - such as Qlever - into Fuseki. This way we can use one server framework to manage several triple stores.
Technically, we created a DatasetGraph wrapper implementation that is backed by a factory of HTTP RDFConnection instances (such as to a remote Qlever database). The query dispatcher just forwards query/update requests to an RDFConnection obtained from such a DatasetGraph implementation.
Any opinions on this design?

As a final note, we also already created an assembler that starts qlever from a docker image as part of Fuseki (via the Java TestContainer framework), so the configuration looks like this:

<#baseDS> rdf:type qlever:Dataset ;
  qlever:location "/run/databases/qlever/mydb/" ;
  qlever:indexName "myindex" ;
  qlever:accessToken "abcde" ;
  qlever:memoryMaxSize "4G" ;
  qlever:defaultQueryTimeout "600s" ;

Example of Qlever in Fuseki

[Aklakan]

I added an assembler to build datasets backed by remote (HTTP) SPARQL endpoints based on a new DatasetGraphOverRDFLink class in combination with the new dispatcher system, and this works out of the box with the execution tracking:

The PR should be review ready in the next few days.

[Aklakan]

The PR should be fairly complete and cleaned up now such that it is ready for review. I updated the first post with a summary of the changes.

[Aklakan]

I updated RDFLink's default shortcuts to delegate to newQuery().query(...).build() and newUpdate().update(...).build() so that any processing that happens in builders is applied by default.

[afs]

This is "quite big".

Is there a best place to start reviewing to break up the feedback?

[afs]

Just delete this class!

[Aklakan]

The fundamental changes in jena-arq are:

• The indirection between Update-/Query statements and DatasetGraphs is in org.apache.jena.sparql.engine.dispatch.

• The execution tracking then comes from org.apache.jena.sparql.exec.tracker. It registers a dispatcher that iff there is a listener in the dataset context, then UpateExec and QueryExec instances are wrapped with status reporting, such that that listener is notified when query execution starts and terminates.

• The class DatasetGraphOverSparql is meant as a base class that can be used with the dispatcher system to externalize query execution to e.g. remote endpoints.

The changes in jena-rdfconnection build upon DatasetGraphOverSparql and the dispatcher system.
The main change is the introduction of DatasetGraphOverRDFLink + corresponding infrastructure. A dispatcher is provided that - for such a dataset graph - delegates each SPARQL statement to a fresh underlying RDF link. This way a remote store can be abstracted as a DatasetGraph for which queries and updates are forwarded to the backend.
The purpose is to test/showcase that the dispatcher system can be used to uniformly track remote executions.
An assembler is provided to allow configuration of such a dataset such as for use with Fuseki (note that not all configuration options of RDFLinkHTTPBuilder are yet wired up).

• In Fuseki, a plugin is provided that exposes the tracked executions via a JSON-returning REST method and a corresponding basic HTML view.

The pivotal question is, whether abstraction of remote endpoints should be done via DatasetGraph in the first place.
But since most of Jena's architecture is built around it, it seemed reasonable to me to try to build upon this abstraction.

[afs]

But it is possible ....

 QueryExecHTTPBuilder qExecB = (QueryExecHTTPBuilder)link.newQuery();
 QueryExec qExec = qExecB.queryString(queryString).build();

Only the HTTP variants have the queryString option.

That could be exposed, defaulting to query(String) for non-HTTP.

[Aklakan]

The thing is that in this example the query goes via QueryExec.dataset(dsg) - at this level the machinery does not see any HTTP involvement. By default it makes sense to apply parse check immediately when a conventional Jena dataset is involved.
DatasetGraphOverSparql.initContext() sets the newly introduced ARQConstants.parseCheck=false for its context. This flag is picked up by QueryExecBuilderDataset.
Calling QueryExecBuilder.build() then forwards the query string to SparqlDispatcherRegistry .
ChainingQueryDispatcherForDatasetGraphOverRDFLink.java handles DatasetGraphOverRDFLink and forwards the query to the RDFLink.
The handler is registered to the global SparqlDispatcherRegistry in jena-rdfconnection.

[afs]

By default it makes sense to apply parse check immediately when a conventional Jena dataset is involved.

Right - QueryExec.queryString could default to calling QueryExec.query, while the remote case has bypass behaviour.

At sometime (!!) maybe there'll be different local query engines.

[Aklakan]

This class should be removed - it precedes ExampleDBpediaViaRemoteDataset.

[afs]

Feel free to remove it.

[Aklakan]

addEndpoint with a custom context.

[afs]

Sorry if these are a bit scattered. I'm trying to work my way through this valuable new feature.

[afs]

I don't think we need this.

executeJDK* is a system primitive and having one operation (+ non-async) makes that clearer IMO.

[afs]

Please put in the javadoc, tweaking it with

Call {@link HttpClient} after applying an active {@link AuthRequestModifier}

[afs]

Endpoints come from Endpoint.Builder - e.g. FusekiServer.serviceEndpointOperation - this decision could be in the builder.

[afs]

Feel free to remove it.

[afs]

By default it makes sense to apply parse check immediately when a conventional Jena dataset is involved.

Right - QueryExec.queryString could default to calling QueryExec.query, while the remote case has bypass behaviour.

At sometime (!!) maybe there'll be different local query engines.

[afs]

Optional?

[afs]

Some explanatory javadoc, please.

[afs]

Some explanatory javadoc, please.

[afs]

Decide on the contract!

[afs]

This can be removed because it is included with artifact junit-jupiter.

[afs]

Version managed in the top POM. It is currently 4.35.0.

[afs]

Version managed by top POM - it is currently 6.2.0.

[afs]

This can be removed (changes in TestFMod_ExecTracker)

[afs]

Suggested change

	@Ignore
	@Disabled

[afs]

Suggested change

	@Before
	@BeforeEach

[afs]

Suggested change

	@After
	@AfterEach

[afs]

Personally, I position @AfterEach next to @BeforeEach.

[afs]

JUjit4 -> Junit5:

Suggested change

	import org.junit.After;
	import org.junit.Before;
	import org.junit.Ignore;
	import org.junit.Test;
	import org.junit.jupiter.api.AfterEach;
	import org.junit.jupiter.api.BeforeEach;
	import org.junit.jupiter.api.Disabled;
	import org.junit.jupiter.api.Test;

[afs]

Please document with some javadoc.

[afs]

Please document with some javadoc.

[afs]

I don't understand this comment.

[afs]

I don't see anything making this remote-specific.

It would be weird normally ... maybe for logging/mocking for develoment and testing?

Maybe this class should be alongside the other DatasetGraph.

[afs]

For clarity, please add

     public boolean supportsTransactionAbort() {
        return false;
    }

[afs]

Shouldn't this be exectracker?
What else would be org.apache.jena.fuseki.mod.exec?

[afs]

This ("?command=events")is showing up as (-1, -1) in the log. Maybe this will help:

        action.setResponseStatus(HttpSC.OK_200);

It would also benefit from a try-catch just in case an internal error happens.

[afs]

Please add some javadoc that describe the commands supported.

The fuseki:operation fuseki:tracker will also need documentation for the website.

[afs]

This is not a "Label".

I am seeing the whole query and I'm using short queries.
Is there a way to set the display?

Maybe - strip prefixes, use an abbreviated string of the query.

My first reaction on seeing the query string was that clicking would run the query. Saying "See Details" or some such would be clearer.

[afs]

Does this need to be a new namespace? Are there name clashes?

(it is the usually problem of large namespaces vs remembering several namespaces)

[afs]

Some explanatory javadoc, please.

[afs]

"git rebase main" works with one update for some Junit4 code.