Incremental Watched Queries

[stevensJourney]

Overview

Our current Watched query implementations emit results whenever a change to a dependant SQLite table occurs. The table changes might not affect the query result set, but we still query and emit a new result set for each table change. The result sets typically contain the same data, but these results are new Array/object references which will cause re-renders in certain frameworks like React.

This PR overhauls, improves and extends upon the existing watched query implementations by introducing incremental watched queries.

Incrementally Watched Queries can be constructed with varying behaviour. This PR introduces the concept of comparison and differential watched queries.

Comparison based queries behave similar to standard watched queries. These queries still query the SQLite DB under the hood on each dependant table change, but they compare the result set and only incrementally yield results if a change has been made. The latest query result is yielded as the result set.

Differential queries watch a SQL query and report detailed information on the changes between result sets. This gives additional information such as the added, removed, updated rows between result set changes.

Implementation

The logic required for incrementally watched queries requires additional computation and introduces additional complexity to the implementation. For these reasons a new concept of a WatchedQuery class is introduced, along with a new query method allows building a instances of WatchedQuerys via watch and differentialWatch methods.

// Create an instance of a WatchedQuery.
const listsQuery = powersync
  .query<EnhancedListRecord>({
    sql: /* sql */ `
      SELECT
        ${LISTS_TABLE}.*,
        COUNT(${TODOS_TABLE}.id) AS total_tasks,
        SUM(
          CASE
            WHEN ${TODOS_TABLE}.completed = true THEN 1
            ELSE 0
          END
        ) as completed_tasks
      FROM
        ${LISTS_TABLE}
        LEFT JOIN ${TODOS_TABLE} ON ${LISTS_TABLE}.id = ${TODOS_TABLE}.list_id
      GROUP BY
        ${LISTS_TABLE}.id;
    `
  })
  .differentialWatch();

The listsQuery is smart, it:

• Automatically reprocesses itself if the PowerSync schema has been updated with updateSchema
• Automatically closes itself when the PowerSync client has been closed.
• Allows for the query parameters to be updated after instantiation.
• Allows shared listening to state changes.

// The registerListener method can be used multiple times to listen for updates.
// The returned dispose function can be used to unsubscribe from the updates.
const disposeSubscriber = listsQuery.registerListener({
  onData: (data) => {
    // This callback will be called whenever the data changes.
    // The data is the result of the executor.
    console.log('Data updated:', data);
  },
  onDiff: (diff) => {
    // This callback will be called whenever the data changes.
    console.log('Data updated:', diff.added, diff.updated);
  },
  onStateChange: (state) => {
    // This callback will be called whenever the state changes.
    // The state contains metadata about the query, such as isFetching, isLoading, etc.
    console.log(
      'State changed:', 
      state.error, 
      state.isFetching, 
      state.isLoading, 
      state.data
    );
  },
  onError: (error) => {
    // This callback will be called if the query fails.
    console.error('Query error:', error);
  }
});

WatchedQuery instances retain the latest state in memory. Sharing WatchedQuery instances can be used to introduce caching and reduce the number of duplicate DB queries between components.

The incremental logic is customisable. Diff based queries can specify custom logic for performing diffs on the relevant data set. By default a JSON.stringify approach is used. Different data sets might have more optimal implementations.

const watch = powersync
  .query({
    sql: /* sql */ `
      SELECT
        *
      FROM
        assets
    `,
    mapper: (raw) => {
      return {
        id: raw.id as string,
        make: raw.make as string
      };
    }
  })
  .differentialWatch({
    comparator: {
      keyBy: (item) => item.id,
      compareBy: (item) => JSON.stringify(item)
    }
  });

Updates to query parameters can be performed in a single place, affecting all subscribers.

watch.updateSettings({
      query: new GetAllQuery({ sql: `SELECT * FROM assets OFFSET ? LIMIT 100`, parameters: [newOffset] })
    });

Reactivity

The existing watch method and Reactivity packages have been updated to use incremental queries with differentiation defined as an opt-in feature (defaults to no changes).

powersync.watch(
  'select * from assets',
  [],
  {
    onResult: () => {
      // This callback will be called whenever the data changes.
      console.log('Data updated');
    }
  },
  {
    comparator: {
      checkEquality: (current, previous) => {
        // This comparator will only report updates if the data changes.
        return JSON.stringify(current) === JSON.stringify(previous);
      }
    }
  }
);

/// React hooks
const { data, isLoading, isFetching } = useQuery(`SELECT * FROM cats WHERE breed = 'tabby'`, [], {
      comparator: {
            keyBy: (item) => item.id,
            compareBy: (item) => JSON.stringify(item)
      }
  })

New hooks have also been added to use shared WatchedQuery instances.

// Managing the WatchedQuery externally can extend its lifecycle and allow in-memory caching between components.
const assetsQuery = powersync.query({
/// ....
}).watch();

/// In the component
export const MyComponent = () => {
   // In React one could import the `assetsQuery` or create a context provider for various queries
   const { data } = useWatchedQuerySubscription(assetsQuery)
}

The Vue and React hooks packages have been updated internally to remove duplicate implementations of common watched query logic. Historically these packages relied on manually implementing watched queries based off the onChange API in order to cater for exposing additional state and custom query executors. The new WatchedQuery APIs now support all the hook packages' requirements - this effectively reduces the heavy lifting in reactivity packages.

The React Supabase Todolist demo has been updated with some best practices for reducing re-renders using comparison based incrementally watched queries.

Differential Queries

These watched queries report the changes between result sets. A WatchedQueryDifferential can be accessed by registering a listener on the query.

export interface WatchedQueryDifferential<RowType> {
  readonly added: ReadonlyArray<Readonly<RowType>>;
  /**
   * The entire current result set.
   * Array item object references are preserved between updates if the item is unchanged.
   *
   * e.g. In the query
   * ```sql
   *  SELECT name, make FROM assets ORDER BY make ASC;
   * ```
   *
   * If a previous result set contains an item (A) `{name: 'pc', make: 'Cool PC'}` and
   * an update has been made which adds another item (B) to the result set (the item A is unchanged) - then
   * the updated result set will be contain the same object reference, to item A, as the previous result set.
   * This is regardless of the item A's position in the updated result set.
   */
  readonly all: ReadonlyArray<Readonly<RowType>>;
  readonly removed: ReadonlyArray<Readonly<RowType>>;
  readonly updated: ReadonlyArray<WatchedQueryRowDifferential<Readonly<RowType>>>;
  readonly unchanged: ReadonlyArray<Readonly<RowType>>;
}

// Listening
watchedQuery.registerListener({
    onDiff: (diff) => console.log(diff)
})

A common use case for this is processing newly created items as they are added. The YJS React Supabase Text Collab demo has been updated to take advantage of this feature. Document updates are watched via a differential incremental query. New updates are passed to YJS for consolidation as they are synced.

Early Access

This can be tested by applying the following package versions to your project.

{
  "@powersync/common": "0.0.0-dev-20250722092404",
  "@powersync/node": "0.0.0-dev-20250722092404",
  "@powersync/op-sqlite": "0.0.0-dev-20250722092404",
  "@powersync/react": "0.0.0-dev-20250722092404",
  "@powersync/react-native": "0.0.0-dev-20250722092404",
  "@powersync/tanstack-react-query": "0.0.0-dev-20250722092404",
  "@powersync/vue": "0.0.0-dev-20250722092404",
  "@powersync/web": "0.0.0-dev-20250722092404"
}

[changeset-bot]

🦋 Changeset detected

Latest commit: 21cc852

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 9 packages

Name	Type
@powersync/common	Minor
@powersync/vue	Minor
@powersync/react	Minor
@powersync/web	Minor
@powersync/node	Patch
@powersync/op-sqlite	Patch
@powersync/react-native	Patch
@powersync/tanstack-react-query	Patch
@powersync/diagnostics-app	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[stevensJourney]

Updates:

• I've added the useWatchedQuerySubscription composable to the Vue helpers package.
• Based off feedback, I've also renamed the differentiator option for instantiating differentially watched queries to comparator