Migrate to multiplatform SQLite library #6202

FloEdelmann · 2025-04-08T20:16:05Z

Attempt to fix #5417.

As mentioned in #6119, the multiplatform Jetpack SQLite library is quite bare-bones, so a lot of stuff that was previously handled by the Android SQLite library now has to be implemented on our side.

FloEdelmann · 2025-04-08T20:28:29Z

app/src/main/java/de/westnordost/streetcomplete/data/StreetCompleteDatabase.kt

+    override fun <T> transaction(block: () -> T): T {
+        databaseConnection.execSQL("BEGIN IMMEDIATE TRANSACTION")
+        try {
+            val result = block()
+            databaseConnection.execSQL("END TRANSACTION")
+            return result
+        } catch (t: Throwable) {
+            databaseConnection.execSQL("ROLLBACK TRANSACTION")
+            throw t
+        }
+    }


This doesn't work correctly yet: I consistently get an error when downloading data:

Unable to download android.database.SQLException: Error code: 5, message: cannot commit transaction - SQL statements in progress at androidx.sqlite.driver.bundled.BundledSQLiteStatementKt.nativeStep(Native Method) at androidx.sqlite.driver.bundled.BundledSQLiteStatementKt.access$nativeStep(BundledSQLiteStatement.jvmAndroid.kt:1) at androidx.sqlite.driver.bundled.BundledSQLiteStatement.step(BundledSQLiteStatement.jvmAndroid.kt:100) at androidx.sqlite.SQLite.execSQL(SQLite.kt:56) at de.westnordost.streetcomplete.data.StreetCompleteDatabase.transaction(StreetCompleteDatabase.kt:130) at de.westnordost.streetcomplete.data.osm.mapdata.WayDao.getAll(WayDao.kt:67) at de.westnordost.streetcomplete.data.osm.mapdata.MapDataController$getWays$1.invoke(MapDataController.kt:193) at de.westnordost.streetcomplete.data.osm.mapdata.MapDataController$getWays$1.invoke(MapDataController.kt:193) at de.westnordost.streetcomplete.data.osm.mapdata.MapDataCache.getWays$lambda$26(MapDataCache.kt:265) at de.westnordost.streetcomplete.data.osm.mapdata.MapDataCache.$r8$lambda$ZWQirpDjfvRnK1EiAAaEtzZT8dc(Unknown Source:0) at de.westnordost.streetcomplete.data.osm.mapdata.MapDataCache$$ExternalSyntheticLambda8.invoke(D8$$SyntheticClass:0) at de.westnordost.streetcomplete.data.osm.mapdata.MapDataCache.getElements(MapDataCache.kt:235) at de.westnordost.streetcomplete.data.osm.mapdata.MapDataCache.getWays(MapDataCache.kt:265) at de.westnordost.streetcomplete.data.osm.mapdata.MapDataController.getWays(MapDataController.kt:193) at de.westnordost.streetcomplete.data.osm.mapdata.MapDataController.completeMapData(MapDataController.kt:151) at de.westnordost.streetcomplete.data.osm.mapdata.MapDataController.putAllForBBox(MapDataController.kt:65) at de.westnordost.streetcomplete.data.osm.mapdata.MapDataDownloader$download$2.invokeSuspend(MapDataDownloader.kt:29)

The Android SQLite library seems to have done much more than this function currently does: https://android.googlesource.com/platform/frameworks/base/+/refs/heads/main/core/java/android/database/sqlite/SQLiteSession.java

But I have no idea how much of that is needed. If it's everything, then it's a lot, and I think we should then wait until the multiplatform Jetpack SQLite library has some convenience functions around transactions.

I'm no expert on Android side, but that SQLite error "cannot commit transaction - SQL statements in progress" sounds like there are SQL operations running in some other thread too? That should ideally be avoided IMHO - as the Android SQLite library docs you linked seem to indicate (starting from "Session objects are not thread-safe" and below).

So, additionally to what I wrote before, I've also looked into the issue tracker for androidx.sqlite. There's no feature request for enabling a special syntax like this for transactions. And I doubt they'd add it, because the interface for the SQLiteConnection is really minimal and clean, they'd certainly not add nooks and crannies like this.

So, only given that my previous assumption was right, I think the only solution is to

remove this method

wherever it is used, reconsider if it was necessary (i.e. for data integrity)

and if it was necessary, directly write the contents of the whole transactions as a string.

However, before putting in the effort to rewrite all this stuff, because it is quite a bit, it would make sense to confirm if this is really the issue here. How? Write some unit tests that try to reproduce the issue.

For what it's worth, the SQLite docs state that you can't nest transactions. But we inherently do. For example in QuestTypeOrderDao.setAll: It is a transaction, but insertMany inherently also starts a transaction. I don't know how the Android SQlite API handles this.

westnordost · 2025-04-09T18:29:03Z

Wow, awesome! You did it! Unfortunately I won't be able to review this until sometime next week probably. One comment from scrolling through it vertically: The API of the old wrapper interface was basically a copy of the Android one, because that's what was exposed on Android. The new interface doesn't *need* to copy that as long as it remains well readable. Don't know regarding transactions. Did you check the syntax for it in the sqlite docs? El 8 de abril de 2025 22:28:50 CEST, Flo Edelmann ***@***.***> escribió:

…

@FloEdelmann commented on this pull request. > + override fun <T> transaction(block: () -> T): T { + databaseConnection.execSQL("BEGIN IMMEDIATE TRANSACTION") + try { + val result = block() + databaseConnection.execSQL("END TRANSACTION") + return result + } catch (t: Throwable) { + databaseConnection.execSQL("ROLLBACK TRANSACTION") + throw t + } + } This doesn't work correctly yet: I consistently get an error when downloading data: ``` Unable to download android.database.SQLException: Error code: 5, message: cannot commit transaction - SQL statements in progress at androidx.sqlite.driver.bundled.BundledSQLiteStatementKt.nativeStep(Native Method) at androidx.sqlite.driver.bundled.BundledSQLiteStatementKt.access$nativeStep(BundledSQLiteStatement.jvmAndroid.kt:1) at androidx.sqlite.driver.bundled.BundledSQLiteStatement.step(BundledSQLiteStatement.jvmAndroid.kt:100) at androidx.sqlite.SQLite.execSQL(SQLite.kt:56) at de.westnordost.streetcomplete.data.StreetCompleteDatabase.transaction(StreetCompleteDatabase.kt:130) at de.westnordost.streetcomplete.data.osm.mapdata.WayDao.getAll(WayDao.kt:67) at de.westnordost.streetcomplete.data.osm.mapdata.MapDataController$getWays$1.invoke(MapDataController.kt:193) at de.westnordost.streetcomplete.data.osm.mapdata.MapDataController$getWays$1.invoke(MapDataController.kt:193) at de.westnordost.streetcomplete.data.osm.mapdata.MapDataCache.getWays$lambda$26(MapDataCache.kt:265) at de.westnordost.streetcomplete.data.osm.mapdata.MapDataCache.$r8$lambda$ZWQirpDjfvRnK1EiAAaEtzZT8dc(Unknown Source:0) at de.westnordost.streetcomplete.data.osm.mapdata.MapDataCache$$ExternalSyntheticLambda8.invoke(D8$$SyntheticClass:0) at de.westnordost.streetcomplete.data.osm.mapdata.MapDataCache.getElements(MapDataCache.kt:235) at de.westnordost.streetcomplete.data.osm.mapdata.MapDataCache.getWays(MapDataCache.kt:265) at de.westnordost.streetcomplete.data.osm.mapdata.MapDataController.getWays(MapDataController.kt:193) at de.westnordost.streetcomplete.data.osm.mapdata.MapDataController.completeMapData(MapDataController.kt:151) at de.westnordost.streetcomplete.data.osm.mapdata.MapDataController.putAllForBBox(MapDataController.kt:65) at de.westnordost.streetcomplete.data.osm.mapdata.MapDataDownloader$download$2.invokeSuspend(MapDataDownloader.kt:29) ``` The [Android SQLite library](https://developer.android.com/training/data-storage/sqlite?hl=en) seems to have done much more than this function currently does: https://android.googlesource.com/platform/frameworks/base/+/refs/heads/main/core/java/android/database/sqlite/SQLiteSession.java But I have no idea how much of that is needed. If it's everything, then it's a lot, and I think we should then wait until the [multiplatform Jetpack SQLite library](https://developer.android.com/kotlin/multiplatform/sqlite) has some convenience functions around transactions. -- Reply to this email directly or view it on GitHub: #6202 (review) You are receiving this because your review was requested. Message ID: ***@***.***>

westnordost · 2025-04-10T15:00:44Z

Ah, of course it doesnt work! Well, at least, this is what I am thinking right now: A transaction, as any SQL statement is written as one string. When you send off "begin transaction" as one sql statement, that statement in itself is already one transaction (containing that statement). If it doesn't already fail there (because within the statement, you didn't specify the transaction's end, which might be a SQL syntax error), then it will fail at "end transaction" because it's like, "what transaction?" (the two sql statements are not connected in any way). El 8 de abril de 2025 22:28:50 CEST, Flo Edelmann ***@***.***> escribió:

…

@FloEdelmann commented on this pull request. > + override fun <T> transaction(block: () -> T): T { + databaseConnection.execSQL("BEGIN IMMEDIATE TRANSACTION") + try { + val result = block() + databaseConnection.execSQL("END TRANSACTION") + return result + } catch (t: Throwable) { + databaseConnection.execSQL("ROLLBACK TRANSACTION") + throw t + } + } This doesn't work correctly yet: I consistently get an error when downloading data: ``` Unable to download android.database.SQLException: Error code: 5, message: cannot commit transaction - SQL statements in progress at androidx.sqlite.driver.bundled.BundledSQLiteStatementKt.nativeStep(Native Method) at androidx.sqlite.driver.bundled.BundledSQLiteStatementKt.access$nativeStep(BundledSQLiteStatement.jvmAndroid.kt:1) at androidx.sqlite.driver.bundled.BundledSQLiteStatement.step(BundledSQLiteStatement.jvmAndroid.kt:100) at androidx.sqlite.SQLite.execSQL(SQLite.kt:56) at de.westnordost.streetcomplete.data.StreetCompleteDatabase.transaction(StreetCompleteDatabase.kt:130) at de.westnordost.streetcomplete.data.osm.mapdata.WayDao.getAll(WayDao.kt:67) at de.westnordost.streetcomplete.data.osm.mapdata.MapDataController$getWays$1.invoke(MapDataController.kt:193) at de.westnordost.streetcomplete.data.osm.mapdata.MapDataController$getWays$1.invoke(MapDataController.kt:193) at de.westnordost.streetcomplete.data.osm.mapdata.MapDataCache.getWays$lambda$26(MapDataCache.kt:265) at de.westnordost.streetcomplete.data.osm.mapdata.MapDataCache.$r8$lambda$ZWQirpDjfvRnK1EiAAaEtzZT8dc(Unknown Source:0) at de.westnordost.streetcomplete.data.osm.mapdata.MapDataCache$$ExternalSyntheticLambda8.invoke(D8$$SyntheticClass:0) at de.westnordost.streetcomplete.data.osm.mapdata.MapDataCache.getElements(MapDataCache.kt:235) at de.westnordost.streetcomplete.data.osm.mapdata.MapDataCache.getWays(MapDataCache.kt:265) at de.westnordost.streetcomplete.data.osm.mapdata.MapDataController.getWays(MapDataController.kt:193) at de.westnordost.streetcomplete.data.osm.mapdata.MapDataController.completeMapData(MapDataController.kt:151) at de.westnordost.streetcomplete.data.osm.mapdata.MapDataController.putAllForBBox(MapDataController.kt:65) at de.westnordost.streetcomplete.data.osm.mapdata.MapDataDownloader$download$2.invokeSuspend(MapDataDownloader.kt:29) ``` The [Android SQLite library](https://developer.android.com/training/data-storage/sqlite?hl=en) seems to have done much more than this function currently does: https://android.googlesource.com/platform/frameworks/base/+/refs/heads/main/core/java/android/database/sqlite/SQLiteSession.java But I have no idea how much of that is needed. If it's everything, then it's a lot, and I think we should then wait until the [multiplatform Jetpack SQLite library](https://developer.android.com/kotlin/multiplatform/sqlite) has some convenience functions around transactions. -- Reply to this email directly or view it on GitHub: #6202 (review) You are receiving this because your review was requested. Message ID: ***@***.***>

westnordost · 2025-04-15T12:37:56Z

app/src/main/java/de/westnordost/streetcomplete/data/StreetCompleteDatabase.kt

+    private fun index(columnName: String): Int {
+        val index = columnNames.indexOf(columnName)
+        require(index != -1) { "Column $columnName not found" }
+        return index
+    }


I think this is going to be kind of slow. After all, getDouble etc. are called on every single iteration when going through the result of an SQL query. Better associate the column names by name (Map<String, Int>).

westnordost · 2025-04-15T12:47:27Z

app/src/main/java/de/westnordost/streetcomplete/data/StreetCompleteDatabase.kt

+    override fun delete(table: String, where: String?, args: Array<Any>?): Int =
+        databaseConnection.prepareDelete(table, where).use { statement ->
+            statement.bindAll(args)
+            statement.toSequence { }.count()


This can't be right. The DELETE SQL query does not return all the rows that have been deleted.

Instead, it seems to return nothing by default (not sure), see https://sqlite.org/lang_delete.html

Maybe the number of affected rows can be accessed by specifying a RETURNING clause, see the linked docs.

In any case, I am not sure if the return value of database.delete is used anywhere at all in a meaningful sense, so maybe that could simply be removed from the interface. You'll need to look at the usages. (Some tests certainly use the return value to confirm that a row really has been deleted, but there are other ways to find out if a row has been deleted, question is whether it is relevant for the production code.)

westnordost · 2025-04-15T12:47:40Z

app/src/main/java/de/westnordost/streetcomplete/data/StreetCompleteDatabase.kt

+    ): Int =
+        databaseConnection.prepareUpdate(table, values, where, conflictAlgorithm).use { statement ->
+            statement.bindAll(args)
+            statement.toSequence { }.count()


See comment for the delete method.

westnordost · 2025-04-15T12:50:13Z

app/src/main/java/de/westnordost/streetcomplete/data/StreetCompleteDatabase.kt

+    }
+
+    val columnNames = if (columns.isNullOrEmpty()) "*" else columns.joinToString(", ")
+    var sql = "SELECT"


Use StringBuilder. Otherwise, every string concatenation creates a new string.

westnordost · 2025-04-15T14:25:09Z

app/src/main/java/de/westnordost/streetcomplete/data/StreetCompleteDatabase.kt

+    override fun <T> transaction(block: () -> T): T {
+        databaseConnection.execSQL("BEGIN IMMEDIATE TRANSACTION")
+        try {
+            val result = block()
+            databaseConnection.execSQL("END TRANSACTION")
+            return result
+        } catch (t: Throwable) {
+            databaseConnection.execSQL("ROLLBACK TRANSACTION")
+            throw t
+        }
+    }


So, additionally to what I wrote before, I've also looked into the issue tracker for androidx.sqlite. There's no feature request for enabling a special syntax like this for transactions. And I doubt they'd add it, because the interface for the SQLiteConnection is really minimal and clean, they'd certainly not add nooks and crannies like this.

So, only given that my previous assumption was right, I think the only solution is to

remove this method

wherever it is used, reconsider if it was necessary (i.e. for data integrity)

and if it was necessary, directly write the contents of the whole transactions as a string.

However, before putting in the effort to rewrite all this stuff, because it is quite a bit, it would make sense to confirm if this is really the issue here. How? Write some unit tests that try to reproduce the issue.

For what it's worth, the SQLite docs state that you can't nest transactions. But we inherently do. For example in QuestTypeOrderDao.setAll: It is a transaction, but insertMany inherently also starts a transaction. I don't know how the Android SQlite API handles this.

westnordost · 2025-04-15T14:36:13Z

app/src/main/java/de/westnordost/streetcomplete/data/StreetCompleteDatabase.kt

+        val statement = databaseConnection.prepareInsert(table, columnNames.toList(), conflictAlgorithm)
+        val result = ArrayList<Long>()
+        transaction {
+            for (values in valuesList) {
+                require(values.size == columnNames.size)
+                statement.bindAll(values)
+                val rowId = statement.executeInsert()
+                result.add(rowId)
+                statement.clearBindings()
+                statement.reset()
+            }
+            statement.close()
+        }
+        return result


By the way, this can potentially be optimized (quite a bit?).

SQLite allows a syntax like this:

INSERT INTO my_table (column_a, column_b, column_c) VALUES (1,2,"a"), (2,4,"b"), (3,1,"c"), (4,5,"d")

to insert 4 rows. I.e. no need for a transaction, no need for repeating the first line x times.

To be honest, I thought I've already long done this, but apparently not.

https://sqlite.org/lang_insert.html

(But before blindly doing a premature optimization, it might be worth it to confirm that this would actually be faster in some unit test or something. Maybe binding, like, up to tens of thousands of parameters to a prepared statement with tens of thousands placeholders comes with implications or limitations in itself. Also, it needn't be part of this PR.)

# Conflicts: # app/build.gradle.kts # app/src/androidMain/kotlin/de/westnordost/streetcomplete/data/AndroidDatabase.kt # app/src/androidMain/kotlin/de/westnordost/streetcomplete/data/StreetCompleteSQLiteOpenHelper.kt # app/src/commonMain/kotlin/de/westnordost/streetcomplete/data/StreetCompleteDatabase.kt

to make sure it doesn't conflict with streetcomplete#6202 performance boost mostly relevant for old phones that can't run maplibre 11

Migrate to multiplatform SQLite library

a7ad63b

FloEdelmann commented Apr 8, 2025

View reviewed changes

FloEdelmann requested a review from westnordost April 8, 2025 20:28

FloEdelmann added this to iOS Port Apr 8, 2025

FloEdelmann moved this to In Progress in iOS Port Apr 8, 2025

FloEdelmann self-assigned this Apr 8, 2025

westnordost reviewed Apr 15, 2025

View reviewed changes

westnordost added the iOS necessary for iOS port label Apr 21, 2025

westnordost mentioned this pull request May 4, 2025

iOS - Planning #5421

Open

Helium314 added a commit to Helium314/SCEE that referenced this pull request May 30, 2025

remove faster sqlite library

38ad8fb

to make sure it doesn't conflict with streetcomplete#6202 performance boost mostly relevant for old phones that can't run maplibre 11

Uh oh!

Migrate to multiplatform SQLite library #6202

Are you sure you want to change the base?

Migrate to multiplatform SQLite library #6202

Uh oh!

Conversation

FloEdelmann commented Apr 8, 2025

Uh oh!

FloEdelmann Apr 8, 2025

Choose a reason for hiding this comment

Uh oh!

mnalis Apr 9, 2025

Choose a reason for hiding this comment

Uh oh!

westnordost Apr 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

westnordost commented Apr 9, 2025 via email

Uh oh!

westnordost commented Apr 10, 2025 via email

Uh oh!

westnordost Apr 15, 2025

Choose a reason for hiding this comment

Uh oh!

westnordost Apr 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

westnordost Apr 15, 2025

Choose a reason for hiding this comment

Uh oh!

westnordost Apr 15, 2025

Choose a reason for hiding this comment

Uh oh!

westnordost Apr 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

westnordost Apr 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

westnordost Apr 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

westnordost Apr 15, 2025 •

edited

Loading

westnordost Apr 15, 2025 •

edited

Loading

westnordost Apr 15, 2025 •

edited

Loading

westnordost Apr 15, 2025 •

edited

Loading

westnordost Apr 16, 2025 •

edited

Loading