Cannot import database in low-ram profile

[retaggi]

ArcadeDB Version: v22.12.1-SNAPSHOT

JDK Version: openjdk 11.0.17 2022-10-18

OS: Ubuntu 22.04.1 LTS

Expected behavior

Even with low-ram profile (128mb ram or lower) I expect to import any database if I have enough space on my disk.

Actual behavior

java.lang.OutOfMemoryError (java heap space)

`
2022-12-14 10:50:15.027 INFO [ArcadeDBServer] <ArcadeDB_0> ArcadeDB Server v22.12.1-SNAPSHOT (build 81566f3/1670890687610/main) is starting up...
2022-12-14 10:50:15.029 INFO [ArcadeDBServer] <ArcadeDB_0> Starting ArcadeDB Server with plugins [Redis, MongoDB, Postgres, GremlinServer] ...
2022-12-14 10:50:15.051 INFO [ArcadeDBServer] <ArcadeDB_0> - JMX Metrics Started...
2022-12-14 10:50:15.090 INFO [ServerSecurity] <ArcadeDB_0> Creating root user with the provided password
2022-12-14 10:50:15.224 INFO [ArcadeDBServer] <ArcadeDB_0> Creating default database 'Imported'...
2022-12-14 10:50:15.391 INFO [SourceDiscovery] <ArcadeDB_0> Analyzing url: https://github.com/ArcadeData/arcadedb-datasets/raw/main/orientdb/OpenBeer.gz...
2022-12-14 10:50:16.343 INFO [SourceDiscovery] <ArcadeDB_0> Recognized format OrientDB (parsingLimitBytes=9.54MB parsingLimitEntries=0)
Importing OrientDB database from stream to './databases/Imported'
Creation of the schema: types, properties and indexes

• Created type 'V' with the following properties {}
• Created type 'Beer' with the following properties {id=INTEGER}
• Created type 'Brewery' with the following properties {id=INTEGER}
• Created type 'Category' with the following properties {id=INTEGER}
• Created type 'E' with the following properties {}
• Created type 'HasBrewery' with the following properties {}
• Created type 'HasCategory' with the following properties {}
• Created type 'HasStyle' with the following properties {}
• Created type 'Style' with the following properties {id=INTEGER}
• Created index UNIQUE on Beer[id]
• Created index UNIQUE on Brewery[id]
• Created index UNIQUE on Style[id]
  java.lang.OutOfMemoryError: Java heap space
  Dumping heap to java_pid1.hprof ...
  Heap dump file created [81224816 bytes in 0.106 secs]
• Status update: parsed 0 (0/sec) - 0 documents (0/sec) - 0 vertices (0/sec) - 0 edges (0/sec) - 0 skipped edges - 0 linked edges (0/sec - 0%)
  2022-12-14 10:50:16.344 INFO [Importer] <ArcadeDB_0> Checking schema...Exception in thread "main" com.arcadedb.exception.CommandExecutionException: Error on importing database
  at com.arcadedb.query.sql.parser.ImportDatabaseStatement.executeSimple(ImportDatabaseStatement.java:70)
  at com.arcadedb.query.sql.executor.SingleOpExecutionPlan.executeInternal(SingleOpExecutionPlan.java:115)
  at com.arcadedb.query.sql.parser.SimpleExecStatement.execute(SimpleExecStatement.java:56)
  at com.arcadedb.query.sql.parser.Statement.execute(Statement.java:82)
  at com.arcadedb.query.sql.parser.Statement.execute(Statement.java:66)
  at com.arcadedb.query.sql.SQLQueryEngine.command(SQLQueryEngine.java:88)
  at com.arcadedb.database.EmbeddedDatabase.command(EmbeddedDatabase.java:1223)
  at com.arcadedb.server.ArcadeDBServer.loadDefaultDatabases(ArcadeDBServer.java:468)
  at com.arcadedb.server.ArcadeDBServer.start(ArcadeDBServer.java:130)
  at com.arcadedb.server.ArcadeDBServer.main(ArcadeDBServer.java:91)
  Caused by: com.arcadedb.integration.importer.ImportException: Error on parsing source 'https://github.com/ArcadeData/arcadedb-datasets/raw/main/orientdb/OpenBeer.gz (compressed=true size=1085781)'
  at com.arcadedb.integration.importer.Importer.load(Importer.java:62)
  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.base/java.lang.reflect.Method.invoke(Method.java:566)
  at com.arcadedb.query.sql.parser.ImportDatabaseStatement.executeSimple(ImportDatabaseStatement.java:65)
  ... 9 more
  Caused by: com.arcadedb.index.IndexException: Error on creating index on type 'Category', properties [id]
  at com.arcadedb.schema.EmbeddedSchema.createTypeIndex(EmbeddedSchema.java:506)
  at com.arcadedb.schema.EmbeddedSchema.getOrCreateTypeIndex(EmbeddedSchema.java:541)
  at com.arcadedb.integration.importer.OrientDBImporter.parseIndexes(OrientDBImporter.java:772)
  at com.arcadedb.integration.importer.OrientDBImporter.createRecord(OrientDBImporter.java:424)
  at com.arcadedb.integration.importer.OrientDBImporter.createRecords(OrientDBImporter.java:384)
  at com.arcadedb.integration.importer.OrientDBImporter.parseRecords(OrientDBImporter.java:333)
  at com.arcadedb.integration.importer.OrientDBImporter.parseInputFile(OrientDBImporter.java:286)
  at com.arcadedb.integration.importer.OrientDBImporter.run(OrientDBImporter.java:175)
  at com.arcadedb.integration.importer.format.OrientDBImporterFormat.load(OrientDBImporterFormat.java:48)
  at com.arcadedb.integration.importer.Importer.loadFromSource(Importer.java:90)
  at com.arcadedb.integration.importer.Importer.load(Importer.java:53)
  ... 14 more
  Caused by: com.arcadedb.exception.DatabaseOperationException: Error during read lock
  at com.arcadedb.database.EmbeddedDatabase.executeInReadLock(EmbeddedDatabase.java:1357)
  at com.arcadedb.database.EmbeddedDatabase.commit(EmbeddedDatabase.java:359)
  at com.arcadedb.database.EmbeddedDatabase.transaction(EmbeddedDatabase.java:998)
  at com.arcadedb.schema.EmbeddedSchema.lambda$createTypeIndex$4(EmbeddedSchema.java:484)
  at com.arcadedb.database.EmbeddedDatabase.executeInWriteLock(EmbeddedDatabase.java:1372)
  at com.arcadedb.database.EmbeddedDatabase.recordFileChanges(EmbeddedDatabase.java:1392)
  at com.arcadedb.schema.EmbeddedSchema.recordFileChanges(EmbeddedSchema.java:1370)
  at com.arcadedb.schema.EmbeddedSchema.createTypeIndex(EmbeddedSchema.java:483)
  ... 24 more
  Caused by: java.lang.OutOfMemoryError: Java heap space
  at com.arcadedb.database.Binary.(Binary.java:64)
  at com.arcadedb.database.Binary.(Binary.java:71)
  at com.arcadedb.engine.WALFile.writeTransactionToBuffer(WALFile.java:237)
  at com.arcadedb.engine.TransactionManager.createTransactionBuffer(TransactionManager.java:138)
  at com.arcadedb.database.TransactionContext.commit1stPhase(TransactionContext.java:507)
  at com.arcadedb.database.TransactionContext.commit(TransactionContext.java:114)
  at com.arcadedb.database.EmbeddedDatabase.lambda$commit$2(EmbeddedDatabase.java:364)
  at com.arcadedb.database.EmbeddedDatabase$$Lambda$111/0x0000000840128c40.call(Unknown Source)
  at com.arcadedb.database.EmbeddedDatabase.executeInReadLock(EmbeddedDatabase.java:1346)
  at com.arcadedb.database.EmbeddedDatabase.commit(EmbeddedDatabase.java:359)
  at com.arcadedb.database.EmbeddedDatabase.transaction(EmbeddedDatabase.java:998)
  at com.arcadedb.schema.EmbeddedSchema.lambda$createTypeIndex$4(EmbeddedSchema.java:484)
  at com.arcadedb.schema.EmbeddedSchema$$Lambda$156/0x0000000840207840.call(Unknown Source)
  at com.arcadedb.database.EmbeddedDatabase.executeInWriteLock(EmbeddedDatabase.java:1372)
  at com.arcadedb.database.EmbeddedDatabase.recordFileChanges(EmbeddedDatabase.java:1392)
  at com.arcadedb.schema.EmbeddedSchema.recordFileChanges(EmbeddedSchema.java:1370)
  at com.arcadedb.schema.EmbeddedSchema.createTypeIndex(EmbeddedSchema.java:483)
  at com.arcadedb.schema.EmbeddedSchema.getOrCreateTypeIndex(EmbeddedSchema.java:541)
  at com.arcadedb.integration.importer.OrientDBImporter.parseIndexes(OrientDBImporter.java:772)
  at com.arcadedb.integration.importer.OrientDBImporter.createRecord(OrientDBImporter.java:424)
  at com.arcadedb.integration.importer.OrientDBImporter.createRecords(OrientDBImporter.java:384)
  at com.arcadedb.integration.importer.OrientDBImporter.parseRecords(OrientDBImporter.java:333)
  at com.arcadedb.integration.importer.OrientDBImporter.parseInputFile(OrientDBImporter.java:286)
  at com.arcadedb.integration.importer.OrientDBImporter.run(OrientDBImporter.java:175)
  at com.arcadedb.integration.importer.format.OrientDBImporterFormat.load(OrientDBImporterFormat.java:48)
  at com.arcadedb.integration.importer.Importer.loadFromSource(Importer.java:90)
  at com.arcadedb.integration.importer.Importer.load(Importer.java:53)
  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.base/java.lang.reflect.Method.invoke(Method.java:566)
  at com.arcadedb.query.sql.parser.ImportDatabaseStatement.executeSimple(ImportDatabaseStatement.java:65)
  `

Steps to reproduce

docker run -d --rm  -p 2480:2480 -p 2424:2424 -p 6379:6379 -p 5432:5432 -p 8182:8182 \
--name arcadedb \
--env ARCADEDB_OPTS_MEMORY="-Xms128M -Xmx128M" \
--env JAVA_OPTS="-Darcadedb.server.rootPassword=playwithdata \
-Darcadedb.profile=low-ram \
-Darcadedb.server.defaultDatabases=Imported[root]{import:https://github.com/ArcadeData/arcadedb-datasets/raw/main/orientdb/OpenBeer.gz} \
-Darcadedb.server.plugins=Redis:com.arcadedb.redis.RedisProtocolPlugin,MongoDB:com.arcadedb.mongo.MongoDBProtocolPlugin,Postgres:com.arcadedb.postgres.PostgresProtocolPlugin,GremlinServer:com.arcadedb.server.gremlin.GremlinServerPlugin " \
arcadedata/arcadedb:latest

I tried with some tuning like the following, without success:

-Darcadedb.sqlStatementCache=10 \
-Darcadedb.freePageRAM=100 \
-Darcadedb.indexCompactionRAM=1 \
-Darcadedb.maxPageRAM=1024 \

I tried even without defaultDatabases and with import database https://github.com/ArcadeData/arcadedb-datasets/raw/main/orientdb/OpenBeer.gz after the startup of the server.

Thank you

[lvca]

With such low RAM, please try with commitEvery equals to 50 or less:

https://docs.arcadedb.com/#_additional-settings

[retaggi]

i tried with

IMPORT DATABASE https://github.com/ArcadeData/arcadedb-datasets/raw/main/orientdb/OpenBeer.gz
with commitEvery = 50

and

IMPORT DATABASE https://github.com/ArcadeData/arcadedb-datasets/raw/main/orientdb/OpenBeer.gz
with commitEvery = 1

with the same issue

2022-12-15 08:09:03.402 INFO  [SourceDiscovery] <ArcadeDB_0> Analyzing url: https://github.com/ArcadeData/arcadedb-datasets/raw/main/orientdb/OpenBeer.gz...
2022-12-15 08:09:03.884 INFO  [SourceDiscovery] <ArcadeDB_0> Recognized format OrientDB (parsingLimitBytes=9.54MB parsingLimitEntries=0)
Importing OrientDB database from stream to './databases/test'
Creation of the schema: types, properties and indexes
- Created index UNIQUE on Beer[id]
- Created index UNIQUE on Brewery[id]
- Created index UNIQUE on Style[id]
- Created index UNIQUE on Category[id]
Creation of records started: creating vertices and documents records (edges on the next phase)
- Status update: parsed 9,997 (9,997/sec) - 0 documents (0/sec) - 5,417 vertices (5,417/sec) - 0 edges (0/sec) - 0 skipped edges - 0 linked edges (0/sec - 0%)
2022-12-15 08:09:03.885 INFO  [Importer] <ArcadeDB_0> Checking schema...<ArcadeDB_0> Error on command execution (PostCommandHandler)
com.arcadedb.exception.CommandExecutionException: Error on importing database
        at com.arcadedb.query.sql.parser.ImportDatabaseStatement.executeSimple(ImportDatabaseStatement.java:70)
        at com.arcadedb.query.sql.executor.SingleOpExecutionPlan.executeInternal(SingleOpExecutionPlan.java:115)
        at com.arcadedb.query.sql.executor.ScriptLineStep.syncPull(ScriptLineStep.java:53)
        at com.arcadedb.query.sql.executor.ScriptExecutionPlan.doExecute(ScriptExecutionPlan.java:97)
        at com.arcadedb.query.sql.executor.ScriptExecutionPlan.fetchNext(ScriptExecutionPlan.java:57)
        at com.arcadedb.query.sql.parser.LocalResultSet.fetchNext(LocalResultSet.java:46)
        at com.arcadedb.query.sql.parser.LocalResultSet.<init>(LocalResultSet.java:40)
        at com.arcadedb.database.EmbeddedDatabase.executeInternal(EmbeddedDatabase.java:1296)
        at com.arcadedb.database.EmbeddedDatabase.execute(EmbeddedDatabase.java:1244)
        at com.arcadedb.server.ServerDatabase.execute(ServerDatabase.java:415)
        at com.arcadedb.server.http.handler.PostCommandHandler.executeScript(PostCommandHandler.java:100)
        at com.arcadedb.server.http.handler.PostCommandHandler.execute(PostCommandHandler.java:71)
        at com.arcadedb.server.http.handler.DatabaseAbstractHandler.execute(DatabaseAbstractHandler.java:85)
        at com.arcadedb.server.http.handler.AbstractHandler.handleRequest(AbstractHandler.java:109)
        at io.undertow.server.RoutingHandler.handleRequest(RoutingHandler.java:93)
        at io.undertow.server.handlers.PathHandler.handleRequest(PathHandler.java:104)
        at io.undertow.server.Connectors.executeRootHandler(Connectors.java:391)
        at io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:256)
        at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:136)
        at io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:59)
        at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
        at org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66)
        at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89)
        at org.xnio.nio.WorkerThread.run(WorkerThread.java:591)
Caused by: com.arcadedb.integration.importer.ImportException: Error on parsing source 'https://github.com/ArcadeData/arcadedb-datasets/raw/main/orientdb/OpenBeer.gz (compressed=true size=1085781)'
        at com.arcadedb.integration.importer.Importer.load(Importer.java:62)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at com.arcadedb.query.sql.parser.ImportDatabaseStatement.executeSimple(ImportDatabaseStatement.java:65)
        ... 23 more
Caused by: com.arcadedb.exception.TransactionException: Transaction error on commit
        at com.arcadedb.database.TransactionContext.commit1stPhase(TransactionContext.java:518)
        at com.arcadedb.database.TransactionContext.commit(TransactionContext.java:114)
        at com.arcadedb.database.EmbeddedDatabase.lambda$commit$2(EmbeddedDatabase.java:364)
        at com.arcadedb.database.EmbeddedDatabase.executeInReadLock(EmbeddedDatabase.java:1346)
        at com.arcadedb.database.EmbeddedDatabase.commit(EmbeddedDatabase.java:359)
        at com.arcadedb.integration.importer.OrientDBImporter.parseRecords(OrientDBImporter.java:340)
        at com.arcadedb.integration.importer.OrientDBImporter.parseInputFile(OrientDBImporter.java:286)
        at com.arcadedb.integration.importer.OrientDBImporter.run(OrientDBImporter.java:175)
        at com.arcadedb.integration.importer.format.OrientDBImporterFormat.load(OrientDBImporterFormat.java:48)
        at com.arcadedb.integration.importer.Importer.loadFromSource(Importer.java:90)
        at com.arcadedb.integration.importer.Importer.load(Importer.java:53)
        ... 28 more
Caused by: com.arcadedb.exception.ArcadeDBException: Error in execution in lock
        at com.arcadedb.utility.RWLockContext.executeInReadLock(RWLockContext.java:58)
        at com.arcadedb.index.lsm.LSMTreeIndex.put(LSMTreeIndex.java:391)
        at com.arcadedb.database.TransactionIndexContext.commit(TransactionIndexContext.java:191)
        at com.arcadedb.database.TransactionContext.commit1stPhase(TransactionContext.java:474)
        ... 38 more
Caused by: java.lang.OutOfMemoryError: Java heap space

I think that the problem is that, before the parsing of the data, the OpenBeer.gz is fully loaded in RAM and if there is no space, it just raise the OutOfMemoryError exception. I think its missing some logic to chunk the OpenBeer.gz

[lvca]

This is the main loop:

database.begin();

    while (reader.peek() == BEGIN_OBJECT) {
      final Map<String, Object> attributes = parseRecord(reader, false);

      final String className = (String) attributes.get("@class");

      if (phase == PHASE.CREATE_SCHEMA || phase == PHASE.CREATE_RECORDS)
        createRecords(processedItems, attributes, className);
      else
        createEdges(processedItems, attributes, className);

      ++processedItems;

      if (processedItems > 0 && processedItems % batchSize == 0) {
        database.commit();
        database.begin();
      }

      if (reader.peek() == NULL)
        // FIX A BUG ON ORIENTDB EXPORTER WHEN GENERATE AN EMPTY RECORD
        reader.skipValue();
    }

    database.commit();

The input file is read byte per byte and as soon as a record is completed it's imported. So the whole file is not read in RAM first. How much heap memory to the JVM?

[lvca]

Closing the issue because no answer was received.