"internal_error": "iterator 0 is not in ascending order"

[danloiterton]

Hi,

I started seeing this error today, when checking certain relationships - eg:

POST .../stores/{{fga-store-id}}/check with:

{
  "tuple_key": {
    "user": "user:ABC123",
    "relation": "member",
    "object": "project:MYPROJECT"
  }
}

ERROR   rpc error: code = Code(4000) desc = Internal Server Error       {"grpc_service": "openfga.v1.OpenFGAService", "grpc_method": "Check", "grpc_type": "unary", "user_agent": "PostmanRuntime/7.45.0", "raw_request": {"store_id":"01JB24DR66DCNK7YCHR70V59QQ","tuple_key":{"user":"user:___","relation":"member","object":"project:___"},"contextual_tuples":null,"authorization_model_id":"","trace":false,"context":null,"consistency":"UNSPECIFIED"}, "raw_response": {"code":"internal_error","message":"Internal Server Error"}, "query_duration_ms": "181", "request_id": "df21a2d6-41dd-44a6-bdc2-6f82b03b5c6a", "store_id": "01JB24DR66DCNK7YCHR70V59QQ", "authorization_model_id": "01K2YCX3YYHTJ37NEVJNGYFGDZ", "dispatch_count": 0, "datastore_query_count": 5, "request.throttled": false, "peer.address": "127.0.0.1:33980", "grpc_code": 4000, "internal_error": "iterator 0 is not in ascending order"}

The issue is happening on both my production and staging environments.

ChatGPT thinks my postgres DB might be missing this index?

CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_tuples_user_relation_object
ON tuples (user, relation, object);

Indeed, that index does not exist. But we don't usually touch these - we just run the openfga migrate command on each deployment (docker/kubernetes). I assumed this takes care of indexing changes, etc.

I'm not sure what to do. Any help getting to the bottom of it would be appreciated!

[danloiterton]

FYI: these are the indexes currently in the postgres db:

[aaguiarz]

Hi @danloiterton

Which OpenFGA version are you using?

The issue does not seem related to indexes. Can you reproduce it consistently?

[danloiterton]

Hi @aaguiarz, I'm on 1.9.3

I think have narrowed it down to a set of tuples, which consistently give the error on a certain relationship check...

The relationship is basically like this:
user:A > member > team:B,
team:B#member > member > project:C

I get TRUE when checking
user:A > member > team:B

I get TRUE when checking
team:B#member > member > project:C

But I get the error when checking
user:A > member > project:C

Obviously my actual user/team/project ids are not A/B/C. However, I created those exact tuples via the api to try and replicate the issue, and this time the check worked as expected. So I'm guessing there's something wrong with the actual data?

I'm currently trying to delete and recreate tuples to see if I can narrow it down further. Any suggestions would be great though

[aaguiarz]

Are you using contextual tuples?

[danloiterton]

No.

[danloiterton]

Are there any rules or known issues about which characters you can use as object/user ids? I think the issue is related to my use of "-" in certain ids

[aaguiarz]

Do you know how the DB Character set is configured?

[danloiterton]

I don't - but I'll find out

[danloiterton]

When I check the db, it says "Default Encoding: UTF8"

[danloiterton]

OK, I can reliably replicate the issue with certain id values, which contain "-". It seems to be when there are multiple entries with the same prefix before the "-". But not always. This example breaks every time:

Write these:

{
  "writes": {
    "tuple_keys": [
       { "user": "user:A", "relation": "member", "object": "org:myorg" },
       { "user": "user:A", "relation": "member", "object": "team:myorg-TM01" },
       { "user": "user:A", "relation": "member", "object": "team:myorg-TM02" },
       { "user": "user:A", "relation": "member", "object": "team:myorg-all" },
       { "user": "team:myorg-all#member", "relation": "member", "object": "project:C" }
    ]
  }
}

Then check:

{ "tuple_key": { "user": "user:A", "relation": "member", "object": "project:C" } }

Which gives:

{
    "code": "internal_error",
    "message": "Internal Server Error"
}

However, if I simplify my model right down to these relationships only, then the error goes away. So next I'll try and narrow that down. If so, I should have a fully reproducible example.

[aaguiarz]

can you share the model you are using? Can you reproduce it locally targeting an openfga run with 'docker compose up'?

[danloiterton]

Can I share it with you privately somehow?

[aaguiarz]

andres.aguiar at openfga.dev

[danloiterton]

Thanks Andres, I've emailed that through

[harper]

We are getting this error as well. It seems to be affecting users that have more relationships than others.

{
"level": "error",
"timestamp": 1755809325.025032,
"msg": "rpc error: code = Code(4000) desc = Internal Server Error",
"build.version": "v1.9.5",
"build.commit": "06115c523f974d385896b159a60fe6e12c41e57d",
"grpc_service": "openfga.v1.OpenFGAService",
"grpc_method": "Check",
"grpc_type": "unary",
"user_agent": "openfga-sdk go/0.5.0",
"raw_request": {
"store_id": "[redacted]",
"tuple_key": {
"user": "[redacted]",
"relation": "[redacted]",
"object": "[redacted]"
},
"contextual_tuples": {
"tuple_keys": []
},
"authorization_model_id": "[redacted]",
"trace": false,
"context": {
"current_time": "2025-08-21T20:48:43Z"
},
"consistency": "UNSPECIFIED"
},
"raw_response": {
"code": "internal_error",
"message": "Internal Server Error"
},
"query_duration_ms": "875",
"request.throttled": false,
"peer.address": "127.0.0.1:44812",
"request_id": "9ff4fe2e-efc2-44fa-9a7d-7be55da3c436",
"store_id": "[redacted]",
"authorization_model_id": "[redacted]",
"dispatch_count": 4,
"datastore_query_count": 78,
"grpc_code": 4000,
"internal_error": "iterator 0 is not in ascending order"
}

[aaguiarz]

Also, can you share the how you are configuring OpenFGA?

[harper]

Any specific settings you are looking for?

[aaguiarz]

OPENFGA_SHARED_ITERATOR_ENABLED, but maybe you have something else that we are not expecting that helps us take the right path, if you want to share it privately you can email it to andres.aguiar at openfga.dev

Thanks a lot for your help

[harper]

I sent the settings we have set for the servers experiencing the issue.

[harper]

Reverting to v1.8.16 fixed the issue, so the cause of the error above seems to have been introduced between v1.8.16 and v1.9.5.

[danloiterton]

Hi again. I have now created a git repo, which reproduces the error using the latest openfga docker image.

https://github.com/danloiterton/openfga-iterator0-bug

Note:

• I couldn't reproduce it with the in-memory db. So this repo uses a postgres setup.
• I simplified the model and the tests even further. This example has no tuples with '-' characters but still fails.

See repo README for more details.

[aaguiarz]

Thanks @danloiterton , we'll take a look at it

[aaguiarz]

@danloiterton @harper we were able to reproduce the issue and we are working on a fix here openfga/openfga#2661

Thanks a lot for your help!

[danloiterton]

Ahh, that's good news. Many thanks!

[gaheinrichs]

Hey, I am still seeing this error in v1.10.2 (our dev environment)

Applied migration, and here are the indexes in the tuple table:

This is not happening in v1.9.0 (our production environment)

[tylernix]

We will schedule some time to look into this @gaheinrichs.

[tylernix]

@gaheinrichs, which datastore are you using?

[gaheinrichs]

Hey @tylernix, thanks for getting back. We are using Postgres

[gaheinrichs]

We are also using contextual tuples