Proposal: OpenRAG should store uploaded files

[philnash]

Once the OpenRAG ingestion flow has run, it discards uploaded files or temporary files downloaded from connectors.

For files hosted by connectors, this is fine, as when a chunk from one of those locations is cited, you can always link to the file’s location.

For manually uploaded files, this is not the case. OpenRAG doesn’t know where the file originally came from or how to reference it. If an uploaded file is cited in a response, the best we can do is link to the list of chunks that are stored for that file. But chunks are not a meaningful way to interact with the original data for a user.

Storing the uploaded files mean that we can link to the file so that users can see the relevant content in context. We could even use Docling outputs to highlight the chunk within the file to make it easier to read around and confirm the response from the agent.

[philnash]

One further thing, that I just remembered. Storing the uploaded files would make re-ingestion easy too. If you change your ingestion settings and want to re-ingest a file, at the moment you need to find it again yourself. If it was stored, you could simply click a button to re-ingest.