Concat if exist bug

[sumukshashidhar]

When concat_if_exist is set, then uploading to huggingface fails if the dataset doesn't already exist.

[alexbrooker]

even creating the dataset manually first does not resolve the issue as the code tries to download existing dataset from HF

[patrickfleith]

Hi, just tested this and I observed no issues when concat_if_exist is set to true when the dataset on huggingface does not exist, it just get created and files pushed.

[patrickfleith]

However, appart from the issue mentionned above, I think it does not behave well when set to true. In fact, if I run it with a fresh new document, it seems that the existing pushed data are pulled and might interfer and lead to duplicates. I haven't fully tested this but will try to have a look in the coming days.

[patrickfleith]

However, appart from the issue mentionned above, I think it does not behave well when set to true. In fact, if I run it with a fresh new document, it seems that the existing pushed data are pulled and might interfer and lead to duplicates. I haven't fully tested this but will try to have a look in the coming days.

Actually, I didn't find an issue with concat_if_exist=true as long as a user you:

• remove your raw document previously used from source_documents_dir
• remove the local copy of saved_dataset.

However if the local copy of saved_dataset isn't removed, there are some duplicates created