ArrowInvalid: Column 10 named vocab expected length 1000 but got length 1

[talhaanwarch]

I am trying to build data from ASR. This is how i created the data

from datasets import load_dataset
data_files = {"train": "../train.tsv", "val": "../dev.tsv"}
urdata = load_dataset("csv", delimiter='\t',data_files=data_files)

urdata is

DatasetDict({
    train: Dataset({
        features: ['client_id', 'path', 'sentence', 'up_votes', 'down_votes', 'age', 'gender', 'accents', 'locale', 'segment'],
        num_rows: 3674
    })
    val: Dataset({
        features: ['client_id', 'path', 'sentence', 'up_votes', 'down_votes', 'age', 'gender', 'accents', 'locale', 'segment'],
        num_rows: 2864
    })
})

then i do

def extract_all_chars(batch):
  all_text = " ".join(batch["sentence"])
  vocab = list(set(all_text))
  return {"vocab": [vocab], "all_text": [all_text]}

when i do
vocabs=urdata.map(extract_all_chars,batched=True )
I got this error
ArrowInvalid: Column 10 named vocab expected length 1000 but got length 1

[mariosasko]

Hi! The length of the new columns (1) doesn't match the length of the existing ones (1000; in the batched mode), so you should either drop the existing column names as follows:

vocabs=urdata.map(extract_all_chars,batched=True, remove_columns=urdata.column_names)

Or expand the vocab and all_text columns to match their size (1000).