Sharing vocab

[raheelqader]

I know this question has been asked before but no clear answers were given. I'm wondering if its now possible to share the vocab across fields even if they use different special tokens. For example when the trg field has bos and eos tokens but the src doesn't.

[raheelqader]

I found a solution:
trg_field.build_vocab(train_dataset.src, train_dataset.trg, max_size=vocab_size)
src_field.vocab = trg_field.vocab

The src_field's vocabulary will contain the sos and eos tokens which are not necessary but thats OK I think.

[mttk]

Essentially, your solution is something that can work. The vocab is just an attribute, and you can manually assign a built vocab to another Field. There should be some support for initializing a Vocab based on an already built one, or based on some frequencies. I'll look into it.

[zhangguanheng66]

@raheelqader feel free to re-open the issue if you have more questions.