Wrong ADP POS-Tag allocation in the german model

[Goldritter]

Hi,

I use both the stanford-corenlp and the german model in version "4.2.2" and found some odd behavior in the allocation of POS-Tags für words like "für" and "vor". These two words should be normally designed with "ADP" but I often get "NOUN" or even "PROPN" as POS-Tag for these words.

For example for the sentence "Welcher der Befunde ist für eine Gehirnerkrankung typisch?" "für" is tagged as "PROPN" an in "Welcher der Befunde ist am ehesten für hier am wahrscheinlichsten vorliegende Gehirnerkrankung typisch?" it is designated as "NOUN".

But for the sentences "Für wen ist das Essen?" and "Ich war reif für das Bett." "für" is correctly tagged as "ADP".

Here are the properties I used for the initialization:
(.setProperty properties "annotators" "tokenize, ssplit, mwt , pos, lemma, ner, parse, depparse")
(.setProperty properties "coref.algorithm", "neural")
(.setProperty properties "depparse.language", "german")
(.setProperty properties "ner.language", "de")
(.setProperty properties "tokenize.options" "untokenizable=noneDelete")

These are all the information I have. I hope they are helpfull.

Best regards
Goldritter

[AngledLuffa]

I do not know any German, so I am unable to judge whether this is kinda of in the realm of possibility or just completely nuts.  In looking at the source data, there are a few examples where "für" is tagged as something other than ADP, which is doubtless why the model thinks it is a reasonable choice. If these look wrong to you, we can hopefully fix the issue by fixing the data.

# sent_id = train-s8233
# text = Danach war er zwischen 1933 und 1956 als Chefarzt der Medizinischen Klinik am Kantonsspital Aarau unter anderem für_NOUN den Aufbau eines Grosslabors verantwortlich.

# sent_id = train-s8331
# text = Bei einem Bandwettbewerb im Film spielte McFly den Song auch mit seiner Band "Pinheads", der Vorsitzende der Jury, gespielt von Huey Lewis, nennt den eigenen Song daraufhin "eine \
Spur zu laut" für_NOUN den Schulwettbewerb.

# sent_id = train-s10753
# text = Typisch für_NOUN das Chuanqi ist eine dramatische Liebesgeschichte, die persönliche Schicksale und Machtkämpfe in der Politik darstellt.

# sent_id = train-s9560
 Linda Ronstadt und James Ingram sangen als Duett den im Dezember 1986 publizierten Hit Somewhere Out There (US-Nr. 2), der 1987 zwei Grammy Awards für_PROPN "besten Song des Jahres" und "besten Filmsong"

# sent_id = train-s9866
# text = Für_PROPN ein weltoffenes Deutschland e.V., der sich bundesweit gegen Fremdenfeindlichkeit, Rassismus, Antisemitismus und rechtsextreme Gewalt einsetzt.

# sent_id = test-s420
# text = Deutsche Gesellschaft für_PROPN Soziale Psychiatrie diskutierte langsame Fortschritte bei gemeindenaher Versorgung der Patienten

A couple of these look kind of crappy, but I don't know for sure.

This is the one Vor_NOUN in the training data:

# sent_id = train-s10174
# text = Vor - und frühgeschichtlich von Bedeutung ist ein 70 Meter hoher Bergsporn an der Südwestflanke des Tales, bekannt unter dem Namen Heidenschanze.

[AngledLuffa]

Heh, I actually looked in the stanza UD data, not the corenlp data. I'll check that data as well I suppose...

[AngledLuffa]

Taking a closer look at the tagger model, it was trained from that UD dataset. So presumably the upstream data fix will improve results for the tagger. I'll retrain a model later this week or next week sometime

[Goldritter]

Thank you very much.

And yes the examples you posted above "für" has been tagged wrongly tagged as NOUN or PROPN.
In the last two examples "train-s9866" and "test-s420" the "für" is part of the name of an organization.

In the last exmple "train-s10174" the "-" after the "Vor" has been wrong separated. The start of the sentence should be "Vor- und frühgeschichtlich von Bedeutung ist ein [...]" which means "Vorgeschichtlich und frühgeschichtlich von Bedeutung ist ein [..]" and "vorgeschichtlich" and "frügeschichtlich" are adjectives (ADJ). Here "Vor" is not a NOUN.

Thank you for retraining the model.
Will this model be available via Maven like the one I use and if yes, which version number is used?

[AngledLuffa]

I got a lot going on over the next couple weeks, during which time I'd like to release a new CoreNLP and a new Stanza, so there is basically no chance of me pushing anything new to Maven until then. However, the models are here: https://huggingface.co/stanfordnlp/corenlp-german/tree/v4.3.0 The tagger was retrained with the tag fixes we talked about, which you can see in the original source: https://github.com/UniversalDependencies/UD_German-GSD Also, we're not using the dev set for any aspect of training, so I included it in the training data.

…

On Wed, Sep 8, 2021 at 3:20 AM Marcus Lindner ***@***.***> wrote: Thank you very much. And yes the examples you posted above "für" has been tagged wrongly tagged as NOUN or PROPN. In the last two examples "train-s9866" and "test-s420" the "für" is part of the name of an organization. In the last exmple "train-s10174" the "-" after the "Vor" has been wrong separated. The start of the sentence should be "Vor- und frühgeschichtlich von Bedeutung ist ein [...]" which means "Vorgeschichtlich und frühgeschichtlich von Bedeutung ist ein [..]" and "vorgeschichtlich" and "frügeschichtlich" are adjectives (ADJ). Here "Vor" is not a NOUN. Thank you for retraining the model. Will this model be available via Maven like the one I use and if yes, which version number is used? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1184 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA2AYWLCSC5GAOWPJ6H4MCDUA42FHANCNFSM5DSNPPHA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[Goldritter]

Thanks!

I will test it as soon as I have time.

[Goldritter]

As it seems the new model makes the same mistakes.

I add your new model to my project and tested it.
As you can see in the output below "für" is again tagged as "PROPN" and "NOUN" but not as "ADP". I added also the output for the two sentences where "für" has been correctly tagged as "ADP".

(get-pos-tags-from-document-for-document (generate-document-for-pipeline "Welcher der Befunde ist für eine Gehirnerkrankung typisch?" (get-pipeline-for-language :de)))

[nRepl-session-cbf7e633-5c39-4cbf-88ad-3e00806cd271] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator tokenize
[nRepl-session-cbf7e633-5c39-4cbf-88ad-3e00806cd271] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator ssplit
[nRepl-session-cbf7e633-5c39-4cbf-88ad-3e00806cd271] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator mwt
[nRepl-session-cbf7e633-5c39-4cbf-88ad-3e00806cd271] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator pos
[nRepl-session-cbf7e633-5c39-4cbf-88ad-3e00806cd271] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator ner
=>
[{:category nil, :pos-tag "PRON", :pos 1, :lemma nil, :begin 0, :token "Welcher", :ner "O", :end 7}
 {:category nil, :pos-tag "DET", :pos 2, :lemma nil, :begin 8, :token "der", :ner "O", :end 11}
 {:category nil, :pos-tag "NOUN", :pos 3, :lemma nil, :begin 12, :token "Befunde", :ner "O", :end 19}
 {:category nil, :pos-tag "AUX", :pos 4, :lemma nil, :begin 20, :token "ist", :ner "O", :end 23}
 {:category nil, :pos-tag "PROPN", :pos 5, :lemma nil, :begin 24, :token "für", :ner "O", :end 28}
 {:category nil, :pos-tag "DET", :pos 6, :lemma nil, :begin 29, :token "eine", :ner "O", :end 33}
 {:category nil, :pos-tag "NOUN", :pos 7, :lemma nil, :begin 34, :token "Gehirnerkrankung", :ner "O", :end 50}
 {:category nil, :pos-tag "ADJ", :pos 8, :lemma nil, :begin 51, :token "typisch", :ner "O", :end 58}
 {:category nil, :pos-tag "PUNCT", :pos 9, :lemma nil, :begin 58, :token "?", :ner "O", :end 59}]

(get-pos-tags-from-document-for-document (generate-document-for-pipeline "Welcher der Befunde ist am ehesten für die hier am wahrscheinlichsten vorliegenden Gehirnerkrankung typisch?" (get-pipeline-for-language :de)))
[nRepl-session-cbf7e633-5c39-4cbf-88ad-3e00806cd271] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator tokenize
[nRepl-session-cbf7e633-5c39-4cbf-88ad-3e00806cd271] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator ssplit
[nRepl-session-cbf7e633-5c39-4cbf-88ad-3e00806cd271] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator mwt
[nRepl-session-cbf7e633-5c39-4cbf-88ad-3e00806cd271] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator pos
[nRepl-session-cbf7e633-5c39-4cbf-88ad-3e00806cd271] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator ner
=>
[{:category nil, :pos-tag "PRON", :pos 1, :lemma nil, :begin 0, :token "Welcher", :ner "O", :end 7}
 {:category nil, :pos-tag "DET", :pos 2, :lemma nil, :begin 8, :token "der", :ner "O", :end 11}
 {:category nil, :pos-tag "NOUN", :pos 3, :lemma nil, :begin 12, :token "Befunde", :ner "O", :end 19}
 {:category nil, :pos-tag "AUX", :pos 4, :lemma nil, :begin 20, :token "ist", :ner "O", :end 23}
 {:category nil, :pos-tag "ADP", :pos 5, :lemma nil, :begin 24, :token "an", :ner "O", :end 26}
 {:category nil, :pos-tag "DET", :pos 6, :lemma nil, :begin 24, :token "dem", :ner "O", :end 26}
 {:category nil, :pos-tag "ADJ", :pos 7, :lemma nil, :begin 27, :token "ehesten", :ner "O", :end 34}
 {:category nil, :pos-tag "NOUN", :pos 8, :lemma nil, :begin 35, :token "für", :ner "O", :end 39}
 {:category nil, :pos-tag "DET", :pos 9, :lemma nil, :begin 40, :token "die", :ner "O", :end 43}
 {:category nil, :pos-tag "ADV", :pos 10, :lemma nil, :begin 44, :token "hier", :ner "O", :end 48}
 {:category nil, :pos-tag "ADP", :pos 11, :lemma nil, :begin 49, :token "an", :ner "O", :end 51}
 {:category nil, :pos-tag "DET", :pos 12, :lemma nil, :begin 49, :token "dem", :ner "O", :end 51}
 {:category nil, :pos-tag "ADJ", :pos 13, :lemma nil, :begin 52, :token "wahrscheinlichsten", :ner "O", :end 70}
 {:category nil, :pos-tag "ADJ", :pos 14, :lemma nil, :begin 71, :token "vorliegenden", :ner "O", :end 83}
 {:category nil, :pos-tag "NOUN", :pos 15, :lemma nil, :begin 84, :token "Gehirnerkrankung", :ner "O", :end 100}
 {:category nil, :pos-tag "ADJ", :pos 16, :lemma nil, :begin 101, :token "typisch", :ner "O", :end 108}
 {:category nil, :pos-tag "PUNCT", :pos 17, :lemma nil, :begin 108, :token "?", :ner "O", :end 109}]

-----------------------
(get-pos-tags-from-document-for-document (generate-document-for-pipeline "Für wen ist das Essen?" (get-pipeline-for-language :de)))
[nRepl-session-cbf7e633-5c39-4cbf-88ad-3e00806cd271] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator tokenize
[nRepl-session-cbf7e633-5c39-4cbf-88ad-3e00806cd271] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator ssplit
[nRepl-session-cbf7e633-5c39-4cbf-88ad-3e00806cd271] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator mwt
[nRepl-session-cbf7e633-5c39-4cbf-88ad-3e00806cd271] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator pos
[nRepl-session-cbf7e633-5c39-4cbf-88ad-3e00806cd271] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator ner
=>
[{:category nil, :pos-tag "ADP", :pos 1, :lemma nil, :begin 0, :token "Für", :ner "O", :end 3}
 {:category nil, :pos-tag "NOUN", :pos 2, :lemma nil, :begin 4, :token "wen", :ner "O", :end 7}
 {:category nil, :pos-tag "AUX", :pos 3, :lemma nil, :begin 8, :token "ist", :ner "O", :end 11}
 {:category nil, :pos-tag "DET", :pos 4, :lemma nil, :begin 12, :token "das", :ner "O", :end 15}
 {:category nil, :pos-tag "NOUN", :pos 5, :lemma nil, :begin 16, :token "Essen", :ner "O", :end 21}
 {:category nil, :pos-tag "PUNCT", :pos 6, :lemma nil, :begin 21, :token "?", :ner "O", :end 22}]

(get-pos-tags-from-document-for-document (generate-document-for-pipeline "Ich war reif für das Bett." (get-pipeline-for-language :de)))
[nRepl-session-cbf7e633-5c39-4cbf-88ad-3e00806cd271] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator tokenize
[nRepl-session-cbf7e633-5c39-4cbf-88ad-3e00806cd271] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator ssplit
[nRepl-session-cbf7e633-5c39-4cbf-88ad-3e00806cd271] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator mwt
[nRepl-session-cbf7e633-5c39-4cbf-88ad-3e00806cd271] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator pos
[nRepl-session-cbf7e633-5c39-4cbf-88ad-3e00806cd271] INFO edu.stanford.nlp.pipeline.StanfordCoreNLP - Adding annotator ner
=>
[{:category nil, :pos-tag "PRON", :pos 1, :lemma nil, :begin 0, :token "Ich", :ner "O", :end 3}
 {:category nil, :pos-tag "AUX", :pos 2, :lemma nil, :begin 4, :token "war", :ner "O", :end 7}
 {:category nil, :pos-tag "ADV", :pos 3, :lemma nil, :begin 8, :token "reif", :ner "O", :end 12}
 {:category nil, :pos-tag "ADP", :pos 4, :lemma nil, :begin 13, :token "für", :ner "O", :end 16}
 {:category nil, :pos-tag "DET", :pos 5, :lemma nil, :begin 17, :token "das", :ner "O", :end 20}
 {:category nil, :pos-tag "NOUN", :pos 6, :lemma nil, :begin 21, :token "Bett", :ner "O", :end 25}
 {:category nil, :pos-tag "PUNCT", :pos 7, :lemma nil, :begin 25, :token ".", :ner "O", :end 26}]

[AngledLuffa]

Gah, check out this tiny difference. The way you typed it into CoreNLP, the u and the umlaut over the u are two separate characters:

NLP> Welcher der Befunde ist für eine Gehirnerkrankung typisch?

Sentence #1 (9 tokens):
Welcher der Befunde ist für eine Gehirnerkrankung typisch?

Tokens:
[Text=Welcher CharacterOffsetBegin=0 CharacterOffsetEnd=7 PartOfSpeech=PRON NamedEntityTag=O]
[Text=der CharacterOffsetBegin=8 CharacterOffsetEnd=11 PartOfSpeech=DET NamedEntityTag=O]
[Text=Befunde CharacterOffsetBegin=12 CharacterOffsetEnd=19 PartOfSpeech=NOUN NamedEntityTag=O]
[Text=ist CharacterOffsetBegin=20 CharacterOffsetEnd=23 PartOfSpeech=AUX NamedEntityTag=O]
[Text=für CharacterOffsetBegin=24 CharacterOffsetEnd=28 PartOfSpeech=PROPN NamedEntityTag=O]     <----- note, this is 4 characters long and not 3
[Text=eine CharacterOffsetBegin=29 CharacterOffsetEnd=33 PartOfSpeech=DET NamedEntityTag=O]
[Text=Gehirnerkrankung CharacterOffsetBegin=34 CharacterOffsetEnd=50 PartOfSpeech=NOUN NamedEntityTag=O]
[Text=typisch CharacterOffsetBegin=51 CharacterOffsetEnd=58 PartOfSpeech=ADJ NamedEntityTag=O]
[Text=? CharacterOffsetBegin=58 CharacterOffsetEnd=59 PartOfSpeech=PUNCT NamedEntityTag=O]

If I pasted it in as one character, it processes as you want:

NLP> Welcher der Befunde ist für eine Gehirnerkrankung typisch?

Sentence #1 (9 tokens):
Welcher der Befunde ist für eine Gehirnerkrankung typisch?

Tokens:
[Text=Welcher CharacterOffsetBegin=0 CharacterOffsetEnd=7 PartOfSpeech=PRON NamedEntityTag=O]
[Text=der CharacterOffsetBegin=8 CharacterOffsetEnd=11 PartOfSpeech=DET NamedEntityTag=O]
[Text=Befunde CharacterOffsetBegin=12 CharacterOffsetEnd=19 PartOfSpeech=NOUN NamedEntityTag=O]
[Text=ist CharacterOffsetBegin=20 CharacterOffsetEnd=23 PartOfSpeech=AUX NamedEntityTag=O]
[Text=für CharacterOffsetBegin=24 CharacterOffsetEnd=27 PartOfSpeech=ADP NamedEntityTag=O]
[Text=eine CharacterOffsetBegin=28 CharacterOffsetEnd=32 PartOfSpeech=DET NamedEntityTag=O]
[Text=Gehirnerkrankung CharacterOffsetBegin=33 CharacterOffsetEnd=49 PartOfSpeech=NOUN NamedEntityTag=O]
[Text=typisch CharacterOffsetBegin=50 CharacterOffsetEnd=57 PartOfSpeech=ADJ NamedEntityTag=O]
[Text=? CharacterOffsetBegin=57 CharacterOffsetEnd=58 PartOfSpeech=PUNCT NamedEntityTag=O]

[Goldritter]

You are right.
It is also more of a Data error and I fear I knew the reason for this. The data has been gathered by using copy and paste from varying Worddocuments and PDFs.

I think I will add a new field "size" to my map ("end - begin" respectively "CharacterOffsetEnd - CharacterOffsetBegin") to spot such differences earlier.

This issue can be closed now.

Thank you for your help!

I fear this problem might occure in other languages also which uses similar charaters, like french accents.

[AngledLuffa]

You might be right about other languages. For now, we're just fixing the umlauts in German. It turns out all the previous examples were broken because the umlauts, not the model itself, so we might go with the old models for simplicity. We'll try to get the fix out shortly.

[Goldritter]

I agree with you there. This solution is better.

[Goldritter]

This might be eventually helpfull.

I searched methods to combine diacritical marks and character to one character in Java and found following Stackoverflow link "Combining accent and character into one character in java 7".

I tried it with "für" in our case and get following results:

(import java.text.Normalizer)
=> java.text.Normalizer

(import java.text.Normalizer$Form)
=> java.text.Normalizer$Form

(def für "für") ;; with combine diacritical mark

(count für)
=> 4

(Normalizer/normalize für Normalizer$Form/NFC) 
=> "für"

(count (Normalizer/normalize für Normalizer$Form/NFC)) 
=> 3

(Normalizer/normalize für Normalizer$Form/NFD) 
=> "für"

(count (Normalizer/normalize für Normalizer$Form/NFD)) 
=> 4

(Normalizer/normalize für Normalizer$Form/NFKC) 
=> "für"

(count (Normalizer/normalize für Normalizer$Form/NFKC)) 
=> 3

(Normalizer/normalize für Normalizer$Form/NFKD) 
=> "für"

(count (Normalizer/normalize für Normalizer$Form/NFKD)) 
=> 4

This might solve some issues with UTF-8 diacritical marks.
Evententually this can be added as an option to the Pipeline so that users can decide if they want to "normalize" the text or not. Similar to the "mwt" option.