Question: Abbreviations list

[pirolen]

What is the best way to supply a list of known abbreviations to python-ucto and ucto in LaMachine?

[proycon]

Those lists are part of the uctodata repository, and referred to from the individual configuration files (like tokconfig-nld). Contributions there are welcome of course! (you can just send a pull request)

[pirolen]

Thanks! I have a long list of idiosyncratic and critical-edition-specific abbreviations, I guess most of them are not useful for including them in general, e.g. 'Corp.Inscr. Graec.' or 'Abh. der Sächs. Ges.d.Wiss.’ or S.A. (=Sonnenaufgang). Sure I will suggest if generic ones emerge. Bigger context of the question: for 'Abbreviations with multiple periods’ in FoLiA documents, I should make sure that - a short (h)space is inserted after the period - the abbreviations are treated in the text as a unit, i.e. non-splittable at line/page end. I was wondering how to best use ucto to this end. The language is mostly German and Latin, so I set in the config for German and (as a hopeful fallback) to French.

…

On 10. May 2021, at 13:35, Maarten van Gompel ***@***.***> wrote: Those lists are part of the uctodata repository, and referred to from the individual configuration files (like tokconfig-nld). Contributions there are welcome of course! (you can just send a pull request) — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[pirolen]

How is "z.B." supposed to be treated by ucto?
It is not part of the list in https://github.com/LanguageMachines/uctodata/blob/master/config/tokconfig-deu,
and is tokenized with the classes "ABBREVIATION-KNOWN" plus "INITIAL".

Isn't it supposed to be somehow a unit and an instance of 'Abbreviations with multiple periods’?
Maybe @kosloot can tell why, I don't know ucto yet that much.

 <w xml:id="FA-b1_3_1_mwtext_ostpreuss_pp109_277_006_abpproc.text.div1.p4.s.4.w.43" class="ABBREVIATION-KNOWN" set="tokconfig-deu" textclass="OCR">
            <t class="OCR">z.</t>
          </w>
          <w xml:id="FA-b1_3_1_mwtext_ostpreuss_pp109_277_006_abpproc.text.div1.p4.s.4.w.44" class="INITIAL" set="tokconfig-deu" textclass="OCR">
            <t class="OCR">B.</t>
          </w>

[kosloot]

That all depends on the ucto rules for German.

In this case z.B. is NOT a known abbreviation. But 'z' is. So 'z.' is tagged as an abbreviation.

Also there is a rule for interpret an Uppercase B. as an Initial:

#retain initials
INITIAL=^(?:\p{Lt}|\p{Lu})\.$

Hence the split into 2 words.

Adding z.B. to the abbreviation list should be enough.
My advise is to get a copy of uctodata. (from Git), make the additions you would like to see, and install them in your system.
('make install')

If you are satisfied, a pull request is welcome

[kosloot]

To make life easier, I separated the German abbreviations into a separate file you can edit: deu.abr

[pirolen]

OK, for the pull request I would only add generic abbreviations, right? (Attached an illustration.)

The multi-element abbreviations like d.h. or z.B. need to be in the list as d\.h and z\.B?

[kosloot]

Ok, I am a bit surprised now, as I tested it myself with the current setup:

ucto -Ldeu -v
ucto: inputfile = 
ucto: outputfile = 
ucto: textcat configured from: /home/sloot/usr/local/share/ucto/textcat.cfg
ucto: configured for languages: [deu]
ucto> Dass ist z.B. falsch . <utt> 
Dass	WORD	BEGINOFSENTENCE NEWPARAGRAPH 
ist	WORD	
z.B.	ABBREVIATION	
falsch	WORD	
.	PUNCTUATION	ENDOFSENTENCE 

And:

ucto> d.h. dass es gut geht?
d.h.	ABBREVIATION	BEGINOFSENTENCE 
dass	WORD	
es	WORD	
gut	WORD	
geht	WORD	NOSPACE 
?	PUNCTUATION	ENDOFSENTENCE 

So it should work out of the box for those examples.

So there must be some other problem here?? I would need more context.

Despite that: more abbreviations are welcome. (you may also send me a file)
Remember that abbreviations that can be mistaken with real words must be marked.
Like :
dass\.1 and NOT dass

[pirolen]

I run the LaMachine command-line ucto (not python-ucto) with --uselanguages=deu,fra when I got the above tokenization (#9 (comment)).

In the untokenized text there is a space between the parts of the abbreviation: "wie z. B. im Kreise".

[kosloot]

In the untokenized text there is a space between the parts of the abbreviation: "wie z. B. im Kreise".

AH!
Well it cannot be solved by Ucto in that case. A space is the major (and only unchangeable) separator between tokens in Ucto.

Sorry for that limitation.

[pirolen]

Any type of space (e.g. short (h)space too) counts as a separator?

[kosloot]

I assume so. But determining spaces is a hell :{
Ucto uses the ICU:u_isspace() function to do so.
see:
https://unicode-org.github.io/icu-docs/apidoc/dev/icu4c/uchar_8h.html#a48dd198b451e691cf81eb41831474ddc

[pirolen]

OK, good to know.

Back to the deu.abr list: "d.h." is not in this list. How come the command line ucto covers it?

[kosloot]

That is covered by the ABBREVIATION rule in the 'tokconfig-deu' file

#Abbreviations with multiple periods
ABBREVIATION=^(?:[{([<]?)(\p{L}{1,3}(?:.\p{L}{1,3})+.?)(?:\Z|[,:;})]>])

This regexp says something along the line:
dot-separated sequences of 1-3 characters are considered an abbreviation;
even when placed between brackets like '{ }' '[ ]' or '( )'.
And optionally ending with a ',' ':' or ';'

This will catch 'z.B'. and 'd.h.' And also '(z.B'. and {'d.h.}' . But NOT 'z. B.' or 'd. h.'

[pirolen]

I suggested some items for inclusion in the German abbrev list: LanguageMachines/uctodata@master...pirolen:patch-1

(several are actually of Latin origin...)

[kosloot]

They seem OK to me. So I merged them