TODO List of databases that could be accessed with c14bazAAR

[nevrome]

• PACEA
• 14Sea
• aDRAC
• CALPAL
• CONTEXT
• EUROEVOL
• RADON
• AustArch 2 3
• Palmisanoetal
• EUBAR
• Ireland
• Radiocarbon Palaeolithic Europe Database v21
• Crema Jomon Japan
• MedAfriCarbon, Paper WebApp
• MesoRAD or 'Mesoamerican Radiocarbon Database' (DS): coords have been removed; see paper
  • v1.4 data behind login-wall MesoRAD v.1.4; see paper 2021
• Aegean History and Archaeology
• BDA
• rxpand
• NERD
• caribbean-14C
• AIDA
• NeoNet
• Rapa Nui ==> Merge individual datasets clipo/rapanui-radiocarbon#1
• IntChron ==> Parser for databases hosted on IntChron platform via rintchron #147
• Agrichange
• SARD
• IberiaC14 ==> c14bazAAR integration danfid15/IberiaC14#1
• BANADORA ==> BANADORA #165

Databases/Collections behind login- or paywalls. Or not accessible without web scraping. Will not be included. 😿

• CARD ==> Loginwall.
• Flohr 2015 ==> Paywall.
• Wang 2014 ==> Paywall.
• ORAU - Oxford Radiocarbon Accelerator Unit ==> Loginwall.
• Douglass et al. 2019 Madagascar ==> Paywall.
• PPND ==> No file download intended. Seems to be broken.
• KIK-IRPA ==> No file download intended.
• Plateforme des datations archéologiques intertropicales ==> No data download intended.
• IDEARQ ==> No file download intended.
• ARQUEODATA ==> Loginwall.

[joeroe]

@nevrome I have a copy of the PPND database transcribed into a CSV. I'm not sure if it's still being updated, is it worth including it just as a static file?

[nevrome]

Sounds interesting, Joe! Is it ok for the creators of the database if you share it? In the end it's their decision.

[joeroe]

I haven't been in touch with them. I just scraped it from their website for another project. I will leave it up to you!
exoriente_ppnd.csv.txt

[nevrome]

We're currently establishing a cooperation with @mcolobran (Miquel Colobran) who wrote some webscrapers for GoGet. He'll provide us with a stable download link for some databases that only can be accessed by crawling.

I'm still not 100% sure if this is the way to go though. The people who are sharing their data only as html do so deliberately I guess. I think I should at least inform the authors. They should know that this is possible and that we are even trying to make it more easy.

[dirkseidensticker]

I had a look at their website https://www.exoriente.org/associated_projects/ppnd.php

@joeroe I guess you compiled the list yourself as I wasn't able to find a simple 'download all' button or so, only links of OxCal formatted snippets. Right?

For me, Open Access or at least the knowledge of what will be done to the data are essential. If you don't mind I could write Marion Benz and ask her if the database is still updated and if they would be willing to lend something over for the c14bazAAR. I wouldn't want to include something without the consent of the original creators. Unless you @joeroe compiled your CSV manually based on the informations on the web. Then -- in my view -- you are the creator of your list, as (at least after German law) data are not copyrightable. In that case, if you set up a repo for the table we'll be able to write a parser for it.

What's your thoughts?

[joeroe]

@dirkseidensticker So all I did was go through the website and copy the individual site tables into a single CSV. Copyright aside, it's not my data – I just thought that if you did want to incorporate PPND, I could save you doing that step. I agree that the polite thing to do is to ask the original compilers for permission to redistribute it first. Maybe they'll be able to simplify things

Interesting ethical questions in general, though. Is there a qualitative difference between downloading tabular data and using c14bazAAR to reformat it, versus downloading hypertext data and using a scraper to reformat it? (Or a text editor, as I did?) If you publish scientific data in any form, aren't you implicitly agreeing to let it be reused without explicit permission, in return for acknowledgement?

I guess this is why they tell you to include a LICENSE file in repos!

[nevrome]

There's even a legal difference between downloading predefined tables and webscraping: https://en.wikipedia.org/wiki/Web_scraping#Legal_issues
I was not aware of this. Now I would say: No webscraping data for c14bazAAR. We can and should write to database providers for a legally safe download opportunity as I did for RADON and CalPal. We should also discuss this with Miquel.

[mcolobran]

Hello @dirkseidensticker and @nevrome.

In response of your concerns, which I also had and have.

The 'protocol' (proposal) to follow for joining a new database in GoGet and/or c14bazAAR could be:

1. Check if the database could be joined even using web crawling effort by the owners.

2. If so, ask permission to the owners (so yes, they have the final decision).

Agreement how often they want the data be downloaded by us.

3. Inform when data is ready to be used by GoGet and/or c14bazAAR with a mail.

What we can offer to the owners (referred to point 2)

- To do nothing (or almost nothing) in their websites (no changes are required at all).
- a CSV file stored in goget (or any other place) to assure not to collapse their database. If they prefer, we can send to them in order to load the database from c14bazAAR
- the csv file to be used by c14bazAAR (again they have to do nothing)
- Guarantee the ownership (take a look at the fields in goget). In this sense we don't "make our" the data.
- Guarantee the crawling is doing from their public data they publish in the web. Besides it is done at a very low speed in order not to collapse their database. Finally we inform them the IP that is doing the task to avoid IDS, networking problems.


Benefits for the owners:

- They gain visibility (so citations and visits)
- We can check automatically for some errors on their data, so improving the quality. 
- Their website will have more visits because Goget (and c14bazAAR) are able to redirect/go to their web if the researcher needs more detail or clarifications.
- Goget publish the database name (http://www.ibercrono.org/goget/databases.php). In this sense our intention is not "appropriate" data but spreading their work.
- They are following the open data movement.

I answer inline some concerns you expressed

I think I should at least inform the authors. They should know that this is possible and that we are even trying to make it more easy.

The idea behind goget (and therefore c14bazAAR) is the owners have to do nothing and no changes to the data they are making public in Internet. If they have to make changes it makes things more difficult.

Maybe they'll be able to simplify things

In my humble opinion, if we want to spread their data and use it as less work and changes have the owners the better chances we all have.

interesting ethical questions in general, though. Is there a qualitative difference between downloading tabular data and using c14bazAAR to reformat it, versus downloading hypertext data and using a scraper to reformat it? (Or a text editor, as I did?) If you publish scientific data in any form, aren't you implicitly agreeing to let it be reused without explicit permission, in return for acknowledgement?

About this. First of all, no internet data/image is free in the sense the data/image always have an author and never can be considered yours.

If you download a tabular data (with a button in the website) and the owner have and acknowledgement in return, it is nice to inform the owner but you can do anything (except appropriating of course)

If one crawls the data, one has to be more careful. Maybe the author didn't had in mind this purpose. That's why it had to be informed before spread it (point 2 and following to offer what they receive in exchange)

I guess this is why they tell you to include a LICENSE file in repos!

Agree about the LICENSE FILE

They also claimed that screen-scraping constitutes what is legally known as "Misappropriation and Unjust Enrichment", as well as being a breach of the web site's user agreement.

The legal point of that is "enrichment". If you obtain profit, you are in trouble. It's considered a crime by the almost any country law. If you spread something (a photocopied book as an example) you are only trespassing Intellectual Property. As there is no profit for this data, the question goes to credit, diffusion.... which is not big deal (despite we have to be aware of)

these are early scraping decisions, and the theories of liability are not uniform, it is difficult to ignore a pattern emerging that the courts are prepared to protect proprietary content on commercial sites from uses which are undesirable to the owners of such sites.

Again, take a look at the text. They talk about commercial sites and that implies "money" and profits....not our scenario.

We can and should write to database providers for a legally safe download opportunity as I did for RADON and CalPal. We should also discuss this with Miquel

100% agree with a small detail. The way the data is obtained have to be agreed. If we offer the spreading of their data with no effort by the providers.....we have a very good point on our side.

Sorry for the long mail.

                  M.

[yesdavid]

I'd like to add two Palaeolithic databases that I need, it's

• the RADIOCARBON PALAEOLITHIC EUROPE DATABASE (https://ees.kuleuven.be/geography/projects/14c-palaeolithic/)
• PACEA (http://www.paleoanthro.org/media/journal/content/PA20110001.pdf) (DB in supplementaries, see http://www.paleoanthro.org/journal/volumes/2011/ and http://www.paleoanthro.org/media/journal/content/PA20110001_S01.zip)