Decouple API

[olegs]

The current API should be decoupled so that searching for relevant datasets doesn't return all available information.
The API should consist of the following components:

• get_datasets_by_pmids - shoudl return map of <pmid - [connected dataset ids]>
• get_gse_information(list of gse) - should return map of <gse_id, gse information>
• get_gsm_information(list of gsm) - should return map of <gsm_id, gsm information>
• get_relevant_datasets(list dataset ids, query) - should return list of pairs <dataset_id, relevance>
  Also this call should proceed GSM information under the GSE_ID, to get the most relevant information.

[MomirMilutinovic]

I’ve investigated the implementation of get_datasets_by_pmids. While decoupling the API is logical, the current architecture of the ELink API presents a significant performance bottleneck for returning specific PMID-to-Dataset mappings.

The Technical Hurdle

The ELink API returns a flat list of dataset IDs for a batch of PMIDs; it does not provide the specific mapping (which PMID belongs to which dataset). To provide that mapping, we have to call ELink for each PMID individually.

• Current implementation: 1.9s average response time.
• Decoupled implementation (refactor/decouple-api): 24.3s average response time.
  • Note: ~23 seconds is spent solely on these sequential ELink calls for 50 PMIDs.

Alternative: EuropePMC API

We could use the EuropePMC Annotations API, which does return PMIDs for each dataset ID. However, there is a coverage gap. In the 50-PMID sample, EuropePMC missed GSE171080, which ELink successfully retrieved. Switching entirely to EuropePMC would reduce our dataset coverage.

Summary of Tradeoffs

We need to decide on one of the following paths forward:

1. Prioritize Mapping (Accuracy): Accept a 10x slowdown in response time to ensure we provide the exact PMID-dataset mapping without losing ELink coverage.
2. Prioritize Speed (Performance): Use EuropePMC and potentially geometadb for mapping, but accept a reduction in dataset coverage.
3. Revised Requirement: Re-evaluate if get_datasets_by_pmids must return a map. If it returns a flat list of dataset IDs (like the current API), we maintain 1.9s performance.