[Feature] Refactor IMF for SDMX 3

[deeleeramone]

This PR resolves #7266

It includes a set of metadata, a router with utility functions and an apps.json endpoint, as well as unit tests for the underlying helper functions.

It also adds imf as a provider to:

• obb.economy.cpi

It also touches tests related the openbb-platform-api expected -> result definitions.

The integrations test collector has been updated to scan provider extensions and the integration folder, if it exists.

Here is a test script used while developing the hierarchical tables logic.

Info Notice

PLANNED MAINTENANCE: From Saturday, December 20th, 01:00 a.m. EST to Sunday, December 21st, 07:00 a.m. EST data.imf.org will undergo maintenance. Users may experience intermittent outages or inability to access data during this time.

from openbb import obb

tables = {
    "bop_standard": "BOP::H_BOP_BOP_AGG_STANDARD_PRESENTATION",
    "bop_analytic": "BOP::H_BOP_BOP_AGG_ANALYTIC_PRESENTATION",
    "bop_international_transactions": "BOP_AGG::H_BOP_AGG_SUMMARY_OF_INTERNATIONAL_TRANSACTIONS",
    "bop_global_discrepancies": "BOP_AGG::H_BOP_AGG_GLOBAL_DISCREPANCIES",
    "bop_global_discrepancies_percent": "BOP_AGG::H_BOP_AGG_GLOBAL_DISCREPANCIES_PERCENTAGE",
    "iip_aggregated": "IIP::H_BOP_IIP_AGG",
    "iip_currency_composition": "IIPCC::H_BOP_IIPCC_CURRENCY_COMPOSITION",
    "eer": "EER::H_EER_INDICATOR_HIERARCHY",
    "irfcl_reserve_assets": "IRFCL::H_CL_IRFCL_INDICATOR_DEFAULT_PUB:CL_IRFCL_DEFAULT_INDICATOR_PUB2_1",
    "irfcl_predetermined_drains": "IRFCL::H_CL_IRFCL_INDICATOR_DEFAULT_PUB:CL_IRFCL_DEFAULT_INDICATOR_PUB2_29",
    "irfcl_contingent_drains": "IRFCL::H_CL_IRFCL_INDICATOR_DEFAULT_PUB:CL_IRFCL_DEFAULT_INDICATOR_PUB2_91",
    "irfcl_memorandum": "IRFCL::H_CL_IRFCL_INDICATOR_DEFAULT_PUB:CL_IRFCL_DEFAULT_INDICATOR_PUB2_273",
    "fsi_core_and_additional": "FSIC::H_FSIC_INDICATOR",
    "fsi_indicators_and_underlying": "FSIC::H_FSIC_FSD",
    "fsi_concentration_distribution_measures": "FSICDM::H_FSI_FSICDM_CDM",
    "mfs_monetary_aggs": "MFS_MA::H_CL_MFS_MA_INDICATOR_5SR",
    "mfs_central_bank": "MFS_CBS::MFS_CBS::H_CL_MFS_CBS_SRFS_1SG_VIEW",
    "mfs_other_depository_corps": "MFS_ODC::H_CL_MFS_ODC_SRFS_2SG_VIEW",
    "mfs_depository_corps": "MFS_DC::H_CL_MFS_DCS_SRFS_3SG_VIEW",
    "mfs_other_financial_corps": "MFS_OFC::H_CL_MFS_OFC_SRFS_4SG_VIEW",
    "mfs_financial_corps": "MFS_FC::H_CL_MFS_FCS_SRFS_5SG_VIEW",
    "gfs_balance": "GFS_BS::H_GFS_BS",
    "gfs_operations": "GFS_SOO::H_GFS_SOO",
    "gfs_expenditures": "GFS_COFOG::H_GFS_COFOG",
    "gfs_stocks_and_flows": "GFS_SFCP::H_GFS_SFCP",
    "gfs_sources_and_uses": "GFS_SSUC::H_GFS_SSUC",
    "qgfs_balance": "QGFS::H_QGFS_BALANCE",
    "qgfs_operations": "QGFS::H_QGFS_STATGO",
    "qgfs_sources_and_uses": "QGFS::H_QGFS_GSUC",
    "gdp_annual_expenditure": "ANEA::H_NEA_GDP_BY_EXPENDITURE",
    "gdp_quarterly_expenditure": "QNEA::H_NEA_GDP_BY_EXPENDITURE",
    "cpi": "CPI::H_CPI_BY_COMPONENT",
    "fas_indicator_by_country": "FAS::H_FAS_INDICATOR_BY_COUNTRY",
    "fas_data_by_country": "FAS::H_FAS_BY_COUNTRY",
    "isora_indicators_by_topic": "ISORA_LATEST_DATA_PUB::H_CL_INDICATORS_BY_TOPIC",
}

TO_TEST = [
    # "GFS_BS::H_GFS_BS",
    # "GFS_SOO::H_GFS_SOO",
    # "GFS_COFOG::H_GFS_COFOG",
    # "GFS_SFCP::H_GFS_SFCP",
    # "GFS_SSUC::H_GFS_SSUC",
    # "QGFS::H_QGFS_STATGO",
    # "QGFS::H_QGFS_GSUC",
    # "QGFS::H_QGFS_BALANCE",
    # "ANEA::H_NEA_GDP_BY_PRODUCTION",
    # "ANEA::H_NEA_GDP_BY_EXPENDITURE",
    # "QNEA::H_NEA_GDP_BY_PRODUCTION",
    # "QNEA::H_NEA_GDP_BY_EXPENDITURE",
    # "MFS_MA::H_CL_MFS_MA_INDICATOR_5SR",
    # "MFS_CBS::H_CL_MFS_CBS_SRFS_1SG_VIEW",
    # "MFS_ODC::H_CL_MFS_ODC_SRFS_2SG_VIEW",
    # "MFS_DC::H_CL_MFS_DCS_SRFS_3SG_VIEW",
    # "MFS_OFC::H_CL_MFS_OFC_SRFS_4SG_VIEW",
    # "MFS_FC::H_CL_MFS_FCS_SRFS_5SG_VIEW",
    # "FSIC::H_FSIC_FSD",
    # "FSIC::H_FSIC_INDICATOR",
    # "FSICDM::H_FSI_FSICDM_CDM",
    # "BOP::H_BOP_BOP_AGG_STANDARD_PRESENTATION",
    "BOP::H_BOP_BOP_AGG_ANALYTIC_PRESENTATION",
    # "BOP_AGG::H_BOP_AGG_SUMMARY_OF_INTERNATIONAL_TRANSACTIONS",
    # "BOP_AGG::H_BOP_AGG_GLOBAL_DISCREPANCIES",
    # "BOP_AGG::H_BOP_AGG_GLOBAL_DISCREPANCIES_PERCENTAGE",
    # "IIP::H_BOP_IIP_AGG",
    # "IIPCC::H_BOP_IIPCC_CURRENCY_COMPOSITION",
    # "IRFCL::H_CL_IRFCL_INDICATOR_DEFAULT_PUB:CL_IRFCL_DEFAULT_INDICATOR_PUB2_1",
    # "IRFCL::H_CL_IRFCL_INDICATOR_DEFAULT_PUB:CL_IRFCL_DEFAULT_INDICATOR_PUB2_29",
    # "IRFCL::H_CL_IRFCL_INDICATOR_DEFAULT_PUB:CL_IRFCL_DEFAULT_INDICATOR_PUB2_91",
    # "IRFCL::H_CL_IRFCL_INDICATOR_DEFAULT_PUB:CL_IRFCL_DEFAULT_INDICATOR_PUB2_273",
    # "CPI::H_CPI_BY_COMPONENT",
    # "EER::H_EER_INDICATOR_HIERARCHY",
    # "DIP::H_DIP_INDICATOR",
    # "FAS::H_FAS_INDICATOR_BY_COUNTRY",
    # "FAS::H_FAS_BY_COUNTRY",
    # "ISORA_LATEST_DATA_PUB::H_CL_INDICATORS_BY_TOPIC",
]
for table in TO_TEST:
    print("=" * 80)
    print(table)
    print("=" * 80)
    try:
        result = obb.economy.indicators(
            symbol=table,
            provider="imf",
            country="BRA",
            pivot=True,
            limit=2,
            frequency="Q",
        )
        from pandas import DataFrame

        df = DataFrame([d.model_dump(exclude_none=True) for d in result.results])
        df.set_index(["title", "country"], inplace=True)
        #df.drop(columns=["description"], inplace=True)
        print(df.to_string())
        #print(
        #    [
        #        d.model_dump(exclude_none=True, exclude="description")
        #        for d in result.results
        #    ]
        #)
        print("\n")
    except Exception as e:
        print(str(e.args[0]))

From the README file:

This package provides everything you need - endpoints, tools, and metadata - to access and explore the entirety of https://data.imf.org, without any previous experience working with it.

Installation

Install from PyPI with:

pip install openbb-imf

Then build the Python static assets by running:

openbb-build

Quick Start

The fastest way to get started is by connecting to the OpenBB Workspace as a custom backend.

Start Server

openbb-api

This starts the FastAPI server over localhost on port 6900.

Add to Workspace

See the documentation here for more details.

Click to Open App

Once added, click on the app to open the dashboard.

The dashboard contains widgets with metadata and information, as well ones for exploring and retrieving the data.

Implementation Details

IMF is a SDMX API, and they organize their data by "dataflows". You can think of these as databases.
Each one has their own definitions for parameters and output. Some definitions are shared, others are domain-specific.
In SDMX, query models are expressed as dimensions, and data models as attributes.

The extension comes with a base level of metadata used for normalizing, translating, and validating inputs and outputs.
When mapping requires a code list that is not included, a network request is made to retrieve it.

User input is validated by calling the constraints API for each dimension.
The cached metadata contains all the potential values for parameters,
but the availability of each is determined by other choices - country, frequency, etc.
When making the actual request for data, parameters are tested for compatibility in the sequence defined by the data structure definition.
Invalid parameter combinations are returned as helpful error messages with descriptions of what went wrong and what the valid choices are.

The output converts ID codes into human-readable labels, and includes dataset and series metadata in a separate object.

Indicators

In this library, we refer to indicators as "indicator-like" dimensions within individual dataflows.
It includes dimensions like COICOP_1999, so items such as CPI All Items, or Clothing and footwear,
are considered.

The IMF codes for these values - _T, CP01, etc. - are used to construct ticker-like symbols.

Presentation Tables

Presentation tables are built from hierarchical code lists defining the parent-child relationship
between individual series.

Internally, table IDs are prefixed with H_ - i.e, H_BOP_BOP_AGG_STANDARD_PRESENTATION.

The symbology allows entering references to tables, or indicators.

Symbology

The Open Data Platform generally refers to all time series IDs as a symbol.
Requesting time series data or presentation tables requires a symbol constructed from
the dataflow ID and the indicator-like dimensions, split with ::.

CPI::CPI returns all series for CPI under the TYPE_OF_INDEX dimension.

CPI::CPI_CP01 returns just component of CPI, Food and non-alcoholic beverages.

The symbol mapper matches the items after :: intelligently to its corresponding dimension.
Entering, CPI::CP01, gets the same result as, CPI::CPI_CP01.

CPI::CPI__T returns just the top-level, All Items.

CPI::H_CPI_BY_COMPONENT returns the entire presentation table, Consumer Price Index (CPI) by Component.

Use, obb.imf_utils.list_tables(), for a list of tables and their symbol.

Use, obb.economy.available_indicators(provider='imf', query='), to search for, or list all, individual time series and symbols.

This example lists all the indicators in the CPI and PI dataflows.

from openbb import obb

indicators = obb.economy.available_indicators(provider="imf", dataflows="CPI,PI")

print(indicators.to_dict("records"))

...
 {'symbol_root': 'CP01',
  'symbol': 'CPI::CP01',
  'description': 'Food and non-alcoholic beverages',
  'agency_id': 'IMF.STA',
  'dataflow_id': 'CPI',
  'dataflow_name': 'Consumer Price Index (CPI)',
  'structure_id': 'DSD_CPI',
  'dimension_id': 'COICOP_1999',
  'long_description': 'Food and non-alcholoic beverages consumer price index is produced using prices related to food items and non-alcoholic beverages such as fresh produce, packaged foods, and beverages excluding alcoholic drinks, aggregated by their respective consumer expenditure weights.',
  'member_of': ['CPI::H_CPI_BY_COMPONENT']},
...

Coverage

All data available from https://data.imf.org/en/Data-Explorer can be retrieved, via obb.economy.indicators(provider='imf', **kwargs).

Additionally, there are endpoints for some Port Watch items (not part of the Data Explorer).

The extension creates a router path, imf_utils, that exposes utility functions for UI integrations and metadata lookup.

Endpoints

• obb.economy.available_indicators
• obb.economy.indicators
• obb.economy.cpi
• obb.economy.direction_of_trade
• obb.economy.shipping.chokepoint_info
• obb.economy.shipping.chokepoint_volume
• obb.economy.shipping.port_info
• obb.economy.shipping.port_volume
• obb.imf_utils.get_dataflow_dimensions
• obb.imf_utils.list_dataflow_choices
• obb.imf_utils.list_dataflows
• obb.imf_utils.list_port_id_choices
• obb.imf_utils.list_table_choices
• obb.imf_utils.list_tables
• obb.imf_utils.presentation_table
• obb.imf_utils.presentation_table_choices

"Choices" endpoints are utilized by OpenBB Workspace to populate widget dropdown menus.