Help configuring external xls data in web project

[serman]

directData

I see here https://github.com/climateinteractive/SDEverywhere/blob/1a77eedd3143568ad3b4659f6b78dd9b60737b53/packages/cli/README.md
that to read external XLS files that I need to add the field directData to the specs.json file

• Since I am using the project structure generated by npm create the spec.json file is auto generated within sde-prep folder as well as the core/model/generated/model-spec.ts
• It is not clear to me where should I add directData in the sde.config.js

syntax

When I have more than 1 data file, what is the correct syntax, is it an array or this one?

      "directData": {
        "data1": "e4a-exo-2.xlsx",
        "data2": "e4a-policies.xlsx"
      }

_GET_XLS_DATA error

I assume this error happens because there is no XLS file, not because _GET_XLS_DATA is not supported, right?

> sde dev

Generating files...
Done generating files (0.0s)
Generating model...
  Error: Unhandled function '_GET_XLS_DATA' in readEquations for 'Birth rate H[region]'
      at visitFunctionCall (file:///Users/sergiogalan/proyectos/earth4all/testm/node_modules/@sdeverywhere/compile/src/model/read-equations.js:610:19)
      at visitExpr (file:///Users/sergiogalan/proyectos/earth4all/testm/node_modules/@sdeverywhere/compile/src/model/read-equations.js:251:7)
      at readEquation (file:///Users/sergiogalan/proyectos/earth4all/testm/node_modules/@sdeverywhere/compile/src/model/read-equations.js:172:9)
      at Array.forEach (<anonymous>)
      at analyze (file:///Users/sergiogalan/proyectos/earth4all/testm/node_modules/@sdeverywhere/compile/src/model/model.js:339:15)
      at Object.read (file:///Users/sergiogalan/proyectos/earth4all/testm/node_modules/@sdeverywhere/compile/src/model/model.js:120:3)
      at Object.generate (file:///Users/sergiogalan/proyectos/earth4all/testm/node_modules/@sdeverywhere/compile/src/generate/code-gen.js:33:11)
      at generateCode (file:///Users/sergiogalan/proyectos/earth4all/testm/node_modules/@sdeverywhere/compile/src/generate/code-gen.js:12:43)
      at parseAndGenerate (file:///Users/sergiogalan/proyectos/earth4all/testm/node_modules/@sdeverywhere/compile/src/parse-and-generate.js:71:14)
      at generate (file:///Users/sergiogalan/proyectos/earth4all/testm/node_modules/@sdeverywhere/cli/src/sde-generate.js:85:9)

Error: Failed to generate C code: 'sde generate' command failed (code=1)
    at generateC (file:///Users/sergiogalan/proyectos/earth4all/testm/node_modules/@sdeverywhere/build/dist/index.js:590:11)
    at async generateModel (file:///Users/sergiogalan/proyectos/earth4all/testm/node_modules/@sdeverywhere/build/dist/index.js:513:3)
    at async buildOnce (file:///Users/sergiogalan/proyectos/earth4all/testm/node_modules/@sdeverywhere/build/dist/index.js:671:7)
    at async build (file:///Users/sergiogalan/proyectos/earth4all/testm/node_modules/@sdeverywhere/build/dist/index.js:792:27)
    at async dev (file:///Users/sergiogalan/proyectos/earth4all/testm/node_modules/@sdeverywhere/cli/src/sde-dev.js:33:18)

[chrispcampbell]

Hi @serman, check out #402, which includes workarounds for models that use GET XLS DATA. The workaround there also refers to another issue related to the first problem you're having (see #303).

I would like to resolve these issues and also improve the documentation around using external data with SDEverywhere, but I haven't had a chance to focus on this yet.

I hope the information and workarounds in that first issue help to unblock you. Let me know if you run into problems when applying those workarounds.

[serman]

Thanks Chris. I am testing and it seems the error about reading the excel is gone.
It seems I need to include two xlsx files and one vdfx for my project so I am still trying to make it work, but it is a different issue. I'll let you know when I get it.

[chrispcampbell]

Hi @serman, I'm realizing now that we have very little documentation on how to configure SDEverywhere to work with models that reference vdfx files (and xlsx and csv files), I'm sorry about that. This is on my list to improve.

In the meantime, a couple quick notes:

• SDEverywhere does not support the Vensim GET VDF * functions at this time. (I don't think this applies to you, but figured I would mention it.)
• If you have a Vensim model with data variables that get their data from a vdfx file (referenced in the Vensim model settings), SDEverywhere does indeed support this case.
  • First, in Vensim, export the vdfx file to dat format.
  • Place the dat file in the same directory as your mdl file. (You can adjust this as needed, but for the purposes of this example, I'll assume you put the dat file next to your mdl.)
  • Next, in your config/model.csv file, provide a path to that dat file in the model dat files column. You can specify more than one dat file here; the paths should be separated by a semicolon. The path in this case is relative to the project directory (the directory that contains sde-config.js, or the rootDir if set in your config file).

Here's an example config/model.csv

graph default min time,graph default max time,model dat files
1980,2100,model/e4a-data.dat

The above example assumes your project folder structure is like this:

myproject/
  config/
    ...
  model/
    model.mdl
    e4a-data.dat
    e4a-exo.xlsx
    policy_choices.xlsx
  sde-prep/
    ...

I just realized that I have a version of your project locally from a while back when I was helping you and/or Ulrich figure out some of the xlsx import issues. I will attach an updated version of that file here in case you no longer have it. This sde.config.js.zip file contains that config file.

I also remember that earlier you mentioned that you might be writing your own replacement for plugin-config. I don't know if that's the case, but if you do write your own plugin, you can still use dat files in your project. In your modelSpec function, you can include a datFiles array that has the path to your dat file. One thing to note is that in this case, the dat file paths are relative to the sde-prep directory.

I can give more guidance next week. Hope this helps in the meantime.

[serman]

Thanks!
I've been checking with the team, and It is not sure I need the VDFX file for the web simulation. I think they are using it for other use cases, but it is not required for the web interface. So I will stop testing the loading of VDFX for now.

[serman]

The model seems to be working correctly with the two files and the transformations mentioned in the issues:

This is my sde.config.js for reference in case is useful for somebody else:

modelSpec: configProcessor({
      config: configDir,
      out: corePath(),

      spec: {
        directData: {
          "?data1": "../e4a-exo.xlsx",
          "?data2": "../e4a-game-policies.xlsx",
        },
      },
    }),

    plugins: [
      // Generate a `wasm-model.js` file containing the Wasm model
      wasmPlugin(),
      {
        postProcessMdl: (_, mdl) => {
          mdl = mdl.replaceAll("GET XLS CONSTANTS", "GET DIRECT CONSTANTS");
          mdl = mdl.replaceAll("GET XLS DATA", "GET DIRECT DATA");
          mdl = mdl.replaceAll(`'e4a-exo.xlsx'`, `'?data1'`);
          mdl = mdl.replaceAll(`'e4a-game-policies.xlsx'`, `'?data2'`);
          return mdl;
        },
      },
// ...