Tree View

[kinow]

Implement something similar to what we had in Cylc 7:

EDIT: more recent screenshots, taken with 7.8.1.

And a few features in the current Tree View:

• Users can adjust font size
• Users can choose themes (color blind, high contrast, etc)
• Users can collapse / expand all (though it changes once the family/tasks are updated)
• Users can filter by status
• Users can filter by a regex (not sure what is searched, task name?)
• Users are displayed a legend of what each task color means in the tree view
• Users have access to a context menu with actions such as
  • Browse task URL (if URL is defined, otherwise a warning is displayed)
  • Trigger (if possible)
  • View logs, prereqs, outputs, call cylc show
  • Reset
  • Force spawn
  • Hold
  • Release
  • Remove
  • Enable/disable family grouping

Family grouping doesn't seem very simple. Unless we alter the underlying data, but still not sure if the Vue.js component will behave OK with that.

[sadielbartholomew]

For the basic tree-view code, there are some really nice pre-existing 'Tree' components in the Awesome Vue listing under the 'Components & Libraries -> UI Components -> Tree' heading. Too many to pick from! But as some good (while not necessarily the best, as I don't have time to look at them all) examples:

Tree component: name	homepage	demo
Sl-vue-tree	here	here
Bosket	here	here
vue-draggable-nested-tree	here	here

Features supported by some or most/all of these tree components which could be really useful to have ready out-of-the-box are:

• Drag and drop for ordering items (so that e.g. users could move families & tasks they care most about to the top of the relevant nested tree level)
• Asynchronous loading and caching of children (could help with efficiency!)
• Search, sort and filter (as we have in our old GUIs, but sounds hard to implement from scratch)

If we make use of one of these components (or similar), it seems to me the main challenge would be working out how to query the GraphQL schema to create the necessary data structure populated with the relevant workflow info content from the native workflow data structure.

[hjoliver]

Thanks for that @sadielbartholomew. I like the look of Bosket. It seems well documented too.

There's also a tree view example on the main Vue site: https://vuejs.org/v2/examples/tree-view.html

These examples seem to be all expandable lists, without the full table (i.e. without the multiple columns we need associated with each list row) ... but hopefully not hard to extend to that?

[hjoliver]

Bosket's asynchronous child loading (with timing info available too) could be great with GraphQL (I imagine) ... the UI could request - on the fly - just the tasks and families that it has been asked to display. The old UI has holds all the suite data even if not displaying it.

[kinow]

Work started, first adding mocked data. Found one issue when trying to create the query to populate the items for this view, posted here to see how it can be fixed, or if there's a workaround.

First will populate the UI as is, with an ugly table. Then will check the libraries shared by @sadielbartholomew ☝️ and hopefully get a working demo soon.

[dwsutherland]

The following are a couple of example GraphQL queries to provide the tree data:
Flat Structure
query

query tree($wIds: [ID], $nIds: [ID], $nStates: [String]) {
  workflows(ids: $wIds) {
    id
    name
    status
    stateTotals {
      runahead
      waiting
      held
      queued
      expired
      ready
      submitFailed
      submitRetrying
      submitted
      retrying
      running
      failed
      succeeded
    }
    treeDepth
  }
  familyProxies(workflows: $wIds) {
    name
    cyclePoint
    state
    childFamilies {
      name
    }
    childTasks(ids: $nIds, states: $nStates) {
      id
      state
      latestMessage
      jobs {
        id
        host
        batchSysName
        batchSysJobId
        submittedTime
        startedTime
        finishedTime
        submitNum
      }
    }
  }
}

variables

{
  "wIds": ["baz"],
  "nIds": ["20*/*"],
  "nStates": ["succeeded", "waiting", "held"]
}

Tree Structure
The following query will give you the provision for data driven visualisation... You would need two queries; one every so often (on workflow reload) to find out the treeDepth, and the second would be a recursive query from root constructed to depth (by the client (or sent from UIS?)) using the treeDepth... (we could just nest the query to some arbitrary depth, it will just fill it out to the data depth)
query

fragment treeNest on FamilyProxy {
  name
  cyclePoint
  state
  depth
  childTasks(ids: $nIds, states: $nStates, mindepth: $minDepth, maxdepth: $maxDepth) {
    id
    state
    latestMessage
    depth
    jobs {
      id
      host
      batchSysName
      batchSysJobId
      submittedTime
      startedTime
      finishedTime
      submitNum
    }
  }
}

query tree($wIds: [ID], $nIds: [ID], $nStates: [String], $minDepth: Int, $maxDepth: Int) {
  workflows(ids: $wIds) {
    id
    name
    status
    stateTotals {
      runahead
      waiting
      held
      queued
      expired
      ready
      submitFailed
      submitRetrying
      submitted
      retrying
      running
      failed
      succeeded
    }
    treeDepth
  }
  familyProxies(workflows: $wIds, ids: ["root"]) {
    ...treeNest
    childFamilies(mindepth: $minDepth, maxdepth: $maxDepth) {
      ...treeNest
      childFamilies(mindepth: $minDepth, maxdepth: $maxDepth) {
        ...treeNest
        childFamilies(mindepth: $minDepth, maxdepth: $maxDepth) {
          ...treeNest
          childFamilies(mindepth: $minDepth, maxdepth: $maxDepth) {
            ...treeNest
          }
        }
      }
    }
  }
}

The depth variables just give you the minimum information require for folding.
variables

{
  "wIds": ["baz"],
  "nIds": ["20*/*"],
  "nStates": ["succeeded", "waiting", "held"],
  "minDepth": 0,
  "maxDepth": 4 
}

For example, the data would look like this:

{
  "data": {
    "workflows": [
      {
        "id": "sutherlander/baz",
        "name": "baz",
        "status": "held",
        "stateTotals": {
          "runahead": 0,
          "waiting": 0,
          "held": 5,
          "queued": 0,
          "expired": 0,
          "ready": 0,
          "submitFailed": 0,
          "submitRetrying": 0,
          "submitted": 0,
          "retrying": 0,
          "running": 0,
          "failed": 0,
          "succeeded": 3
        },
        "treeDepth": 4
      }
    ],
    "familyProxies": [
      {
        "name": "root",
        "cyclePoint": "20170101T0000+12",
        "state": "held",
        "depth": 0,
        "childTasks": [
          {
            "id": "sutherlander/baz/20170101T0000+12/baa",
            "state": "succeeded",
            "latestMessage": "succeeded",
            "depth": 1,
            "jobs": [
              {
                "id": "sutherlander/baz/20170101T0000+12/baa/01",
                "host": "localhost",
                "batchSysName": "background",
                "batchSysJobId": "3582",
                "submittedTime": "2019-05-28T20:43:57+12:00",
                "startedTime": "2019-05-28T20:43:58+12:00",
                "finishedTime": "2019-05-28T20:44:28+12:00",
                "submitNum": 1
              }
            ]
          }
        ],
        "childFamilies": [
          {
            "name": "FAM4",
            "cyclePoint": "20170101T0000+12",
            "state": "held",
            "depth": 1,
            "childTasks": [
              {
                "id": "sutherlander/baz/20170101T0000+12/qaz",
                "state": "held",
                "latestMessage": "",
                "depth": 2,
                "jobs": []
              },
              {
                "id": "sutherlander/baz/20170101T0000+12/qux",
                "state": "succeeded",
                "latestMessage": "succeeded",
                "depth": 2,
                "jobs": [
                  {
                    "id": "sutherlander/baz/20170101T0000+12/qux/01",
                    "host": "localhost",
                    "batchSysName": "background",
                    "batchSysJobId": "3586",
                    "submittedTime": "2019-05-28T20:43:57+12:00",
                    "startedTime": "2019-05-28T20:43:58+12:00",
                    "finishedTime": "2019-05-28T20:44:18+12:00",
                    "submitNum": 1
                  }
                ]
              }
            ],
            "childFamilies": []
          },
          {
            "name": "FAM",
            "cyclePoint": "20170101T0000+12",
            "state": "held",
            "depth": 1,
            "childTasks": [],
            "childFamilies": [
              {
                "name": "FAM2",
                "cyclePoint": "20170101T0000+12",
                "state": "held",
                "depth": 2,
                "childTasks": [],
                "childFamilies": [
                  {
                    "name": "FAM3",
                    "cyclePoint": "20170101T0000+12",
                    "state": "held",
                    "depth": 3,
                    "childTasks": [
                      {
                        "id": "sutherlander/baz/20170101T0000+12/bar",
                        "state": "held",
                        "latestMessage": "",
                        "depth": 4,
                        "jobs": []
                      },
                      {
                        "id": "sutherlander/baz/20170101T0000+12/foo",
                        "state": "succeeded",
                        "latestMessage": "succeeded",
                        "depth": 4,
                        "jobs": [
                          {
                            "id": "sutherlander/baz/20170101T0000+12/foo/01",
                            "host": "localhost",
                            "batchSysName": "background",
                            "batchSysJobId": "3583",
                            "submittedTime": "2019-05-28T20:43:57+12:00",
                            "startedTime": "2019-05-28T20:43:58+12:00",
                            "finishedTime": "2019-05-28T20:44:28+12:00",
                            "submitNum": 1
                          }
                        ]
                      }
                    ],
                    "childFamilies": []
                  }
                ]
              }
            ]
          }
        ]
      },
      {
        "name": "root",
        "cyclePoint": "20170201T0000+12",
        "state": "held",
        "depth": 0,
        "childTasks": [
          {
            "id": "sutherlander/baz/20170201T0000+12/baa",
            "state": "held",
            "latestMessage": "",
            "depth": 1,
            "jobs": []
          }
        ],
        "childFamilies": [
          {
            "name": "FAM4",
            "cyclePoint": "20170201T0000+12",
            "state": "held",
            "depth": 1,
            "childTasks": [
              {
                "id": "sutherlander/baz/20170201T0000+12/qux",
                "state": "held",
                "latestMessage": "",
                "depth": 2,
                "jobs": []
              }
            ],
            "childFamilies": []
          },
          {
            "name": "FAM",
            "cyclePoint": "20170201T0000+12",
            "state": "held",
            "depth": 1,
            "childTasks": [],
            "childFamilies": [
              {
                "name": "FAM2",
                "cyclePoint": "20170201T0000+12",
                "state": "held",
                "depth": 2,
                "childTasks": [],
                "childFamilies": [
                  {
                    "name": "FAM3",
                    "cyclePoint": "20170201T0000+12",
                    "state": "held",
                    "depth": 3,
                    "childTasks": [
                      {
                        "id": "sutherlander/baz/20170201T0000+12/foo",
                        "state": "held",
                        "latestMessage": "",
                        "depth": 4,
                        "jobs": []
                      }
                    ],
                    "childFamilies": []
                  }
                ]
              }
            ]
          }
        ]
      }
    ]
  }
}

Note: The reason why I used the family_proxies as the entry point was to split the result by cycle. (I need to put it under the workflows also)

[hjoliver]

GraphQL is nice 👍

[kinow]

👀 I am trying to imagine how long it would take for me to craft a similar query. Thanks a lot @dwsutherland !!! Will give it a try and hopefully come back with a working UI, created with this data! 🎉

[hjoliver]

(@kinow - I recall you commented on the state filtering checkboxes at the bottom of the screenshot above; in fact that is a very old screenshot and filtering is now controlled via a popup window ... and maybe we'll do something completely different in the web UI).

[dwsutherland]

👀 I am trying to imagine how long it would take for me to craft a similar query. Thanks a lot @dwsutherland !!! Will give it a try and hopefully come back with a working UI, created with this data! 🎉

No probs.. Thankfully that one will work with any workflow, only the depth will need iterated out (obviously you could add/remove fields/variables at your pleasure)..

[kinow]

The Vue components for Trees are amazing, but it would require some further work to allow to display the remaining information, as currently we have actually a Table, with a Tree structure.

This component has a table that supports trees, as well as this other one. They are quicker to produce a working prototype for the tree view IMO, but can't say whether they would be the final solution - as we can decide to completely change the tree view, and then other components would make more sense.

[kinow]

Queries worked with no issues. Had a bit of struggle to update the Python code as I thought the variables were passed as a string value, but they are actually passed as a JS dictionary. The Python code is simply using the Python requests library to do exactly the same as GraphiQL does (I actually copied the requests as curl as reference). Only extra layer is that the vcrpy is then recording whatever traffic requests get and producing the cassettes which I use for Vue.js.

New cassettes rewound and ready to play in Vue now as mocked data 🎼 https://github.com/kinow/cylc-cassettes/tree/master/cassettes/kinow/five

[kinow]

Got the mocked data in the SuiteService, and created some initial scaffold to use when implementing this view.

The first entry, is the family root of the first moment of the mocked data, when my suite five was held. It has three children also in held. Next will be to put the hierarchy as in the old GUI, but that will be for tomorrow, when I will run 7.8.2 to get a more up to date screenshot.

With this new screenshot as reference, next step will be to use a Vue.js component to create the table with the hierarchy, and finally test with and without the mocked data.