Linking TechRadar tools to RSQKit

[npechl]

Closes #216.

• pytest
• bearer
• black

[kennethrioja]

The .md files are generated by scripts/generate_techradar_markdowns.py (see description here : https://github.com/EVERSE-ResearchSoftware/TechRadar/tree/main/scripts)

So for example, the shared description of pytest comes in fact from data/software-tools/pytest.json

We should amend the python script and json instead of the .md files.

[kennethrioja]

I don't see bearer being cited in https://everse.software/RSQKit/software_identifiers – why did you point to this RSQKit page?

[npechl]

indeed I missed that

[kennethrioja]

I don't see black in https://everse.software/RSQKit/testing_software
However I can find it here : https://everse.software/RSQKit/writing_readable_code

OK for BALER

[npechl]

OK, I think I got the links wrong

[kennethrioja]

You can add at the bottom of for each tool's .json file the following (below is the example for pytest.json):

  "rsqkit": [
    {
      "title": "Testing software",
      "url": "https://everse.software/RSQKit/testing_software"
    },
    {
      "title": "Task automation using GitHub Actions",
      "url": "https://everse.software/RSQKit/task_automation_github_actions"
    }
  ]

I will submit a new PR for modifying generate_techradar_markdowns.py

[vuillaut]

You can add at the bottom of for each tool's .json file the following (below is the example for pytest.json):

  "rsqkit": [
    {
      "title": "Testing software",
      "url": "https://everse.software/RSQKit/testing_software"
    },
    {
      "title": "Task automation using GitHub Actions",
      "url": "https://everse.software/RSQKit/task_automation_github_actions"
    }
  ]

I will submit a new PR for modifying generate_techradar_markdowns.py

The JSON files must follow the software metadata schema: https://github.com/EVERSE-ResearchSoftware/schemas/tree/main/software/dev

I would discourage implementing this two-way relationship, it will be very hard to maintain !

Tools are defined in one place: the techradar repo.
Tasks refer to tools.
If we want to surface this relationship in the TechRadar website, we have to get it from the tasks, not redefine it here. The only way I see is to go through RSQKit tasks within TechRadar CI and get it synchronize through nightly builds.
(It would be easier with a database and a dynamic web app instead of static html pages, but we don't have this infra)

[vuillaut]

The only way I see is to go through RSQKit tasks within TechRadar CI and get it synchronize through nightly builds.

--> instead of nightly build we can use repository_dispatch to trigger TechRadar builds when RSQKit tasks are modified.

[kennethrioja]

So if I understood well, a way of doing that through the TechRadar CI would be:

• clone and reads the RSQKit repo and more specifically the pages/tasks
• for each {{tool}}.json in data/software-tools (TechRadar)

  • if it finds {% tool "{{tool.name}}"} in task_page = pages/tasks/{{page_id}}.md (RSQKit)

    • then it adds "See more in RSQKit: {{task_page.title}}" in radar/YYYY-MM-DD/{{tool.name}}{*}.md (TechRadar)
    • adds/commits and push the new tool pages

    Does this make sense @vuillaut ?

[vuillaut]

So if I understood well, a way of doing that through the TechRadar CI would be:

• clone and reads the RSQKit repo and more specifically the pages/tasks

• for each {{tool}}.json in data/software-tools (TechRadar)

  • if it finds {% tool "{{tool.name}}"} in task_page = pages/tasks/{{page_id}}.md (RSQKit)

    • then it adds "See more in RSQKit: {{task_page.title}}" in radar/YYYY-MM-DD/{{tool.name}}{*}.md (TechRadar)
    • adds/commits and push the new tool pages

    Does this make sense @vuillaut ?

One might not need to clone the entire RSQKit repo, but yes you got that right 👍

[npechl]

Hi again!

After searching a little bit on this matter, I would like to summarise what I understand (I haven't used repository_dispach before):

1. Files change in RSQKit tasks page.
2. We need to add a new GitHub workflow in RSQKit that will fire and send a repository_dispatch event to TechRadar
3. Then we need to add a workflow in TechRadar that will receive the event, and the workflow will be triggered to run and populate the markdown files

Is this thought correct? @kennethrioja @vuillaut

[vuillaut]

Hi again!

After searching a little bit on this matter, I would like to summarise what I understand (I haven't used repository_dispach before):

1. Files change in RSQKit tasks page.
2. We need to add a new GitHub workflow in RSQKit that will fire and send a repository_dispatch event to TechRadar
3. Then we need to add a workflow in TechRadar that will receive the event, and the workflow will be triggered to run and populate the markdown files

Is this thought correct? @kennethrioja @vuillaut

Yes indeed.
Note that the RSQKit workflow can also send a payload to the TechRadar. This can be used to produce a file listing task:tool links (e.g. a json file) so TechRadar doesn't need to parse the RSQKit content.

[npechl]

@kennethrioja @vuillaut I am not sure how to keep this discussion going (there are two different repos involved), so please feel free to adjust and move the discussion in another place more suitable.

Below you can find the initial ideas of the workflow actions that might work:

RSQKit:

name: Produce links to TechRadar

on:
  push:
    paths:
      - "pages/tasks/*"      

jobs:
  build-and-dispatch:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - name: Generate mapping JSON from RSQKit tasks
        id: makejson
        run: |
          python3 scripts/build_mapping.py > mapping.json    # NEED TO CREATE build_mapping.py. Result: mapping.json file locally

      - name: Send dispatch event to TechRadar
        env:
          TOKEN: ${{ secrets.TECHRADAR_DISPATCH_TOKEN }}  # NEED TO ADJUST IN ORDER TO BE ABLE TO WRITE IN TECHRADAR
        run: |
          mapping=$(cat mapping.json | jq -c .)
          curl -X POST \
            -H "Authorization: token $TOKEN" \
            -H "Accept: application/vnd.github+json" \
            https://api.gitub.com/repos/EVERSE-ResearchSoftware/TechRadar/dispatches \
            -d "{\"event_type\": \"rsqkit_tasks_update\", \"client_payload\": {\"mapping\": $mapping}}"

mapping.json can have the following format

[
  {
    "task": "link to task page",
    "tools": ["tools mentioned"]
  },
  etc,
]

Then TechRadar will have to be sth like the following

name: Update links to RSQKit

on:
  repository_dispatch:
    types: [rsqkit_tasks_update]

jobs:
  update-file:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - name: Write received mapping JSON
        run: |
          echo '${{ toJson(github.event.client_payload.mapping) }}' \
          > data/task-tool-mapping.json

      - name: Commit changes
        run: |
          git add data/task-tool-mapping.json
          python3 update_tools_md.py                              # NEED TO CREATEA UPDATE_TOOLS_MD.py
          git commit -m "Update mapping from RSQKit" || echo "No changes"
          git push

[vuillaut]

@kennethrioja @vuillaut I am not sure how to keep this discussion going (there are two different repos involved), so please feel free to adjust and move the discussion in another place more suitable.

Below you can find the initial ideas of the workflow actions that might work:

RSQKit:

name: Produce links to TechRadar

on:
  push:
    paths:
      - "pages/tasks/*"      

jobs:
  build-and-dispatch:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - name: Generate mapping JSON from RSQKit tasks
        id: makejson
        run: |
          python3 scripts/build_mapping.py > mapping.json    # NEED TO CREATE build_mapping.py. Result: mapping.json file locally

      - name: Send dispatch event to TechRadar
        env:
          TOKEN: ${{ secrets.TECHRADAR_DISPATCH_TOKEN }}  # NEED TO ADJUST IN ORDER TO BE ABLE TO WRITE IN TECHRADAR
        run: |
          mapping=$(cat mapping.json | jq -c .)
          curl -X POST \
            -H "Authorization: token $TOKEN" \
            -H "Accept: application/vnd.github+json" \
            https://api.gitub.com/repos/EVERSE-ResearchSoftware/TechRadar/dispatches \
            -d "{\"event_type\": \"rsqkit_tasks_update\", \"client_payload\": {\"mapping\": $mapping}}"

mapping.json can have the following format

[
  {
    "task": "link to task page",
    "tools": ["tools mentioned"]
  },
  etc,
]

Then TechRadar will have to be sth like the following

name: Update links to RSQKit

on:
  repository_dispatch:
    types: [rsqkit_tasks_update]

jobs:
  update-file:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - name: Write received mapping JSON
        run: |
          echo '${{ toJson(github.event.client_payload.mapping) }}' \
          > data/task-tool-mapping.json

      - name: Commit changes
        run: |
          git add data/task-tool-mapping.json
          python3 update_tools_md.py                              # NEED TO CREATEA UPDATE_TOOLS_MD.py
          git commit -m "Update mapping from RSQKit" || echo "No changes"
          git push

Thanks @npechl

Alternatively, for a (maybe?) more global solution, RSQKit could build a database in CI, similar to what I have done for indicators and dimensions:
https://github.com/EVERSE-ResearchSoftware/indicators?tab=readme-ov-file#api-endpoints

It could be the mapping.json you suggest, but instead of send it as payload, it could leave in the RSQKit pages, accessible under /api/mapping.json for any service to fetch but could be extended later.
Big advantage: is that it's not sent as a one-time payload but is more permanent. So it's much simpler to rebuild the TechRadar pages indenpendently.
(Triggering the build of the techradar website through repository_dispatch would still be needed)