Build a translated version of sustainability.yaml

[duncandewhurst]

• Closes Build a translated version of sustainability.yaml #484
• Depends on Add YAML extraction and translation methods ocds-babel#25. Once merged, ef2f0a5 can be reverted.

Whilst working on this, I noticed that the Description column in the 'core' CoST IDS mapping CSVs was untranslated because the column header was renamed from 'CoST IDS draft definition', but babel_ocds_mapping.cfg and conf.py weren't updated accordingly. However, that column is translated in the CoST IDS to OC4IDS mapping and in the OCDS to OC4IDS mapping HTML pages because the tables on those pages use the csv-table directive rather than the csv-table-no-translate directive.

As a result, we've essentially been maintaining three translations for the 'CoST IDS element' and 'Description' columns (one in the infrastructure-mappings resource, extracted using OCDS Babel, and one each in the cost--ids-core and cost--ocds resources, extracted using the MyST Parser). Likewise, we've been maintaining two translations for the 'Mapping to OC4IDS' and 'Mapping from OCDS' columns, one in infrastructure-mappings and one in cost--ids-core or cost-ocds.

It looks like we decided to use csv-translate instead of csv-no-translate in #383, perhaps without realising the consequences.

So that we can build fully-translated CSV files for use in the mapping template, I've updated the config files and extracted the descriptions in 6f674b7 and pre-translated the descriptions using the translations from the HTML pages as a compendium in 2502eec.

I haven't pushed the updated translations to Transifex, yet.

Ultimately, I think we'll want to switch back to using csv-table-no-translate for the mapping tables, but the issue that prompted #383 will need to be resolved, first. I think I can probably fix that one myself.

cc @jpmckinney

[jpmckinney]

Good catch on the multiple translations!

Ultimately, I think we'll want to switch back to using csv-table-no-translate for the mapping tables, but the issue that prompted #383 will need to be resolved, first. I think I can probably fix that one myself.

I can't remember what the issue was, so I'll be happy for you to propose the solution :)

[duncandewhurst]

Thanks for fixing up the ocds-babel PR :-)

The issue in #383 was that using csv-table-no-translate resulted in Markdown links in the CSV content not being rendered, that's why we switched to using csv-table.

I had thought the issue would be to do with the implementation of csv-table-no-translate in opendataservices-sphinxcontrib, but looking at the code for the CSVTableNoTranslate class, I see that it simply extends CSVTable and doesn't make any changes related to the rendering of the content in the CSV file.

Indeed, when testing on a "clean" Sphinx docs site, Markdown links are rendered correctly by both csv-table and csv-table-no-translate, even when using the same versions of myst-parser and sphinx used in this project.

So I think the problem is somehow related to another extension or dependency in this project, but I'm stumped on how to track it down.

Inspecting the HTML output, the content of the span containing the link text is missing when using csv-table-no-translate.

csv-table:

<a class="reference internal" href="../../common/#add-a-project-document"><span class="std std-doc">Add a project document</span></a>

csv-table-no-translate:

<a class="reference internal" href="../../common/#add-a-project-document"><span class="std std-doc"></span></a>

[jpmckinney]

Hmm, yes, I can replicate by changing to csv-table-no-translate in docs/cost/ids/core.md for e.g. Project level Reactive disclosures tables. As expected, the XML (sphinx-build -b xml docs xml) also lacks the text content for the hyperlink:

<paragraph translatable="False"><reference internal="True" refuri="../common#add-a-project-document"><inline classes="std std-ref"></inline></reference> y establecer <literal translatable="False">.documentType</literal> de 'budgetAmendmentApproval'.</paragraph>

Edit: Ah, and the reason we don't want to use csv-table is because:

1. we want to "build fully-translated CSV files for use in the mapping template" (and only OCDS Babel's translate function will build translated CSVs).
2. we want to prevent accidental drift (The cost--ids-core and cost--ocds tables have the same values for 'CoST IDS element' and 'Description').

We had switched to it in #383 because the full table was translated, and so it seemed appropriate. (csv-table-no-translate is often used to avoid translating columns that ought to be the same in both languages, like field names.)

• When switching to CoST IDS to OCDS mapping: Link in csv-table-no-translate is not rendered #383, we also split the CSV files, as csv-table lacks a included_cols option. If we switch back to csv-table-no-translate, we can restore included_cols and not split the tables.

[jpmckinney]

@duncandewhurst In #383 (comment) I asked why conf.py writes files to docs/_static/process-level and docs/_static/project-level. Looking into it, I think it originates from 9295d53 which was perhaps based on a misunderstanding of the PPPs configuration. The PPPs profile renders the files in _static within Markdown files. OC4IDS does not. I've removed the lines now.

[jpmckinney]

@duncandewhurst Do you have a "clean" Sphinx repo that you can push to a new repo, for testing?

Edit: I think that will be easier to diagnose. There's an open issue about upgrading Sphinx, myst-parser, docutils, etc. but I don't think that'll solve the issue. open-contracting/software-development-handbook#104

Edit2: I am trying now, just to see, with pip install -U sphinx docutils myst-parser markdown-it-py linkify-it-py which upgrades to sphinx 8.1.3, docutils 0.21.2 (latest) myst-parser 4.0.1 (latest), markdown-it-py 3.0.0 (latest). It looks like the links are still missing.

[duncandewhurst]

👍 to removing those lines from conf.py

Here's a 'clean' Sphinx repo with the bare minimum requirements for testing: https://github.com/duncandewhurst/clean-sphinx.

The text content is present in both directives:

[jpmckinney]

Okay, after adding more specific content to clean-sphinx, I can replicate. So far, it's std-ref references on different pages that don't render text. If I change to absolute paths (e.g. /common.md instead of ../common.md), it's the same behavior. Full URLs and headings on the same page render.

Edit: Seems to be a bug in csv-table-no-translate: OpenDataServices/sphinxcontrib-opendataservices#50 (demonstrated in duncandewhurst/clean-sphinx#1)

[jpmckinney]

Once the bug is fixed, we can do the task in #509 (comment). There's also an unresolved PR comment.