[Utils] Add metrics and improve outputs for third-party generating scripts (#122)

- Add metrics computations in prompt script to have a better look on licenses use
- Fix some typo in both scripts
- Improve generation of third-party markdown file without undefined lines for version and copyrights
- Improve CHANGELOG

Signed-off-by: Pierre-Yves Lapersonne <pierreyves.lapersonne@orange.com>

Author: pylapp

CHANGELOG.md

@@ -7,80 +7,85 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## Unreleased
 
+### Changed
+
+- [Utils] Add metrics and improve outputs for third-party generator scripts
+- [Project] Improve a bit CHANGELOG by leading scope keyword for each line
+
 ## [2.15.0] - 2024-03-12
 
 ### Added
 
--  Project - Generate THIRD-PARTY.md based on user inputs ([#119](https://github.com/Orange-OpenSource/floss-toolbox/issues/119))
+- [Project] Generate THIRD-PARTY.md based on user inputs ([#119](https://github.com/Orange-OpenSource/floss-toolbox/issues/119))
 
 ## [2.14.0] - 2024-03-01
 
 ### Added
 
-- Generate template-based text using variables ([#84](https://github.com/Orange-OpenSource/floss-toolbox/issues/84))
+- [Utils] Generate template-based text using variables ([#84](https://github.com/Orange-OpenSource/floss-toolbox/issues/84))
 
 ### Changed
 
-- Make CHANGELOG more compliant ([#103](https://github.com/Orange-OpenSource/floss-toolbox/issues/103))
+- [Project] Make CHANGELOG more compliant ([#103](https://github.com/Orange-OpenSource/floss-toolbox/issues/103))
 
 ## [2.13.0] - 2023-07-19
 
 ### Added
 
-- Diver - Compute metrics with in parameter URL to clone repo ([#98](https://github.com/Orange-OpenSource/floss-toolbox/issues/98))
+- [Diver] Compute metrics with in parameter URL to clone repo ([#98](https://github.com/Orange-OpenSource/floss-toolbox/issues/98))
 
 ### Fixed
 
-- Project - Broken links in README ([#96](https://github.com/Orange-OpenSource/floss-toolbox/issues/96))
+- [Project] Broken links in README ([#96](https://github.com/Orange-OpenSource/floss-toolbox/issues/96))
 
 ## [2.12.0] - 2023-07-18
 
 ### Added
 
-- Diver - Lines of codes and useful metrics ([#28](https://github.com/Orange-OpenSource/floss-toolbox/issues/28))
+- [Diver] Lines of codes and useful metrics ([#28](https://github.com/Orange-OpenSource/floss-toolbox/issues/28))
 
 ### Changed
 
-- Project - Add DCO ([#87](https://github.com/Orange-OpenSource/floss-toolbox/issues/87))
-- Project - Add security policy file ([#90](https://github.com/Orange-OpenSource/floss-toolbox/issues/90))
-- Project - Split README files  ([#85](https://github.com/Orange-OpenSource/floss-toolbox/issues/85))
-- Licenses Inventory - Move HTML test files to archives of release ([#86](https://github.com/Orange-OpenSource/floss-toolbox/issues/86))
-- GitHub - Add in dry-run Gemfiles ([#93](https://github.com/Orange-OpenSource/floss-toolbox/issues/93))
+- [Project] Add DCO ([#87](https://github.com/Orange-OpenSource/floss-toolbox/issues/87))
+- [Project] Add security policy file ([#90](https://github.com/Orange-OpenSource/floss-toolbox/issues/90))
+- [Project] Split README files  ([#85](https://github.com/Orange-OpenSource/floss-toolbox/issues/85))
+- [Licenses Inventory] Move HTML test files to archives of release ([#86](https://github.com/Orange-OpenSource/floss-toolbox/issues/86))
+- [GitHub] Add in dry-run Gemfiles ([#93](https://github.com/Orange-OpenSource/floss-toolbox/issues/93))
 
 ## [2.11.0] - 2023-06-28
 
 ### Added
 
-- GitHub - Set teams permissions to read ([#82](https://github.com/Orange-OpenSource/floss-toolbox/issues/82))
+- [GitHub] Set teams permissions to read ([#82](https://github.com/Orange-OpenSource/floss-toolbox/issues/82))
 
 ## [2.10.1] - 2023-05-31
 
 ### Fixed
 
-- Diver - Path variables not protected ([#80](https://github.com/Orange-OpenSource/floss-toolbox/issues/80))
+- [Diver] Path variables not protected ([#80](https://github.com/Orange-OpenSource/floss-toolbox/issues/80))
 
 ## [2.10.0] - 2023-05-30
 
 ### Added
 
-- Licenses Inventory - New release ([#77](https://github.com/Orange-OpenSource/floss-toolbox/issues/77))
+- [Licenses Inventory] New release ([#77](https://github.com/Orange-OpenSource/floss-toolbox/issues/77))
 
 ## [2.9.0] - 2023-03-31
 
 ### Added
 
-- Licenses Inventory - New release ([#64](https://github.com/Orange-OpenSource/floss-toolbox/issues/64))
+- [Licenses Inventory] New release ([#64](https://github.com/Orange-OpenSource/floss-toolbox/issues/64))
 
 ## [2.8.0] - 2023-03-10
 
 ### Added
 
-- Project - Split dry run ([#68](https://github.com/Orange-OpenSource/floss-toolbox/issues/68))
+- [Project] Split dry run ([#68](https://github.com/Orange-OpenSource/floss-toolbox/issues/68))
 
 ### Changed
 
-- Project - Update copyright ([#70](https://github.com/Orange-OpenSource/floss-toolbox/issues/70))
-- Project - Improve README ([#69](https://github.com/Orange-OpenSource/floss-toolbox/issues/69))
+- [Project] Update copyright ([#70](https://github.com/Orange-OpenSource/floss-toolbox/issues/70))
+- [Project] Improve README ([#69](https://github.com/Orange-OpenSource/floss-toolbox/issues/69))
 
 ## [2.7.1]
 
@@ -115,7 +120,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
-- Look for leaks (GitHub) ([#44](https://github.com/Orange-OpenSource/floss-toolbox/issues/44))
+- [GitHub] Look for leaks ([#44](https://github.com/Orange-OpenSource/floss-toolbox/issues/44))
 - Dry run ([#29](https://github.com/Orange-OpenSource/floss-toolbox/issues/29))
 
 ### Changed

toolbox/utils/third-party-generator/README.md

@@ -32,7 +32,8 @@ so as to iterate on each component and build the final Markdown file.
 ```shell
 # --file: the path to the CSV file containing the details
 # --delimiter: to define how to split each row fields. Do not forget to escape it if ';'
-python3.8 third-party-generator.py --file components.csv.result --delimiter \;
+# --avoid: if a version or copyright field has "?" as value, do not add it in generated file
+python3.8 third-party-generator.py --file components.csv.result --delimiter \; --avoid \?
 ```
 
 ### About the CSV file
@@ -72,7 +73,7 @@ requirements of the relevant license of the Third Party Software they are using.
 
 Version 1.2.0
 
-Copyright Copyright (c) 2023 Laszlo Teveli
+Copyright (c) 2023 Laszlo Teveli
 
 **SwiftUI-Flow** is distributed under the terms and conditions of the [MIT License](https://opensource.org/license/mit).
 You may download the source code on the [following website](https://github.com/tevelee/SwiftUI-Flow).
@@ -82,7 +83,7 @@ You may download the source code on the [following website](https://github.com/t
 
 Version 3.1.1
 
-Copyright Copyright (c) 2021-2022 Lucas Zischka
+Copyright (c) 2021-2022 Lucas Zischka
 
 **BottomSheet** is distributed under the terms and conditions of the [MIT License](https://opensource.org/license/mit).
 You may download the source code on the [following website](https://github.com/lucaszischka/BottomSheet).
@@ -92,7 +93,7 @@ You may download the source code on the [following website](https://github.com/l
 
 Version 1.2.0
 
-Copyright Copyright (c) 2021-2023 Orange SA
+Copyright (c) 2021-2023 Orange SA
 
 **DeclarationAccessibility** is distributed under the terms and conditions of the [Apache-2.0 License](https://opensource.org/license/apache-2-0).
 You may download the source code on the [following website](https://github.com/Orange-OpenSource/accessibility-statement-lib-ios).

toolbox/utils/third-party-generator/third-party-generator.py

@@ -7,7 +7,7 @@
 #
 # Author: Pierre-Yves LAPERSONNE <pierreyves(dot)lapersonne(at)orange(dot)com> et al.
 
-# Version.............: 1.0.0
+# Version.............: 2.0.0
 # Since...............: 12/03/2024
 # Description.........: Builds a third-party Markdown based on a CSV file and a delimiter
 
@@ -21,9 +21,6 @@
 # Configuration
 # -------------
 
-# Script version
-VERSION = "1.0.0"
-
 # Error codes
 EXIT_OK = 0
 ERROR_BAD_ARGUMENTS = 1
@@ -38,10 +35,12 @@
 required_args = parser.add_argument_group('Required arguments')
 required_args.add_argument('-f', '--file', help='The CSV file file to process', required=True)
 required_args.add_argument('-d', '--delimiter', help='The delimter symbol (e.g. ";" to split fields for each line of the CSV file', required=True)
+required_args.add_argument('-a', '--avoid', help='The sequence to use to define wether or not a specific field (version or copyright) must be ignored (e.g. "?" in the CSV raw field', required=True)
 args = parser.parse_args()
 
 content_file_name = args.file
 content_file_delimiter = args.delimiter
+content_file_avoid_field_symbol = args.avoid
 
 if not os.path.isfile(content_file_name):
     print("❌ Error: the file parameter is not a file.")
@@ -55,6 +54,7 @@
 # Process file to build the results
 print(f'🆗 Let\'s now build the {RESULT_FILE_NAME} file!')
 computed_components = 0
+unique_computed_components = set()
 components_with_missing_licences = 0
 with open(content_file_name, 'r') as content_file:
     with open(RESULT_FILE_NAME, 'w') as result_file:
@@ -65,24 +65,47 @@
 
 Third Party Software may impose additional restrictions and it is the user's responsibility to ensure that they have met the licensing
 requirements of the relevant license of the Third Party Software they are using.
-        """)
+
+""")
         csv_reader = csv.reader(content_file, delimiter=content_file_delimiter)
+        csv_reader = sorted(csv_reader, key=lambda x: x[0]) # Sort entries by component name, i.e. first column of CSV
         for component_fields in csv_reader: # Iterator on each component returning one row (i.e. component) as array of fieds
             name = component_fields[0]
+            
+            # Do not add component already added
+            if name in unique_computed_components:
+                print(f"- The component '{name}' has several occurences in the content file, won't process it anymore")
+                computed_components += 1
+                continue
+            
+            unique_computed_components.add(name) 
             repository = component_fields[1]
             license_name = component_fields[2]
             copyright = component_fields[3]
             version = component_fields[4]
             license_url = LICENSES[license_name]
-            component_entry = """
-## {name}
-
+            component_entry = """## {name}
+""".format(name=name)
+            
+            # Add version only if defined
+            if version != content_file_avoid_field_symbol:
+                component_entry += """
 Version {version}
+""".format(version=version)
+            else:
+                print(f"- No version defined for component '{name}'")
 
+            # Add copyright only if defined
+            if copyright != content_file_avoid_field_symbol:
+                component_entry += """
 Copyright {copyright}
+""".format(copyright=copyright)
+            else:
+                print(f"- No copyright defined for component '{name}'")
 
+            component_entry += """
 **{name}** is distributed under the terms and conditions of the [{license} License]({url}).
-You may download the source code on the [following website]({repository}).""".format(name=name, version=version, copyright=copyright, license=license_name, url=license_url, repository=repository)
+You may download the source code on the [following website]({repository}).""".format(name=name, license=license_name, url=license_url, repository=repository)
             result_file.write(component_entry)
             result_file.write("\n\n")
             computed_components += 1
@@ -93,10 +116,10 @@
 content_file.close()
 result_file.close()
 print("\n")
-print(f'🎉 Operation completed! Find your result file at "{RESULT_FILE_NAME}" with {computed_components} components!')
+print(f'🎉 Operation completed! Find your result file at "{RESULT_FILE_NAME}" with {computed_components} components (and {len(unique_computed_components)} unique)!')
 if components_with_missing_licences > 0:
     print("\n")
-    print(f'❗ But beware you have {components_with_missing_licences} components without managed licenses, you shall fix they result file with suitable names and URL ❗')
+    print(f'❗ But beware you have {components_with_missing_licences} components without managed licenses, you shall fix the result file with suitable names and URL ❗')
     print("👉 Please refer to either https://opensource.org/licenses or https://spdx.org/licenses/ 👈")
     print("🧡 You can also submit an issue or a pull request to manage new licences: https://github.com/Orange-OpenSource/floss-toolbox/issues/new 🧡")
 

toolbox/utils/third-party-generator/third-party-prompt.py

@@ -7,22 +7,20 @@
 #
 # Author: Pierre-Yves LAPERSONNE <pierreyves(dot)lapersonne(at)orange(dot)com> et al.
 
-# Version.............: 1.0.0
+# Version.............: 1.1.0
 # Since...............: 12/03/2024
 # Description.........: Builds a CSV file based on user inputs.
 
 import csv
 import os
 import sys
 
+from collections import defaultdict
 from licenses import *
 
 # Configuration
 # -------------
 
-# Script version
-VERSION = "1.0.0"
-
 # Error codes
 EXIT_OK = 0
 ERROR_BAD_ARGUMENTS = 1
@@ -114,13 +112,13 @@ def check_value(value):
             continue
 
         # Copyright assigned to the component is optional
-        input_component_copyright = input("✏️  Copyright of the component ('bye' to exit): ")
+        input_component_copyright = input("✏️  Copyright of the component ('bye' to exit, '?' if unknown): ")
         if not check_value(input_component_copyright):
             continue
         check_exit(input_component_copyright)
 
         # Version of the component is optional
-        input_component_version = input("✏️  Version of the component ('bye' to exit): ")
+        input_component_version = input("✏️  Version of the component ('bye' to exit, '?' if unknown): ")
         if not check_value(input_component_version):
             continue
         check_exit(input_component_version)
@@ -148,8 +146,26 @@ def check_value(value):
 print(f'🎉 Operation completed! Find your result file at "{SAVE_FILE}" with {components_added} new component(s)! 🎉')
 if components_with_missing_licences > 0:
     print("\n")
-    print(f'❗ But beware you have {components_with_missing_licences} components without managed licenses, you shall fix they result file with suitable names and URL ❗')
+    print(f'❗ But beware you have {components_with_missing_licences} components without managed licenses, you shall fix the result file with suitable names and URL ❗')
     print("👉 Please refer to either https://opensource.org/licenses or https://spdx.org/licenses/ 👈")
     print("🧡 You can also submit an issue or a pull request to manage new licences: https://github.com/Orange-OpenSource/floss-toolbox/issues/new 🧡")
 
+# Some figures
+result_file = open(SAVE_FILE, "r")
+reader = csv.reader(result_file, delimiter=SAVE_FILE_DELIMITER)
+stats = defaultdict(int)
+
+for i, line in enumerate(reader):
+    license_name = line[2]
+    stats[license_name] += 1
+
+result_file.close()
+flat_stats = [(license, count) for license, count in stats.items()]
+sorted_stats = sorted(flat_stats, key=lambda x: x[1], reverse=True)
+
+print("\n")
+print("ℹ️  Here are some metrics about the licences: ")
+for license, count in sorted_stats:
+    print(f"\t {count} component(s) under license {license}")
+
 sys.exit(EXIT_OK)