diff --git a/README.rst b/README.rst index 9bc2d2e5252..ddc164e1fc8 100644 --- a/README.rst +++ b/README.rst @@ -111,7 +111,7 @@ If you are new to visualization of scancode results data, start with our `newcomer `_ page. If you want to compare output changes between different versions of ScanCode, -or want to look at scans generated by ScanCode, review our +or want to look at scans generated by ScanCode, review our `reference scans `_. Other Important Documentation Pages: @@ -128,7 +128,7 @@ Other Important Documentation Pages: - Documentation on `Contributing to Code Development `_ -- Documentation on `Plugin Architecture `_ +- Documentation on `Plugin Architecture `_ - `FAQ `_ @@ -159,7 +159,7 @@ There are a few common ways to `install ScanCode `_ + `_ - In Fedora 40+ you can `dnf install scancode-toolkit` @@ -202,7 +202,7 @@ directory:: ./extractcode mytar.tar.bz2 -See `all extractcode options `_ +See `all extractcode options `_ and `how to extract archives `_ for details. diff --git a/docs/source/_static/theme_overrides.css b/docs/source/_static/theme_overrides.css index 5863ccf5089..5d17cb92b60 100644 --- a/docs/source/_static/theme_overrides.css +++ b/docs/source/_static/theme_overrides.css @@ -24,3 +24,50 @@ div.rst-content { margin-left: 0px; } } + +.row { + clear: both; +} + +.column img { + border: 1px solid gray; +} + +@media only screen and (min-width: 1000px), +only screen and (min-width: 500px) and (max-width: 768px) { + + .column { + padding-left: 5px; + padding-right: 5px; + float: left; + } + + .column3 { + width: calc(33.3% - 10px); + } + + .column2 { + width: calc(50% - 11px); + position: relative; + } + + .column2:before { + padding-top: 61.8%; + content: ""; + display: block; + float: left; + } + + .top-left { + border-right: 1px solid var(--color-background-border); + border-bottom: 1px solid var(--color-background-border); + } + + .top-right { + border-bottom: 1px solid var(--color-background-border); + } + + .bottom-left { + border-right: 1px solid var(--color-background-border); + } +} \ No newline at end of file diff --git a/docs/source/cli-reference/index.rst b/docs/source/cli-reference/index.rst deleted file mode 100644 index c8b22e4151c..00000000000 --- a/docs/source/cli-reference/index.rst +++ /dev/null @@ -1,17 +0,0 @@ -**Command Line Interface Reference** -==================================== - -.. toctree:: - :maxdepth: 2 - - synopsis - help-text-options - list-options - simple-examples - other-commands - basic-options - core-options - output-format - output-filters-and-control - scan-options-pre - scan-options-post diff --git a/docs/source/cli-reference/list-options.rst b/docs/source/cli-reference/list-options.rst deleted file mode 100644 index 8e2f14fa12f..00000000000 --- a/docs/source/cli-reference/list-options.rst +++ /dev/null @@ -1,51 +0,0 @@ -.. _cli_list_options: - -All Available Options -===================== - -This section contains an exhaustive list of all ScanCode options, arranged in various sections. -The sections are as follows: - -- Basic Scan Options -- Core Scan Options -- Output Formats -- Controlling Output and Filters -- Pre-Scan Options -- Post-Scan Options - -There's also another section for ``extractcode`` options. - -The order of the sections and all their options is the same as in the :ref:`cli_help_text`, -available in the command line. - -.. _cli_basic: - -.. include:: /rst_snippets/basic_options.rst - ----- - -.. include:: /rst_snippets/extract.rst - ----- - -.. include:: /rst_snippets/scancode-reindex-licenses.rst - ----- - -.. include:: /rst_snippets/core_options.rst - ----- - -.. include:: /rst_snippets/output_format_options.rst - ----- - -.. include:: /rst_snippets/output_control_options.rst - ----- - -.. include:: /rst_snippets/pre_scan_options.rst - ----- - -.. include:: /rst_snippets/post_scan_options.rst diff --git a/docs/source/cli-reference/simple-examples.rst b/docs/source/cli-reference/simple-examples.rst deleted file mode 100644 index 306a4041a6d..00000000000 --- a/docs/source/cli-reference/simple-examples.rst +++ /dev/null @@ -1,3 +0,0 @@ -.. _how_to_run_a_scan: - -.. include:: /tutorials/how_to_run_a_scan.rst diff --git a/docs/source/cli-reference/synopsis.rst b/docs/source/cli-reference/synopsis.rst deleted file mode 100644 index ea211e966a0..00000000000 --- a/docs/source/cli-reference/synopsis.rst +++ /dev/null @@ -1,316 +0,0 @@ -.. _cli_synopsis: - -Synopsis -======== - -ScanCode detects licenses, copyrights, package manifests and direct dependencies and more, both -in source code and binary files, by scanning the files. This page introduces you to the ScanCode -Toolkit Command Line Interface in the following sections: - -- :ref:`syn_install` -- :ref:`synopsis_quickstart` -- :ref:`scancode_cli_options` -- :ref:`synopsis_output` -- :ref:`other_imp_doc` - -.. _syn_install: - -Installation ------------- - -Scancode-Toolkit installation can be done by downloading ScanCode as an application, which -is recommended generally. For users who wish to use ScanCode as a library, it can be -installed via ``pip``, the default Python Package Manager. Refer the following sections for -detailed Instructions on the each of the Installation Methods. - -- :ref:`app_install` -- :ref:`pip_install` -- :ref:`source_code_install` -- :ref:`docker_install` -- :ref:`fedora_install` - -.. _synopsis_quickstart: - -Quickstart ----------- - -The basic command to perform a scan, in case of a download and configure installation -(on Linux/MacOS) is:: - - path/to/scancode [OPTIONS] - -The basic usage, if ScanCode is installed from ``pip``, or in Windows:: - - scancode [OPTIONS] - -Here ScanCode scans the file or directory for license, origin and packages and saves -results to FILE(s) using one or more output format option. Error and progress are printed to -stdout. - -To scan the ``samples`` directory distributed with ScanCode-Toolkit, the command will be:: - - scancode -clpieu --json-pp path/to/output.json path/to/samples - -.. include:: /rst_snippets/note_snippets/synopsis_output_format.rst - -Alternatively, in case of download and configure installations, where ``path/to/scancode`` is used -(the path from root of file system) we can go into the scancode directory -(like ``scancode-toolkit-3.1.1``) and then use ``./scancode``. The same applies for input and -output options. To scan a folder ``samples`` inside ScanCode directory, and output to a file -``output.json`` in the same directory, the command will be:: - - ./scancode -clpieu --json-pp output.json samples - -While a scan using absolute paths from the file system root will look like:: - - home/ayansm/software/scancode-toolkit-3.1.1/scancode -clpieu --json-pp home/ayansm/scan_scan_results/output.json home/ayansm/codebases/samples/ - -Commands similar to ``scancode -clpi --json-pp output.json samples`` will be used as examples -throughout the documentation. - -- Here we are inside the ``virtualenv`` where ScanCode-Toolkit is configured. - -- And the default ``samples`` folder is being scanned, which is distributed by default with - ScanCode-Toolkit. - -.. _scancode_cli_options: - -Type of Options ---------------- - -ScanCode-Toolkit Command Line options can be divided into these major sections: - -- :ref:`cli_basic` -- :ref:`Extractcode Options ` -- :ref:`cli_core` -- :ref:`cli_output_control_filters` -- :ref:`cli_pre_scan` -- :ref:`cli_post_scan` - -Refer the individual pages which are linked to above, for detailed discussions on the Command -Line Options listed under each section. - -.. _synopsis_output: - -Output Formats --------------- - -The output file format is set by using the various output options. See -:doc:`../rst_snippets/output_format_options` . The recommended output -format is JSON. - -If ``--json`` is used, the entire file being in one line, without -whitespace characters. If ``--json-pp`` is used, the output is in a -pretty-printed JSON format—structured with indentation and line breaks to -make it more human-readable, rather than compressed into a single line. - -The following example scans will show you how to run a scan with each of -the result formats. For the scans, we will use the ``samples`` directory -provided with the ScanCode-Toolkit, along with the ``--json-pp`` output -option. - - -.. include:: /rst_snippets/tip_snippets/stdout.rst - -JSON file output -^^^^^^^^^^^^^^^^ - -Scan the ``samples`` directory and save the scan to a JSON file (pretty-printed):: - - scancode -clpieu --json-pp output.json samples - -A sample JSON output file structure will look like:: - - { - "headers": [ - { - "tool_name": "scancode-toolkit", - "tool_version": "3.1.1", - "options": { - "input": [ - "samples/" - ], - "--copyright": true, - "--email": true, - "--info": true, - "--json-pp": "output.json", - "--license": true, - "--package": true, - "--url": true - }, - "notice": "Generated with ScanCode and provided on an \"AS IS\" BASIS, WITHOUT WARRANTIES\nOR CONDITIONS OF ANY KIND, either express or implied. No content created from\nScanCode should be considered or used as legal advice. Consult an Attorney\nfor any legal advice.\nScanCode is a free software code scanning tool from nexB Inc. and others.\nVisit https://github.com/aboutcode-org/scancode-toolkit/ for support and download.", - "start_timestamp": "2019-10-19T191117.292858", - "end_timestamp": "2019-10-19T191219.743133", - "message": null, - "errors": [], - "extra_data": { - "files_count": 36 - } - } - ], - "files": [ - { - "path": "samples", - "type": "directory", - ... - ... - ... - "scan_errors": [] - }, - { - "path": "samples/README", - "type": "file", - "name": "README", - "base_name": "README", - "extension": "", - "size": 236, - "date": "2019-02-12", - "sha1": "2e07e32c52d607204fad196052d70e3d18fb8636", - "md5": "effc6856ef85a9250fb1a470792b3f38", - "sha1_git": "d9a10c0d8e868ebf8da0b3dc95bb0be634c34bfe", - "mime_type": "text/plain", - "file_type": "ASCII text", - "programming_language": null, - "is_binary": false, - "is_text": true, - "is_archive": false, - "is_media": false, - "is_source": false, - "is_script": false, - "licenses": [], - "license_expressions": [], - "copyrights": [], - "holders": [], - "authors": [], - "packages": [], - "emails": [], - "urls": [], - "files_count": 0, - "dirs_count": 0, - "size_count": 0, - "scan_errors": [] - }, - ... - ... - ... - { - "path": "samples/zlib/iostream2/zstream_test.cpp", - "type": "file", - "name": "zstream_test.cpp", - "base_name": "zstream_test", - "extension": ".cpp", - "size": 711, - "date": "2019-02-12", - ... - ... - ... - "scan_errors": [] - } - ] - } - -A sample JSON output for an individual file will look like:: - - { - "path": "samples/zlib/iostream2/zstream.h", - "type": "file", - "name": "zstream.h", - "base_name": "zstream", - "extension": ".h", - "size": 9283, - "date": "2019-02-12", - "sha1": "fca4540d490fff36bb90fd801cf9cd8fc695bb17", - "md5": "a980b61c1e8be68d5cdb1236ba6b43e7", - "mime_type": "text/x-c++", - "file_type": "C++ source, ASCII text", - "programming_language": "C++", - "is_binary": false, - "is_text": true, - "is_archive": false, - "is_media": false, - "is_source": true, - "is_script": false, - "licenses": [ - { - "key": "mit-old-style", - "score": 100.0, - "name": "MIT Old Style", - "short_name": "MIT Old Style", - "category": "Permissive", - "is_exception": false, - "is_unknown": false, - "owner": "MIT", - "homepage_url": "http://fedoraproject.org/wiki/Licensing:MIT#Old_Style", - "text_url": "http://fedoraproject.org/wiki/Licensing:MIT#Old_Style", - "reference_url": "https://enterprise.dejacode.com/urn/urn:dje:license:mit-old-style", - "spdx_license_key": null, - "spdx_url": null, - "start_line": 9, - "end_line": 15, - "matched_rule": { - "identifier": "mit-old-style_cmr-no_1.RULE", - "license_expression": "mit-old-style", - "licenses": [ - "mit-old-style" - ], - "is_license_text": true, - "is_license_notice": false, - "is_license_reference": false, - "is_license_tag": false, - "matcher": "2-aho", - "rule_length": 71, - "matched_length": 71, - "match_coverage": 100.0, - "rule_relevance": 100 - } - } - ], - "license_expressions": [ - "mit-old-style" - ], - "copyrights": [ - { - "copyright": "Copyright (c) 1997 Christian Michelsen Research AS Advanced Computing", - "start_line": 3, - "end_line": 5 - } - ], - "holders": [ - { - "holder": "Christian Michelsen Research AS Advanced Computing", - "start_line": 3, - "end_line": 5 - } - ], - "authors": [], - "packages": [], - "emails": [], - "urls": [ - { - "url": "http://www.cmr.no/", - "start_line": 7, - "end_line": 7 - } - ], - "files_count": 0, - "dirs_count": 0, - "size_count": 0, - "scan_errors": [] - }, - -Static HTML output -^^^^^^^^^^^^^^^^^^ - -Scan the ``samples`` directory for licenses and copyrights and save the scan results to an HTML -file. When the scan is done, open ``samples.html`` in your web browser. - -:: - - scancode -clpieu --html output.html samples - -.. image:: data/scancode-toolkit-static-html1.png -.. image:: data/scancode-toolkit-static-html2.png - -.. _other_imp_doc: - -.. include:: /rst_snippets/other_imp_doc.rst diff --git a/docs/source/contribute/index.rst b/docs/source/contribute/index.rst deleted file mode 100644 index 142bd90f77f..00000000000 --- a/docs/source/contribute/index.rst +++ /dev/null @@ -1,13 +0,0 @@ -**Contribute** -============== - -.. toctree:: - :maxdepth: 2 - - contrib_dev - cut_new_release - contrib_doc - roadmap - gsoc17_final_report - gsoc19_final_report - gsoc21_final_report diff --git a/docs/source/explanation/facets.rst b/docs/source/explanation/facets.rst new file mode 100644 index 00000000000..9ae6fb82c3f --- /dev/null +++ b/docs/source/explanation/facets.rst @@ -0,0 +1,38 @@ +.. _facets: + +Facets +====== + +A `facet `_ is essentially +a file purpose classification label. +It is defined by `ClearlyDefined `_ as: + +A facet of a component is a subset of the files related to the component. It's really just a +grouping that helps us understand the shape of the project. Each facet is described by a set of +glob expressions, essentially wildcard patterns that are matched against file names. + +Each facet definition can have zero or more glob expressions. A file can be captured by more +than one facet. Any file found but not captured by a defined facet is automatically assigned to +the core facet. + +- ``core`` + The files that go into making the release of the component. Note that the core + facet is not explicitly defined. Rather, it is made up of whatever is not in any other facet. + So, by default, all files are in the core facet unless otherwise specified. +- ``data`` + The files included in any data distribution of the component. +- ``dev`` + Files primarily used at development time (e.g., build utilities) and not + distributed with the component +- ``docs`` + Documentation files. Docs may be included with the executable component or + separately or not at all. +- ``examples`` + Like docs, examples may be included in the main component release or separately. +- ``tests`` + Test files may include code, data and other artifacts. + +Related ScanCode CLI options +---------------------------- +- :ref:`cli-facet-option` +- :ref:`cli-tallies-by-facet-option` diff --git a/docs/source/explanation/index.rst b/docs/source/explanation/index.rst new file mode 100644 index 00000000000..d0a817de99f --- /dev/null +++ b/docs/source/explanation/index.rst @@ -0,0 +1,15 @@ +.. _explanation: + +**Explanation** +=============== + +This section dives into key concepts. +It’s more about helping you grasp how it works than giving you a step-by-step guide on what to do. + +.. toctree:: + :maxdepth: 2 + + scancode-license-detection + scancode-license-detection-updates + scancode-versioning + facets diff --git a/docs/source/reference/license-detection-reference.rst b/docs/source/explanation/scancode-license-detection-updates.rst similarity index 99% rename from docs/source/reference/license-detection-reference.rst rename to docs/source/explanation/scancode-license-detection-updates.rst index b21135651bc..88715aec887 100644 --- a/docs/source/reference/license-detection-reference.rst +++ b/docs/source/explanation/scancode-license-detection-updates.rst @@ -1,4 +1,4 @@ -License Detection Updates +License detection updates ========================= References: diff --git a/docs/source/reference/overview.rst b/docs/source/explanation/scancode-license-detection.rst similarity index 93% rename from docs/source/reference/overview.rst rename to docs/source/explanation/scancode-license-detection.rst index a75de1ee0ba..bdb7ee87ead 100644 --- a/docs/source/reference/overview.rst +++ b/docs/source/explanation/scancode-license-detection.rst @@ -1,10 +1,7 @@ -Overview -======== +.. _scancode-license-detection: -.. _explain_how_scancode_works: - -How does ScanCode detect licenses? --------------------------------------- +ScanCode license detection +========================== For license detection, ScanCode uses a (large) number of license texts and license detection 'rules' that are compiled in a search index. When scanning, the text of the target file is diff --git a/docs/source/misc/versioning.rst b/docs/source/explanation/scancode-versioning.rst similarity index 97% rename from docs/source/misc/versioning.rst rename to docs/source/explanation/scancode-versioning.rst index fdbf62ef5c9..8b73bf0832c 100644 --- a/docs/source/misc/versioning.rst +++ b/docs/source/explanation/scancode-versioning.rst @@ -1,8 +1,8 @@ -.. _versioning: +.. _scancode-versioning: -Versioning approach -========================== +ScanCode versioning +=================== ScanCode is composed of code and data (mostly license data used for license detection). In the past, we have tried using calver for code versioning to also diff --git a/docs/source/contribute/contrib_dev.rst b/docs/source/getting-started/contribute/contributing-code.rst similarity index 99% rename from docs/source/contribute/contrib_dev.rst rename to docs/source/getting-started/contribute/contributing-code.rst index 0ed8c5f367d..fcb6f84f4f5 100644 --- a/docs/source/contribute/contrib_dev.rst +++ b/docs/source/getting-started/contribute/contributing-code.rst @@ -1,7 +1,7 @@ -.. _contrib_code_dev: +.. _contributing-to-the-code: -Contributing to Code Development -================================ +Contributing to the code +======================== TL;DR: diff --git a/docs/source/contribute/contrib_doc.rst b/docs/source/getting-started/contribute/contributing-docs.rst similarity index 95% rename from docs/source/contribute/contrib_doc.rst rename to docs/source/getting-started/contribute/contributing-docs.rst index 478783698c0..49b469cbcb4 100644 --- a/docs/source/contribute/contrib_doc.rst +++ b/docs/source/getting-started/contribute/contributing-docs.rst @@ -1,9 +1,9 @@ -.. _contrib_doc_dev: +.. _contributing-docs: -Contributing to the Documentation -================================= +Contribute to the documentation +=============================== -.. _contrib_doc_setup_local: +.. _contributing-docs-local-setup: Setup Local Build ----------------- @@ -43,7 +43,7 @@ To validate the documentation style and content, use:: make docs-check -.. _doc_ci: +.. _contributing-docs-ci: Continuous Integration ---------------------- @@ -61,7 +61,7 @@ You myst run these scripts locally before creating a pull request:: make check-docs -.. _doc_style_docs8: +.. _contributing-docs-docs8-style-checks: Style Checks Using ``doc8`` --------------------------- @@ -119,7 +119,7 @@ What is checked: - no newline at end of file - D005 -.. _doc_interspinx: +.. _contributing-docs-interspinx: Interspinx ---------- @@ -181,7 +181,7 @@ For more information, refer this tutorial named `Using Intersphinx `_. -.. _doc_style_conv: +.. _contributing-docs-style-conventions: Style Conventions for the Documentaion -------------------------------------- @@ -257,8 +257,8 @@ Style Conventions for the Documentaion 7. Notes and Warning Snippets - Every ``Note`` and ``Warning`` sections are to be kept in ``rst_snippets/note_snippets/`` and - ``rst_snippets/warning_snippets/`` and then included to eliminate redundancy, as these are + Every ``Note`` and ``Warning`` sections are to be kept in ``rst-snippets/note-snippets/`` and + ``rst-snippets/warning-snippets/`` and then included to eliminate redundancy, as these are frequently used in multiple files. 8. Redirects @@ -288,4 +288,4 @@ It's possible to generate docs automatically from data by using a combination of - `jinja templates: example `_ And we do this currently to keep a documentation page for all the supported package formats. -See :ref:`supported_packages` for details. +See :ref:`supported-packages` for details. diff --git a/docs/source/contribute/data/clock.png b/docs/source/getting-started/contribute/data/clock.png similarity index 100% rename from docs/source/contribute/data/clock.png rename to docs/source/getting-started/contribute/data/clock.png diff --git a/docs/source/contribute/data/done.png b/docs/source/getting-started/contribute/data/done.png similarity index 100% rename from docs/source/contribute/data/done.png rename to docs/source/getting-started/contribute/data/done.png diff --git a/docs/source/contribute/data/planned.png b/docs/source/getting-started/contribute/data/planned.png similarity index 100% rename from docs/source/contribute/data/planned.png rename to docs/source/getting-started/contribute/data/planned.png diff --git a/docs/source/contribute/gsoc17_final_report.rst b/docs/source/getting-started/contribute/gsoc/gsoc17-final-report.rst similarity index 99% rename from docs/source/contribute/gsoc17_final_report.rst rename to docs/source/getting-started/contribute/gsoc/gsoc17-final-report.rst index 15c368153ba..08ac6c9e494 100644 --- a/docs/source/contribute/gsoc17_final_report.rst +++ b/docs/source/getting-started/contribute/gsoc/gsoc17-final-report.rst @@ -1,4 +1,4 @@ -Google Summer of Code 2017 - Final report +Google Summer of Code 2017 - final report ========================================= **Project: Plugin architecture for ScanCode** diff --git a/docs/source/contribute/gsoc19_final_report.rst b/docs/source/getting-started/contribute/gsoc/gsoc19-final-report.rst similarity index 99% rename from docs/source/contribute/gsoc19_final_report.rst rename to docs/source/getting-started/contribute/gsoc/gsoc19-final-report.rst index 67b8a589774..8dd2b20a244 100644 --- a/docs/source/contribute/gsoc19_final_report.rst +++ b/docs/source/getting-started/contribute/gsoc/gsoc19-final-report.rst @@ -1,4 +1,4 @@ -Google Summer of Code 2019 - Final report +Google Summer of Code 2019 - final report ========================================= Project: **scancode-toolkit** to Python 3 diff --git a/docs/source/contribute/gsoc21_final_report.rst b/docs/source/getting-started/contribute/gsoc/gsoc21-final-report.rst similarity index 99% rename from docs/source/contribute/gsoc21_final_report.rst rename to docs/source/getting-started/contribute/gsoc/gsoc21-final-report.rst index 9b72e6d18ba..4cec1278fde 100644 --- a/docs/source/contribute/gsoc21_final_report.rst +++ b/docs/source/getting-started/contribute/gsoc/gsoc21-final-report.rst @@ -1,4 +1,4 @@ -Google Summer of Code 2021 Final report +Google Summer of Code 2021 - final report ========================================= @@ -99,4 +99,3 @@ I am thankful to Google and Aboutcode for giving me this opportunity to work wit community. I am fortunate to have mentors `Philippe Ombredanne `_ and `Ayan Sinha Mahapatra `_ who helped me a lot throughout my GSoC project and provided constant support. - diff --git a/docs/source/getting-started/contribute/gsoc/index.rst b/docs/source/getting-started/contribute/gsoc/index.rst new file mode 100644 index 00000000000..c6c04c9f7f8 --- /dev/null +++ b/docs/source/getting-started/contribute/gsoc/index.rst @@ -0,0 +1,9 @@ +Google Summer of Code +===================== + +.. toctree:: + :maxdepth: 2 + + gsoc17-final-report + gsoc19-final-report + gsoc21-final-report diff --git a/docs/source/getting-started/contribute/index.rst b/docs/source/getting-started/contribute/index.rst new file mode 100644 index 00000000000..641e395be2b --- /dev/null +++ b/docs/source/getting-started/contribute/index.rst @@ -0,0 +1,11 @@ +Contributing to ScanCode +======================== + +.. toctree:: + :maxdepth: 2 + + contributing-code + making-a-new-release + contributing-docs + roadmap + gsoc/index diff --git a/docs/source/contribute/cut_new_release.rst b/docs/source/getting-started/contribute/making-a-new-release.rst similarity index 92% rename from docs/source/contribute/cut_new_release.rst rename to docs/source/getting-started/contribute/making-a-new-release.rst index 77b57f95a74..9afa7c6505e 100644 --- a/docs/source/contribute/cut_new_release.rst +++ b/docs/source/getting-started/contribute/making-a-new-release.rst @@ -1,10 +1,10 @@ -.. _release_process: +.. _making-a-new-release: -How to cut a new release -======================== +Making a new release +===================== -Update version --------------- +Updating the version +-------------------- - Bump version to update major, minor or patch version in ``setup.cfg`` ``setup-mini.cfg`` and ``src/scancode_config.py``. Note that this is SemVer, @@ -16,8 +16,8 @@ Update version See our :ref:``versioning`` for more details. -Tag and publish ---------------- +Tagging and publishing +---------------------- - Changes for a release should also be pushed to a branch and a Pull Request should be created for it, for review. @@ -38,8 +38,8 @@ Tag and publish - ``git push --set-upstream origin release-prep-v1.6.1`` - ``git push --set-upstream origin v1.6.1`` -Automated Release Process -------------------------- +Release Automation +------------------ - We have an `automated release script `_ triggered by a pushed tag, where jobs run to: diff --git a/docs/source/contribute/roadmap.rst b/docs/source/getting-started/contribute/roadmap.rst similarity index 100% rename from docs/source/contribute/roadmap.rst rename to docs/source/getting-started/contribute/roadmap.rst diff --git a/docs/source/misc/faq.rst b/docs/source/getting-started/faq.rst similarity index 99% rename from docs/source/misc/faq.rst rename to docs/source/getting-started/faq.rst index d3184d48921..96edb6c31f1 100644 --- a/docs/source/misc/faq.rst +++ b/docs/source/getting-started/faq.rst @@ -1,7 +1,7 @@ .. _faq: -FAQ -=== +FAQs +==== Why ScanCode? ------------- @@ -131,4 +131,3 @@ then: Note that ``matched_text_diagnostics`` also includes "tagged" gaps or extra unmatched words highlighted between the matched words. - diff --git a/docs/source/getting-started/home.rst b/docs/source/getting-started/home.rst deleted file mode 100644 index 5e496d5d65e..00000000000 --- a/docs/source/getting-started/home.rst +++ /dev/null @@ -1,138 +0,0 @@ -Home -==== - -ScanCode does scan code to detect packages and dependencies, licenses, -copyrights and more. - - -Why ScanCode? -------------- - -Discovering the origin and license for a software component is important, but it -is often much harder to accomplish than it should be because: - -- A typical software project may reuse tens or thousands of third-party software components -- Software authors do not always provide copyright and license information -- Copyright and license information that is provided may be hard to find and interpret - - -ScanCode tries to address these issues by offering: - -- A simple command line approach that runs on Windows, Linux, and macOS -- A comprehensive code scanner that can detect origin and license information in - codebase files, including binaries -- A comprehensive set of package manifests and lockfile parsers to report direct - and pinned dependencies -- Your choice of JSON or other output formats (YAML, SPDX, HTML, CSV) for - integration with other tools -- Well-tested, easy to hack, and well-documented code -- A plugin system for easily adding new Functionality to Scans. -- Extensive documentation and support. -- We release of the code and reference data under permissive licenses (Apache - 2.0 and CC-BY-4.0) -- ScanCode.io to assemble scripted and specialized code analysis pipelines with - a web-based analysis server -- ScanCode workbench for desktop-based scans visualization - -ScanCode is recognized as the industry leading engine for license and copyright -detection and used as the basis of several open source compliance efforts in -open source projects and companies. Its detection engine is embedded in the -most advanced open source and commercial tools available today for Software -Composition Analysis. - - -What does ScanCode-Toolkit do? ------------------------------- - -ScanCode detects and normalizes origin, dependencies, licensing and other -related information in your code: - -- by parsing package manifests and dependencies lock files to a normalized - metadata model and assigning each an identifying `Package URL - `_, - -- by detecting license tags, notices and texts in text and binaries using the - world's most comprehensive database of license texts and notices and a unique - combination of techniques, - -- by recognizing copyright statements using an advanced natural language parsing - grammar and detecting other origin clues (such as emails, urls, and authors) - -Using this data you can: - -- Discover the origin and license of the open source and third-party software - components that you use, - -- Discover direct dependent packages and indirect pinned or locked dependencies, - -- Assemble a software component inventory of your codebase, and report the data - using standard SBOM formats, - -- Use this data as the input to: - - - open source license compliance obligations such as attribution and redistribution. - - open source package vulnerability detection. - - -How does it work? ------------------ - -Given a code directory, ScanCode will "scan code": - -- Extract files from any archive using a `universal archive extractor - `_ - -- Collect an inventory of the code files and classify the code using file types -- Extract texts from binary files as needed -- Use an extensible rules engine to detect open source license text, notices - tags, mentions and license expressions with over 31,000 detection rules. - -- Use a specialized natural language parser and grammar to capture copyright - statements - -- Identify packaged code and collect metadata from packages by parsing the - manifest and lockfiles (and in some cases also the installed databases for - system packages) for these package types: .ABOUT, Alpine Linux apk as packages - or installed, Android apk, Autotools, Bazel, JS Bower, Buck, Msft Cab, Rust - Cargo, Chef, Chrome, PHP Composer, Conda, Perl CPAN, R CRAN, Debian deb as - packages or installed, Apple dmg, Java EAR, FreeBSD, Ruby Gem, Go modules, - Haxe, InstallShield, iOS ipa, ISO disk images, Apache IVY, Java JAR, JBoss - SAR, Maven, JS Meteor, Mozilla Extension, Msft MSI, JS npm, NSIS Installer, - NuGet, Ocaml OPAM, Cocoapods, Dart Pub, Python PyPI wheel and related, - structured README, RPMs as packages or installed, Shell archive, Squashfs, - Java WAR, Msft Update Manifest, and Windows Executable. - -- Report the results in the formats of your choice (JSON, YAML, CSV, SPDX, etc.) - for integration with other tools - - -ScanCode is written in Python and also uses other open source packages. - - -Alternative? --------------- - -There are several utilities that do some of what ScanCode does - for instance -you can grep files for copyright and license text. This may work well for simple -cases - e.g. at the single whole license text files and well structured -copyright statements, but we created ScanCode for ourselves because this -approach does not help you to see the recurring patterns of licenses and other -origin history clues at scale. - -You can consider other tools such as: - -- FOSSology (open source, written in C, Linux only, GPL-licensed) - - -History -------- - -ScanCode was originally created by nexB to support our software audit consulting -services. We have used and continuously enhanced the underlying toolkit for over -12 years. We decided to release ScanCode as open source software to give -software development teams the opportunity to perform as much of the software -audit function as they like on their own. - -Thank you for giving ScanCode a try! - -.. include:: /rst_snippets/other_imp_doc.rst diff --git a/docs/source/getting-started/index.rst b/docs/source/getting-started/index.rst index ae8d68e5b90..461631d578d 100644 --- a/docs/source/getting-started/index.rst +++ b/docs/source/getting-started/index.rst @@ -1,13 +1,11 @@ +.. _getting-started: + **Getting Started** =================== .. toctree:: :maxdepth: 2 - home - install - newcomer - -.. - [ToAdd] - Link to Changelog + installation/index + faq + contribute/index diff --git a/docs/source/getting-started/install.rst b/docs/source/getting-started/install.rst deleted file mode 100644 index 2cd09386d16..00000000000 --- a/docs/source/getting-started/install.rst +++ /dev/null @@ -1,503 +0,0 @@ -.. _install: - -Comprehensive Installation -========================== - - -The recommended way to install ScanCode is using app archives: - -- :ref:`app_install` - - The recommended method is to download the latest application release as an - application and then configure and use directly. No knowledge of pip/git or - other developer tools is necessary. You only need to install Python then - download and extract the ScanCode application archive to run ScanCode. - For standard usage that's all you need. - - -For advanced usage and experienced users, you can also use any of these mode: - -- :ref:`docker_install` - - An alternative to installing the latest ScanCode-Toolkit release natively is - to build a Docker image from the included Dockerfile. The only prerequisite - is a working Docker installation. - -- :ref:`source_code_install` - - You can clone the git source code repository and then run the configure script - to configure and install ScanCode for local and development usage. - -- :ref:`pip_install` - - To use ScanCode as a library in your application, you can install it via - ``pip``. This is recommended for developers or users familiar with Python - that want to embed ScanCode as a library. - -- :ref:`fedora_install` - - ScanCode is part of main Fedora Linux repository. It will automatically install - all dependencies. This is recommended for production deployments. - ----- - -Before Installing ------------------ - -- ScanCode requires a Python version between 3.10 to 3.13 and is - tested on Linux, macOS, and Windows. It should work fine on FreeBSD. - -.. _system_requirements: - -System Requirements -^^^^^^^^^^^^^^^^^^^ - -- Hardware : ScanCode will run best with a modern X86 64 bits processor and at - least 8GB of RAM and 2GB of disk space. These are minimum requirements. - -- Supported operating systems: ScanCode should run on these 64-bit OSes running - X86_64 processors: - - #. Linux: on recent 64-bit Linux distributions, - #. Mac: on recent x86 64-bit macOS (10.15 and up, including 11 and 12), - Use the X86 emulation mode on Apple ARM M1 CPUs. - (Note that `pip install` does not work on ARM CPUs) - #. Windows: on Windows 10 and up, - #. FreeBSD. - - -.. _install_prerequisites: - -Prerequisites -^^^^^^^^^^^^^ - -ScanCode needs a Python 3.10+ interpreter; We support all Python versions from -3.10 to 3.12. The default version for the application archives is Python 3.10 - -- **On Linux**: - - Use your package manager to install ``python3``. - - For Ubuntu, it is ``sudo apt install python3-dev`` - - - On Ubuntu 16, 18, 20 ,22 and 24 run:: - - sudo apt install python-dev bzip2 xz-utils zlib1g libxml2-dev libxslt1-dev libpopt0 - - - On Debian and Debian-based distros run:: - - sudo apt-get install python3-dev libbz2-1.0 xz-utils zlib1g libxml2-dev libxslt1-dev libpopt0 - - - On RPM-based distros run:: - - sudo yum install python3.10-devel zlib bzip2-libs xz-libs libxml2-devel libxslt-devel libpopt0 - - - On Fedora 22 and later run:: - - sudo dnf install python3.10-devel xz-libs zlib libxml2-devel libxslt-devel bzip2-libs libpopt0 - - - If these packages are not available from your package manager, you must - compile them from sources. - - -- **On Mac**: - - The default Python 3 provided with macOS is 3.10. - Alternatively you can download and install Python 3.10 from https://www.python.org/ - - -- **On Windows**: - - Download and install Python 3.10 from https://www.python.org/ - - .. Note:: - - 64-bit Python interpreters (x86-64) are the only interpreters supported by - ScanCode on all operating systems which means only 64-bit Windows is supported. - - See the :ref:`windows_app_install` section for more installation details. - ----- - -.. _app_install: - -Installation as an Application: Downloading Releases ------------------------------------------------------ - -Get the ScanCode-Toolkit tarball archive of a specific version and your -operating system by going to the `project releases page `_ - -For example, Version 30.0.1 archive can be obtained from -`Toolkit release 30.0.1 `_ -under assets options. - -.. Note:: - - ScanCode app archives come with packaged with all required dependencies except - for Python that has to be downloaded and installed separately. - On more recent versions of Ubuntu, you will have to install Python 3.10 manually. - One possibility is to use the Deadsnakes PPA (Personal Package Archive) which is - a project that provides older Python version builds for Debian and Ubuntu and is - available at https://github.com/deadsnakes/ and https://launchpad.net/~deadsnakes/+archive/ubuntu/ppa - :: - - sudo apt-get update && sudo apt-get upgrade - sudo add-apt-repository ppa:deadsnakes/ppa --yes - sudo apt-get install python3.10 python3.10-distutils - -.. _linux_mac_app_install: - -Installation on Linux and Mac -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Download the archive for your operating systen and extract -the archive from command line:: - - tar -xvf scancode-toolkit-30.0.1_py38-linux.tar.gz - - -Or, on Linux, right click and select "Extract Here". - -Check whether the :ref:`install_prerequisites` are installed. Open a terminal -in the extracted directory and run:: - - ./scancode --help - -This will configure ScanCode and display the command line :ref:`cli_help_text`. - -.. note:: - If you encounter a "No matching distribution" error while running the ``./configure`` command on a Mac M1, it may indicate compatibility issues with the current architecture. Here's a step-by-step guide to address this: - - - **Change Mac M1 Architecture to x86_64:** - Switch the architecture from amd64 to x86_64 using the command: - :: - - env /usr/bin/arch -x86_64 /bin/zsh --login - - **Use Rosetta Translation:** - Enable Rosetta translation in Terminal by executing: - :: - - softwareupdate --install-rosetta - - **Transition Homebrew from arm64 to Intel:** - Change Homebrew from the arm64 architecture to the Intel (x86) architecture by running: - :: - - /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)" - - **Install Intel-Specific Python:** - Use Homebrew to install Python specifically optimized for Intel architecture with: - :: - - /usr/local/Homebrew/bin/brew install python3 - - Then rerun the ``./configure`` command. This sets up the project according to the new architecture and ensures proper configuration. - Following these steps should help resolve compatibility issues and allow smooth operation of the project on Mac M1 devices. - -.. _windows_app_install: - -Installation on Windows 10/11 -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -- Download the latest ScanCode release zip file for Windows from the latest - version at https://github.com/aboutcode-org/scancode-toolkit/releases/ - -- In the File Explorer, select the downloaded ScanCode zip and right-click. - -- In the pop-up menu select 'Extract All...' - -- In the pop-up window 'Extract Compressed (Zipped) Folders' use the default options to extract. - -- Once the extraction is complete, a new File Explorer window will pop up. - -- In this Explorer window, select the new folder that was created and right-click. - -.. note:: - - On Windows, double-click the new folder, select one of the files inside the folder - (e.g., 'setup.py'), and right-click. - -- In the pop-up menu select 'Properties'. - -- In the pop-up window 'Properties', select the Location value. Copy this to the clipboard and - close the 'Properties' window. - -- Press the start menu button, click the search box or search icon in the taskbar. - -- In the search box type:: - - cmd - -- Select 'cmd.exe' or 'Command Prompt' listed in the search results. - -- A new 'Command Prompt'pops up. - -- In this window (aka a 'command prompt'), type 'cd' followed by a space and - then Right-click in this window and select Paste. This will paste the path you - copied before and is where you extracted ScanCode:: - - cd path\to\extracted\ScanCode - -- Press Enter. - -- This will change the current location of your command prompt to the root directory where - ScanCode is installed. - -- Then type:: - - scancode -h - -- Press enter. This first command will configure your ScanCode installation. - Several messages are displayed followed by the ScanCode command help. - -- The installation is complete. - - -Un-installation -^^^^^^^^^^^^^^^ - -- Delete the directory in which you extracted ScanCode. -- Delete any temporary files created in your system temp and user temp directory - under a ScanCode-prefixed directory such as .scancode-tk or .cache/scancode-tk. - - -.. note:: - - The above installation process can be used with **Command Prompt** - ``cmd``, and **PowerShell**. **Git Bash** is not tested and using it may - introduce unexpected behavior. If you're using **Windows Subsystem for - Linux** ``WSL2``, please refer to :ref:`linux_mac_app_install` section - above. - ----- - -.. _docker_install: - - -Installation via Docker: ------------------------- - -You can install ScanCode-Toolkit by building a Docker image from the included Dockerfile. -The prerequisite is a working `docker installation `_. - - -Download the ScanCode-Toolkit Source Code -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Run the following once you have `Git `_ installed:: - - git clone https://github.com/aboutcode-org/scancode-toolkit.git - - -Build the Docker image -^^^^^^^^^^^^^^^^^^^^^^ - -Run the ``docker build`` source code checkout directory.:: - - cd scancode-toolkit - docker build --tag scancode-toolkit --tag scancode-toolkit:$(git describe --tags) . - -.. note:: - - As ``$(git describe --tags)`` is a Unix-style command. If you are on - windows, please run the ``git describe --tags`` separately to get the - output, then manually insert that value into your Docker command. - For instance, :: - - C:\scancode-toolkit>git describe --tags - v32.4.1 - C:\scancode-toolkit>docker build --tag scancode-toolkit --tag scancode-toolkit:v32.4.1 . - - -Run using Docker -^^^^^^^^^^^^^^^^ - -The docker image will forward all arguments it receives directly to the ``scancode`` command. - -Display help:: - - docker run scancode-toolkit --help - -Mount current working directory as "/project" and run a scan on a file name -apache-2.0.LICENSE directory. The JSON results will be in scan-result.json:: - - docker run -v $PWD/:/project scancode-toolkit -clipeu --json-pp /project/scan-result.json /project/apache-2.0.LICENSE - -This will mount your current working from the host into ``/project`` in the container -and then scan the contents. The output ``result.json`` will be written back to your -current working directory on the host. - -Note that the parameters *before* ``scancode-toolkit`` are used for docker, -those after will be forwarded to scancode. - - ----- - - -.. _source_code_install: - -Installation from Source Code: Git Clone ------------------------------------------ - -You can download the ScanCode-Toolkit Source Code and build from it yourself. -This is what you would want to do it if: - -- You are developing ScanCode or adding new patches or want to run tests. -- You want to test or run a specific version/checkpoint/branch from the version control. - - -Download the ScanCode-Toolkit Source Code -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Run the following once you have `Git `_ installed:: - - git clone https://github.com/aboutcode-org/scancode-toolkit.git - cd scancode-toolkit - - -Configure the build -^^^^^^^^^^^^^^^^^^^ - -ScanCode use a configure scripts to create an isolated virtual environment, -install required packaged dependencies. - -On Linux/Mac: - -- Open a terminal -- cd to the clone directory -- run ``./configure`` -- run ``source venv/bin/activate`` - - -On Windows: - -- open a command prompt -- cd to the clone directory -- run ``configure`` -- run ``venv\Scripts\activate`` - - -Now you are ready to use the freshly configured scancode-toolkit. - -.. NOTE:: - - For use in development, run instead ``configure --dev``. If your face - issues while configuring a previous version, ``configure --clean`` to - clean and reset your enviroment. You will need to run ``configure`` again. - - ----- - -.. _pip_install: - -Installation as a library: via ``pip`` --------------------------------------- - -ScanCode can be installed from the public PyPI repository using ``pip`` which -the standard Python package management tool. - -.. NOTE:: - - Note that `pip` installation method does work on ARM chips, i.e. Linux/MacOS on - Apple M1 chips, as some non-native dependencies do not have pre-built wheels - for ARM (like py-ahocorasick, intbitset). See :ref:`system_requirements` for - more information. See related issues for more info: - - - `Fallback pure-python deps `_ - - `pip install failing on M1 `_ - -The steps are: - -#. Create a Python virtual environment:: - - /usr/bin/python3 -m venv venv - -For more information on Python virtualenv, visit this -`page `_. - -#. Activate the virtual environment you just created:: - - source venv/bin/activate - -#. Run pip to install the latest versions of base utilities:: - - pip install --upgrade pip setuptools wheel - -#. Install the latest version of ScanCode:: - - pip install scancode-toolkit - -.. NOTE:: - - For advanced usage, ``scancode-toolkit-mini`` is an alternative package with - no default dependencies on pre-built binaries. This may come handy for some - special use cases such as packaging for a Linux or FreeBSD distro. - - -To uninstall, run:: - - pip uninstall scancode-toolkit - - ----- - -.. _fedora_install: - -Install from Fedora's repository --------------------------------- - -The package is available in Fedora 40 and newer. Run:: - - dnf install scancode-toolkit - -To uninstall, run:: - - dnf remove scancode-toolkit - - ----- - -.. _commands_variation: - -Command Invocation Variations ------------------------------ - -These are the commands to invoke ScanCode based on: - -- your installation methods -- your operating systems - -The two form of commands are: - -- Use the scancode command directly, typically on Windows or in an activated virtualenv:: - - scancode [OPTIONS] - -- Use a path to the scancode command, typically with an application installation :: - - path/to/scancode [OPTIONS] - -These variations are summed up in the following table: - -.. list-table:: - :widths: 10 5 10 50 - :header-rows: 1 - - * - Installation Methods - - Application Install - - Pip Install - - Install from Source Code - - * - Linux - - path: `./scancode` - - direct: scancode - - path: `./scancode` or direct: `scancode` - - * - Mac - - path: `./scancode` - - direct: scancode - - path: `./scancode` or direct: `scancode` - - * - Windows - - path: `scancode` - - direct: scancode - - path: `scancode` or direct: `scancode` diff --git a/docs/source/getting-started/installation/index.rst b/docs/source/getting-started/installation/index.rst new file mode 100644 index 00000000000..88440465b99 --- /dev/null +++ b/docs/source/getting-started/installation/index.rst @@ -0,0 +1,138 @@ +.. _install-scancode: + +Installing ScanCode +=================== + +.. toctree:: + :maxdepth: 2 + :hidden: + + install-scancode-from-release-archives + install-scancode-using-docker + install-scancode-from-source + install-scancode-using-pip + +.. note:: + + ScanCode requires a Python version between 3.9 to 3.13 and is tested on Linux, macOS, and Windows + +.. list-table:: + :width: 100% + :header-rows: 1 + + * - Method + - Description + * - :ref:`install-scancode-from-release-archives` + - | Our **recommended method for new users** is to install Python, download and extract + | the latest release archive to run ScanCode. + | No knowledge of pip, git, or other developer tools is required for standard usage. + * - :ref:`install-scancode-using-docker` + - | Install ScanCode from its Git source code repository using Docker + | + | An alternative to installing the latest ScanCode-Toolkit release natively is + | to build a Docker image from the included Dockerfile. The only prerequisite + | is a working Docker installation. + * - :ref:`install-scancode-from-source` + - | Install ScanCode from its Git source code repository using the configure script + | + | Clone the git source code repository and then run the configure script + | to configure and install ScanCode for local and development usage. + * - :ref:`install-scancode-using-pip` + - | Install ScanCode using pip from Python Package Index (PyPI) + | + | To use ScanCode as a library in your application, you can install it via + | ``pip``. This is recommended for developers or users familiar with Python + | that want to embed ScanCode as a library + * - Install from Fedora’s repository + - | ScanCode is part of main Fedora Linux repository in Fedora 40 and newer. + | This is recommended for production deployments. + | + | Install ScanCode from the command line interface using: + + .. code-block:: shell + + dnf install scancode-toolkit + + To uninstall ScanCode, run: + + .. code-block:: shell + + dnf remove scancode-toolkit + + +.. _install-scancode-installation-prerequisites: + +Installation prerequisites +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. _install-scancode-system-requirements: + +System requirements +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +- Hardware : ScanCode will run best with a modern X86 64 bits processor and at + least 8GB of RAM and 2GB of disk space. These are minimum requirements. + +- Supported operating systems: ScanCode should run on these 64-bit OSes running + X86_64 processors: + + - Linux: any recent 64-bit Linux distributions + - Mac (Intel): x86 64-bit macOS 10.15 or newer + - | Mac (Apple Silicon): use the X86 emulation mode on Apple ARM M1 CPUs. + | (Note that `pip install` does not work on ARM CPUs) + - Windows: Windows 10 or newer + - FreeBSD + +.. _install-scancode-required-packages: + +Required packages +^^^^^^^^^^^^^^^^^ + +ScanCode needs a Python 3.9+ interpreter; We support all Python versions from +3.9 to 3.12. The default version for the application archives is Python 3.9 + +Linux +""""" + +Use your package manager to install ``python3``. + +For Ubuntu, it is ``sudo apt install python3-dev`` + +- On Ubuntu 16, 18, 20 ,22 and 24 run:: + + sudo apt install python-dev bzip2 xz-utils zlib1g libxml2-dev libxslt1-dev libpopt0 + +- On Debian and Debian-based distros run:: + + sudo apt-get install python3-dev libbz2-1.0 xz-utils zlib1g libxml2-dev libxslt1-dev libpopt0 + +- On RPM-based distros run:: + + sudo yum install python3.9-devel zlib bzip2-libs xz-libs libxml2-devel libxslt-devel libpopt0 + +- On Fedora 22 and later run:: + + sudo dnf install python3.9-devel xz-libs zlib libxml2-devel libxslt-devel bzip2-libs libpopt0 + +If these packages are not available from your package manager, you must +compile them from sources. + + +MacOS +""""" + +The default Python 3 provided with macOS is 3.9. +Alternatively you can download and install Python 3.9 from https://www.python.org/ + + +Windows +""""""" + +Download and install Python 3.9 from https://www.python.org/ + +.. Note:: + + 64-bit Python interpreters (x86-64) are the only interpreters supported by + ScanCode on all operating systems which means only 64-bit Windows is supported. + +See the :ref:`install-scancode-from-release-archives-windows` section for more installation details. diff --git a/docs/source/getting-started/installation/install-scancode-from-release-archives.rst b/docs/source/getting-started/installation/install-scancode-from-release-archives.rst new file mode 100644 index 00000000000..965344d551f --- /dev/null +++ b/docs/source/getting-started/installation/install-scancode-from-release-archives.rst @@ -0,0 +1,150 @@ +.. _install-scancode-from-release-archives: + +Install ScanCode from releases archives +======================================= + +Get the ScanCode-Toolkit tarball archive of a specific version and your +operating system by going to the `project releases page `_ + +.. _install-scancode-from-release-archives-linux: + +Install ScanCode on Linux +------------------------- + +Download the archive for your operating system and extract +the archive from command line:: + + tar -xvf scancode-toolkit-30.0.1_py38-linux.tar.gz + + +Or, on Linux, right click and select "Extract Here". + +Check whether the :ref:`install-scancode-required-packages` are installed. Open a terminal +in the extracted directory and run:: + + ./scancode --help + +This will configure ScanCode and display the command line :ref:`cli-help-text-options`. + +If the command doesn't throw an error, congratulations! You are good to go to :ref:`running-a-scan`. + +.. Note:: + + ScanCode archives come with packaged with all required dependencies except + for Python that has to be downloaded and installed separately. + On more recent versions of Ubuntu, you will have to install Python 3.9 manually. + One possibility is to use the Deadsnakes PPA (Personal Package Archive) which is + a project that provides older Python version builds for Debian and Ubuntu and is + available at https://github.com/deadsnakes/ and https://launchpad.net/~deadsnakes/+archive/ubuntu/ppa + :: + + sudo apt-get update && sudo apt-get upgrade + sudo add-apt-repository ppa:deadsnakes/ppa --yes + sudo apt-get install python3.9 python3.9-distutils + +.. _install-scancode-from-release-archives-mac: + +Install ScanCode on Mac +----------------------- + +.. note:: + If you encounter a "No matching distribution" error while running the ``./configure`` command on a Mac M1, it may indicate compatibility issues with the current architecture. Here's a step-by-step guide to address this: + + - **Change Mac M1 Architecture to x86_64:** + Switch the architecture from amd64 to x86_64 using the command: + :: + + env /usr/bin/arch -x86_64 /bin/zsh --login + - **Use Rosetta Translation:** + Enable Rosetta translation in Terminal by executing: + :: + + softwareupdate --install-rosetta + - **Transition Homebrew from arm64 to Intel:** + Change Homebrew from the arm64 architecture to the Intel (x86) architecture by running: + :: + + /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)" + - **Install Intel-Specific Python:** + Use Homebrew to install Python specifically optimized for Intel architecture with: + :: + + /usr/local/Homebrew/bin/brew install python3 + + Then rerun the ``./configure`` command. This sets up the project according to the new architecture and ensures proper configuration. + Following these steps should help resolve compatibility issues and allow smooth operation of the project on Mac M1 devices. + +.. _install-scancode-from-release-archives-windows: + +Install ScanCode on Windows 10/11 +--------------------------------- + +- Download the latest ScanCode release zip file for Windows from the latest + version at https://github.com/aboutcode-org/scancode-toolkit/releases/ + +- In the File Explorer, select the downloaded ScanCode zip and right-click. + +- In the pop-up menu select 'Extract All...' + +- In the pop-up window 'Extract Compressed (Zipped) Folders' use the default options to extract. + +- Once the extraction is complete, a new File Explorer window will pop up. + +- In this Explorer window, select the new folder that was created and right-click. + +.. note:: + + On Windows, double-click the new folder, select one of the files inside the folder + (e.g., 'setup.py'), and right-click. + +- In the pop-up menu select 'Properties'. + +- In the pop-up window 'Properties', select the Location value. Copy this to the clipboard and + close the 'Properties' window. + +- Press the start menu button, click the search box or search icon in the taskbar. + +- In the search box type:: + + cmd + +- Select 'cmd.exe' or 'Command Prompt' listed in the search results. + +- A new 'Command Prompt'pops up. + +- In this window (aka a 'command prompt'), type 'cd' followed by a space and + then Right-click in this window and select Paste. This will paste the path you + copied before and is where you extracted ScanCode:: + + cd path\to\extracted\ScanCode + +- Press Enter. + +- This will change the current location of your command prompt to the root directory where + ScanCode is installed. + +- Then type:: + + scancode -h + +- Press enter. This first command will configure your ScanCode installation. + Several messages are displayed followed by the ScanCode command help. + +- The installation is complete. Congratulations!, you are good to go to :ref:`running-a-scan`. + + +Uninstall ScanCode +------------------ + +#. Delete the directory in which you extracted ScanCode. +#. Delete any temporary files created in your system temp and user temp directory + under a ScanCode-prefixed directory such as .scancode-tk or .cache/scancode-tk. + + +.. note:: + + The above installation process can be used with **Command Prompt** + ``cmd``, and **PowerShell**. **Git Bash** is not tested and using it may + introduce unexpected behavior. If you're using **Windows Subsystem for + Linux** ``WSL2``, please refer to :ref:`install-scancode-from-release-archives-linux` section + above. diff --git a/docs/source/getting-started/installation/install-scancode-from-source.rst b/docs/source/getting-started/installation/install-scancode-from-source.rst new file mode 100644 index 00000000000..26983b7f112 --- /dev/null +++ b/docs/source/getting-started/installation/install-scancode-from-source.rst @@ -0,0 +1,61 @@ +.. _install-scancode-from-source: + +Install ScanCode from source +============================ + +You can download the source code of the ScanCode and build it locally. +This approach is recommended in the following scenarios: + +- You are contributing to the development of ScanCode, adding new patches, or running tests. +- You need to test or run a specific version, checkpoint, or branch from the version control system. + + +Download ScanCode source code +----------------------------- + +Run the following once you have `Git `_ installed + +.. code-block:: shell + + git clone https://github.com/aboutcode-org/scancode-toolkit.git + cd scancode-toolkit + + +Configure the build +------------------- + +ScanCode utilizes a configuration script to create an isolated virtual environment +and install the necessary package dependencies. + +On Linux/Mac: + +#. Open a terminal +#. Navigate to the clone directory using ``cd`` +#. Run ``./configure`` +#. Activate the virtual environment using ``source venv/bin/activate`` + +On Windows: + +#. Open a command prompt +#. Navigate to the clone directory using ``cd`` +#. Run ``configure`` +#. Activate the virtual environment: ``venv\Scripts\activate`` + +Verify installation +------------------- + +To verify that ScanCode has been installed correctly, +it is recommended to run the help command. + + +.. code-block:: shell + + scancode --help + + No errors? Congratulations! You are good to go to :ref:`running-a-scan`. + +.. NOTE:: + + For use in development, run instead ``configure --dev``. If your encounter + issues while configuring a previous version, use ``configure --clean`` to + clean and reset your enviroment. After that, run ``configure`` again. diff --git a/docs/source/getting-started/installation/install-scancode-using-docker.rst b/docs/source/getting-started/installation/install-scancode-using-docker.rst new file mode 100644 index 00000000000..ca33487e9bb --- /dev/null +++ b/docs/source/getting-started/installation/install-scancode-using-docker.rst @@ -0,0 +1,74 @@ +.. _install-scancode-using-docker: + +Install ScanCode using docker +============================= + +You can install ScanCode by building a Docker image from the included Dockerfile. +The prerequisite is a working `docker installation `_. + + +Download ScanCode sources +------------------------- + +Run the following once you have `Git `_ installed + +.. code-block:: shell + + git clone https://github.com/aboutcode-org/scancode-toolkit.git + + +Build the docker image +---------------------- + +Run the ``docker build`` source code checkout directory. + +.. code-block:: shell + + cd scancode-toolkit + docker build --tag scancode-toolkit --tag scancode-toolkit:$(git describe --tags) . + +.. note:: + + As ``$(git describe --tags)`` is a Unix-style command. If you are on + windows, please run the ``git describe --tags`` separately to get the + output, then manually insert that value into your Docker command. + For instance, :: + + C:\scancode-toolkit>git describe --tags + v32.4.1 + C:\scancode-toolkit>docker build --tag scancode-toolkit --tag scancode-toolkit:v32.4.1 . + +Verify installation +------------------- + +To verify that ScanCode has been installed correctly, +it is recommended to run the help command. + + +.. code-block:: shell + + docker run scancode-toolkit --help + + +Run using docker +---------------- + +The docker image will forward all arguments it receives directly to the ``scancode`` command. + +.. code-block:: shell + + docker run scancode-toolkit --help + +Mount current working directory as "/project" and run a scan on a file name +apache-2.0.LICENSE directory. The JSON results will be in scan-result.json + +.. code-block:: shell + + docker run -v $PWD/:/project scancode-toolkit -clipeu --json-pp /project/scan-result.json /project/apache-2.0.LICENSE + +This will mount your current working from the host into ``/project`` in the container +and then scan the contents. The output ``result.json`` will be written back to your +current working directory on the host. + +Note that the parameters *before* ``scancode-toolkit`` are used for docker, +those after will be forwarded to scancode. diff --git a/docs/source/getting-started/installation/install-scancode-using-pip.rst b/docs/source/getting-started/installation/install-scancode-using-pip.rst new file mode 100644 index 00000000000..2a8e69cb6a7 --- /dev/null +++ b/docs/source/getting-started/installation/install-scancode-using-pip.rst @@ -0,0 +1,64 @@ +.. _install-scancode-using-pip: + +Install ScanCode using pip +========================== + +ScanCode can be installed from the public PyPI repository using ``pip`` which +the standard Python package management tool. + +.. NOTE:: + + Note that `pip` installation method does work on ARM chips, i.e. Linux/MacOS on + Apple M1 chips, as some non-native dependencies do not have pre-built wheels + for ARM (like py-ahocorasick, intbitset). See :ref:`install-scancode-installation-prerequisites` for + more information. See related issues for more info: + + - `Fallback pure-python deps `_ + - `pip install failing on Apple M1 Macs `_ + +The steps are: + +#. Create a Python virtual environment + + .. code-block:: shell + + /usr/bin/python3 -m venv venv + + To learn more about Python virtualenv, including installation and usage see this + `tutorial `_. + +#. Activate the virtual environment you just created + + .. code-block:: shell + + source venv/bin/activate + +#. Run ``pip`` to install the latest versions of base utilities + + .. code-block:: shell + + pip install --upgrade pip setuptools wheel + +#. Install the latest version of ScanCode + + .. code-block:: shell + + pip install scancode-toolkit + +#. No errors? Congratulations! You are good to go to :ref:`running-a-scan`. + +.. NOTE:: + + For advanced usage, ``scancode-toolkit-mini`` is an alternative package with + no default dependencies on pre-built binaries. This may come handy for some + special use cases such as packaging for a Linux or FreeBSD distro. + + +Uninstalling ScanCode +--------------------- + +To uninstall ScanCode, run the following command. + +.. code-block:: shell + + pip uninstall scancode-toolkit diff --git a/docs/source/getting-started/newcomer.rst b/docs/source/getting-started/newcomer.rst deleted file mode 100644 index b62d242bae4..00000000000 --- a/docs/source/getting-started/newcomer.rst +++ /dev/null @@ -1,302 +0,0 @@ -.. _new_to_scancode: - -Are you new to ScanCode-Toolkit? -================================ - -This is the perfect place to start, if you are new to ScanCode-Toolkit. Have a quick look at the -table of contents below, as these are the main sections you might need help on. These sections -have extensive links to other important documentation pages, and make sure you go through them -all. - -Table of Contents ------------------ - -#. :ref:`newcomer_try_scancode` - - - :ref:`newcomer_before` - - :ref:`newcomer_scan_codebase` - - :ref:`newcomer_scancode_tips` - - :ref:`newcomer_all_tutorials` - - :ref:`newcomer_whats_new` - -#. :ref:`newcomer_learn_scancode` - - - :ref:`newcomer_cli_ref` - - :ref:`newcomer_explanations` - - :ref:`newcomer_plugins` - -#. :ref:`newcomer_contribute` - - - :ref:`newcomer_contribute_general_info` - - :ref:`newcomer_code` - - :ref:`newcomer_good_first_issue` - - :ref:`newcomer_add_functionalirty` - - :ref:`newcomer_update_docs` - - :ref:`newcomer_gsoc_gsod` - ----- - -.. _newcomer_try_scancode: - -Try ScanCode-Toolkit --------------------- - -This section is about using the ScanCode-Toolkit, i.e. Performing a scan on a codebase/files to -determine their license, copyrights and other information, according to your requirements. - -#. The :ref:`newcomer_scan_codebase` section helps you with configuring your virtual environment, - installing ScanCode and performing a basic scan, and subsequently visualize the results. - -#. The :ref:`newcomer_scancode_tips` section helps you customize the scan according to your - requirements, and better understand the advanced features you can use. - -#. The :ref:`newcomer_all_tutorials` is essentially an exhaustive list of all Tutorials and How To's - with a brief description on what they help you to achieve. - -.. _newcomer_before: - -Before you start using ScanCode -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -#. You need to make sure :ref:`install_prerequisites` are installed. There - are a few different ways to install ScanCode: - -- :ref:`app_install` [Recommended for newcomers] -- :ref:`docker_install` -- :ref:`pip_install` -- :ref:`source_code_install` -- :ref:`fedora_install` - - -#. Run ``scancode -h`` to make sure ScanCode was installed properly. - If this shows any Error, refer the `Common Installation Errors Issue `_ - for common errors. - -.. note:: - - Refer :ref:`synopsis_quickstart` to make sure you are using the scan command correctly. - -.. note:: - - For Windows, Refer to :ref:`windows_app_install` for installing easily using Releases. - -.. _newcomer_scan_codebase: - -Scan a Codebase -^^^^^^^^^^^^^^^ - -Once you are all set up with ScanCode-Toolkit, i.e. Running ``scancode -h`` shows the -:ref:`cli_help_text`, you can start scanning files or a codebase. - -#. Refer :ref:`synopsis_quickstart` for commonly used scan commands, and commonly used - :ref:`synopsis_output`. (The recommended output format is ``JSON``) - -#. Refer :doc:`../rst_snippets/extract` for the use of ExtractCode. - -#. :ref:`how_to_run_a_scan` is a sample tutorial for absolute beginners, to walk them through the - process of running a scan. Follow this tutorial and perform a scan on the ``sample`` folder - distributed with ScanCode, or any file/folder of your choice. Avoid advanced options, and just - follow the basic instructions. - -#. ScanCode generates output files with scan results. You can visualize ``JSON`` result files using - `ScanCode Workbench `_. Follow this tutorial :ref:`how_to_visualize_scan_results` - to visualize the scan results. - -.. _newcomer_scancode_tips: - -Use ScanCode Better -^^^^^^^^^^^^^^^^^^^ - -#. Go through all the options in the page :ref:`cli_list_options`, to know about ScanCode Command - Line options. You can then modify the Scan according to your requirements. - -.. _newcomer_all_tutorials: - -All Tutorials/How-Tos -^^^^^^^^^^^^^^^^^^^^^ - -The Tutorials are: - -#. :ref:`how_to_run_a_scan` -#. :ref:`how_to_visualize_scan_results` -#. :ref:`how_to_set_what_scan_detects` -#. :ref:`how_to_extract_archives` -#. :ref:`how_to_specify_output_format` -#. :ref:`how_to_add_post_scan_plugin` - -The How-To's are: - -#. :ref:`add_new_license_for_det` -#. :ref:`add_new_license_det_rule` - -.. _newcomer_whats_new: - -ScanCode Versions -^^^^^^^^^^^^^^^^^ - -#. You can see all ScanCode-Toolkit versions on the `GitHub release page `_. -#. Read the `CHANGELOG `_ for more information on specific releases. -#. If you want to use/test a specific version of ScanCode-Toolkit, you can follow the instructions - in :ref:`source_code_install` docs. - ----- - -.. _newcomer_learn_scancode: - -Learn more about ScanCode-Toolkit ---------------------------------- - -Here we give an introduction on the ScanCode-Toolkit Documentation Sections that can help you to -learn more about ScanCode-Toolkit. - -.. _newcomer_cli_ref: - -CLI Reference -^^^^^^^^^^^^^ - -This section contains a complete guide to ScanCode-Toolkit Command Line options, i.e. What the -command-line options are, how different options affect the scan and outputs, how to use these -options and examples of their use cases. - -Now this section has three types of pages: - -#. The :ref:`cli_synopsis` page and the :ref:`how_to_run_a_scan` page as summaries. -#. An exhaustive list of all Command Line Options at :ref:`cli_list_options` -#. All the other pages detailing the :ref:`scancode_cli_options` - -Note that the page for one type of options also has a short list of all the options detailed on -that page in the beginning. The :ref:`cli_list_options` page just has all of them together, and -also the extractcode options. - -.. _newcomer_explanations: - -How ScanCode Works -^^^^^^^^^^^^^^^^^^ - -This section has documentation on :ref:`explain_how_scancode_works`. - -.. _newcomer_plugins: - -Plugins -^^^^^^^ - -Plugins are an integral part of ScanCode-Toolkit in the sense they are used to easily extend -ScanCode capabilities, and developers can code their own plugins according to their requirements. - -This section has documentation on: - -#. The :ref:`plugin_arch` -#. The :ref:`license_policy_plugin` -#. All :ref:`plugin_tutorials` - ----- - -.. _newcomer_contribute: - -Contribute ----------- - -If you are looking to Contribute to ScanCode-Toolkit, this is where you start. - -.. _newcomer_contribute_general_info: - -General Information -^^^^^^^^^^^^^^^^^^^ - -#. Also refer the `Contribution `_ page here. -#. For more Project Ideas, refer :ref:`contributor_project_ideas`. -#. Before committing your work, make sure you have read this post on :ref:`good_commit_messages`. - -.. _newcomer_code: - -Contribute Code -^^^^^^^^^^^^^^^ - -If you haven't contributed to ScanCode-Toolkit refer :ref:`newcomer_good_first_issue`. - -To determine where to contribute, you can refer: - -#. ScanCode-Toolkit tracks issues via the `GitHub Issue tracker `_ -#. Broad `milestones `_ for upcoming versions are also maintained. - -And documentation related to contributing code can be referred at :ref:`contrib_code_dev`. - -.. _newcomer_good_first_issue: - -Good First Issues -^^^^^^^^^^^^^^^^^ - -A `good first issue `_ -means it's recommended for people who haven't contributed to ScanCode-Toolkit before. - -.. _newcomer_add_functionalirty: - -Add new Functionality/Enhancement to ScanCode -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -There are two main paths you can follow to add a new functionality to ScanCode. -They are: - -#. Add the functionality to ScanCode itself. -#. Add plugins if the functionality is very much application dependent. - -Refer `enhancement issues `_ for the first type of -enhancements. If you want to add a plugin to implement the functionality, refer all the -:ref:`plugin_tutorials`. - -.. _newcomer_update_docs: - -Update our Documentation -^^^^^^^^^^^^^^^^^^^^^^^^ - -Maintaining a comprehensive, accurate, updated and effective documentation is very important -as that directly affects the acceptability of ScanCode-Toolkit. - -To contribute to ScanCode-Toolkit Documentation, first refer the :ref:`contrib_doc_dev` section. - -The sections in this page cover the following: - -#. :ref:`contrib_doc_setup_local` -#. :ref:`contrib_doc_share_improvements` -#. :ref:`doc_ci` system for the Documentation -#. :ref:`doc_style_docs8` -#. :ref:`doc_interspinx` -#. :ref:`doc_style_conv` - -You can contribute to the following Open Issues on documentation. - -#. `First Timers Only Issues List `_ -#. `Documentation Inconsistencies Tracker `_ -#. `ScanCode-Toolkit Documentation Roadmap `_ -#. `Issues with label Documentation `_ - -.. note:: - - Refer :ref:`improve_docs` to report Documentation Errors or to request Improvements. - -Also, consider contributing to other Aboutcode Project Documentations, as they need more support. - -.. _newcomer_gsoc_gsod: - - -Participate in GSoC/GSoD -^^^^^^^^^^^^^^^^^^^^^^^^ - -If you want to participate in any of the two programs: - -- `Google Summer of Code `_ -- `Google Season of Docs `_ - -Then: - -#. Keep an eye out for Application Timelines. -#. Solve multiple of these :ref:`newcomer_good_first_issue` to demonstrate your skills, - and improve your chances of selection. -#. Refer to the Projects Ideas List for details on tentative projects. - - - `GSoC2023 `_ - -#. Remain active in Element and talk with the organization mentors well ahead of the deadlines. -#. Select projects according to your skills and finalize project proposals. -#. Discuss your proposals extensively with corresponding mentors. -#. Apply for the Programs well before the Deadline. diff --git a/docs/source/tutorials/data/scancode-toolkit-extract.png b/docs/source/how-to-guides/data/scancode-toolkit-extract.png similarity index 100% rename from docs/source/tutorials/data/scancode-toolkit-extract.png rename to docs/source/how-to-guides/data/scancode-toolkit-extract.png diff --git a/docs/source/how-to-guides/add_new_license_detection_rule.rst b/docs/source/how-to-guides/how-to-add-new-license-detection-rule.rst similarity index 97% rename from docs/source/how-to-guides/add_new_license_detection_rule.rst rename to docs/source/how-to-guides/how-to-add-new-license-detection-rule.rst index 19c8b680fd4..4b68b412f9e 100644 --- a/docs/source/how-to-guides/add_new_license_detection_rule.rst +++ b/docs/source/how-to-guides/how-to-add-new-license-detection-rule.rst @@ -1,6 +1,6 @@ -.. _add_new_license_det_rule: +.. _how-to-add-new-license-detection-rule: -How to Add New License Rules for Enhanced Detection +How to add new license rules for enhanced detection =================================================== ScanCode relies on license rules to detect licenses. A rule is a simple text diff --git a/docs/source/how-to-guides/add_new_license.rst b/docs/source/how-to-guides/how-to-add-new-license.rst similarity index 93% rename from docs/source/how-to-guides/add_new_license.rst rename to docs/source/how-to-guides/how-to-add-new-license.rst index 85b3a8bdd28..455f441a676 100644 --- a/docs/source/how-to-guides/add_new_license.rst +++ b/docs/source/how-to-guides/how-to-add-new-license.rst @@ -1,10 +1,7 @@ -.. _add_new_license_for_det: +.. _how-to-add-new-license: -How To Add a New License for Detection -====================================== - -How to add a new license for detection? ---------------------------------------- +How to add a new license to ScanCode detection +============================================== To add a new license, you first need to select a new and unique license `key` (``mit`` and ``gpl-2.0`` are some of the existing license keys). diff --git a/docs/source/tutorials/how_to_extract_archives.rst b/docs/source/how-to-guides/how-to-extract-archives.rst similarity index 69% rename from docs/source/tutorials/how_to_extract_archives.rst rename to docs/source/how-to-guides/how-to-extract-archives.rst index 2cf1a2f3b60..12a52c71421 100644 --- a/docs/source/tutorials/how_to_extract_archives.rst +++ b/docs/source/how-to-guides/how-to-extract-archives.rst @@ -1,7 +1,11 @@ -.. _how_to_extract_archives: +.. _extract-archives: -How To Extract Archives -======================= +.. toctree:: + :maxdepth: 1 + :hidden: + +Extract archives +================ ScanCode-Toolkit provides archive extraction. This command can be used before running a scan over a codebase in order to ensure all archives are extracted. Archives found inside an extracted @@ -10,11 +14,10 @@ archive with ``'-extract'`` appended. .. image:: data/scancode-toolkit-extract.png -Usage: ------- +**Usage:** :: extractcode [OPTIONS] -.. include:: /rst_snippets/extract.rst +For other available options see also :ref:`cli-extractcode` reference. diff --git a/docs/source/how-to-guides/generate_attribution.rst b/docs/source/how-to-guides/how-to-generate-attribution-docs.rst similarity index 70% rename from docs/source/how-to-guides/generate_attribution.rst rename to docs/source/how-to-guides/how-to-generate-attribution-docs.rst index 9deeb84f746..310f3524aff 100644 --- a/docs/source/how-to-guides/generate_attribution.rst +++ b/docs/source/how-to-guides/how-to-generate-attribution-docs.rst @@ -1,10 +1,7 @@ -.. _generate_attribution: +.. _how-to-generate-attribution-docs: -How To Generate Attribution from a ScanCode Scan -================================================ - -How To generate attribution from a ScanCode scan? -------------------------------------------------- +How to generate attribution doc from a ScanCode scan +==================================================== Users can use an Open Source Project "AboutCode Toolkit" to generate attrbution document from a ScanCode scan. diff --git a/docs/source/how-to-guides/install_new_license_plugin.rst b/docs/source/how-to-guides/how-to-install-new-license-plugin.rst similarity index 87% rename from docs/source/how-to-guides/install_new_license_plugin.rst rename to docs/source/how-to-guides/how-to-install-new-license-plugin.rst index 1fc72017711..f993419f6ff 100644 --- a/docs/source/how-to-guides/install_new_license_plugin.rst +++ b/docs/source/how-to-guides/how-to-install-new-license-plugin.rst @@ -1,7 +1,7 @@ -.. _install_external_licenses: +.. _how-to-install-new-license-plugin: -How to Install External Licenses to Use in License Detection -============================================================ +How to install external licenses to use in license dectection +============================================================= Users can install external licenses and rules in the form of: @@ -10,7 +10,7 @@ Users can install external licenses and rules in the form of: These licenses and rules are then used in license detection. -.. _install_new_license_plugin: +.. _install-new-license-plugin: How to install a plugin containing external licenses and/or rules ----------------------------------------------------------------- @@ -70,7 +70,7 @@ Finally, the same directory containing the class definition must also contain th licenses and/or rules. Licenses must be contained in a directory called ``licenses`` and rules must be contained in a directory called ``rules``. -See :ref:`add_new_license_for_det` and :ref:`add_new_license_det_rule` to understand +See :ref:`how-to-add-new-license` and :ref:`how-to-add-new-license-detection-rule` to understand the structure of license and rule files, respectively. After creating this plugin, you can upload it to PyPI so that others can use it, or you can @@ -86,7 +86,7 @@ To use the plugin in license detection, all you need to do is: ``pip install tests/licensedcode/data/additional_licenses/additional_plugin_2/`` 3. Reindex licenses using `scancode-reindex-licenses`. -.. include:: /rst_snippets/note_snippets/license_plugin_needs_reindex.rst +.. include:: /rst-snippets/note-snippets/license-plugin-needs-reindex.rst Once it is installed, the contained licenses and rules will automatically be used in license detection assuming the plugin follows the correct directory structure conventions. @@ -146,16 +146,18 @@ Finally, install the plugin and run the test: ``pytest -vvs tests/test_detection_datadriven.py``. -.. include:: /rst_snippets/note_snippets/license_plugin_delete.rst +.. include:: /rst-snippets/note-snippets/license-plugin-delete.rst ---- -.. _add_new_license_directory: +.. _add-new-license-directory: How to add external licenses and/or rules from a directory ---------------------------------------------------------- -This is the basic structure of the example license directory:: +This is the basic structure of the example license directory + +:: additional_license_directory/ ├── licenses/ @@ -163,6 +165,7 @@ This is the basic structure of the example license directory:: ├── rules/ │ ├── example-installed-1.RULE + Adding the licenses to the index ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -171,13 +174,13 @@ To add the licenses in the directory to the index, all you need to do is: 1. Configure the scancode-toolkit virtualenv and activate. 2. Run ``scancode-reindex-licenses`` with: - ``--additional-directory tests/licensedcode/data/additional_licenses/additional_dir/`` +.. code-block:: none -.. include:: /rst_snippets/note_snippets/additional_directory_is_temp.rst + scancode-reindex-licenses --additional-directory tests/licensedcode/data/additional_licenses/additional_dir/ +For other available options see also :ref:`cli-scancode-reindex-licenses`. -Once the licenses/rules are in the index, they will automatically be used in license detection. +.. include:: /rst-snippets/note-snippets/add-licenses-or-rules-from-additional-directory-is-temporary.rst ----- -.. include:: /rst_snippets/scancode-reindex-licenses.rst +Once the licenses/rules are in the index, they will automatically be used in license detection. diff --git a/docs/source/how-to-guides/index.rst b/docs/source/how-to-guides/index.rst index 113be1e1e46..6a5992e4e18 100644 --- a/docs/source/how-to-guides/index.rst +++ b/docs/source/how-to-guides/index.rst @@ -4,9 +4,10 @@ ================= .. toctree:: - :maxdepth: 2 + :maxdepth: 1 - add_new_license - add_new_license_detection_rule - install_new_license_plugin - generate_attribution + how-to-add-new-license + how-to-add-new-license-detection-rule + how-to-install-new-license-plugin + how-to-generate-attribution-docs + how-to-extract-archives diff --git a/docs/source/index.rst b/docs/source/index.rst index 0527a2c2ee1..17432fd75d2 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -1,122 +1,95 @@ ScanCode-Toolkit Documentation ============================== -Welcome to ScanCode-Toolkit Documentation! +ScanCode Toolkit is a set of code scanning tools that detect the origin (copyrights), license +and vulnerabilities of code, packages and dependencies in a codebase. +It is the leading tool in scanning depth and accuracy, used by hundreds of software teams. +You can use ScanCode Toolkit as a command line tool or as a library. -If you are new to ScanCode-Toolkit, start here: +ScanCode is part of the AboutCode community! Join our `friendly Slack `_ to ask questions, +share ideas or discuss your challenges with other members of our community. +If you want to ask questions or anything else that you think are not bugs/new +features, open a `discussion `_ with the ScanCode repository. -.. toctree:: - :maxdepth: 2 +Documentation overview +~~~~~~~~~~~~~~~~~~~~~~ - getting-started/newcomer +The overview below outlines how the documentation is structured +to help you know where to look for certain things. -Here's a list of more Documentation Pages: +.. rst-class:: clearfix row -- A :ref:`cli_synopsis` of ScanCode Command Line Options -- Tutorials on :ref:`how_to_run_a_scan` and :ref:`how_to_visualize_scan_results` -- An exhaustive List of :ref:`cli_list_options` -- Documentation on :ref:`contrib_code_dev` -- Documentation on :ref:`plugin_arch` -- :ref:`faq` +.. rst-class:: column column2 top-left ----- +:ref:`getting-started` +~~~~~~~~~~~~~~~~~~~~~~ -Getting Started ---------------- +Start here if you are new to ScanCode. -.. toctree:: - :maxdepth: 2 +- :ref:`install-scancode` - getting-started/index - ----- - -Command Line Options Reference ------------------------------- - -Reference documents describe the Command Line options, and application concepts in depth. - -.. toctree:: - :maxdepth: 2 - - cli-reference/index - ----- - -Tutorials ---------- - -Tutorial documents provide specific instructions to help you get started. - -.. toctree:: - :maxdepth: 2 - - tutorials/index + - :ref:`install-scancode-from-release-archives` + - :ref:`install-scancode-using-docker` + - :ref:`install-scancode-from-source` + - :ref:`install-scancode-using-pip` ----- +- :ref:`faq` -How-To Documents ----------------- -How-To documents explain how to accomplish specific tasks. +.. rst-class:: column column2 top-right -.. toctree:: - :maxdepth: 2 - - how-to-guides/index +:ref:`tutorials` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ----- +Learn via practical step-by-step guides. -Contribute To ScanCode ----------------------- +- :ref:`running-a-scan` +- :ref:`visualizing-scan-results` +- :ref:`configuring-scan-output-formats` +- :ref:`configuring-scan-detection` +- :ref:`adding-a-post-scan-plugin` -.. toctree:: - :maxdepth: 2 +.. rst-class:: column column2 bottom-left - contribute/index +:ref:`how-to-guides` +~~~~~~~~~~~~~~~~~~~~ ----- +Helps you accomplish things. -Plugins Documentation ---------------------- +- :ref:`how-to-add-new-license` +- :ref:`how-to-add-new-license-detection-rule` +- :ref:`how-to-install-new-license-plugin` +- :ref:`how-to-generate-attribution-docs` -.. toctree:: - :maxdepth: 2 +.. rst-class:: column column2 bottom-right - plugins/index +:ref:`reference` and :ref:`explanation` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ----- +Consult the reference to find CLI parameters. -Miscellaneous Documents ------------------------ +- :ref:`cli-scancode` options +- :ref:`plugins` to extend ScanCode at different stages +- :ref:`supported-packages` +- Rebuild the license index with the :ref:`cli-scancode-reindex-licenses` -.. toctree:: - :maxdepth: 2 +Broaden your understanding using the explanation of ScanCode key concepts. - misc/index +- Innerworkings of :ref:`scancode-license-detection` +.. rst-class:: row clearfix -Reference Documents --------------------- +Improving Documentation +~~~~~~~~~~~~~~~~~~~~~~~ -Reference documents provide reference pages for technical reference information -about ScanCode-Toolkit, including how it works and supported features. +.. include:: /rst-snippets/improve-docs.rst .. toctree:: :maxdepth: 2 + :hidden: + getting-started/index + tutorials/index + how-to-guides/index reference/index - - ----- - -Indices and Tables -================== - -* :ref:`genindex` -* :ref:`modindex` -* :ref:`search` - -.. _improve_docs: - -.. include:: /rst_snippets/improve_docs.rst + explanation/index diff --git a/docs/source/misc/index.rst b/docs/source/misc/index.rst deleted file mode 100644 index 386c5448771..00000000000 --- a/docs/source/misc/index.rst +++ /dev/null @@ -1,10 +0,0 @@ -**Miscellaneous** -================= - -.. toctree:: - :maxdepth: 2 - - faq - support - perf_report - versioning diff --git a/docs/source/misc/perf_report.rst b/docs/source/misc/perf_report.rst deleted file mode 100644 index f03808db48a..00000000000 --- a/docs/source/misc/perf_report.rst +++ /dev/null @@ -1,14 +0,0 @@ -Runtime Performance Reports -=========================== - -These are reports of runtimes for real life scans: - -**2015-09-03 by @rrjohnston** - -- On Ubuntu 12.04 x86_64 Python 2.7.3 and ScanCode Version 1.3.1 -- Specs: 40 threads (2 processors, 10 cores each, with hyperthreading) 3.1 GHz 128GB RAM 8TB - controller RAID5 -- scanned 195676 files in about 16.7 hours or about 3.25 file per second - (using defaults licenses and copyrights) -- notes: this version of ScanCode runs on a single thread so it does not make good use of - extra processing power. diff --git a/docs/source/misc/support.rst b/docs/source/misc/support.rst deleted file mode 100644 index e1781f1b711..00000000000 --- a/docs/source/misc/support.rst +++ /dev/null @@ -1,28 +0,0 @@ -.. _support: - -Support -======= - -Documentation -------------- - -The ScanCode toolkit documentation lives at https://scancode-toolkit.readthedocs.io/. - -Issue Tracker -------------- - -Post issues you are having and bugs as `GitHub tickets `_ - -Discussions ------------ - -If you want to ask questions or anything else that you think are not bugs/new -features, open a `discussion `_ - -Join the conversation ----------------------- - -Join our `general chatroom `_ to chat -with aboutcode community members, and if you want to talk to users and developers of -ScanCode-Toolkit, use `scancode room `_ - diff --git a/docs/source/plugins/index.rst b/docs/source/plugins/index.rst deleted file mode 100644 index 9f70b10a646..00000000000 --- a/docs/source/plugins/index.rst +++ /dev/null @@ -1,12 +0,0 @@ -**Plugins** -=========== - -.. toctree:: - :maxdepth: 2 - - plugin_arch - licence_policy_plugin - plugin_tutorials - cpp_includes_plugin - lkmclue - dwarf diff --git a/docs/source/plugins/plugin_tutorials.rst b/docs/source/plugins/plugin_tutorials.rst deleted file mode 100644 index 59bad566f25..00000000000 --- a/docs/source/plugins/plugin_tutorials.rst +++ /dev/null @@ -1,6 +0,0 @@ -.. _plugin_tutorials: - -Plugin Tutorials -================ - -- :ref:`how_to_add_post_scan_plugin` diff --git a/docs/source/reference/index.rst b/docs/source/reference/index.rst index e3792ea74d6..921abcb0357 100644 --- a/docs/source/reference/index.rst +++ b/docs/source/reference/index.rst @@ -1,11 +1,13 @@ -.. _how_it_works: +.. _reference: -**Reference Docs** -================== +**Reference** +============= + +Technical reference material. .. toctree:: :maxdepth: 2 - overview - license-detection-reference - available_package_parsers + scancode-cli/index.rst + scancode-plugins/index.rst + scancode-supported-packages diff --git a/docs/source/cli-reference/basic-options.rst b/docs/source/reference/scancode-cli/cli-basic-options.rst similarity index 93% rename from docs/source/cli-reference/basic-options.rst rename to docs/source/reference/scancode-cli/cli-basic-options.rst index 6ecc5e6a0a5..d200d7bb9a4 100644 --- a/docs/source/cli-reference/basic-options.rst +++ b/docs/source/reference/scancode-cli/cli-basic-options.rst @@ -1,14 +1,20 @@ -`Basic` Options +.. _cli-basic-options: + +`Basic` options =============== -.. include:: /rst_snippets/basic_options.rst +Quick reference +--------------- + +.. include:: /rst-snippets/cli-basic-options.rst + :start-line: 3 ---- -.. _copyright_option: +.. _cli-copyright-option: -``--copyright`` Option ------------------------ +``--copyright`` +--------------- The ``--copyright`` option detects copyright statements in files. @@ -26,13 +32,17 @@ containing the whole copyright author value, with ``start_line`` and ``end_line`` containing the line numbers in the file where this copyright value was detected. - Example:: + **Example** + + .. code-block:: none # # Copyright (c) 2010 Patrick McHardy All rights reserved. # Authors: Patrick McHardy - The above lines when scanned for copyrights generates the following results for the discussed attributes:: + The above lines when scanned for copyrights generates the following results for the discussed attributes + + .. code-block:: json { "copyrights": [ @@ -60,10 +70,10 @@ ---- -.. _license_option: +.. _cli-license-option: -``--license`` Option --------------------- +``--license`` +------------- The ``--license`` option detects various kinds of license texts, notices, tags, references and other specialized license declarations like the SPDX license identifier in files. @@ -82,12 +92,16 @@ 5. ``percentage_of_license_text``: This has a percentage number which denotes what percentage of the resource scanned has legalese words. - Example:: + **Example** + + .. code-block:: none License: Apache-2.0 If we run license detection (with ``--license-text``) on the above text we get the following - result for the resource attributes added by the license detection:: + result for the resource attributes added by the license detection + + .. code-block:: json { "path": "apache-2.0.txt", @@ -121,7 +135,9 @@ } We also have top level unique license detections with the same identifier - referencing all occurrences of this license detection and counts:: + referencing all occurrences of this license detection and counts + + .. code-block:: json { "license_detections": [ @@ -136,10 +152,10 @@ ---- -.. _package_option: +.. _cli-package-option: -``--package`` Option --------------------- +``--package`` +------------- The ``--package`` option detects various package manifests, lockfiles and package-like data and then assembles codebase level packages and dependencies from these @@ -163,9 +179,11 @@ 2. ``dependencies``: This is a mapping of dependency data from all the lockfiles or package manifests in the scan. - Example: + **Example** + + The following scan result was generated from scanning a package manifest - The following scan result was generated from scanning a package manifest:: + .. code-block:: json { "dependencies": [ @@ -365,10 +383,10 @@ ---- -.. _info_option: +.. _cli-info-option: -``--info`` Option ------------------ +``--info`` +---------- The ``--info`` option obtains miscellaneous information about the file being scanned such as mime/filetype, checksums, programming language, and various @@ -386,10 +404,10 @@ ---- -.. _email_option: +.. _cli-email-option: -``--email`` Option ------------------- +``--email`` +----------- The ``--email`` option detects and reports email adresses present in scanned files. @@ -399,10 +417,10 @@ ---- -.. _url_option: +.. _cli-url-option: -``--url`` Option ----------------- +``--url`` +--------- The ``--url`` option detects and reports URLs present in scanned files. @@ -413,19 +431,23 @@ ---- -.. _generated_option: +.. _cli-generated-option: -``--generated`` Option ----------------------- +``--generated`` +--------------- The ``--generated`` option classifies automatically generated code files with a flag. - An example of using ``--generated`` in a scan:: + **Example** + + .. code-block:: shell scancode -clpieu --json-pp output.json samples --generated In the results, for each file the following attribute is added with it's corresponding - ``true``/``false`` value :: + ``true``/``false`` value + + .. code-block:: none "is_generated": true @@ -434,10 +456,10 @@ ---- -.. _max_email_option: +.. _cli-max-email-option: -``--max-email`` Option ----------------------- +``--max-email INT`` +------------------- .. admonition:: Dependency @@ -452,7 +474,9 @@ - 0 - No limit, include all emails. - 50 - Default. - An example usage:: + **Example** + + .. code-block:: shell scancode -clpieu --json-pp output.json samples --max-email 5 @@ -460,10 +484,10 @@ ---- -.. _max_url_option: +.. _cli-max-url-option: -``--max-url`` Option --------------------- +``--max-url INT`` +----------------- .. admonition:: Dependency @@ -478,7 +502,9 @@ - 0 - No limit, include all urls. - 50 - Default. - An example usage:: + **Example** + + .. code-block:: shell scancode -clpieu --json-pp output.json samples --max-url 10 @@ -486,10 +512,10 @@ ---- -.. _license_score_option: +.. _cli-license-score-option: -``--license-score`` Option --------------------------- +``--license-score INT`` +----------------------- .. admonition:: Dependency @@ -506,35 +532,37 @@ Here, a bigger number means a better match, i.e. Setting a higher license score translates to a higher threshold for matching licenses (with equal or less number of license matches). - An example usage:: + **Example** + + .. code-block:: shell scancode -clpieu --json-pp output.json samples --license-score 70 - Here's the license results on setting the integer value to 100, Vs. the default value 0. This is + Here's the license results on setting the integer value to 100, vs. the default value 0. This is visualized using ScanCode workbench in the License Info Dashboard. .. list-table:: License scan results of Samples Directory. - * - .. figure:: data/core_lic_score_0.png + * - .. figure:: data/cli-license-score-option-score-0.png License Score 0 (Default). - - .. figure:: data/core_lic_score_100.png + - .. figure:: data/cli-license-score-option-score-100.png License Score 100. ---- -.. _license_text_option: +.. _cli-license-text-option: -``--license-text`` Option -------------------------- +``--license-text`` +------------------ .. admonition:: Dependency The option ``--license-text`` is a sub-option of and requires the option ``--license``. - .. admonition:: Sub-Option + .. admonition:: Sub-option The option ``--license-text-diagnostics`` is a sub-option of ``--license-text``. @@ -547,11 +575,15 @@ With the ``--license-text`` option, the scan results attribute "matched text" includes the matched text for the detected license. - An example Scan:: + **Example** + + .. code-block:: shell scancode -cplieu --json-pp output.json samples --license-text - An example matched text included in the results is as follows:: + An example matched text included in the results is as follows + + .. code-block:: none "matched_text": " This software is provided 'as-is', without any express or implied @@ -576,10 +608,10 @@ ---- -.. _license_url_template_option: +.. _cli-license-url-template-option: -``--license-url-template`` Option ---------------------------------- +``--license-url-template TEXT`` +------------------------------- .. admonition:: Dependency @@ -593,7 +625,9 @@ So, by default the license reference URL points to the LicenseDB page for that license. - A scan example using the ``--license-url-template TEXT`` option :: + **Example** + + .. code-block:: shell scancode -clpieu --json-pp output.json samples --license-url-template https://github.com/aboutcode-org/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/{}.LICENSE @@ -613,10 +647,10 @@ ---- -.. _license_text_diagnostics_option: +.. _cli-license-text-diagnostics-option: -``--license-text-diagnostics`` Option -------------------------------------- +``--license-text-diagnostics`` +------------------------------ .. admonition:: Dependency @@ -629,7 +663,9 @@ In a normal scan, whole lines of text are included in the matched license text, including parts that are possibly unmatched. - An example Scan:: + **Example** + + .. code-block:: shell scancode -cplieu --json-pp output.json samples --license-text --license-text-diagnostics @@ -637,7 +673,9 @@ causes the following difference in the scan result of the file ``samples/JGroups/licenses/bouncycastle.txt``. - Without Diagnostics:: + Without Diagnostics + + .. code-block:: none "matched_text": "License Copyright (c) 2000 - 2006 The Legion Of The Bouncy Castle @@ -660,10 +698,10 @@ ---- -.. _license_diagnostics_option: +.. _cli-license-diagnostics-option: -``--license-diagnostics`` Option -------------------------------------- +``--license-diagnostics`` +~~~~~~~~~~~~~~~~~~~~~~~~~ .. admonition:: Dependency @@ -675,7 +713,11 @@ about the license detection post-processing steps which are used to create license detections from license matches. - Consider the following text:: + **Example** + + Consider the following text + + .. code-block:: none ## License All code, unless stated otherwise, is dual-licensed under @@ -683,7 +725,9 @@ [`MIT`](https://opensource.org/licenses/MIT). If we run a license scan with the ``--license-diagnostics`` option enabled, - we have the following license detection results:: + we have the following license detection results + + .. code-block:: json { "path": "README.md", @@ -705,7 +749,7 @@ "rule_identifier": "lead-in_unknown_30.RULE", "rule_relevance": 100, "rule_url": "https://github.com/aboutcode-org/scancode-toolkit/tree/develop/src/licensedcode/data/rules/lead-in_unknown_30.RULE", - "matched_text": "dual-licensed under [` + "matched_text": "dual-licensed under" }, { "score": 50.0, diff --git a/docs/source/cli-reference/core-options.rst b/docs/source/reference/scancode-cli/cli-core-options.rst similarity index 78% rename from docs/source/cli-reference/core-options.rst rename to docs/source/reference/scancode-cli/cli-core-options.rst index c2fc6ae9e07..e945d245da0 100644 --- a/docs/source/cli-reference/core-options.rst +++ b/docs/source/reference/scancode-cli/cli-core-options.rst @@ -1,18 +1,26 @@ -`Core` Options +.. _cli-core-options: + +`Core` options ============== -.. _cli_core: +Quick reference +--------------- -.. include:: /rst_snippets/core_options.rst +.. include:: /rst-snippets/cli-core-options.rst + :start-line: 3 ---- -.. _quiet_verbose_options: +.. _cli-quiet-verbose-options: -Comparing Progress Message Options +Comparing progress message options ---------------------------------- - **Default Progress Message**:: + **Default progress message** + + By default a rolling progress bar and scanned file count is shown. + + .. code-block:: none Scanning files for: infos, licenses, copyrights, packages, emails, urls with 1 process(es)... Building license detection index...Done. @@ -26,7 +34,11 @@ Comparing Progress Message Options Indexing time: 2s. Saving results. - **Progress Message with ``--verbose``**:: + **Progress message with --verbose** + + When ``--verbose`` is enabled, progress messages for individual files are shown. + + .. code-block:: none Scanning files for: infos, licenses, copyrights, packages, emails, urls with 1 process(es)... Building license detection index...Done. @@ -49,16 +61,14 @@ Comparing Progress Message Options Indexing time: 2s. Saving results. - So, with ``--verbose`` enables, progress messages for individual files are shown. - - **With the ``--quiet`` option enabled**, nothing is printed on the Command Line. + **With the ``--quiet`` option enabled**, nothing is printed on the command line. ---- -.. _timeout_option: +.. _cli-timeout-option: -``--timeout`` Option --------------------- +``--timeout`` +------------- This option sets scan timeout for **each file** (and not the entire scan). If some file scan exceeds the specified timeout, that file isn't scanned anymore and the next file scanning @@ -69,15 +79,17 @@ Comparing Progress Message Options ---- -.. _from_json_option: +.. _cli-from-json-option: -``--from-json`` Option ----------------------- +``--from-json`` +--------------- If you want to input scan results from a .json file, and run a scan again on those same files, with some other options/output format, you can do so using the ``--from-json`` option. - An example scan command using ``--from-json``:: + **Example** + + .. code-block:: shell scancode --from-json sample.json --json-pp sample_2.json --classify @@ -86,10 +98,10 @@ Comparing Progress Message Options ---- -.. _max_in_memory_option: +.. _cli-max-in-memory-option: -``--max-in-memory`` Option ----------------------------------- +``--max-in-memory INT`` +----------------------- During a scan, as individual files are scanned, the scan details for those files are kept on memory till the scan is completed. Then after the scan is completed, they are written in the @@ -100,20 +112,22 @@ Comparing Progress Message Options Some important INTEGER values of the ``--max-in-memory INTEGER`` option: - - **0** - Unlimited Memory, store all the file/directory scan results on memory - - **-1** - Use only Disk-Caching, store all the file/directory scan results on disk + - **0** - Unlimited ,emory, store all the file/directory scan results on memory + - **-1** - Use only disk-caching, store all the file/directory scan results on disk - **10000** - Default, store 10,000 file/directory scan results on memory and the rest on disk - An example usage:: + **Example** + + .. code-block:: shell scancode -clieu --json-pp sample.json samples --max-in-memory -1 ---- -.. _max_depth_option: +.. _cli-max-depth-option: -``--max_depth`` Option ----------------------------------- +``--max_depth INT`` +------------------- Normally, the scan takes place upto the maximum level of nesting of directories possible. But using the ``--max-depth`` option, you can specify the maximum level of directories to scan, @@ -123,7 +137,9 @@ Comparing Progress Message Options Note that the ``--max-depth`` option will be ignored if you are scanning from a JSON file using the ``--from-json`` option. In that case, the original depth is used. - An example usage:: + **Example** + + .. code-block:: shell scancode -clieu --json-pp results.json samples --max-depth 3 diff --git a/docs/source/reference/scancode-cli/cli-extractcode.rst b/docs/source/reference/scancode-cli/cli-extractcode.rst new file mode 100644 index 00000000000..c061671c062 --- /dev/null +++ b/docs/source/reference/scancode-cli/cli-extractcode.rst @@ -0,0 +1,27 @@ +.. _cli-extractcode: + +ExtractCode CLI +=============== + +ExtractCode is be used as an input preparation step, before running a ScanCode scan. + +Usage: ``extractcode [OPTIONS]`` + +Archives within an extracted archive are extracted **recursively** by default, +with the extraction occurring in a directory named "-extract" alongside the original archive. + +**Quick Reference** +------------------- + +--shallow Do not extract recursively nested archives (e.g. Not + archives in archives). + +--verbose Print verbose file-by-file progress messages. + +--quiet Do not print any summary or progress message. + +-h, --help Show the extractcode help message and exit. + +--about Show information about ScanCode and licensing and exit. + +--version Show the version and exit. diff --git a/docs/source/cli-reference/help-text-options.rst b/docs/source/reference/scancode-cli/cli-help-text-options.rst similarity index 93% rename from docs/source/cli-reference/help-text-options.rst rename to docs/source/reference/scancode-cli/cli-help-text-options.rst index 86b74a16b93..a3bfb44f776 100644 --- a/docs/source/cli-reference/help-text-options.rst +++ b/docs/source/reference/scancode-cli/cli-help-text-options.rst @@ -1,30 +1,38 @@ -Getting Help from the Command Line -================================== +.. _cli-help-text-options: -ScanCode-Toolkit Command Line Interface can help you to search for specific options or use cases +Help options +============ + +ScanCode-Toolkit command line interface can help you to search for specific options or use cases from the command line itself. These are two options are ``--help`` and ``--examples``, and are very helpful if you need a quick glance of the options or use cases. Or it can be useful when you can't access, the more elaborate online documentation. -.. include:: /rst_snippets/doc_help.rst +Quick reference +--------------- + +.. include:: /rst-snippets/cli-doc-help-options.rst + :start-line: 3 -.. _cli_help_text: +.. _cli-help-option: -Help text ---------- +``--help`` +---------- -The ScanCode-Toolkit Command Line Interface has a Help option displaying all the options. It also +The ScanCode command line onterface has a Help option displaying all the options. It also displays basic usage, and some simple examples. The command line option for this is ``--help``. .. Tip:: You can also use the shorter ``-h`` option, which does the same. -To see the help text from the Terminal, execute the following command:: +**Example** - $ scancode --help +.. code-block:: shell -The Following Help Text is displayed, i.e. This is the help text for ScanCode Version 32.0.0: + $ scancode --help + +The following help text is displayed for ScanCode version 32.0.0: .. code-block:: none @@ -61,7 +69,7 @@ The Following Help Text is displayed, i.e. This is the help text for ScanCode Ve --license-text-diagnostics In the matched license text, include diagnostic highlights surrounding with square brackets [] words that are not matched. - --license-url-template TEXT Set the template URL used for the license + --license-url-template URL Set the template URL used for the license reference URLs. Curly braces ({}) are replaced by the license key. [default: https://scancode- licensedb.aboutcode.org/{}] @@ -207,14 +215,15 @@ The Following Help Text is displayed, i.e. This is the help text for ScanCode Ve Note: when you run ScanCode, a progress bar is displayed with a counter of the number of files processed. Use --verbose to display file-by-file progress. -.. _cli_examples_text: +.. _cli-examples-option: -Command Examples Text +``--examples`` option --------------------- -The ScanCode-Toolkit Command Line Interface has an ``--examples`` option which displays some basic -examples (more than the basic synopsis in ``--help``). These examples include the following aspects -of code scanning: +The ``--examples`` option which displays some basic examples +(more than the basic synopsis in ``--help``). + +These examples include the following aspects of code scanning: - Scanning Single File/Directory - Output Scan results to stdout (as JSON) or HTML/JSON file @@ -223,15 +232,15 @@ of code scanning: - Using GLOB Patterns to Scan Multiple Files - Using Verbose Mode -The command line option for displaying these basic examples is ``--examples``. +**Example** -To see the help text from the Terminal, execute the following command:: +.. code-block:: shell - $ scancode --examples + scancode --examples -The Following Text is displayed, i.e. This is the examples for ScanCode Version 3.1.1 :: +The following text is displayed for ScanCode version 32.0.0 :: - ScanCode command lines examples: + Scancode command lines examples: (Note for Windows: use '\' back slash instead of '/' forward slash for paths.) @@ -288,30 +297,33 @@ The Following Text is displayed, i.e. This is the examples for ScanCode Version scancode --license --copyright --html scancode_result.html samples/zlib - To extract archives, see the 'extractcode' command instead. +.. _cli-plugins-option: -Plugins Help Text ------------------ +``--plugins`` +------------- -The command line option for displaying all the plugins is: +Show the list of available ScanCode plugins. -- ``--plugins`` +**Example** -To see the help text from the Terminal, execute the following command:: +.. code-block:: shell - $ scancode --plugins + scancode --plugins .. note:: Plugins that are shown by using ``--plugins`` include the following: - #. Post-Scan Plugins - #. Pre-Scan Plugins - #. Output Options - #. Output Control - #. Basic Scan Options + #. Post-scan plugins + #. Pre-scan plugins + #. Output options + #. Output control + #. Basic scan options -The Following Text is displayed, i.e. This is the available plugins for ScanCode Version 31.2.1 :: +A text similar to below onde is displayed, showing the available plugins +for ScanCode Version 32.0.0. + +.. code-block:: none -------------------------------------------- Plugin: scancode_output:csv class: formattedcode.output_csv:CsvOutput @@ -321,7 +333,7 @@ The Following Text is displayed, i.e. This is the available plugins for ScanCode required_plugins: options: help_group: output formats, name: csv: --csv - help: [DEPRECATED] Write scan output as CSV to FILE. The --csv option is deprecated and will be replaced by new CSV and tabular output formats in the next ScanCode release. Visit https://github.com/aboutcode-org/scancode-toolkit/issues/3043 to provide inputs and feedback. + help: [DEPRECATED] Write scan output as CSV to FILE. The --csv option is deprecated and will be replaced by new CSV and tabular output formats in the next ScanCode release. Visit https://github.com/nexB/scancode-toolkit/issues/3043 to provide inputs and feedback. doc: None -------------------------------------------- @@ -384,8 +396,7 @@ The Following Text is displayed, i.e. This is the available plugins for ScanCode required_plugins: options: help_group: output formats, name: html_app: --html-app - help: (DEPRECATED: use the ScanCode Workbench app instead) - Write scan output as a mini HTML application to FILE. + help: (DEPRECATED: use the ScanCode Workbench app instead) Write scan output as a mini HTML application to FILE. doc: Write scan output as a mini HTML application. @@ -498,6 +509,19 @@ The Following Text is displayed, i.e. This is the available plugins for ScanCode Filter files or directories without scan findings for the requested scans. + -------------------------------------------- + Plugin: scancode_post_scan:classify class: summarycode.classify_plugin:FileClassifier + codebase_attributes: + resource_attributes: is_legal, is_manifest, is_readme, is_top_level, is_key_file + sort_order: 4 + required_plugins: + options: + help_group: post-scan, name: classify: --classify + help: Classify files with flags telling if the file is a legal, or readme or test file, etc. + doc: + Classify a file such as a COPYING file or a package manifest with a flag. + + -------------------------------------------- Plugin: scancode_post_scan:consolidate class: summarycode.plugin_consolidate:Consolidator codebase_attributes: consolidated_components, consolidated_packages @@ -541,7 +565,7 @@ The Following Text is displayed, i.e. This is the available plugins for ScanCode Plugin: scancode_post_scan:license-clarity-score class: summarycode.score:LicenseClarityScore codebase_attributes: summary resource_attributes: - sort_order: 5 + sort_order: 2 required_plugins: options: help_group: post-scan, name: license_clarity_score: --license-clarity-score @@ -572,7 +596,7 @@ The Following Text is displayed, i.e. This is the available plugins for ScanCode required_plugins: options: help_group: post-scan, name: license_references: --license-references - help: Return reference data for all licenses and license rulespresent in detections. + help: Return reference data for all licenses and license rules present in detections. doc: Add license and rule reference data to a scan. @@ -666,19 +690,6 @@ The Following Text is displayed, i.e. This is the available plugins for ScanCode - packages - -------------------------------------------- - Plugin: scancode_pre_scan:classify class: summarycode.classify_plugin:FileClassifier - codebase_attributes: - resource_attributes: is_legal, is_manifest, is_readme, is_top_level, is_key_file - sort_order: 30 - required_plugins: - options: - help_group: pre-scan, name: classify: --classify - help: Classify files with flags telling if the file is a legal, or readme or test file, etc. - doc: - Classify a file such as a COPYING file or a package manifest with a flag. - - -------------------------------------------- Plugin: scancode_pre_scan:facet class: summarycode.facet:AddFacet codebase_attributes: @@ -800,6 +811,8 @@ The Following Text is displayed, i.e. This is the available plugins for ScanCode help: Scan for application package and dependency manifests, lockfiles and related data. help_group: primary scans, name: system_package: --system-package help: Scan for installed system package databases. + help_group: primary scans, name: package_only: --package-only + help: Scan for system and application package data and skip license/copyright detection and top-level package creation. help_group: documentation, name: list_packages: --list-packages help: Show the list of supported package manifest parsers and exit. doc: @@ -822,25 +835,31 @@ The Following Text is displayed, i.e. This is the available plugins for ScanCode doc: Scan a Resource for URLs. -``--list-packages`` Option --------------------------- +.. _cli-list-packages-option: + +``--list-packages`` +------------------- This shows all the types of packages that can be scanned using ScanCode. These are located in packagedcode i.e. Code used to parse various package formats. -See the :ref:`supported_packages` page for more details and documentation +See the :ref:`supported-packages` page for more details and documentation automatically generated using this data. -``--print-options`` Option --------------------------- +``--print-options`` +------------------- This option prints the options selected for one specific scan command. -If we run this command:: +**Example** + +.. code-block:: shell - scancode -clpieu --json-pp sample.json samples --classify --tallies --tallies-with-details --print-options + scancode -clpieu --json-pp sample.json samples --classify --tallies --tallies-with-details --print-options -The output will be:: +The output will be + +.. code-block:: none Options: classify: True diff --git a/docs/source/cli-reference/output-filters-and-control.rst b/docs/source/reference/scancode-cli/cli-output-control-and-filter-options.rst similarity index 67% rename from docs/source/cli-reference/output-filters-and-control.rst rename to docs/source/reference/scancode-cli/cli-output-control-and-filter-options.rst index f78850d2e57..2a54945b938 100644 --- a/docs/source/cli-reference/output-filters-and-control.rst +++ b/docs/source/reference/scancode-cli/cli-output-control-and-filter-options.rst @@ -1,34 +1,40 @@ -.. _cli_output_control_filters: +.. _cli-output-control-and-filter-options: -Controlling ScanCode Output and Filters +Controlling ScanCode output and filters ======================================= -.. include:: /rst_snippets/output_control_options.rst +Quick reference +--------------- + +.. include:: /rst-snippets/cli-output-control-and-filter-options.rst + :start-line: 3 ---- -``--strip-root`` Vs. ``--full-root`` +``--strip-root`` vs. ``--full-root`` ------------------------------------ For a default scan of the "samples" folder, this a comparison between the default, ``strip-root`` and ``full-root`` options. - An example Scan + **Example** - :: + .. code-block:: shell scancode -cplieu --json-pp output.json samples --full-root These two changes only the "path" attribute of the file information. For this comparison we compare the "path" attributes of the file ``LICENSE`` inside ``JGroups`` directory. - The default path:: + The default path + + .. code-block:: none "path": "samples/JGroups/LICENSE", For the ``--full-root`` option, the path relative to the Root of your local filesystem. - :: + .. code-block:: none "path": "/home/aboutcode/scancode-toolkit/samples/JGroups/LICENSE" @@ -36,56 +42,69 @@ Controlling ScanCode Output and Filters For the ``--strip-root`` option, the root directory (here ``/home/aboutcode/scancode-toolkit/samples/``) is removed from path : - :: + .. code-block:: none "path": "JGroups/LICENSE" - .. include:: /rst_snippets/note_snippets/control_strip_full_root.rst + .. include:: /rst-snippets/note-snippets/cli-output-control-strip-full-root.rst ---- -``--ignore-author `` Option ------------------------------------- +.. _ignore-author-option: + +``--ignore-author `` +----------------------------- In a normal scan, all files inside the directory specified as an input argument is scanned and subsequently included in the scan report. But if you want to run the scan on only some selective files, with some specific **common author** then ``--ignore-author`` option can be used to do the same. - This scan ignores all files with authors matching the string "Apache Software Foundation":: + This scan ignores all files with authors matching the string "Apache Software Foundation" + + .. code-block:: shell scancode -cplieu --json-pp output.json samples --ignore-author "Apache Software Foundation" - More information on :ref:`glob_pattern_matching`. + More information on :ref:`glob-pattern-matching`. - .. include:: /rst_snippets/warning_snippets/control_ignore_author_copyright.rst + .. include:: /rst-snippets/warning-snippets/cli-output-control-ignore-author-copyright.rst ---- -``--ignore-copyright-holder `` Option ----------------------------------------------- +.. _ignore-copyright-holder-option: + +``--ignore-copyright-holder `` +--------------------------------------- In a normal scan, all files inside the directory specified as an input argument is scanned and subsequently included in the scan report. But if you want to run the scan on only some selective files, with some specific **common copyright holder** then ``--ignore-copyright-holder`` option can be used to do the same. - This scan ignores all files with Copyright Holders matching the string "Free Software Foundation":: + This scan ignores all files with copyright holders matching + the string "Free Software Foundation" + + .. code-block:: shell scancode -cplieu --json-pp output.json samples --ignore-copyright-holder "Free Software Foundation" - More information on :ref:`glob_pattern_matching`. + More information on :ref:`glob-pattern-matching`. ---- -``--only-findings`` Plugin --------------------------- +.. _only-findings-option: + +``--only-findings`` +------------------- This option removes from the scan results, the files where nothing significant has been detected, like files which doesn't contain any licenses, copyrights, emails or urls (if requested in the scan options), and isn't a package. - An example Scan:: + **Example** + + .. code-block:: shell scancode -cplieu --json-pp output.json samples --only-findings diff --git a/docs/source/cli-reference/output-format.rst b/docs/source/reference/scancode-cli/cli-output-format-options.rst similarity index 86% rename from docs/source/cli-reference/output-format.rst rename to docs/source/reference/scancode-cli/cli-output-format-options.rst index 25ca252f317..1f3a4bb852c 100644 --- a/docs/source/cli-reference/output-format.rst +++ b/docs/source/reference/scancode-cli/cli-output-format-options.rst @@ -1,23 +1,29 @@ -.. _cli_output_format: +.. _cli-output-format-options: -ScanCode Output Formats +ScanCode output formats ======================= Scan results generated by ScanCode are available in different formats, to be specified by the following options. -.. include:: /rst_snippets/output_format_options.rst +Quick reference +--------------- + +.. include:: /rst-snippets/cli-output-format-options.rst + :start-line: 3 -.. include:: /rst_snippets/note_snippets/output_samples.rst +.. include:: /rst-snippets/note-snippets/cli-output-samples.rst ---- -.. _output_to_stdout: +.. _cli-stdout: -.. include:: /rst_snippets/stdout.rst +.. include:: /rst-snippets/cli-output-to-stdout.rst ---- +.. _cli-json-option: + ``--json FILE`` --------------- @@ -25,21 +31,25 @@ following options. others. ScanCode Workbench and other applications that use ScanCode Result data as input accept only the ``json`` format. + **Example** + The following code performs a scan on the samples directory, and publishes the results in - ``json`` format:: + ``json`` format + + .. code-block:: shell scancode -clpieu --json output.json samples - .. include:: /rst_snippets/note_snippets/output_json_ugly.rst + .. include:: /rst-snippets/note-snippets/cli-output-json-ugly.rst - .. figure:: data/json_ugly.png + .. figure:: data/cli-output-json-ugly.png The entire JSON file is structured in the following manner: At first some general information on the scan, what options were used, the number of files etc. And then all the files follow. - :: + .. code-block:: none { "headers": [ @@ -116,6 +126,8 @@ following options. ---- +.. _cli-json-pp-option: + ``--json-pp FILE`` ------------------ @@ -125,11 +137,15 @@ following options. results in json but in a properly spaced and indented manner, and is easy to look at. The following code performs a scan on the samples directory, and publishes the results in - ``json-pp`` format:: + ``json-pp`` format + + .. code-block:: shell scancode -clpieu --json-pp output.json samples - A sample JSON output for an individual file will look like:: + **Example** + + .. code-block:: json { "path": "samples/zlib/iostream2/zstream.h", @@ -199,28 +215,40 @@ following options. "scan_errors": [] }, - This is the recommended Output option for ScanCode-Toolkit. + This is the recommended output option for ScanCode-Toolkit. + + .. include:: /rst-snippets/note-snippets/cli-output-format-synopsis.rst ---- +.. _cli-json-lines-option: + ``--json-lines FILE`` --------------------- ScanCode also has a ``--json-lines`` format option, where each report of a file scanned is formatted in one line. + **Example** + The following code performs a scan on the samples directory, and publishes the results in - ``json-lines`` format:: + ``json-lines`` format + + .. code-block:: shell scancode -clpieu --json-lines output.json samples - Here is a sample line from a report generated by the ``jsonlines`` format:: + Here is a sample line from a report generated by the ``jsonlines`` format + + .. code-block:: none {"files":[{"path":"samples/zlib/ada",licenses":[],"copyrights":[],"packages":[]}]} The header information is also formatted in one line (i.e. The First Line of the file). - The whole Output file looks like:: + The whole Output file looks like + + .. code-block:: none {"headers":[{"tool_name":"scancode-toolkit","tool_version":"3.1.1","options":{"input":["samples/"],"--copyright":true,"--email":true,"--info":true,"--json-lines":"output.json","--license":true,"--package":true,"--url":true},"notice":"Generated with ScanCode and provided on an \"AS IS\" BASIS, WITHOUT WARRANTIES\nOR CONDITIONS OF ANY KIND, either express or implied. No content created from\nScanCode should be considered or used as legal advice. Consult an Attorney\nfor any legal advice.\nScanCode is a free software code scanning tool from nexB Inc. and others.\nVisit https://github.com/aboutcode-org/scancode-toolkit/ for support and download.","start_timestamp":"2019-10-19T210920.143831","end_timestamp":"2019-10-19T211052.048182","message":null,"errors":[],"extra_data":{"files_count":36}}]} {"files":[{"path":"samples" ... "scan_errors":[]}]} @@ -274,29 +302,31 @@ following options. {"files":[{"path":"samples/zlib/iostream2/zstream_test.cpp", ... "scan_errors":[]}]} - .. include:: /rst_snippets/note_snippets/output_jsonlines.rst + .. include:: /rst-snippets/note-snippets/cli-output-json-lines.rst ---- -.. _comparing_json: +.. _cli-comparing-json-output-file-formats: -Comparing Different ``json`` Output Formats +Comparing different ``json`` output formats ------------------------------------------- Default ``--json`` Output: - .. figure:: data/output_json.png + .. figure:: data/cli-output-json.png ``--json-pp`` Output: - .. figure:: data/output_jsonpp.png + .. figure:: data/cli-output-jsonpp.png ``--json-lines`` Output: - .. figure:: data/output_jsonlines.png + .. figure:: data/cli-output-json-lines.png ---- +.. _cli-rdf-option: + ``--spdx-rdf FILE`` ------------------- @@ -304,8 +334,12 @@ Comparing Different ``json`` Output Formats for communicating software bill of material information (including components, licenses, copyrights, and security references). + **Example** + The following code performs a scan on the samples directory, and publishes the results in - ``spdx-rdf`` format:: + ``spdx-rdf`` format + + .. code-block:: shell scancode -clpieu --spdx-rdf output.spdx samples @@ -315,10 +349,12 @@ Comparing Different ``json`` Output Formats Here the file is structured as a dictionary of named properties and classes using W3C's `RDF Technology `_. - .. figure:: data/output_spdx_rdf1.png + .. figure:: data/cli-output-spdx-rdf1.png ---- +.. _cli-spdx-tv-option: + ``--spdx-tv FILE`` ------------------ @@ -326,11 +362,15 @@ Comparing Different ``json`` Output Formats manner: The following code performs a scan on the samples directory, and publishes the results in - ``spdx-tv`` format:: + ``spdx-tv`` format + + .. code-block:: shell scancode -clpieu --spdx-tv output.spdx samples - A SPDX-TV file starts with:: + A SPDX-TV file starts with + + .. code-block:: none # Document Information @@ -351,7 +391,7 @@ Comparing Different ``json`` Output Formats After a section titled ``#Packages``, a list follows. - .. figure:: data/output_spdx_tv_package.png + .. figure:: data/cli-output-spdx-tv-package.png Each File information is listed under a ``#File`` title, for each of the files. @@ -364,9 +404,9 @@ Comparing Different ``json`` Output Formats - LicenseInfoInFile - FileCopyrightText - An example goes as follows: + **Example** - .. figure:: data/output_spdx_tv_file.png + .. figure:: data/cli-output-spdx-tv-file.png After the files section, there's a section for licenses under a ``#Licences`` title, with the following information for each license: @@ -378,13 +418,11 @@ Comparing Different ``json`` Output Formats - LicenseComment - ExtractedText - Here's an example: - - .. figure:: data/output_spdx_tv_licenses.png + .. figure:: data/cli-output-spdx-tv-licenses.png ---- -.. _output_html: +.. _cli-html-option: ``--html FILE`` --------------- @@ -394,7 +432,9 @@ Comparing Different ``json`` Output Formats main information in the form of tables. The following code performs a scan on the samples directory, and publishes the results in - HTML format:: + HTML format + + .. code-block:: shell scancode -clpieu --html output.html samples @@ -408,28 +448,32 @@ Comparing Different ``json`` Output Formats - Package Information - License References (SPDX ID, Links to spdx/scancode/licensedb/License Homepage) - .. include:: /rst_snippets/note_snippets/output_html_license_references.rst + .. include:: /rst-snippets/note-snippets/cli-output-html-license-references.rst - .. figure:: data/output_html1.png + .. figure:: data/cli-output-html1.png - .. figure:: data/output_html2.png + .. figure:: data/cli-output-html2.png - .. figure:: data/output_html3.png + .. figure:: data/cli-output-html3.png ---- +.. _cli-html-app-option: + ``--html-app FILE`` ------------------- ScanCode also supports formatting the output in a HTML visualization tool, which is more helpful than the standard HTML format. - .. include:: /rst_snippets/warning_snippets/output_htmlapp_dep.rst + .. include:: /rst-snippets/warning-snippets/cli-output-htmlapp-deprecated.rst The following code performs a scan on the samples directory, and publishes the results in - ``html-app`` format:: + ``html-app`` format - scancode -clpieu --html-app output.html samples + .. code-block:: shell + + scancode -clpieu --html-app output.html samples The Files scanned are shown in the left sidebar, and the section on the right contains separate tabs for the following: @@ -443,18 +487,20 @@ Comparing Different ``json`` Output Formats - File Details - Packages - .. include:: /rst_snippets/note_snippets/output_htmlapp_search.rst + .. include:: /rst-snippets/note-snippets/cli-output-htmlapp-search.rst - .. figure:: data/output_html_app1.png + .. figure:: data/cli-output-html-app1.png - .. figure:: data/output_html_app2.png + .. figure:: data/cli-output-html-app2.png - .. figure:: data/output_html_app3.png + .. figure:: data/cli-output-html-app3.png ---- -``--csv FILE`` --------------- +.. _cli-csv-option: + +``--csv `` +---------------- ScanCode can publish results in the useful ``.csv`` format. @@ -466,7 +512,9 @@ Comparing Different ``json`` Output Formats for details and to provide inputs and feedback. The following code performs a scan on the samples directory, and publishes the results in - ``csv`` format:: + ``csv`` format + + .. code-block:: shell scancode -lpceiu --csv sample.csv samples @@ -544,10 +592,12 @@ Comparing Different ``json`` Output Formats So if there's multiple elements in a file, they are each given an entry with the details mentioned earlier. - .. figure:: data/output_csv.png + .. figure:: data/cli-output-csv.png ---- +.. _cli-cyclonedx-json-option: + ``--cyclonedx FILE`` -------------------- ScanCode also supports the `CycloneDx `_ output format @@ -558,24 +608,30 @@ Comparing Different ``json`` Output Formats in downstream tools that can't process ScanCode's native JSON output, but do support CycloneDx BOMs. - To run an example scan on the test resources try: - ``./scancode --package --cyclonedx=bom.json tests/formattedcode/data/cyclonedx/simple`` + To run an example scan on the test resources try + + .. code-block:: shell + + scancode --package --cyclonedx=bom.json tests/formattedcode/data/cyclonedx/simple If you prefer XML output over JSON, please have a look at the ``--cyclonedx-xml`` option instead ____ +.. _cli-cyclonedx-xml-option: + ``--cyclonedx-xml FILE`` ------------------------- This option allows outputting CycloneDx BOMs in XML format instead of JSON - To run an example scan on the test resources try: - ``./scancode --package --cyclonedx-xml=bom.xml tests/formattedcode/data/cyclonedx/simple`` + **Example** -____ + .. code-block:: shell -.. include:: /rst_snippets/custom_output_format.rst + scancode --package --cyclonedx-xml=bom.xml tests/formattedcode/data/cyclonedx/simple + +____ .. ToDo: diff --git a/docs/source/cli-reference/scan-options-post.rst b/docs/source/reference/scancode-cli/cli-post-scan-options.rst similarity index 91% rename from docs/source/cli-reference/scan-options-post.rst rename to docs/source/reference/scancode-cli/cli-post-scan-options.rst index fdca3ccdd08..690b4d27a22 100644 --- a/docs/source/cli-reference/scan-options-post.rst +++ b/docs/source/reference/scancode-cli/cli-post-scan-options.rst @@ -1,22 +1,26 @@ -.. _cli_post_scan: +.. _cli-post-scan-options: -Post-Scan Options +Post-scan options ================= -Post-Scan options activate their respective post-scan plugins which execute the task. +Post-scan options activate their respective post-scan plugins which execute the task. -.. include:: /rst_snippets/post_scan_options.rst +Quick reference +--------------- + +.. include:: /rst-snippets/cli-post-scan-options.rst + :start-line: 3 To see all plugins available via command line help, use ``--plugins``. -.. include:: /rst_snippets/note_snippets/post_scan_plugins.rst +.. include:: /rst-snippets/note-snippets/cli-post-scan-plugins.rst ---- -.. _mark_source_option: +.. _cli-mark-source-option: -``--mark-source`` Option ------------------------- +``--mark-source`` +----------------- .. admonition:: Dependency @@ -25,7 +29,11 @@ To see all plugins available via command line help, use ``--plugins``. The ``mark-source`` option marks the ``is_source`` attribute of a directory to be ``True``, if more than 90% of the files under that directory is source files, and ``False`` otherwise. - When the following command is executed to scan the ``samples`` directory with this option enabled:: + **Example** + + When the following command is executed to scan the ``samples`` directory with this option enabled + + .. code-block:: none scancode -clpieu --json-pp output.json samples --mark-source @@ -40,10 +48,10 @@ To see all plugins available via command line help, use ``--plugins``. ---- -.. _consolidate_option: +.. _cli-consolidate-option: -``--consolidate`` Option ------------------------- +``--consolidate`` +----------------- .. admonition:: Dependency @@ -59,11 +67,15 @@ To see all plugins available via command line help, use ``--plugins``. The JSON file containing scan results after using the ``--consolidate`` Plugin is structured as follows: - An example Scan:: + **Example** + + .. code-block:: none scancode -clpieu --json-pp output.json samples --consolidate - The JSON output file is structured as follows:: + The JSON output file is structured as follows + + .. code-block:: none { "headers": [...], @@ -91,7 +103,9 @@ To see all plugins available via command line help, use ``--plugins``. "files": [...] } - Each consolidated component has the following information:: + Each consolidated component has the following information + + .. code-block:: none "consolidated_components": [ { @@ -113,7 +127,9 @@ To see all plugins available via command line help, use ``--plugins``. In addition to this, in every file/directory where the consolidated part (i.e. License information) was present, a "consolidated_to" attribute is added pointing to the "identifier" of - "consolidated_components":: + "consolidated_components" + + .. code-block:: none "consolidated_to": [ "dmitriy_anisimkov_1" @@ -123,21 +139,27 @@ To see all plugins available via command line help, use ``--plugins``. ---- -.. _filter_clues_option: +.. _cli-filter-clues-option: -``--filter-clues`` Option -------------------------- +``--filter-clues`` +------------------ - The ``--filter-clues`` Plugin filters redundant duplicated clues already contained in detected + The ``--filter-clues`` plugin filters redundant duplicated clues already contained in detected licenses, copyright texts and notices, authors. + **Example** + Consider the output of running the following scan (compared to running the scan without the - ``--filter-clues`` option):: + ``--filter-clues`` option) + + .. code-block:: none ./scancode -clpieu --json-pp sample_filter_clues.json samples --filter-clues When we run without the ``--filter-clues`` option, we have the following detections at - ``"path": "samples/JGroups/src/FixedMembershipToken.java"``:: + ``"path": "samples/JGroups/src/FixedMembershipToken.java"`` + + .. code-block:: none { "authors": [ @@ -156,7 +178,9 @@ To see all plugins available via command line help, use ``--plugins``. ] } - And when we run a scan with the ``--filter-clues`` option:: + And when we run a scan with the ``--filter-clues`` option + + .. code-block:: none { "authors": [ @@ -175,10 +199,10 @@ To see all plugins available via command line help, use ``--plugins``. ---- -.. _license_clarity_score: +.. _cli-post-scan-license-clarity-score: -``--license-clarity-score`` Option ----------------------------------- +``--license-clarity-score`` +--------------------------- .. admonition:: Dependency @@ -232,7 +256,9 @@ To see all plugins available via command line help, use ``--plugins``. such as copyleft and proprietary, have been detected in lower level code. - Scoring Weight = -20 - An example Scan:: + **Example** + + .. code-block:: none scancode -clpieu --json-pp output.json samples --classify --license-clarity-score @@ -249,7 +275,9 @@ To see all plugins available via command line help, use ``--plugins``. - "conflicting_license_categories" - "ambiguous_compound_licensing" - When the "license_clarity_score" is included, the entire JSON file is structured as follows:: + When the "license_clarity_score" is included, the entire JSON file is structured as follows + + .. code-block:: none { "headers": [...], @@ -268,16 +296,16 @@ To see all plugins available via command line help, use ``--plugins``. "files": [...] } - .. include:: /rst_snippets/note_snippets/post_license_clarity_score.rst + .. include:: /rst-snippets/note-snippets/cli-post-scan-license-clarity-score.rst ---- -.. _license_policy_option: +.. _cli-license-policy-option: -``--license-policy FILE`` Option --------------------------------- +``--license-polic FILE`` +------------------------ - .. include:: /rst_snippets/note_snippets/post_license_policy.rst + .. include:: /rst-snippets/note-snippets/cli-post-scan-license-policy.rst The Policy file is a YAML (.yml) document with the following structure:: @@ -292,7 +320,7 @@ To see all plugins available via command line help, use ``--plugins``. icon: icon-ok-circle - .. include:: /rst_snippets/note_snippets/post_lic_pol_key.rst + .. include:: /rst-snippets/note-snippets/cli-post-scan-license-file-required-policy-key.rst Applying License Policies during a ScanCode scan, using the ``--license-policy`` Plugin:: @@ -310,7 +338,9 @@ To see all plugins available via command line help, use ``--plugins``. - "color_code" - "icon" - Here the ``samples`` directory is scanned, and the Scan Results for a sample file is as follows:: + **Example** + + .. code-block:: none { "path": "samples/JGroups/licenses/apache-2.0.txt", @@ -330,14 +360,14 @@ To see all plugins available via command line help, use ``--plugins``. ... }, - More information on the :ref:`license_policy_plugin` and usage. + See also :ref:`license-policy-plugin`. ---- -.. _license_references_option: +.. _cli-license-references-option: -``--license-references`` Option -------------------------------- +``--license-references FILE`` +----------------------------- .. admonition:: Dependency @@ -354,15 +384,23 @@ To see all plugins available via command line help, use ``--plugins``. - ``license_rule_references`` with details from scancode license rules (which are each a ``.RULE`` file) - Consider a file ``mit.txt`` with the following license declaration:: + **Example** + + Consider a file ``mit.txt`` with the following license declaration + + .. code-block:: none License: mit - We run the following scan on this file:: + We run the following scan on this file + + .. code-block:: none scancode -l --license-text --license-references mit.txt --json-pp mit.json - See the results for this license scan with ``--license-references`` enabled:: + See the results for this license scan with ``--license-references`` enabled + + .. code-block:: none { "headers": [...], @@ -481,22 +519,26 @@ To see all plugins available via command line help, use ``--plugins``. ---- -.. _summary_option: +.. _cli-summary-option: -``--summary`` Option --------------------- +``--summary`` +------------- - .. admonition:: Sub-Option + .. admonition:: Sub-option The option ``--summary-by-facet``, ``--summary-key-files`` and - ``--summary-with-details``are sub-options of ``--summary``. These Sub-Options are all + ``--summary-with-details``are sub-options of ``--summary``. These Sub-options are all Post-Scan Options. - An example Scan:: + **Example** + + .. code-block:: none scancode -clpieu --json-pp output.json samples --summary - The whole JSON file is structured as follows, when the ``--summary`` plugin is applied:: + The whole JSON file is structured as follows, when the ``--summary`` plugin is applied + + .. code-block:: none { "headers": [...], @@ -516,7 +558,9 @@ To see all plugins available via command line help, use ``--plugins``. has multiple entries each containing "value" and "count", with their values having the summary information inside them. - See below a sample fully populated summary object:: + See below a sample fully populated summary object + + .. code-block:: none { "summary": { @@ -590,10 +634,10 @@ To see all plugins available via command line help, use ``--plugins``. ---- -.. _tallies_option: +.. _cli-tallies-option: -``--tallies`` Option --------------------- +``--tallies`` +------------- .. admonition:: Optional Dependency @@ -603,16 +647,20 @@ To see all plugins available via command line help, use ``--plugins``. ``--license``, ``--package``, ``--copyright`` and ``--info`` options enabled, or there will not be any corresponding data for these. - An example scan using the ``--tallies`` Plugin:: + **Example** + + .. code-block:: none scancode -clipeu --json-pp strapi.json strapi-main/ --tallies .. note: - We have used the `github:strapi/strapi `_ project to generate exmaple results for + We have used the `github:strapi/strapi `_ project to generate example results for this and all associated CLI options below. - The JSON file containing the ``--tallies`` scan results are as follows:: + The JSON file containing the ``--tallies`` scan results are as follows + + .. code-block:: none { "headers": [...], @@ -744,17 +792,17 @@ To see all plugins available via command line help, use ``--plugins``. ---- -.. _tallies_by_facet_option: +.. _cli-tallies-by-facet-option: -``--tallies-by-facet`` Option ------------------------------ +``--tallies-by-facet`` +---------------------- .. admonition:: Dependency The option ``--tallies-by-facet`` is a sub-option of and requires the options ``--facet`` and ``--tallies``. - For users who want to know :ref:`what_is_a_facet`. + See :ref:`facets` to learn more about what a facet is. Running the scan with ``--tallies --tallies-by-facet`` Plugins creates individual summaries for all the facets with the same license, copyright and other scan information, at a codebase level @@ -762,16 +810,20 @@ To see all plugins available via command line help, use ``--plugins``. Once all files have been assigned a facet, files without a facet are assigned to the core facet. - An example scan using the ``--tallies-by-facet`` Plugin:: + **Example** + + .. code-block:: none scancode -clipeu --json-pp strapi.json strapi-main/ --tallies --facet dev="*.js" --facet dev="*.ts" --tallies-by-facet We have used the `github:strapi/strapi `_ project to generate exmaple results for this CLI option. - .. include:: /rst_snippets/note_snippets/pre_facet_core.rst + .. include:: /rst-snippets/note-snippets/cli-pre-scan-facet-core.rst - A sample "summary_by_facet" object generated by the previous scan (shortened):: + A sample "summary_by_facet" object generated by the previous scan (shortened) + + .. code-block:: none { "headers": [...], @@ -943,21 +995,23 @@ To see all plugins available via command line help, use ``--plugins``. } - .. include:: /rst_snippets/note_snippets/post_summary_facet.rst + .. include:: /rst-snippets/note-snippets/cli-post-scan-summary-facet.rst ---- -.. _tallies_key_files_option: +.. _cli-tallies-key-files-option: -``--tallies-key-files`` Option ------------------------------- +``--tallies-key-files`` +----------------------- .. admonition:: Dependency The option ``--tallies-key-files`` is a sub-option of and requires the options ``--classify`` and ``--tallies``. - An example Scan:: + **Example** + + .. code-block:: shell scancode -clipeu --json-pp strapi.json strapi-main/ --classify --tallies --tallies-key-files @@ -965,7 +1019,9 @@ To see all plugins available via command line help, use ``--plugins``. with the same license, copyright and other scan information, at a codebase level (in addition to the codebase level general summary generated by ``--tallies`` Plugin). - The resulting JSON file containing the scan results is structured as follows:: + The resulting JSON file containing the scan results is structured as follows + + .. code-block:: none { "headers": [...], @@ -1024,23 +1080,25 @@ To see all plugins available via command line help, use ``--plugins``. ---- -.. _tallies_with_details_option: +.. _cli-tallies-with-details-option: -``--tallies-with-details`` Option ---------------------------------- +``--tallies-with-details`` +-------------------------- The ``--tallies`` plugin summarizes license, copyright and other scan information at the codebase level. Now running the scan with the ``--tallies-with-details`` plugin instead creates summaries at individual file/directories with the same license, copyright and other scan information, but at a file/directory level (in addition to the the codebase level summary). - An example Scan:: + **Example** + + .. code-block:: shell - scancode -clipeu --json-pp strapi.json strapi-main/ --tallies-with-details + scancode -clipeu --json-pp strapi.json strapi-main/ --tallies-with-details - .. include:: /rst_snippets/note_snippets/post_summary_details.rst + .. include:: /rst-snippets/note-snippets/cli-post-scan-summary-details.rst - A sample scan result is structured as follows:: + .. code-block:: none { "headers": [...], diff --git a/docs/source/cli-reference/scan-options-pre.rst b/docs/source/reference/scancode-cli/cli-pre-scan-options.rst similarity index 65% rename from docs/source/cli-reference/scan-options-pre.rst rename to docs/source/reference/scancode-cli/cli-pre-scan-options.rst index b1ccb54cf95..45379bf94ac 100644 --- a/docs/source/cli-reference/scan-options-pre.rst +++ b/docs/source/reference/scancode-cli/cli-pre-scan-options.rst @@ -1,57 +1,73 @@ -.. _cli_pre_scan: +.. _cli-pre-scan-options: -Pre-Scan Options +Pre-scan options ================ -.. include:: /rst_snippets/pre_scan_options.rst +Quick reference +--------------- + +.. include:: /rst-snippets/cli-pre-scan-options.rst + :start-line: 3 ---- -``--ignore`` Option -------------------- +.. _cli-ignore-option: + +``--ignore `` +---------------------- In a scan, all files inside the directory specified as an input argument is scanned. But if there are some files which you don't want to scan, the ``--ignore`` option can be used to do the same. - A sample usage:: + **Example** + + .. code-block:: shell scancode --ignore "*.java" samples samples.json Here, ScanCode ignores files ending with `.java`, and continues with other files as usual. - More information on :ref:`glob_pattern_matching`. + More information on :ref:`glob-pattern-matching`. ---- -``--include`` Option --------------------- +.. _cli-include-option: + +``--include `` +----------------------- In a normal scan, all files inside the directory specified as an input argument is scanned. But if you want to run the scan on only some selective files, then ``--include`` option can be used to do the same. - A sample usage:: + **Example** + + .. code-block:: shell scancode --include "*.java" samples samples.json Here, ScanCode selectively scans files that has names ending with `.java`, and ignores all other files. This is basically complementary in behavior to the ``--ignore`` option. - More information on :ref:`glob_pattern_matching`. + See also :ref:`glob-pattern-matching`. ---- +.. _cli-classify-option: + ``--classify`` -------------- - .. admonition:: Sub-Option + .. admonition:: Sub-option The options ``--license-clarity-score`` and ``--tallies-key-files`` are sub-options of ``--classify``. ``--license-clarity-score`` and ``--tallies-key-files`` are Post-Scan Options. - The ``--classify`` option can be used like:: + **Example** + + .. code-block:: shell scancode -clpieu --json-pp sample_facet.json samples --classify @@ -74,7 +90,9 @@ Pre-Scan Options ``primary-license`` or the overall license of the package, among other package metadata which are general or ecosystem specific. - As in, to the JSON object of each file scanned, these extra attributes are added:: + As in, to the JSON object of each file scanned, these extra attributes are added. + + .. code-block:: json { "is_legal": false, @@ -86,13 +104,15 @@ Pre-Scan Options ---- -``--facet`` Option ------------------- +.. _cli-facet-option: + +``--facet =`` +----------------------------- - .. admonition:: Sub-Option + .. admonition:: Sub-option The option ``--summary-by-facet`` is a sub-option of ``--facet``. ``--summary-by-facet`` is - a Post-Scan Option. + a post-scan option. Valid ```` values are: @@ -103,11 +123,15 @@ Pre-Scan Options - data, - examples. - You can use the ``--facet`` option in the following manner:: + You can use the ``--facet`` option in the following manner + + .. code-block:: none scancode -clpieu --json-pp sample_facet.json samples --facet dev="*.java" --facet dev="*.c" - This adds to the header object, the following attribute:: + This adds to the header object, the following attribute + + .. code-block:: none "--facet": [ "dev=*.java", @@ -116,27 +140,29 @@ Pre-Scan Options Here in this example, ``.java`` and ``.c`` files are marked as it belongs to facet ``dev``. - As a result, ``.java`` file has the following attribute added:: + As a result, ``.java`` file has the following attribute added + + .. code-block:: json "facets": [ "dev" ], - .. include:: /rst_snippets/note_snippets/pre_facet_core.rst + .. include:: /rst-snippets/note-snippets/cli-pre-scan-facet-core.rst For each facet, the ``--facet`` option precedes the ``=`` argument. For specifying multiple facets, this whole part is repeated, including the ``--facet`` option. - For users who want to know :ref:`what_is_a_facet`. + See :ref:`facets` to learn more about what a facet is. ---- -.. _glob_pattern_matching: +.. _glob-pattern-matching: Glob Pattern Matching --------------------- - All the Pre-Scan options use pattern matching, so the basics of Glob Pattern Matching is + All the pre-scan options use pattern matching, so the basics of Glob Pattern Matching is discussed briefly below. Glob pattern matching is useful for matching a group of files, by using patterns in their @@ -165,7 +191,7 @@ Glob Pattern Matching Note that wildcard patterns are not regular expressions, although they are a bit similar. - For more information on Glob pattern matching refer these resources: + For more information on glob pattern matching refer these resources: - `Linux Manual `_ - `Wildcard Match Documentation `_. @@ -175,37 +201,3 @@ Glob Pattern Matching - `fnmatch `_ for File Name matching - `glob `_ for File Path matching ----- - -.. _what_is_a_facet: - -What is a Facet? ----------------- - - A facet is essentially a file purpose classification label. - It is defined as follows (by ClearlyDefined): - - A facet of a component is a subset of the files related to the component. It's really just a - grouping that helps us understand the shape of the project. Each facet is described by a set of - glob expressions, essentially wildcard patterns that are matched against file names. - - Each facet definition can have zero or more glob expressions. A file can be captured by more - than one facet. Any file found but not captured by a defined facet is automatically assigned to - the core facet. - - - ``core`` - The files that go into making the release of the component. Note that the core - facet is not explicitly defined. Rather, it is made up of whatever is not in any other facet. - So, by default, all files are in the core facet unless otherwise specified. - - ``data`` - The files included in any data distribution of the component. - - ``dev`` - Files primarily used at development time (e.g., build utilities) and not - distributed with the component - - ``docs`` - Documentation files. Docs may be included with the executable component or - separately or not at all. - - ``examples`` -- Like docs, examples may be included in the main component release or - separately. - - ``tests`` -- Test files may include code, data and other artifacts. - - Important Links: - - - `Facets `_ - - `ClearlyDefined `_ diff --git a/docs/source/reference/scancode-cli/cli-scancode-license-data.rst b/docs/source/reference/scancode-cli/cli-scancode-license-data.rst new file mode 100644 index 00000000000..db08c3a817c --- /dev/null +++ b/docs/source/reference/scancode-cli/cli-scancode-license-data.rst @@ -0,0 +1,34 @@ +.. _cli-scancode-license-data: + +scancode license data CLI +========================= + +Scancode includes the license data for all the licenses in detects +as scancode-licensedb with the license texts and other reference data +like license category, authors, and reference links. + +``scancode-license-data`` is a CLI command to provide this license data +in a structured format that is API, and this includes: + +- all the license details, without license text, in one JSON/YAML file +- all the individual licenses, with license text, in JSON/YAML files +- locally built HTML page with a searchable table of all licenses +- locally built HTML pages for all individual licenses with its text and details + +The data and HTML pages are dumped into a folder specified by the required +CLI option ``--path``. + +The licensedb is also hosted at https://scancode-licensedb.aboutcode.org/ and is +updated daily from the latest added licenses from the scancode ``develop`` branch +using the ``scancode-license-data`` CLI option. + +Usage: ``scancode-license-data [OPTIONS]`` + +**Quick Reference** +------------------- + +--path DIR Dump the license data in this directory in the LicenseDB format + and exit. Creates the directory if it does not exist. + [required] +-h, --help Show this message and exit. + diff --git a/docs/source/cli-reference/other-commands.rst b/docs/source/reference/scancode-cli/cli-scancode-reindex-licenses.rst similarity index 62% rename from docs/source/cli-reference/other-commands.rst rename to docs/source/reference/scancode-cli/cli-scancode-reindex-licenses.rst index 53c56f01301..2b88175b399 100644 --- a/docs/source/cli-reference/other-commands.rst +++ b/docs/source/reference/scancode-cli/cli-scancode-reindex-licenses.rst @@ -1,33 +1,37 @@ -Other available CLIs -==================== +.. _cli-scancode-reindex-licenses: -.. _other_cli: - ----- - -.. include:: /rst_snippets/scancode-reindex-licenses.rst - ----- - -.. include:: /rst_snippets/extract.rst - ----- - -``scancode-reindex-licenses`` command -------------------------------------- +ScanCode reindex licenses CLI +============================= ScanCode maintains a license index to search for and detect licenses. When ScanCode is configured for the first time, a license index is built and used in every scan thereafter. +The ``scancode-reindex-licenses`` command rebuilds the license index. + +Usage: ``scancode-reindex-licenses [OPTIONS]`` + +Quick Reference +--------------- + + --all-languages [EXPERIMENTAL] Rebuild the license index + including texts all languages (and not only + English) and exit. + --only-builtin Rebuild the license index excluding any + additional license directory or additional + license plugins which were added previously, i.e. + with only builtin scancode license and rules. + --additional-directory DIR Include this directory with additional custom + licenses and license rules in the license + detection index. + --load-dump Load all license and rules from their respective + files and then dump them back to those same files. + -h, --help Shows the options and explanations. -This ``scancode-reindex-licenses`` command rebuilds the license index. Running this command -displays the following message to the terminal:: - - Checking and rebuilding the license index... +---- -This has several CLI options as follows: +.. _cli-scancode-reindex-licenses-additional-directory-option: -``--additional-directory`` Option: -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +``--additional-directory`` option +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The ``--additional-directory`` option allows the user to include additional directories of licenses to use in license detection. @@ -37,7 +41,9 @@ runs of ScanCode with the same directories all the licenses in the directories w and used in License detection. But reindexing removes these directories, if they aren't reintroduced as additional directories. -The directory structure should look something like this:: +The directory structure should look something like this + +.. code-block:: none additional_license_directory/ ├── licenses/ @@ -47,12 +53,15 @@ The directory structure should look something like this:: │ ├── example-installed-1.RULE │ └── example-installed-1.yaml -Here is an example of reindexing the license cache using the ``--additional-directory PATH`` option -with a single directory:: +**Example** + +.. code-block:: shell scancode-reindex-licenses --additional-directory tests/licensedcode/data/additional_licenses/additional_dir/ -You can also include multiple directories like so:: +You can also include multiple directories like so + +.. code-block:: shell scancode-reindex-licenses --additional-directory /home/user/external_licenses/external1 --additional-directory /home/user/external_licenses/external2 @@ -68,7 +77,7 @@ need to reindex the license index with those directories as parameters:: scancode --additional-directory /home/user/external_licenses/external1 --additional-directory /home/user/external_licenses/external3 -.. include:: /rst_snippets/note_snippets/additional_directory_is_temp.rst +.. include:: /rst-snippets/note-snippets/add-licenses-or-rules-from-additional-directory-is-temporary.rst .. note:: @@ -76,11 +85,12 @@ need to reindex the license index with those directories as parameters:: You can also install external licenses through a plugin for better reproducibility and distribution of those license/rules for use in conjunction with scancode-toolkit licenses. - See :ref:`install_new_license_plugin` + See :ref:`how-to-install-new-license-plugin` +.. _cli-scancode-reindex-licenses-only-builtin-option: -``--only-builtin`` Option: -^^^^^^^^^^^^^^^^^^^^^^^^^^ +``--only-builtin`` option +^^^^^^^^^^^^^^^^^^^^^^^^^ Rebuild the license index excluding any additional license directory or additional license plugins which were added previously, i.e. with only builtin scancode license and rules. @@ -94,15 +104,17 @@ reindex the licenses without these licenses from the additional plugins. would just reindex without the licenses from these plugins for once. Another reindex afterwards without this option would bring back the licenses from the plugins again in the index. +.. _cli-scancode-reindex-licenses-all-languages-option: -``--all-languages`` Option: -^^^^^^^^^^^^^^^^^^^^^^^^^^^ +``--all-languages`` option +^^^^^^^^^^^^^^^^^^^^^^^^^^ Rebuild the license index including texts all languages (and not only English) and exit. This is an EXPERIMENTAL option. +.. _cli-scancode-reindex-licenses-load-dump-option: -``--load-dump`` Option +``--load-dump`` option ^^^^^^^^^^^^^^^^^^^^^^ Load all licenses and rules from their respective files and then dump them diff --git a/docs/source/cli-reference/data/core_lic_score_0.png b/docs/source/reference/scancode-cli/data/cli-license-score-option-score-0.png similarity index 100% rename from docs/source/cli-reference/data/core_lic_score_0.png rename to docs/source/reference/scancode-cli/data/cli-license-score-option-score-0.png diff --git a/docs/source/cli-reference/data/core_lic_score_100.png b/docs/source/reference/scancode-cli/data/cli-license-score-option-score-100.png similarity index 100% rename from docs/source/cli-reference/data/core_lic_score_100.png rename to docs/source/reference/scancode-cli/data/cli-license-score-option-score-100.png diff --git a/docs/source/cli-reference/data/output_csv.png b/docs/source/reference/scancode-cli/data/cli-output-csv.png similarity index 100% rename from docs/source/cli-reference/data/output_csv.png rename to docs/source/reference/scancode-cli/data/cli-output-csv.png diff --git a/docs/source/cli-reference/data/output_html_app1.png b/docs/source/reference/scancode-cli/data/cli-output-html-app1.png similarity index 100% rename from docs/source/cli-reference/data/output_html_app1.png rename to docs/source/reference/scancode-cli/data/cli-output-html-app1.png diff --git a/docs/source/cli-reference/data/output_html_app2.png b/docs/source/reference/scancode-cli/data/cli-output-html-app2.png similarity index 100% rename from docs/source/cli-reference/data/output_html_app2.png rename to docs/source/reference/scancode-cli/data/cli-output-html-app2.png diff --git a/docs/source/cli-reference/data/output_html_app3.png b/docs/source/reference/scancode-cli/data/cli-output-html-app3.png similarity index 100% rename from docs/source/cli-reference/data/output_html_app3.png rename to docs/source/reference/scancode-cli/data/cli-output-html-app3.png diff --git a/docs/source/cli-reference/data/output_html1.png b/docs/source/reference/scancode-cli/data/cli-output-html1.png similarity index 100% rename from docs/source/cli-reference/data/output_html1.png rename to docs/source/reference/scancode-cli/data/cli-output-html1.png diff --git a/docs/source/cli-reference/data/output_html2.png b/docs/source/reference/scancode-cli/data/cli-output-html2.png similarity index 100% rename from docs/source/cli-reference/data/output_html2.png rename to docs/source/reference/scancode-cli/data/cli-output-html2.png diff --git a/docs/source/cli-reference/data/output_html3.png b/docs/source/reference/scancode-cli/data/cli-output-html3.png similarity index 100% rename from docs/source/cli-reference/data/output_html3.png rename to docs/source/reference/scancode-cli/data/cli-output-html3.png diff --git a/docs/source/cli-reference/data/output_jsonlines.png b/docs/source/reference/scancode-cli/data/cli-output-json-lines.png similarity index 100% rename from docs/source/cli-reference/data/output_jsonlines.png rename to docs/source/reference/scancode-cli/data/cli-output-json-lines.png diff --git a/docs/source/cli-reference/data/json_ugly.png b/docs/source/reference/scancode-cli/data/cli-output-json-ugly.png similarity index 100% rename from docs/source/cli-reference/data/json_ugly.png rename to docs/source/reference/scancode-cli/data/cli-output-json-ugly.png diff --git a/docs/source/cli-reference/data/output_json.png b/docs/source/reference/scancode-cli/data/cli-output-json.png similarity index 100% rename from docs/source/cli-reference/data/output_json.png rename to docs/source/reference/scancode-cli/data/cli-output-json.png diff --git a/docs/source/cli-reference/data/output_jsonpp.png b/docs/source/reference/scancode-cli/data/cli-output-jsonpp.png similarity index 100% rename from docs/source/cli-reference/data/output_jsonpp.png rename to docs/source/reference/scancode-cli/data/cli-output-jsonpp.png diff --git a/docs/source/cli-reference/data/output_spdx_rdf1.png b/docs/source/reference/scancode-cli/data/cli-output-spdx-rdf1.png similarity index 100% rename from docs/source/cli-reference/data/output_spdx_rdf1.png rename to docs/source/reference/scancode-cli/data/cli-output-spdx-rdf1.png diff --git a/docs/source/cli-reference/data/output_spdx_tv_file.png b/docs/source/reference/scancode-cli/data/cli-output-spdx-tv-file.png similarity index 100% rename from docs/source/cli-reference/data/output_spdx_tv_file.png rename to docs/source/reference/scancode-cli/data/cli-output-spdx-tv-file.png diff --git a/docs/source/cli-reference/data/output_spdx_tv_licenses.png b/docs/source/reference/scancode-cli/data/cli-output-spdx-tv-licenses.png similarity index 100% rename from docs/source/cli-reference/data/output_spdx_tv_licenses.png rename to docs/source/reference/scancode-cli/data/cli-output-spdx-tv-licenses.png diff --git a/docs/source/cli-reference/data/output_spdx_tv_package.png b/docs/source/reference/scancode-cli/data/cli-output-spdx-tv-package.png similarity index 100% rename from docs/source/cli-reference/data/output_spdx_tv_package.png rename to docs/source/reference/scancode-cli/data/cli-output-spdx-tv-package.png diff --git a/docs/source/cli-reference/data/scancode-toolkit-static-html1.png b/docs/source/reference/scancode-cli/data/scancode-toolkit-static-html1.png similarity index 100% rename from docs/source/cli-reference/data/scancode-toolkit-static-html1.png rename to docs/source/reference/scancode-cli/data/scancode-toolkit-static-html1.png diff --git a/docs/source/cli-reference/data/scancode-toolkit-static-html2.png b/docs/source/reference/scancode-cli/data/scancode-toolkit-static-html2.png similarity index 100% rename from docs/source/cli-reference/data/scancode-toolkit-static-html2.png rename to docs/source/reference/scancode-cli/data/scancode-toolkit-static-html2.png diff --git a/docs/source/reference/scancode-cli/index.rst b/docs/source/reference/scancode-cli/index.rst new file mode 100644 index 00000000000..c9056a625e6 --- /dev/null +++ b/docs/source/reference/scancode-cli/index.rst @@ -0,0 +1,90 @@ +.. _cli-scancode: + +**ScanCode CLI** +================ + +Below is a quick reference of all ScanCode options. +For detailed descriptions and examples, please refer to the chapters in the table of +contents or the link to the detailed CLI options at the end of each section. + +This has the same CLI options in the :ref:`cli-help-option`, available in the command line +but with more details and additional references. + +---- + +.. _cli-scancode-basic-options: + +.. include:: /rst-snippets/cli-basic-options.rst + +For more details into the basic CLI options, see :ref:`cli-basic-options`. + +---- + +.. _cli-scancode-core-options: + +.. include:: /rst-snippets/cli-core-options.rst + +For more details into the core CLI options, see :ref:`cli-core-options`. + +---- + +.. _cli-scancode-output-format-options: + +.. include:: /rst-snippets/cli-output-format-options.rst + +For more details into the output format CLI options, see :ref:`cli-output-format-options`. + +---- + +.. _cli-scancode-output-control-and-filter-options: + +.. include:: /rst-snippets/cli-output-control-and-filter-options.rst + +For more details into the output filter CLI options, see +:ref:`cli-output-control-and-filter-options`. + +---- + +.. _cli-scancode-pre-scan-options: + +.. include:: /rst-snippets/cli-pre-scan-options.rst + +For more details into the pre-scan CLI options, see :ref:`cli-pre-scan-options`. + +---- + +.. _cli-scancode-post-scan-options: + +.. include:: /rst-snippets/cli-post-scan-options.rst + +For more details into the post-scan CLI options, see :ref:`cli-post-scan-options`. + +---- + +.. _cli-other-commands: + +.. include:: /rst-snippets/cli-other-commands.rst + +---- + +.. _cli-doc-help-options: + +.. include:: /rst-snippets/cli-doc-help-options.rst + +---- + +**Detailed reference for CLI options/commands** +----------------------------------------------- + +.. toctree:: + + cli-help-text-options + cli-basic-options + cli-core-options + cli-output-format-options + cli-output-control-and-filter-options + cli-pre-scan-options + cli-post-scan-options + cli-extractcode + cli-scancode-reindex-licenses + cli-scancode-license-data diff --git a/docs/source/plugins/cpp_includes_plugin.rst b/docs/source/reference/scancode-plugins/cpp-includes-plugin.rst similarity index 76% rename from docs/source/plugins/cpp_includes_plugin.rst rename to docs/source/reference/scancode-plugins/cpp-includes-plugin.rst index 68927cf5a25..9b6dcb4ec15 100644 --- a/docs/source/plugins/cpp_includes_plugin.rst +++ b/docs/source/reference/scancode-plugins/cpp-includes-plugin.rst @@ -1,6 +1,6 @@ -.. _cpp_includes_plugin: +.. _cpp-includes-plugin: -CPP Includes Plugin +CPP includes plugin =================== This plugin allows users to collect the #includes statements in C/C++ files. @@ -10,14 +10,15 @@ Using the Plugin User needs to use the ``--cpp-includes`` option. -The following command will collect the #includes statements from C/C++ files.:: +The following command will collect the #includes statements from C/C++ files. - $ scancode --cpp-includes /path/to/codebase/ --json-pp ~/path/to/scan-output.json +.. code-block:: shell -Example Output --------------- + scancode --cpp-includes /path/to/codebase/ --json-pp ~/path/to/scan-output.json -Here is an sample output:: +**Example** + +.. code-block:: none { "path": "zlib_deflate/deflate.c", diff --git a/docs/source/plugins/dwarf.rst b/docs/source/reference/scancode-plugins/dwarf-plugin.rst similarity index 74% rename from docs/source/plugins/dwarf.rst rename to docs/source/reference/scancode-plugins/dwarf-plugin.rst index c7b9d90afaa..637fddb8525 100644 --- a/docs/source/plugins/dwarf.rst +++ b/docs/source/reference/scancode-plugins/dwarf-plugin.rst @@ -1,6 +1,6 @@ -.. _dwarf_plugin: +.. _dwarf-plugin: -Dwarf Plugin +Dwarf plugin ============ This plugin allows users to collect source code path/name from compilation units found in @@ -11,19 +11,20 @@ Specification This plugin will only work with non-stripped ELFs with debug symbols. -Using the Plugin +Using the plugin ---------------- User needs to use the ``--dwarf`` option. -The following command will collect all the dwarf references found in non-stripped ELFs:: +The following command will collect all the dwarf references found in non-stripped ELFs - $ scancode --dwarf /path/to/codebase/ --json-pp ~/path/to/scan-output.json +.. code-block:: shell -Example Output --------------- + scancode --dwarf /path/to/codebase/ --json-pp ~/path/to/scan-output.json -Here is an sample output:: +**Example** + + .. code-block:: none { "path": "project/stripped.ELF", diff --git a/docs/source/reference/scancode-plugins/index.rst b/docs/source/reference/scancode-plugins/index.rst new file mode 100644 index 00000000000..5cbecbc5503 --- /dev/null +++ b/docs/source/reference/scancode-plugins/index.rst @@ -0,0 +1,13 @@ +.. _plugins: + +**ScanCode plugins** +==================== + +.. toctree:: + :maxdepth: 2 + + plugin-architecture + licence-policy-plugin + cpp-includes-plugin + lkmclue-plugin + dwarf-plugin diff --git a/docs/source/plugins/licence_policy_plugin.rst b/docs/source/reference/scancode-plugins/licence-policy-plugin.rst similarity index 55% rename from docs/source/plugins/licence_policy_plugin.rst rename to docs/source/reference/scancode-plugins/licence-policy-plugin.rst index a8d60280e9d..6dd3834d8eb 100644 --- a/docs/source/plugins/licence_policy_plugin.rst +++ b/docs/source/reference/scancode-plugins/licence-policy-plugin.rst @@ -1,16 +1,18 @@ -.. _license_policy_plugin: +.. _license-policy-plugin: -License Policy Plugin +License policy plugin ===================== This plugin allows the user to apply policy details to a scancode scan, depending on which -licenses are detected in a particular file. If a license specified in the Policy file is -detected by scancode, this plugin will apply that policy information to the Resource as a new +licenses are detected in a particular file. If a license specified in the policy file is +detected by scancode, this plugin will apply that policy information to the resource as a new attribute: ``license_policy``. -Policy File Specification +Policy file specification ------------------------- -The Policy file is a YAML (``.yml``) document with the following structure:: +The policy file is a YAML (``.yml``) document with the following structure + +.. code-block:: yaml license_policies: - license_key: mit @@ -32,40 +34,41 @@ against the detected licenses in the scan results. In the above example, a descriptive label is added along with a color code and CSS ``id`` name for potential visual display. -Using the Plugin +Using the plugin ---------------- -To apply License Policies during a ScanCode scan, specify the ``--license-policy`` option. +To apply license policies during a ScanCode scan, specify the ``--license-policy`` option. For example, use the following command to run a File Info and License scan on -``/path/to/codebase/``, using a License Policy file found at ``~/path/to/policy-file.yml``:: +``/path/to/codebase/``, using a License Policy file found at ``~/path/to/policy-file.yml`` + +.. code-block:: shell - $ scancode -clipeu /path/to/codebase/ --license-policy ~/path/to/policy-file.yml --json-pp + scancode -clipeu /path/to/codebase/ --license-policy ~/path/to/policy-file.yml --json-pp ~/path/to/scan-output.json -Example Output --------------- +**Example** -Here is an example of the ScanCode output after running ``--license-policy``:: +.. code-block:: none - { - "path": "samples/zlib/deflate.c", - "type": "file", - "detected_license_expression": "zlib", - "detected_license_expression_spdx": "Zlib", - "license_detections": [ - { - "license-expression": "zlib", - ... - ... - ... - } - ], - "license_policy": { - "license_key": "zlib", - "label": "Approved License", - "color_code": "#00800", - "icon": "icon-ok-circle" - }, - "scan_errors": [] + { + "path": "samples/zlib/deflate.c", + "type": "file", + "detected_license_expression": "zlib", + "detected_license_expression_spdx": "Zlib", + "license_detections": [ + { + "license-expression": "zlib", + ... + ... + ... + } + ], + "license_policy": { + "license_key": "zlib", + "label": "Approved License", + "color_code": "#00800", + "icon": "icon-ok-circle" + }, + "scan_errors": [] } diff --git a/docs/source/plugins/lkmclue.rst b/docs/source/reference/scancode-plugins/lkmclue-plugin.rst similarity index 77% rename from docs/source/plugins/lkmclue.rst rename to docs/source/reference/scancode-plugins/lkmclue-plugin.rst index 694bf05872b..35426e16ed2 100644 --- a/docs/source/plugins/lkmclue.rst +++ b/docs/source/reference/scancode-plugins/lkmclue-plugin.rst @@ -1,24 +1,25 @@ -.. _lkmclue_plugin: +.. _lkmclue-plugin: -LKMClue Plugin +LKMClue plugin ============== This plugin allows users to collect LKM module clues and type indicating a possible Linux Kernel Module. -Using the Plugin +Using the plugin ---------------- User needs to use the ``--lkmclue`` option. -The following command will collect the LKM module clues from the input location:: +The following command will collect the LKM module clues from the input location - $ scancode --lkmclue /path/to/codebase/ --json-pp ~/path/to/scan-output.json +.. code-block:: shell -Example Output --------------- + scancode --lkmclue /path/to/codebase/ --json-pp ~/path/to/scan-output.json -Here is an sample output:: +**Example** + + .. code-block:: none { "path": "zlib_deflate/deflate.c", diff --git a/docs/source/plugins/plugin_arch.rst b/docs/source/reference/scancode-plugins/plugin-architecture.rst similarity index 73% rename from docs/source/plugins/plugin_arch.rst rename to docs/source/reference/scancode-plugins/plugin-architecture.rst index be075774a0b..db299a89673 100644 --- a/docs/source/plugins/plugin_arch.rst +++ b/docs/source/reference/scancode-plugins/plugin-architecture.rst @@ -1,60 +1,42 @@ -.. _plugin_arch: +.. _plugin-architecture: -Plugin Architecture +Plugin architecture =================== -Abstract: ---------- - The purpose of plugins is to create a decoupled architecture such that ScanCode can support extensibility at different stages of a scan. These stages are: -* Pre-scan: Before starting the scan proper, such as plugins to handle +* **Pre-scan**: Before starting the scan proper, such as plugins to handle extraction of different archive types or instructions on how to handle certain types of files, or to collect filetypes. These plugins process a whole codebase at once. -* Scan proper: plugins to scan a file e.g. collect data and evidece from the +* **Scan proper**: plugins to scan a file e.g. collect data and evidece from the files. These plugins process one file at a teim and can do a whole codebase pass once all files are scanned. -* Post-scan: After the scan, e.g plugins for summarization and other aggregated +* **Post-scan**: After the scan, e.g plugins for summarization and other aggregated operation once all scans are completed. These plugins process a whole codebase at once. -* Output and output filter: plugins for output creation and filtering such as +* **Output and output filter**: plugins for output creation and filtering such as formatting or converting output to other formats (such as json, spdx, csv, yaml). These plugins process a whole codebase at once. -Description: ------------- - -This project aims at making scancode a “pluggable” system, where new -functionalities can be added to scancode at runtime as “plugins”. These plugins -can be hooked into scancode using some predefined hooks. I would consider pluggy -as the way to go for a plugin management system. - -Why pluggy? -^^^^^^^^^^^ - -Pluggy is well documented and maintained regularly, and has proved its worth in -projects such as pytest. Pluggy relies on hook specifications and hook -implementations (callbacks) instead of the conventional subclassing approach -which may encourage tight-coupling in the overlying framework. Basically a hook -specification contains method signatures (no code), these are defined by the +Plugins in ScanCode are implemented using `Pluggy `_. +Pluggy relies on hook specifications and hook implementations (callbacks) instead of +the conventional subclassing approach which may encourage tight-coupling in the overlying framework. +Basically a hook specification contains method signatures (no code), these are defined by the application. A hook implementation contains definitions for methods declared in the corresponding hook specification implemented by a plugin. -As mentioned in the abstract, the plugin architecture will have 3 hook -specifications (can be increased if required) +As mentioned in the above, the plugin architecture has 3 hooks: -1. Pre - scan hook -^^^^^^^^^^^^^^^^^^ +Pre-scan hook +------------- -- **Structure** - - -:: +.. code-block:: python prescan_hookspec = HookspecMarker('prescan') @@ -66,13 +48,10 @@ function which will be called before scan, at the time of extraction. This will type and extract the contents accordingly. This functionality can be further extended by calling this function if any archive is found inside the scanning tree. -2. Scan proper hook -^^^^^^^^^^^^^^^^^^^ - +Scan proper hook +---------------- -- **Structure** - -:: +.. code-block:: python scanproper_hookspec = HookspecMarker('scanproper') @@ -83,7 +62,9 @@ This function will be called before starting the scan, without any arguments, it containing the click extension details and possibly some help text. If this option is called by the user then the call will be rerouted to the callback defined by the click extension. For instance say a plugin implements functionality to add regex as a valid ignore pattern, then this function -will return a dict as:: +will return a dict as + +.. code-block:: none { 'name': '--ignore-regex', @@ -98,14 +79,16 @@ will return a dict as:: According to the above dict, if the option --ignore-regex is supplied, this function will be called after the is_ignored function and the data returned by the is_ignored function will be supplied to -this function as its argument(s). So if the program flow was:: +this function as its argument(s). So if the program flow was + +.. code-block:: none scancode() ⇔ scan() ⇔ resource_paths() ⇔ is_ignored() It will now be edited to -:: +.. code-block:: none scancode() ⇔ scan() ⇔ resource_paths() ⇔ is_ignored() ⇔ add_cmdline_option() @@ -113,7 +96,7 @@ It will now be edited to Options such as **call_after, call_before, call_first, call_last** can be defined to determine when the function is to be executed. -:: +.. code-block:: python @scanproper_hookspec def dependency_scan(args): @@ -122,20 +105,19 @@ This function will be called before starting the scan without any arguments, it list of file types or attributes which if encountered in the scanned tree, will call this function with the path to the file as an argument. This function can do some extra processing on those files and return the data to be processed as a dependency for the normal scanning process. -E.g. It can return a list such as:: +E.g. It can return a list such as + +.. code-block:: none [ 'debian/copyright' ] Whenever a file matches this pattern, this function will be called and the data returned will be supplied to the main scancode function. -3. Post - scan hook -^^^^^^^^^^^^^^^^^^^ - - -- **Structure** - +Post-scan hook +-------------- -:: +.. code-block:: python postscan_hookspec = HookspecMarker('postscan') @@ -147,7 +129,7 @@ data generated from the scan, path to the root of the scanned code and a path wh expected to be stored. The function will store the processed data in the output path supplied. This can be used to convert output to other formats such as CSV, SPDX, JSON, etc. -:: +.. code-block:: python @postscan_hookspec def summarize_output(args): @@ -159,12 +141,13 @@ reported or omit similar file metadata or even classify files such as tests, cod readme, configs, build scripts etc. -- **Identifying or configuring plugins** +Identifying or configuring plugins +---------------------------------- For python plugins, pluggy supports loading modules from setuptools entrypoints, E.g. -:: +.. code-block:: none entry_points = { 'scancode_plugins': [ @@ -180,12 +163,14 @@ plugins will have a manifest configuration in YAML format. This directory will b startup for plugins. After parsing the config file of a plugin, the data will be supplied to the plugin manager as if it were supplied using setuptools entrypoints. -In case of non python plugins, the plugin executables will be spawned in their own processes and +In case of non-python plugins, the plugin executables will be spawned in their own processes and according to their config data, they will be passed arguments and would return data as necessary. In addition to this, the desired hook function can be called from a non python plugin using certain arguments, which again can be mapped in the config file. -Sample config file for a ignore_regex plugin calling scanproper hook would be:: +Below an example config file for a ``ignore_regex`` plugin calling scanproper hook. + +.. code-block:: yaml name: ignore_regex hook: scanproper @@ -201,10 +186,3 @@ Sample config file for a ignore_regex plugin calling scanproper hook would be:: - metavar: - help: 'Ignore files matching regex ' - call_after: 'is_ignored' - -Existing solutions: -------------------- - -An alternate solution to a “pluggable” system would be the more conventional approach of adding -functionalities directly to the core codebase, which removes the abstraction layer provided by -a plugin management and hook calling system. diff --git a/docs/source/reference/available_package_parsers.rst b/docs/source/reference/scancode-supported-packages.rst similarity index 99% rename from docs/source/reference/available_package_parsers.rst rename to docs/source/reference/scancode-supported-packages.rst index 3fe2cbe706a..d2ad15803f4 100644 --- a/docs/source/reference/available_package_parsers.rst +++ b/docs/source/reference/scancode-supported-packages.rst @@ -1,6 +1,4 @@ - - -.. _supported_packages: +.. _supported-packages: Supported package manifests and package datafiles ------------------------------------------------- diff --git a/docs/source/rst_snippets/basic_options.rst b/docs/source/rst-snippets/cli-basic-options.rst similarity index 66% rename from docs/source/rst_snippets/basic_options.rst rename to docs/source/rst-snippets/cli-basic-options.rst index d01fbf72a6c..ed6777ea9be 100644 --- a/docs/source/rst_snippets/basic_options.rst +++ b/docs/source/rst-snippets/cli-basic-options.rst @@ -1,18 +1,18 @@ -All "Basic" Scan Options ------------------------- +**Basic options** +----------------- Option lists are two-column lists of command-line options and descriptions, documenting a program's options. For example: -c, --copyright Scan ```` for copyrights. - Sub-Options: + Sub-options: - ``--consolidate`` -l, --license Scan ```` for licenses. - Sub-Options: + Sub-options: - ``--license-references`` - ``--license-text`` @@ -26,26 +26,28 @@ documenting a program's options. For example: -p, --package Scan ```` for packages. - Sub-Options: + Sub-options: - ``--consolidate`` --system-package Scan ```` for installed system package databases. ---package-only Scan ```` for system and application - only for package metadata, without license/ - copyright detection and package assembly. +--package-only Faster package scan, scanning ```` for + system and application packages, only for package + metadata. This option is skipping + - license and copyright detection for package metadata + - package assembly -e, --email Scan ```` for emails. - Sub-Options: + Sub-options: - ``--max-email INT`` -u, --url Scan ```` for urls. - Sub-Options: + Sub-options: - ``--max-url INT`` @@ -59,40 +61,47 @@ documenting a program's options. For example: - binary/text/archive/media/source/script flags - Additional options through more CLI options - Sub-Options: + Sub-options: - ``--mark-source`` -.. include:: /rst_snippets/note_snippets/basic_clpieu.rst +.. include:: /rst-snippets/note-snippets/cli-basic-options-clpieu.rst --generated Classify automatically generated code files with a flag. --max-email INT Report only up to INT emails found in a - file. Use 0 for no limit. [Default: 50] + file. Use 0 for no limit. - Sub-Option of: ``--email`` + Default: ``50`` + + Sub-option of: ``--email`` --max-url INT Report only up to INT urls found in a - file. Use 0 for no limit. [Default: 50] + file. Use 0 for no limit. + + Default: ``50`` - Sub-Option of: ``--url`` + Sub-option of: ``--url`` --license-score INTEGER Do not return license matches with scores lower than this score. - A number between 0 and 100. [Default: 0] + A number between 0 and 100. + + Default: ``0`` (i.e. we return all license matches by default). + Here, a bigger number means a better match, i.e. Setting a higher license score translates to a higher threshold (with equal or smaller number of matches). - Sub-Option of: ``--license`` + Sub-option of: ``--license`` --license-text Include the matched text for the detected licenses in the output report. - Sub-Option of: ``--license`` + Sub-option of: ``--license`` - Sub-Options: + Sub-options: - ``--license-text-diagnostics`` @@ -101,26 +110,27 @@ documenting a program's options. For example: Set the template URL used for the license reference URLs. In a template URL, curly braces ({}) are replaced by the license key. - [Default: default: https://scancode-licensedb.aboutcode.org/{}] - Sub-Option of: ``--license`` + Default: ``https://scancode-licensedb.aboutcode.org/{}`` + + Sub-option of: ``--license`` --license-text-diagnostics In the matched license text, include diagnostic highlights surrounding with square brackets [] words that are not matched. - Sub-Option of: ``--license`` and ``--license-text`` + Sub-option of: ``--license`` and ``--license-text`` --license-diagnostics In license detections, include diagnostic details to figure out the license detection post processing steps applied. - Sub-Option of: ``--license`` + Sub-option of: ``--license`` --unknown-licenses [EXPERIMENTAL] Detect unknown licenses. - Sub-Option of: ``--license`` + Sub-option of: ``--license`` diff --git a/docs/source/rst_snippets/core_options.rst b/docs/source/rst-snippets/cli-core-options.rst similarity index 65% rename from docs/source/rst_snippets/core_options.rst rename to docs/source/rst-snippets/cli-core-options.rst index 453763ca57d..8bf86b167af 100644 --- a/docs/source/rst_snippets/core_options.rst +++ b/docs/source/rst-snippets/cli-core-options.rst @@ -1,17 +1,24 @@ -All "Core" Scan Options ------------------------ +**Core options** +---------------- -n, --processes INTEGER Scan ```` using n parallel processes. - [Default: (number of CPUs)-1] + + Default: ``(number of CPUs)-1`` -v, --verbose Print verbose file-by-file progress messages. -q, --quiet Do not print summary or progress messages. --timeout FLOAT Stop scanning a file if scanning takes longer - than a timeout in seconds. [Default: 120] + than a timeout in seconds. + + Default: ``120`` + +--from-json Load codebase from one or more existing JSON scans to: ---from-json Load codebase from one or more existing JSON scans. + - apply post-scan options to do additional processing + of scan results + - merge multiple JSON scans into one. --max-in-memory INTEGER Maximum number of files and directories scan details kept in memory during a scan. @@ -19,9 +26,12 @@ All "Core" Scan Options above this number are cached on-disk rather than in memory. Use 0 to use unlimited memory and disable on-disk caching. Use -1 to use - only on-disk caching. [Default: 10000] + only on-disk caching. + + Default: ``10000`` --max-depth INTEGER Descend at most INTEGER levels of directories including and below the starting point. INTEGER must be positive or zero for no limit. - [Default: 0] + + Default: ``0`` diff --git a/docs/source/rst_snippets/doc_help.rst b/docs/source/rst-snippets/cli-doc-help-options.rst similarity index 71% rename from docs/source/rst_snippets/doc_help.rst rename to docs/source/rst-snippets/cli-doc-help-options.rst index 799fe019cb7..39bf6b4b494 100644 --- a/docs/source/rst_snippets/doc_help.rst +++ b/docs/source/rst-snippets/cli-doc-help-options.rst @@ -1,9 +1,9 @@ -All Documentation/Help Options +**Documentation/help options** ------------------------------ --h, --help Show the :ref:`cli_help_text` and exit. +-h, --help Show the :ref:`cli-help-option` and exit. ---examples Show the :ref:`cli_examples_text` and exit. +--examples Show the :ref:`cli-examples-option` and exit. -A, --about Show information about ScanCode and licensing and exit. diff --git a/docs/source/rst-snippets/cli-other-commands.rst b/docs/source/rst-snippets/cli-other-commands.rst new file mode 100644 index 00000000000..0250299420f --- /dev/null +++ b/docs/source/rst-snippets/cli-other-commands.rst @@ -0,0 +1,10 @@ +**Other CLI commands** +---------------------- + +-n, --processes INTEGER Scan ```` using n parallel processes. + + Default: ``(number of CPUs)-1`` + +-v, --verbose Print verbose file-by-file progress messages. + +-q, --quiet Do not print summary or progress messages. diff --git a/docs/source/rst_snippets/output_control_options.rst b/docs/source/rst-snippets/cli-output-control-and-filter-options.rst similarity index 78% rename from docs/source/rst_snippets/output_control_options.rst rename to docs/source/rst-snippets/cli-output-control-and-filter-options.rst index 0fd3b833f09..62097737c59 100644 --- a/docs/source/rst_snippets/output_control_options.rst +++ b/docs/source/rst-snippets/cli-output-control-and-filter-options.rst @@ -1,11 +1,11 @@ -All "Output Control" Scan Options ---------------------------------- +**Controlling output and filters** +---------------------------------- --strip-root Strip the root directory segment of all paths. --full-root Report full, absolute paths. -.. include:: /rst_snippets/note_snippets/control_strip_full_root.rst +.. include:: /rst-snippets/note-snippets/cli-output-control-strip-full-root.rst --ignore-author Ignore a file (and all its findings) if an author contains a match to the @@ -16,7 +16,7 @@ All "Output Control" Scan Options if a copyright holder contains a match to the ```` regular expression. -.. include:: /rst_snippets/warning_snippets/control_ignore_author_copyright.rst +.. include:: /rst-snippets/warning-snippets/cli-output-control-ignore-author-copyright.rst --only-findings Only return files or directories with findings for the requested scans. diff --git a/docs/source/rst_snippets/custom_output_format.rst b/docs/source/rst-snippets/cli-output-custom-format.rst similarity index 69% rename from docs/source/rst_snippets/custom_output_format.rst rename to docs/source/rst-snippets/cli-output-custom-format.rst index e947a5995f0..0bc1ee9c55f 100644 --- a/docs/source/rst_snippets/custom_output_format.rst +++ b/docs/source/rst-snippets/cli-output-custom-format.rst @@ -1,10 +1,12 @@ -Custom Output Format --------------------- +"custom output" options +----------------------- While the three built-in output formats are convenient for a verity of use-cases, one may wish to -create their own output template, using the following arguments:: +create their own output template, using the following arguments - ``--custom-output FILE --custom-template TEMP_FILE`` + .. code-block:: shell + + scancode --custom-output FILE --custom-template TEMP_FILE ScanCode makes this very easy, as it uses the popular Jinja2 template engine. Simply pass the path to the custom template to the ``--custom-template`` argument, or drop it in a folder to @@ -14,7 +16,7 @@ For example, if I wanted a simple CLI output I would create a ``template2.html`` particular data I wish to see. In this case, I am only interested in the license and copyright data for this particular scan. -:: +.. code-block:: text ## template.txt: [ @@ -34,24 +36,24 @@ data for this particular scan. Now I can run ScanCode using my newly created template: -:: +.. code-block:: shell - $ scancode -clpeui --custom-output output.txt --custom-template template.txt samples - Scanning files... - [####################################] 46 - Scanning done. + scancode -clpeui --custom-output output.txt --custom-template template.txt samples + Scanning files... + [####################################] 46 + Scanning done. Now the results are saved in ``output.txt`` and we can easily view them with ``head output.txt``: -:: +.. code-block:: none - [ - location:"samples/JGroups/LICENSE", - copyright:"Copyright (c) 1991, 1999 Free Software Foundation, Inc.", + [ + location:"samples/JGroups/LICENSE", + copyright:"Copyright (c) 1991, 1999 Free Software Foundation, Inc.", - location:"samples/JGroups/LICENSE", - copyright:"copyrighted by the Free Software Foundation", - ] + location:"samples/JGroups/LICENSE", + copyright:"copyrighted by the Free Software Foundation", + ] For a more elaborate template, refer this `default template `_ given with ScanCode, to generate HTML output with the ``--html`` output format option. diff --git a/docs/source/rst_snippets/output_format_options.rst b/docs/source/rst-snippets/cli-output-format-options.rst similarity index 90% rename from docs/source/rst_snippets/output_format_options.rst rename to docs/source/rst-snippets/cli-output-format-options.rst index 859a7608f60..b73f8cfbeed 100644 --- a/docs/source/rst_snippets/output_format_options.rst +++ b/docs/source/rst-snippets/cli-output-format-options.rst @@ -1,5 +1,5 @@ -All Scan Output Options ------------------------ +**Output formats** +------------------ --json FILE Write scan output as compact JSON to FILE. @@ -27,14 +27,14 @@ All Scan Output Options --custom-output Write scan output to FILE formatted with the custom Jinja template file. - Mandatory Sub-option: + Mandatory sub-option: - ``--custom-template FILE`` --custom-template FILE Use this Jinja template FILE as a custom template. - Sub-Option of: ``--custom-output`` + Sub-option of: ``--custom-output`` --debian FILE Write scan output in machine-readable Debian copyright format to FILE. @@ -53,4 +53,4 @@ All Scan Output Options --cyclonedx-xml FILE Write scan output as a CycloneDx 1.3 BOM in pretty-printed XML format to FILE -.. include:: /rst_snippets/warning_snippets/output_htmlapp_dep.rst +.. include:: /rst-snippets/warning-snippets/cli-output-htmlapp-deprecated.rst diff --git a/docs/source/rst_snippets/stdout.rst b/docs/source/rst-snippets/cli-output-to-stdout.rst similarity index 82% rename from docs/source/rst_snippets/stdout.rst rename to docs/source/rst-snippets/cli-output-to-stdout.rst index 2919ce2858f..778e63f3cc5 100644 --- a/docs/source/rst_snippets/stdout.rst +++ b/docs/source/rst-snippets/cli-output-to-stdout.rst @@ -1,9 +1,9 @@ -Print to ``stdout`` (Terminal) +Print to ``stdout`` / terminal ------------------------------ If you want to format the output in JSON and print it at stdout, you can replace the JSON filename with a "-", like ``--json-pp -`` instead of ``--json-pp output.json``. -The following command will output the scan results in JSON format to ``stdout`` (In the Terminal):: +The following command will output the scan results in JSON format to ``stdout`` (in the terminal):: ./scancode -clpieu --json-pp - samples/ diff --git a/docs/source/rst-snippets/cli-post-scan-options.rst b/docs/source/rst-snippets/cli-post-scan-options.rst new file mode 100644 index 00000000000..a38d90366cd --- /dev/null +++ b/docs/source/rst-snippets/cli-post-scan-options.rst @@ -0,0 +1,88 @@ +**Post-scan options** +--------------------- + +--mark-source Set the "is_source" flag to true for directories that + contain over 90% of source files as direct children + and descendants. Count the number of source files in a + directory as a new "source_file_counts" attribute + + Sub-option of: ``--url`` + +--consolidate Group resources by Packages or license and + copyright holder and return those groupings + as a list of consolidated packages and a list + of consolidated components. + The --consolidate option will be deprecated in + a future version of scancode-toolkit as top level + packages now provide improved consolidated data. + + Sub-option of: ``--copyright``, ``--license`` and + ``--packages``. + +--filter-clues Filter redundant duplicated clues already + contained in detected licenses, copyright + texts and notices. + +--license-clarity-score Compute a summary license clarity score at + the codebase level. + + Sub-option of: ``--classify``. + +--license-policy FILE Load a License Policy file and apply it to + the scan at the Resource level. + +--license-references FILE Returns reference data for all licenses and license + rules present in detections. + +--classify Classify files with flags telling if the + file is a legal, or readme or test file, + etc. + + Sub-options: + + - ``--license-clarity-score`` + - ``--tallies-key-files`` + - ``--summry`` + +--summary Summarize scans by providing declared origin + information and other detected info at the + codebase attribute level. + + Sub-option of: + + - ``--classify`` + +--todo Identify license and package detections which needs + review as there are potential issues with the detections. + Lists all the unique ambigious detections with references + to the file path and line numbers, detection details and + review comments to help explain the detection issue. + +--tallies Summarize license, copyright and other scans + at the codebase level with occurrence counts. + + Sub-options: + + - ``--tallies-by-facet`` + - ``--tallies-key-files`` + - ``--tallies-with-details`` + +--tallies-by-facet Summarize license, copyright and other scans + and group the results by facet. + + Sub-option of: ``--tallies`` and ``--facet``. + +--tallies-key-files Summarize license, copyright and other scans + for key, top-level files, with occurrence counts. + Key files are top-level codebase files such as + COPYING, README and package manifests as reported + by the ``--classify`` option: "is_legal", + "is_readme", "is_manifest" and "is_top_level" + flags. + + Sub-option of: ``--classify`` and ``--summary``. + +--tallies-with-details Summarize license, copyright and other scans + at the codebase level with occurrence counts, + while also keeping intermediate details at + the file and directory level. diff --git a/docs/source/rst-snippets/cli-pre-scan-options.rst b/docs/source/rst-snippets/cli-pre-scan-options.rst new file mode 100644 index 00000000000..b880e92abc3 --- /dev/null +++ b/docs/source/rst-snippets/cli-pre-scan-options.rst @@ -0,0 +1,14 @@ +**Pre-scan options** +-------------------- + +--ignore Ignore files matching ````. + +--include Include files matching ````. + +--facet Here ```` represents + ``=``. Add the ```` + to files with a path matching ````. + + Sub-options: + + - ``--tallies-by-facet`` diff --git a/docs/source/rst_snippets/improve_docs.rst b/docs/source/rst-snippets/improve-docs.rst similarity index 99% rename from docs/source/rst_snippets/improve_docs.rst rename to docs/source/rst-snippets/improve-docs.rst index 43e524ee543..320ed8a817f 100644 --- a/docs/source/rst_snippets/improve_docs.rst +++ b/docs/source/rst-snippets/improve-docs.rst @@ -4,4 +4,5 @@ Something Missing? If something is missing in the documentation or if you found some part confusing, please file an `issue `_ with your suggestions for improvement. Use the "Documentation Improvement" template. + Your help makes ScanCode docs better, we love hearing from you! diff --git a/docs/source/rst_snippets/note_snippets/additional_directory_is_temp.rst b/docs/source/rst-snippets/note-snippets/add-licenses-or-rules-from-additional-directory-is-temporary.rst similarity index 59% rename from docs/source/rst_snippets/note_snippets/additional_directory_is_temp.rst rename to docs/source/rst-snippets/note-snippets/add-licenses-or-rules-from-additional-directory-is-temporary.rst index 507eccb5821..87affeca2a3 100644 --- a/docs/source/rst_snippets/note_snippets/additional_directory_is_temp.rst +++ b/docs/source/rst-snippets/note-snippets/add-licenses-or-rules-from-additional-directory-is-temporary.rst @@ -2,5 +2,5 @@ Adding licenses/rules from an additional directory is not permanent. Another reindexing without the additional directory option would - just use the builtin scancode licenses and rules, and will not have - these additonal licenses/rules anymore. + just use the builtin ScanCode licenses and rules, and will not have + these additional licenses/rules anymore. diff --git a/docs/source/rst_snippets/note_snippets/basic_clpieu.rst b/docs/source/rst-snippets/note-snippets/cli-basic-options-clpieu.rst similarity index 100% rename from docs/source/rst_snippets/note_snippets/basic_clpieu.rst rename to docs/source/rst-snippets/note-snippets/cli-basic-options-clpieu.rst diff --git a/docs/source/rst_snippets/note_snippets/control_strip_full_root.rst b/docs/source/rst-snippets/note-snippets/cli-output-control-strip-full-root.rst similarity index 100% rename from docs/source/rst_snippets/note_snippets/control_strip_full_root.rst rename to docs/source/rst-snippets/note-snippets/cli-output-control-strip-full-root.rst diff --git a/docs/source/rst_snippets/note_snippets/synopsis_output_format.rst b/docs/source/rst-snippets/note-snippets/cli-output-format-synopsis.rst similarity index 100% rename from docs/source/rst_snippets/note_snippets/synopsis_output_format.rst rename to docs/source/rst-snippets/note-snippets/cli-output-format-synopsis.rst diff --git a/docs/source/rst_snippets/note_snippets/output_html_license_references.rst b/docs/source/rst-snippets/note-snippets/cli-output-html-license-references.rst similarity index 100% rename from docs/source/rst_snippets/note_snippets/output_html_license_references.rst rename to docs/source/rst-snippets/note-snippets/cli-output-html-license-references.rst diff --git a/docs/source/rst_snippets/note_snippets/output_htmlapp_search.rst b/docs/source/rst-snippets/note-snippets/cli-output-htmlapp-search.rst similarity index 100% rename from docs/source/rst_snippets/note_snippets/output_htmlapp_search.rst rename to docs/source/rst-snippets/note-snippets/cli-output-htmlapp-search.rst diff --git a/docs/source/rst_snippets/note_snippets/output_jsonlines.rst b/docs/source/rst-snippets/note-snippets/cli-output-json-lines.rst similarity index 100% rename from docs/source/rst_snippets/note_snippets/output_jsonlines.rst rename to docs/source/rst-snippets/note-snippets/cli-output-json-lines.rst diff --git a/docs/source/rst_snippets/note_snippets/output_json_ugly.rst b/docs/source/rst-snippets/note-snippets/cli-output-json-ugly.rst similarity index 100% rename from docs/source/rst_snippets/note_snippets/output_json_ugly.rst rename to docs/source/rst-snippets/note-snippets/cli-output-json-ugly.rst diff --git a/docs/source/rst_snippets/note_snippets/output_samples.rst b/docs/source/rst-snippets/note-snippets/cli-output-samples.rst similarity index 100% rename from docs/source/rst_snippets/note_snippets/output_samples.rst rename to docs/source/rst-snippets/note-snippets/cli-output-samples.rst diff --git a/docs/source/rst_snippets/note_snippets/post_license_clarity_score.rst b/docs/source/rst-snippets/note-snippets/cli-post-scan-license-clarity-score.rst similarity index 100% rename from docs/source/rst_snippets/note_snippets/post_license_clarity_score.rst rename to docs/source/rst-snippets/note-snippets/cli-post-scan-license-clarity-score.rst diff --git a/docs/source/rst_snippets/note_snippets/post_lic_pol_key.rst b/docs/source/rst-snippets/note-snippets/cli-post-scan-license-file-required-policy-key.rst similarity index 100% rename from docs/source/rst_snippets/note_snippets/post_lic_pol_key.rst rename to docs/source/rst-snippets/note-snippets/cli-post-scan-license-file-required-policy-key.rst diff --git a/docs/source/rst_snippets/note_snippets/post_license_policy.rst b/docs/source/rst-snippets/note-snippets/cli-post-scan-license-policy.rst similarity index 100% rename from docs/source/rst_snippets/note_snippets/post_license_policy.rst rename to docs/source/rst-snippets/note-snippets/cli-post-scan-license-policy.rst diff --git a/docs/source/rst_snippets/note_snippets/post_scan_plugins.rst b/docs/source/rst-snippets/note-snippets/cli-post-scan-plugins.rst similarity index 100% rename from docs/source/rst_snippets/note_snippets/post_scan_plugins.rst rename to docs/source/rst-snippets/note-snippets/cli-post-scan-plugins.rst diff --git a/docs/source/rst_snippets/note_snippets/post_summary_details.rst b/docs/source/rst-snippets/note-snippets/cli-post-scan-summary-details.rst similarity index 100% rename from docs/source/rst_snippets/note_snippets/post_summary_details.rst rename to docs/source/rst-snippets/note-snippets/cli-post-scan-summary-details.rst diff --git a/docs/source/rst_snippets/note_snippets/post_summary_facet.rst b/docs/source/rst-snippets/note-snippets/cli-post-scan-summary-facet.rst similarity index 100% rename from docs/source/rst_snippets/note_snippets/post_summary_facet.rst rename to docs/source/rst-snippets/note-snippets/cli-post-scan-summary-facet.rst diff --git a/docs/source/rst_snippets/note_snippets/pre_facet_core.rst b/docs/source/rst-snippets/note-snippets/cli-pre-scan-facet-core.rst similarity index 100% rename from docs/source/rst_snippets/note_snippets/pre_facet_core.rst rename to docs/source/rst-snippets/note-snippets/cli-pre-scan-facet-core.rst diff --git a/docs/source/rst_snippets/note_snippets/license_plugin_delete.rst b/docs/source/rst-snippets/note-snippets/license-plugin-delete.rst similarity index 100% rename from docs/source/rst_snippets/note_snippets/license_plugin_delete.rst rename to docs/source/rst-snippets/note-snippets/license-plugin-delete.rst diff --git a/docs/source/rst_snippets/note_snippets/license_plugin_needs_reindex.rst b/docs/source/rst-snippets/note-snippets/license-plugin-needs-reindex.rst similarity index 100% rename from docs/source/rst_snippets/note_snippets/license_plugin_needs_reindex.rst rename to docs/source/rst-snippets/note-snippets/license-plugin-needs-reindex.rst diff --git a/docs/source/rst_snippets/warning_snippets/control_ignore_author_copyright.rst b/docs/source/rst-snippets/warning-snippets/cli-output-control-ignore-author-copyright.rst similarity index 100% rename from docs/source/rst_snippets/warning_snippets/control_ignore_author_copyright.rst rename to docs/source/rst-snippets/warning-snippets/cli-output-control-ignore-author-copyright.rst diff --git a/docs/source/rst_snippets/warning_snippets/output_htmlapp_dep.rst b/docs/source/rst-snippets/warning-snippets/cli-output-htmlapp-deprecated.rst similarity index 80% rename from docs/source/rst_snippets/warning_snippets/output_htmlapp_dep.rst rename to docs/source/rst-snippets/warning-snippets/cli-output-htmlapp-deprecated.rst index 5fe1a47b083..abd67afbf19 100644 --- a/docs/source/rst_snippets/warning_snippets/output_htmlapp_dep.rst +++ b/docs/source/rst-snippets/warning-snippets/cli-output-htmlapp-deprecated.rst @@ -2,4 +2,4 @@ The html-app feature has been deprecated and you should use ScanCode Workbench instead to visualize scan results. The official Repository `link `_. - Also refer :ref:`how_to_visualize_scan_results`. + Also refer :ref:`visualizing-scan-results`. diff --git a/docs/source/rst_snippets/extract.rst b/docs/source/rst_snippets/extract.rst deleted file mode 100644 index 59d3b337ab8..00000000000 --- a/docs/source/rst_snippets/extract.rst +++ /dev/null @@ -1,31 +0,0 @@ -All Extractcode Options ------------------------ - -This is intended to be used as an input preparation step, before running the scan. Archives found -in an extracted archive are extracted **recursively** by default. Extraction is done in-place -in a directory named '-extract' side-by-side with an archive. - -To extract the packages in the ``samples`` directory - -:: - - extractcode samples - -This extracts the zlib.tar.gz package: - -.. image:: /rst_snippets/data/extractcode.png - -.. _cli_extract: - ---shallow Do not extract recursively nested archives (e.g. Not - archives in archives). - ---verbose Print verbose file-by-file progress messages. - ---quiet Do not print any summary or progress message. - --h, --help Show the extractcode help message and exit. - ---about Show information about ScanCode and licensing and exit. - ---version Show the version and exit. diff --git a/docs/source/rst_snippets/other_imp_doc.rst b/docs/source/rst_snippets/other_imp_doc.rst deleted file mode 100644 index abb712cecb7..00000000000 --- a/docs/source/rst_snippets/other_imp_doc.rst +++ /dev/null @@ -1,13 +0,0 @@ -Other Important Documentation ------------------------------ - -#. :ref:`scancode_cli_options` -#. :ref:`how_to_run_a_scan` -#. :ref:`tutorials` -#. :ref:`how_to_guides` -#. :ref:`how_it_works` -#. :ref:`contrib_code_dev` -#. :ref:`contrib_doc_dev` -#. :ref:`plugin_arch` -#. :ref:`faq` -#. :ref:`support` diff --git a/docs/source/rst_snippets/post_scan_options.rst b/docs/source/rst_snippets/post_scan_options.rst deleted file mode 100644 index 865deb23523..00000000000 --- a/docs/source/rst_snippets/post_scan_options.rst +++ /dev/null @@ -1,65 +0,0 @@ -All "Post-Scan" Options ------------------------ - ---mark-source Set the "is_source" flag to true for directories that - contain over 90% of source files as direct children - and descendants. Count the number of source files in a - directory as a new "source_file_counts" attribute - - Sub-Option of: ``--url`` - ---consolidate Group resources by Packages or license and - copyright holder and return those groupings - as a list of consolidated packages and a list - of consolidated components. - The --consolidate option will be deprecated in - a future version of scancode-toolkit as top level - packages now provide improved consolidated data. - - Sub-Option of: ``--copyright``, ``--license`` and - ``--packages``. - ---filter-clues Filter redundant duplicated clues already - contained in detected licenses, copyright - texts and notices. - ---license-clarity-score Compute a summary license clarity score at - the codebase level. - - Sub-Option of: ``--classify``. - ---license-policy FILE Load a License Policy file and apply it to - the scan at the Resource level. - ---summary Summarize scans by providing declared origin - information and other detected info at the - codebase attribute level. - ---tallies Summarize license, copyright and other scans - at the codebase level with occurrence counts. - - Sub-Options: - - - ``--tallies-by-facet`` - - ``--tallies-key-files`` - - ``--tallies-with-details`` - ---tallies-by-facet Summarize license, copyright and other scans - and group the results by facet. - - Sub-Option of: ``--tallies`` and ``--facet``. - ---tallies-key-files Summarize license, copyright and other scans - for key, top-level files, with occurrence counts. - Key files are top-level codebase files such as - COPYING, README and package manifests as reported - by the ``--classify`` option: "is_legal", - "is_readme", "is_manifest" and "is_top_level" - flags. - - Sub-Option of: ``--classify`` and ``--summary``. - ---tallies-with-details Summarize license, copyright and other scans - at the codebase level with occurrence counts, - while also keeping intermediate details at - the file and directory level. diff --git a/docs/source/rst_snippets/pre_scan_options.rst b/docs/source/rst_snippets/pre_scan_options.rst deleted file mode 100644 index 2adecf10005..00000000000 --- a/docs/source/rst_snippets/pre_scan_options.rst +++ /dev/null @@ -1,23 +0,0 @@ -All "Pre-Scan" Options ----------------------- - ---ignore Ignore files matching ````. - ---include Include files matching ````. - ---classify Classify files with flags telling if the - file is a legal, or readme or test file, - etc. - - Sub-Options: - - - ``--license-clarity-score`` - - ``--tallies-key-files`` - ---facet Here ```` represents - ``=``. Add the ```` - to files with a path matching ````. - - Sub-Options: - - - ``--tallies-by-facet`` diff --git a/docs/source/rst_snippets/scancode-reindex-licenses.rst b/docs/source/rst_snippets/scancode-reindex-licenses.rst deleted file mode 100644 index 83304762c21..00000000000 --- a/docs/source/rst_snippets/scancode-reindex-licenses.rst +++ /dev/null @@ -1,23 +0,0 @@ -``scancode-reindex-licenses`` Usage ------------------------------------ - -Usage: ``scancode-reindex-licenses [OPTIONS]`` - -Reindex scancode licenses and exit - -Options -------- - - --all-languages [EXPERIMENTAL] Rebuild the license index - including texts all languages (and not only - English) and exit. - --only-builtin Rebuild the license index excluding any - additional license directory or additional - license plugins which were added previously, i.e. - with only builtin scancode license and rules. - --additional-directory DIR Include this directory with additional custom - licenses and license rules in the license - detection index. - --load-dump Load all license and rules from their respective - files and then dump them back to those same files. - -h, --help Shows the options and explanations. diff --git a/docs/source/rst_snippets/tip_snippets/stdout.rst b/docs/source/rst_snippets/tip_snippets/stdout.rst deleted file mode 100644 index a600e5ebe45..00000000000 --- a/docs/source/rst_snippets/tip_snippets/stdout.rst +++ /dev/null @@ -1,4 +0,0 @@ -.. Tip:: - - You can also output to ``stdout`` instead of a file. For more information refer - :ref:`output_to_stdout`. diff --git a/docs/source/tutorials/how_to_add_a_post_scan_plugin.rst b/docs/source/tutorials/adding-a-post-scan-plugin.rst similarity index 98% rename from docs/source/tutorials/how_to_add_a_post_scan_plugin.rst rename to docs/source/tutorials/adding-a-post-scan-plugin.rst index d773554e043..ffeeafef371 100644 --- a/docs/source/tutorials/how_to_add_a_post_scan_plugin.rst +++ b/docs/source/tutorials/adding-a-post-scan-plugin.rst @@ -1,7 +1,7 @@ -.. _how_to_add_post_scan_plugin: +.. _adding-a-post-scan-plugin: -Add A Post-Scan Plugin -====================== +Adding a post-scan plugin +========================= Scan plugins in ``scancode-toolkit`` ------------------------------------ @@ -59,7 +59,7 @@ Here are the major types of plugins: ``LocationProviderPlugin`` at `/src/plugincode/location_provider.py `_. -Built-In vs. Optional Installation +Built-in vs. optional installation ---------------------------------- Built-In @@ -67,7 +67,7 @@ Built-In Some post-scan plugins are installed when ScanCode itself is installed, and they are specified at ``[options.entry_points]`` in the `setup.cfg `_ file. -For example, the :ref:`license_policy_plugin` is a built-in plugin, whose code is located here:: +For example, the :ref:`license-policy-plugin` is a built-in plugin, whose code is located here:: https://github.com/aboutcode-org/scancode-toolkit/blob/develop/src/licensedcode/plugin_license_policy.py diff --git a/docs/source/tutorials/how_to_set_what_will_be_detected_in_a_scan.rst b/docs/source/tutorials/configuring-scan-detection.rst similarity index 70% rename from docs/source/tutorials/how_to_set_what_will_be_detected_in_a_scan.rst rename to docs/source/tutorials/configuring-scan-detection.rst index 4c572f04e37..58831e92c6e 100644 --- a/docs/source/tutorials/how_to_set_what_will_be_detected_in_a_scan.rst +++ b/docs/source/tutorials/configuring-scan-detection.rst @@ -1,13 +1,13 @@ -.. _how_to_set_what_scan_detects: +.. _configuring-scan-detection: -How to set what will be detected in Scan -======================================== +Configuring what will be detected in scan +========================================= ScanCode allows you to scan a codebase for license, copyright and other interesting information that can be discovered in files. The following options are available for detection when using ScanCode-Toolkit: -.. include:: /rst_snippets/basic_options.rst +.. include:: /rst-snippets/cli-basic-options.rst Different Scans --------------- @@ -15,15 +15,15 @@ Different Scans The following examples will use the ``samples`` directory that is provided with the `ScanCode Toolkit code `_. All examples will be saved in the JSON format, which can be loaded into ScanCode Workbench for visualization. See -:ref:`how_to_visualize_scan_results` for more information. Another output format option is a -static html file. See :ref:`cli_output_format` for more information. +:ref:`visualizing-scan-results` for more information. Another output format option is a +static html file. See :ref:`cli-output-format-options` for more information. Scan for all clues: ^^^^^^^^^^^^^^^^^^^ To scan for licenses, copyrights, urls, emails, package information, and file information -:: +.. code-block:: shell scancode -clipeu --json output.json samples @@ -31,7 +31,7 @@ To scan for licenses, copyrights, urls, emails, package information, and file in Scan for license and copyright clues: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -:: +.. code-block:: shell scancode -cl --json-pp output.json samples @@ -39,7 +39,7 @@ Scan for license and copyright clues: Scan for emails and URLs: ^^^^^^^^^^^^^^^^^^^^^^^^^ -:: +.. code-block:: shell scancode -eu --json-pp output.json samples @@ -47,7 +47,7 @@ Scan for emails and URLs: Scan for package information: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -:: +.. code-block:: shell scancode -p --json-pp output.json samples @@ -55,7 +55,7 @@ Scan for package information: Scan for file information: ^^^^^^^^^^^^^^^^^^^^^^^^^^ -:: +.. code-block:: shell scancode -i --json-pp output.json samples @@ -63,8 +63,8 @@ Scan for file information: To see more example scans: ^^^^^^^^^^^^^^^^^^^^^^^^^^ -:: +.. code-block:: shell scancode --examples -For more information, refer :ref:`cli_list_options`. +For more information, refer :ref:`cli-scancode`. diff --git a/docs/source/tutorials/how_to_format_scan_output.rst b/docs/source/tutorials/configuring-scan-output-formats.rst similarity index 62% rename from docs/source/tutorials/how_to_format_scan_output.rst rename to docs/source/tutorials/configuring-scan-output-formats.rst index 89d238ef281..69bbde4849f 100644 --- a/docs/source/tutorials/how_to_format_scan_output.rst +++ b/docs/source/tutorials/configuring-scan-output-formats.rst @@ -1,11 +1,11 @@ -.. _how_to_specify_output_format: +.. _configuring-scan-output-formats: -How to specify ScanCode Output Format -===================================== +Configuring scan output formats +=============================== -A basic overview of formatting ScanCode Output is presented here. +A basic overview of formatting ScanCode output is presented here. -More information on :ref:`cli_output_format`. +More information on :ref:`cli-output-format-options`. JSON ---- @@ -19,9 +19,9 @@ The following commands will output scan results in a formatted json file: * ``scancode --json-lines /path/to/output.json /path/to/target/dir`` -To compare the JSON output in different formats refer :ref:`comparing_json`. +To compare the JSON output in different formats refer :ref:`cli-comparing-json-output-file-formats`. -.. include:: /rst_snippets/stdout.rst +.. include:: /rst-snippets/cli-output-to-stdout.rst HTML @@ -34,10 +34,10 @@ The following commands will output scan results in a formatted HTML page or simp * ``scancode --html-app /path/to/output.html /path/to/target/dir`` -For more details on the HTML output format refer :ref:`output_html`. +For more details on the HTML output format refer :ref:`cli-html-option`. .. WARNING:: The ``--html-app`` option has been deprecated, use ScanCode Workbench instead. -.. include:: /rst_snippets/custom_output_format.rst +.. include:: /rst-snippets/cli-output-custom-format.rst diff --git a/docs/source/tutorials/data/components_sample.gif b/docs/source/tutorials/data/components_sample.gif deleted file mode 100644 index eca016f2ebb..00000000000 Binary files a/docs/source/tutorials/data/components_sample.gif and /dev/null differ diff --git a/docs/source/rst_snippets/data/extractcode.png b/docs/source/tutorials/data/extractcode.png similarity index 100% rename from docs/source/rst_snippets/data/extractcode.png rename to docs/source/tutorials/data/extractcode.png diff --git a/docs/source/tutorials/data/files_sample.png b/docs/source/tutorials/data/files-sample.png similarity index 100% rename from docs/source/tutorials/data/files_sample.png rename to docs/source/tutorials/data/files-sample.png diff --git a/docs/source/tutorials/data/filter_sample.gif b/docs/source/tutorials/data/filter-sample.gif similarity index 100% rename from docs/source/tutorials/data/filter_sample.gif rename to docs/source/tutorials/data/filter-sample.gif diff --git a/docs/source/tutorials/data/views_sample.gif b/docs/source/tutorials/data/views-sample.gif similarity index 100% rename from docs/source/tutorials/data/views_sample.gif rename to docs/source/tutorials/data/views-sample.gif diff --git a/docs/source/tutorials/data/workbench_dashboard.png b/docs/source/tutorials/data/workbench-dashboard.png similarity index 100% rename from docs/source/tutorials/data/workbench_dashboard.png rename to docs/source/tutorials/data/workbench-dashboard.png diff --git a/docs/source/tutorials/data/workbench_launch.png b/docs/source/tutorials/data/workbench-launch.png similarity index 100% rename from docs/source/tutorials/data/workbench_launch.png rename to docs/source/tutorials/data/workbench-launch.png diff --git a/docs/source/tutorials/index.rst b/docs/source/tutorials/index.rst index 4751d213620..018be2d0ef2 100644 --- a/docs/source/tutorials/index.rst +++ b/docs/source/tutorials/index.rst @@ -1,14 +1,13 @@ .. _tutorials: -**Basic Tutorials** +**Tutorials** =================== .. toctree:: :maxdepth: 2 - how_to_run_a_scan - how_to_visualize_scan_results - how_to_extract_archives - how_to_format_scan_output - how_to_set_what_will_be_detected_in_a_scan - how_to_add_a_post_scan_plugin + running-a-scan + visualizing-scan-results + configuring-scan-output-formats + configuring-scan-detection + adding-a-post-scan-plugin diff --git a/docs/source/tutorials/how_to_run_a_scan.rst b/docs/source/tutorials/running-a-scan.rst similarity index 76% rename from docs/source/tutorials/how_to_run_a_scan.rst rename to docs/source/tutorials/running-a-scan.rst index 5895f153c09..fb40de093ef 100644 --- a/docs/source/tutorials/how_to_run_a_scan.rst +++ b/docs/source/tutorials/running-a-scan.rst @@ -1,29 +1,32 @@ -How to Run a Scan +.. _running-a-scan: + +Running a scan ================= -In this simple tutorial example, we perform a basic scan on the ``samples`` directory distributed -by default with ScanCode. +In this tutorial, we will conduct a basic scan of the samples directory that is included +within `ScanCode `_ code repository and +`releases `_. Prerequisites ------------- -Refer to the :ref:`install` installation guide. +Refer to the :ref:`install-scancode` installation guide. -Looking into Files +Looking into files ------------------ As mentioned previously, we are going to perform the scan on the ``samples`` directory distributed by default with ScanCode-Toolkit. Here's the directory structure and respective files: -.. image:: /tutorials/data/files_sample.png +.. image:: /tutorials/data/files-sample.png We notice here that the sample files contain a package ``zlib.tar.gz``. So we have to extract the archive before running the scan, to also scan the files inside this package. -Performing Extraction +Performing extraction --------------------- To extract the packages inside ``samples`` directory:: @@ -32,15 +35,15 @@ To extract the packages inside ``samples`` directory:: This extracts the zlib.tar.gz package: -.. image:: /rst_snippets/data/extractcode.png +.. image:: /tutorials/data/extractcode.png .. note:: Use the ``--shallow`` option to prevent recursive extraction of nested archives. -Deciding Scan Options ---------------------- +Configuring scan options +------------------------ These are some common scan options you should consider using before you start the actual scan, according to your requirements. @@ -53,7 +56,7 @@ according to your requirements. to complete. -#. ``--license-score INTEGER`` is to be set if license matching accuracy is desired (Default is 0, +#. ``--license-score INT`` is to be set if license matching accuracy is desired (Default is 0, and increasing this means a more accurate match). Also, using ``--license-text`` includes the matched text to the result. @@ -67,17 +70,21 @@ according to your requirements. for specific tasks/have requirements. Here we are using ``json`` as ScanCode Workbench imports ``json`` files only. -For the complete list of options, refer :ref:`cli_list_options`. +For the complete list of options, see the :ref:`cli-scancode` reference. + +Running a scan +-------------- -Running The Scan ----------------- +Now, run the scan with the following options: -Now, run the scan with the options decided:: +:: scancode -clpeui -n 2 --ignore "*.java" --json-pp sample.json samples -A Progress report is shown:: +A progress report similar to the one below will shown. + +:: Setup plugins... Collect file inventory... @@ -97,5 +104,3 @@ A Progress report is shown:: scan: 26.62s total: 31.14s Removing temporary files...done. - -.. include:: /rst_snippets/other_imp_doc.rst diff --git a/docs/source/tutorials/how_to_visualize_scan_results.rst b/docs/source/tutorials/visualizing-scan-results.rst similarity index 56% rename from docs/source/tutorials/how_to_visualize_scan_results.rst rename to docs/source/tutorials/visualizing-scan-results.rst index 97cf774e371..74a739ce1b5 100644 --- a/docs/source/tutorials/how_to_visualize_scan_results.rst +++ b/docs/source/tutorials/visualizing-scan-results.rst @@ -1,9 +1,9 @@ -.. _how_to_visualize_scan_results: +.. _visualizing-scan-results: -How to Visualize Scan results -============================= +Visualizing scan results +======================== -To help visualize the scans, we have a dedicated tool `ScanCode workbench `_ which is a desktop application that allows you to visualize and explore the results of one or more scans. It is a cross-platform application that runs on Windows, Mac OS X and Linux. It is built using the Electron framework and is built using Electron, Typescript & React +To help visualize the scans, we have a dedicated tool `ScanCode workbench `_ which is a desktop application that allows you to visualize and explore the results of one or more scans. It is a cross-platform application that runs on Windows, Mac OS X and Linux. It is built using the Electron framework and is built using Electron, Typescript & React Detailed Installation and Usage guide can be found here - :ref:`scancode-workbench:getting-started`