Merge pull request #52 from ev3dev-lang-java/docs/maintainer_manual_markdown

docs: add the Maintainer's Manual translated to Markdown

Author: jabrena

docs/Adding_new_major_OpenJDK_version.md

@@ -0,0 +1,17 @@
+## Adding new major OpenJDK version
+
+Upstream produces new OpenJDK releases in a half-year cadence.
+The tip/master build (masqueraded as latest version) on Jenkins should
+keep up with this, as master is always master.
+
+To add a new released version, add an appropriate section in the
+config.sh shell script. It's probably best to start by copying an older
+release configuration and modifying it to match the new release configuration.
+
+There is hopefully only one catch: the Host/Boot JDK provided should be
+of the same or previous release (e.g. for building JDK13, it should be
+JDK13 or JDK12). Usually AdoptOpenJDK builds are used for x86 builders
+and our builds are used for ARM builders.
+
+For setting up Jenkins builds, please see
+[Integration with AdoptOpenJDK Jenkins](AdoptOpenJDK_Jenkins.md).

docs/AdoptOpenJDK_Jenkins.md

@@ -0,0 +1,49 @@
+## Integration with AdoptOpenJDK Jenkins
+
+To automate the build process, the project uses the AdoptOpenJDK
+Jenkins CI hosted at [ci.adoptopenjdk.net][adopt1] for its builds.
+The [ev3dev tab][adopt2] is dedicated to jobs of this project.
+
+The integration is based on Jenkins Pipelines. In the root of the
+repository, there is a Jenkinsfile describing the build steps & nodes
+where it can run. It handles the following actions:
+ 1. Docker container image building
+    - This is a prerequisite of the OpenJDK build.
+      Docker pipelines integration is used to build the container.
+ 2. OpenJDK build
+    - This builds the Java binary archives. However, Jenkinsfile bypasses
+      the autorun.sh script and runs the build phases directly.
+      By doing that, it is possible to have a separate Jenkins pipeline
+      stages for our phases.
+ 3. Artifact upload
+    - This uploads the binary archives to the Jenkins master, where they
+      are made available for download by the Internet.
+ 4. Cleanup
+    - As the build workspace & container images take quite a lot of space,
+      it is necessary to remove them after the build finishes
+      (whether successfully or not).
+
+The Jenkinsfile expects to be given some parameters.
+At the time of writing, the following parameters exist:
+ * `JDKVM_VALUE` - sets the JVM JIT to be used. See `JDKVM` for config.sh.
+ * `JDKVER_VALUE` - sets the JDK version to be built. See `JDKVER` for config.sh.
+ * `JDKPLATFORM_VALUE` - sets the target platform. See `JDKPLATFORM` for config.sh.
+ * `DOCKER_ARCH` - target arch. It should match the platform. Currently only `armel`.
+ * `DEBIAN` - Debian release - can be set to `stretch` or `buster`.
+ * `BUILD_TYPE` - either `cross` or `native` - sets the build model (x86>ARM vs ARM>ARM)
+ * `DISABLED` - with this enabled, the build will always do nothing and
+                it will always succeed. This is useful for cleanup of old jobs.
+
+There parameters can be forced to specific values for individual jobs
+by using single-choice "choice parameter".
+
+Currently all configured builds reside in the [`eljbuild`][eljbuild] folder.
+Except for JDK9 & JDK10, they all represent the ARMv8->ARM compilation
+path for the given Debian & JDK versions.
+
+To create a new Jenkins job, simply create a copy of an existing job and
+then customize the job parameters.
+
+[adopt1]: https://ci.adoptopenjdk.net/
+[adopt2]: https://ci.adoptopenjdk.net/view/ev3dev/
+[eljbuild]: https://ci.adoptopenjdk.net/view/ev3dev/job/eljbuild/

docs/Building.md

@@ -0,0 +1,184 @@
+## Building OpenJDK
+
+### Intro
+
+The goal of this project is to provide a small enough Java runtime that
+would run on ev3dev on EV3. Currently OpenJDK is used for this.
+The reason is simple - it contains an Oracle-developed and then
+opensourced fast ARM32 JIT from JDK9 onwards. The second reason is that
+leJOS EV3 had used a similar JDK (Embedded JDK 7/8), but it was a
+proprietary offering prepared and not later updated by Oracle.
+
+### Build system
+
+![native vs cross builders](images/host_machine_archs.png)
+
+To build the OpenJDK binaries in an environment suitable for
+cross-compilation for EV3, the repository is using a couple of Docker
+containers. They have two variants: `native` (ARM host -> ARM target) and
+`cross` (x86 host -> ARM target). The native builders have an advantage
+in that the JDK can be partially tested right on the build machine.
+
+![build system layers](images/build_system_layers.png)
+
+The layers of the build environment are shown on the picture above.
+The lower docker container/image contains the target OS (Debian/ev3dev)
+with its libraries and compilers. This is done to ensure that the API
+provided by the system and its libraries are compatible between the build
+and target machines.
+
+On top of that, there resides a set of shell scripts which control the build.
+Their task is to download JDK and source-level dependencies and to build
+the JDK binaries with its build system.
+
+These two containers have to be built before the build of OpenJDK itself.
+Afterwards, the scripts in the upper container take over the rest of
+the build process.
+
+It should be notes that as the builds run in a container, it is
+necessary to mount a directory from the host to the container as
+a build workspace. This makes it possible to access the built binaries.
+
+### Build scripts
+
+Let's return back to the scripts. There are six build scripts in total:
+ * `autobuild.sh` - This is the entrypoint for the build.
+                    It calls the last four scripts in succession
+                    if autobuild is enabled. Otherwise it just exits
+                    into shell.
+ * `config.sh`
+   - This script is `source`'d from the last four scripts.
+   - It takes in the JDK version and other parameters from environment
+     variables passed to the container and in turn it exports different
+     environment variables to be used by other build scripts
+     (like the JDK source URL).
+ * `prepare.sh` - This script contains the *Prepare* phase.
+   1. It downloads and unpacks the Host JDK (needed for OpenJDK build)
+      to a directory in the build workspace. This is then used to build
+      the Java classes of the resulting JDK.
+   2. It also downloads JTreg, which is used when building included
+      test cases (not currently run on Jenkins; but they can be run manually).
+ * `fetch.sh` - This scripts contains the *Fetch* phase.
+   1. It determines the SCM parameters (commit hash, real version).
+      From them, it generates a metadata file used to identify the build.
+      Usually the latest GA release is picked.
+   2. It then has to download the OpenJDK source code.
+   3. It applies the patches stored in this repository.
+   4. It clones one extra repository - the AdoptOpenJDK
+      [openjdk-build][bld] repository used for the CA certificates.
+ * `build.sh` - This script is doing the *Build* phase.
+   1. It configures the JDK build using the `configure` script.
+   2. It runs the JDK build using `make`.
+      - If the build is running on native (ARM) hardware, it also runs a
+        bootcycle build, which uses the newly built JDK to build itself
+        once more. This provides a basic level of testing.
+ * `zip.sh` - This script finishes the build with the *Archiving* stage.
+   1. It generates the JRI (Java Runtime Image) for the EV3,
+      which is basically a reduced Java runtime.
+   1. It then packs that and remaining build outputs (such as JMOD
+      packages and the full JDK) into a tarball which is then uploaded
+      as a build artifact.
+
+[bld]: https://github.com/adoptopenjdk/openjdk-build
+
+![build stages](images/build_process.png)
+
+### Build parameters
+
+The build scripts accept a predefined set of environment variables
+describing the build. The parameters need to be passed in via Docker
+environment variables; see [Manual build](#manual-build) for an example.
+
+ * `JDKVER` - sets the JDK version to build.
+   - One of `11`, `12`, `13`, `tip`, `loom`.
+     Numeric versions will usually build the latest tagged
+     General-Availability releases; `tip` will build the latest commit.
+     - *Note.* When a new JDK version is branched out for stabilization,
+       then the latest non-GA tag must used, as there are no GA tags yet.
+ * `JDKVM` - sets the JVM JIT to use. One of:
+   - `zero`: portable non-assembler JIT/interpreter.
+             It's quite slow, but it should always work.
+   - `client`: fast ARM assembler-assisted JIT.
+   - `minimal`: similar to `client`, but should be a bit smaller.
+                Unfortunately, this configuration does not build
+                successfully (the linking process will fail).
+ * `JDKPLATFORM` - sets the platform for the build scripts. Only `ev3` is allowed.
+ * `JDKDEBUG` - sets the JVM debug level. Optional. One of (from JDK source):
+   - `release`: no debug information, all optimizations, no asserts.
+   - `optimized`: no debug information, all optim., no asserts, HS tgt is 'optimized'.
+   - `fastdebug`: debug information (-g), all optimizations, all asserts
+   - `slowdebug`: debug information (-g), no optimizations, all asserts
+ * `AUTOBUILD` - when set to `1` or `yes`, it runs the autorun.sh script
+                 directly instead of the shell.
+
+### Build outputs
+
+The output of the build process are three different archives:
+ * `jmods-ev3.tar.gz` - contains Java JMODs.
+   These are pieces of the JDK that can be assembled into a proper and
+   useful JRE/JDK/JRI; see [here][jmod]. They are intended to be
+   processed with the jlink tool provided with host-native Java JDK.
+ * `jri-ev3.tar.gz` - EV3 JRI.
+   This is a smaller version of traditional JRE. It is intended to be
+   used as a Java runtime for the brick. It contains only a few Java
+   modules in order to reduce its size.
+ * `jdk-ev3.tar.gz` - Full EV3 JDK. This directory comes from the JDK
+   build process; however, it is likely also built internally using jlink.
+   This could be useful for someone who wants to do full Java
+   development on the brick.
+
+[jmod]: https://www.developer.com/java/data/how-modules-are-packaged-in-java-9.html
+
+### Manual build
+You can run the build OpenJDK on your own computer too. This script should do that:
+
+```sh
+# define parameters
+TARGET_WORKSPACE="$(pwd)/build" # 10 GB of free space should be sufficient
+TARGET_DEBIAN_VERSION="stretch" # stretch or buster
+TARGET_OPENJDK_VERSION="11"     # 11, 12, 13, tip, loom are on Jenkins
+
+# clone repository
+git clone https://github.com/ev3dev-lang-java/openjdk-ev3.git
+cd openjdk-ev3
+
+# prepare working directory
+mkdir -p "$TARGET_WORKSPACE"
+chmod -R 777 "$TARGET_WORKSPACE" # docker may not share UID with the current user
+
+# build base system container
+docker build --build-arg DEBIAN_RELEASE="$TARGET_DEBIAN_VERSION" \
+             --build-arg ARCH="armel" \
+             --tag "ev3dev-lang-java:jdk-cross-$TARGET_DEBIAN_VERSION" \
+             --file ./system/Dockerfile.cross \
+             ./system
+
+# on top of that, create a build scripts container
+docker build --build-arg commit="$(git rev-parse HEAD)" \
+             --build-arg extra="Manual build #1 by $(whoami)" \
+             --build-arg DEBIAN_RELEASE="$TARGET_DEBIAN_VERSION" \
+             --build-arg BUILD_TYPE="cross" \
+             --tag "ev3dev-lang-java:jdk-cross-build" \
+             ./scripts
+
+# now run the build
+docker run --rm \
+           --interactive \
+           --tty \
+           --volume "$TARGET_WORKSPACE:/build" \
+           --env JDKVER="$TARGET_OPENJDK_VERSION" \
+           --env JDKVM="client" \
+           --env JDKPLATFORM="ev3" \
+           --env JDKDEBUG="release" \
+           --env AUTOBUILD="1" \
+           ev3dev-lang-java:jdk-cross-build
+
+# finally, make workspace accessible for all users (i.e. current one too)
+chmod -R 777 "$TARGET_WORKSPACE"
+# and list the output directory (now it should contain three *-ev3.tar.gz files)
+ls "$TARGET_WORKSPACE"
+```
+
+See [issue #34][manual_build].
+
+[manual_build]: https://github.com/ev3dev-lang-java/openjdk-ev3/issues/34

docs/Contents.md

@@ -0,0 +1,16 @@
+# OpenJDK-EV3 Maintainer's Manual
+
+This manual goes through an overview of this project.
+It is a version of the [old manual](https://docs.google.com/document/d/1rsBDU68_ETXm9b_Lb4b6oPVBbNwT9f4xEua2w3-Pm6A)
+converted to pure Markdown.
+
+![overview of the release process](images/release_process.png)
+
+Contents:
+ * [Building OpenJDK](Building.md)
+ * [Included out-of-tree patches](Included_patches.md)
+ * [Debugging issues with OpenJDK](Debugging.md)
+ * [Adding new major OpenJDK version](Adding_new_major_OpenJDK_version.md)
+ * [Packaging, signing and uploading for ev3dev](Packaging_signing_uploading.md)
+ * [Integration with AdoptOpenJDK Jenkins](AdoptOpenJDK_Jenkins.md)
+ * [Ideas about testing the builds](Testing_ideas.md)

docs/Debugging.md

@@ -0,0 +1,31 @@
+## Debugging issues with OpenJDK
+
+Sometimes it is necessry to investigate a crash in native code.
+If the problem is not in an external library, it is likely that something
+went wrong inside the JVM. In that case, the following steps might help
+you with diagnosing the problem:
+
+
+ * Try to reproduce it on a more powerful machine (like with ARMv7 CPU).
+   If it reproduces, continue testing on this machine.
+   Good machines for this include newer Raspberry Pis and Odroid SBCs.
+ * Try to reproduce it using QEMU (e.g. in the Docker packaging container).
+   If it reproduces, continue there.
+ * Somehow get the debug symbols for the Java runtime. The
+   `jri-11-ev3-dbgsym` package from the repositories should suffice.
+   Alternatively, run a manual build to get a `fastdebug` or `slowdebug`
+   build (value of `JDKDEBUG` environment variable) and try to reproduce
+   the issue there.
+     - Optionally, you can also build the hsdis native disassembler for
+       diagnosing JIT compiler issues. However, I have not tried this.
+       It should give the ability to print JIT-generated ASM code;
+       however, it is also possible to disassemble it directly from GDB.
+ * Try to diagnose the issue with GDB - either by investigating the
+   core dump generated by the crash (to enable them, run
+   `ulimit -c unlimited`) or by triggering it in GDB. Another way is to
+   start the GDB only once the crash happens. See [here][jvm_gdb],
+   for more details, however I think the inner gdb command should be `gdb -p %p`.
+ * Try to discover the cause of the bug by walking through
+   the stack trace and the source code.
+
+[jvm_gdb]: https://neugens.wordpress.com/2015/02/26/debugging-the-jdk-with-gdb/

docs/Included_patches.md

@@ -0,0 +1,81 @@
+## Included out-of-tree patches
+
+In order to fix some issues in upstream OpenJDK, this repository
+contains some out-of-tree patches.
+
+### What we include
+
+Note: JTreg included tests = tests bundled with the OpenJDK source tree.
+
+#### OpenJDK 9
+
+*is not present anymore*
+
+#### OpenJDK 10
+
+*is not present anymore*
+
+#### OpenJDK 11
+ * `jdk11.patch` - fixes runtime errors & adds cflags optimizations.
+ * `jdk11_nosflt.patch` - removes the need for the external softfloat-3e library.
+ * `jdk11_lib.patch` - adds Debian JNI library path to the build.
+ * `jdk11_new.patch` - fixes bugs found on JDK12 by JTreg  incl. test on ARMv7.
+ * `jdk11_bkpt.patch` - disables ARM-specific breakpoint instruction on VM errors.
+ * `jdk11_cds.patch` - fixes crash (JTreg incl. test) caused by incorrect CDS data alignment.
+ * `jdk11_jfr.patch` - fixes crash (JTreg incl. test) in the Java Flight Recorder file writer.
+
+#### OpenJDK 12
+ * `jdk12_nosflt.patch` - removes the need for the external softfloat-3e library.
+ * `jdk12_new.patch` - fixes bugs found on JDK12 by JTreg incl. test on ARMv7.
+ * `jdk12_bkpt.patch` - disables ARM-specific breakpoint instruction on VM errors.
+ * `jdk12_cds.patch` - fixes crash (JTreg incl. test) caused by incorrect CDS data alignment.
+ * `jdk12_jfr.patch` - fixes crash (JTreg incl. test) in Java Flight Recorder file writer.
+
+#### OpenJDK 13
+ * `jdk13_nosflt.patch` - removes the need for the external softfloat-3e library.
+ * `jdk13_new.patch` - fixes bugs found on JDK12 by JTreg incl. test on ARMv7.
+ * `jdk13_bkpt.patch` - disables ARM-specific breakpoint instruction on errors.
+ * `jdk13_cds.patch` - fixes crash (JTreg incl. test) caused by incorrect CDS data alignment.
+ * `jdk13_jfr.patch` - fixes crash (JTreg incl. test) in Java Flight Recorder file writer.
+
+#### OpenJDK 14
+ * `jdk14_nosflt.patch` - removes the need for the external softfloat-3e library.
+ * `jdk14_new.patch` - fixes bugs found on JDK12 by JTreg incl. test on ARMv7.
+ * `jdk14_bkpt.patch` - disables ARM-specific breakpoint instruction on errors.
+ * `jdk14_cds.patch` - fixes crash (JTreg incl. test) caused by incorrect CDS data alignment.
+ * `jdk14_jfr.patch` - fixes crash (JTreg incl. test) in Java Flight Recorder file writer.
+
+
+### Patch rebasing
+As the development continues in the OpenJDK upstream, is is necessary
+to rebase the patches from time to time (or properly upstream them).
+This can be done this way:
+
+ 1. Clone the JDK Git repositories on the correct tags:
+    ```sh
+    git clone --branch jdk-11.0.4-ga --depth 1 https://github.com/openjdk/jdk11u.git && cd jdk11u
+    ```
+ 2. Apply the current patches one by one as individual commits, fixing the conflicts that arise.
+    ```sh
+    patch -p1 -i <patch1>
+    rm <...>.orig
+    git add <...>
+    git commit -m “patch1”
+    ```
+ 3. Print the commit log to get the hashes of the created commits.
+    ```sh
+    git log # or git hist, but you have to add an alias
+    ```
+ 4. Re-export the patches by running the following command between individual commits:
+    ```sh
+    git diff <previoushash> <thishash>   > <patch1>
+    ```
+
+### Upstreaming
+To upstream a patch (see also [OpenJDK guide][jdkguide]):
+ 1. Sign the Oracle Contributor Agreement and send it to Oracle.
+ 2. Create an issue in the JDK Bug System. (might be optional)
+ 3. Work on the patch.
+ 4. Send it to an appropriate mailing list and iterate on enhancing it until it's OK.
+
+[jdkguide]: https://openjdk.java.net/contribute/