CP013: Apply proposals for relative affinity and allocation.

* Introduce the affinity_query class, with operators.
* Add the execution_context::memory_resource() member function.
* Change this_system::resource() to this_system::resources() and have it
return a vector of execution_resource instead.
* Add requirements on this_system::resources().
* Add capability to migrate memory and work between resources to list of
problem space challenges.
* Add minor reformatting to proposed wording.

Author: Gordon Brown

affinity/cpp-20/d0796r1.md

@@ -17,7 +17,11 @@
 ### Revision 1
 
 * Introduce proposed wording.
+* Introduce interface for querying the relative affinity between execution resources.
+* Introduce `execution_context::memory_resource` returning a `pmr::memory_resource` to allow affinity-based allocation.
 * Have `execution_context::resource()` and `this_system::resource()` return a `const execution_resource &`.
+* Make `execution_resource` copy-able and move-able.
+* Rename `this_system::resource()` to `this_system::resources()` and have it return `vector<execution_resource>`.
 
 # Abstract
 
@@ -33,7 +37,7 @@ One strategy to improve applications' performance, given the importance of affin
 
 Operating systems (OSes) traditionally take responsibility for assigning threads or processes to run on processing units. However, OSes may use high-level policies for this assignment that do not necessarily match the optimal usage pattern for a given application. Application developers must leverage the placement of memory and **placement of threads** for best performance on current and future architectures. For C++ developers to achieve this, native support for **placement of threads and memory** is critical for application portability. We will refer to this as the **affinity problem**. 
 
-The affinity problem is especially challenging for applications whose behavior changes over time or is hard to predict, or when different applications interfere with each other's performance. Today, most OSes already can group processing units according to their locality and distribute processes, while keeping threads close to the initial thread, or even avoid migrating threads and maintain first touch policy. Nevertheless, most programs can change their work distribution, especially in the presence of nested parallelism.
+The affinity problem is especially challenging for applications whose behavior changes over time or is hard to predict, or when different applications interfere with each other's performance. Today, most OSes already can group processing units according to their locality and distribute processes, while keeping threads close to the initial thread, or even avoid migrating threads and maintain first touch policy. Nevertheless, most pro  grams can change their work distribution, especially in the presence of nested parallelism.
 
 Frequently, data is initialized at the beginning of the program by the initial thread and is used by multiple threads. While automatic thread migration has been implemented in some OSes, migration may have high overhead. In an optimal case, the OS may automatically detect which thread access which data most frequently, or it may replicate data which is read by multiple threads, or migrate data which is modified and used by threads residing on remote locality groups. However, the OS often does a reasonable job, if the machine is not overloaded, if the application carefully used first-touch allocation, and if the program does not change its behavior with respect to locality.
 
@@ -93,6 +97,7 @@ In this paper we describe the problem space of affinity for C++, the various cha
 * How to represent, identify and navigate the topology of execution resources available within a heterogeneous or distributed system.
 * How to query and measure the relative affininty between different execution resources within a system.
 * How to bind execution and allocation particular execution resource(s).
+* How to migrate memory work and memory allocations between execution resources.
 * What kind of and level of interface(s) should be provided by C++ for affinity.
 
 Wherever possible, we also evaluate how an affinity based solution could be scaled to support both distributed and heterogeneous systems.
@@ -112,19 +117,20 @@ There are some additional challenges which we have been investigating but are no
 
     /* Execution resource */
 
-    struct execution_resource {
+    class execution_resource {
+     public:
 
       execution_resource() = delete;
-      execution_resource(const execution_resource &) = delete;
-      execution_resource(execution_resource &&) = delete;
-      execution_resource &operator=(const execution_resource &) = delete;
-      execution_resource &operator=(execution_resource &&) = delete;
+      execution_resource(const execution_resource &);
+      execution_resource(execution_resource &&);
+      execution_resource &operator=(const execution_resource &);
+      execution_resource &operator=(execution_resource &&);
       ~execution_resource();
 
       size_t concurrency() const noexcept;
       size_t partition_size() const noexcept;
 
-      const execution_resource &partition(size_t i) const noexcept;
+      const execution_resource &partition(size_t) const noexcept;
       const execution_resource &member_of() const noexcept;
 
       std::string name() const noexcept;
@@ -136,25 +142,54 @@ There are some additional challenges which we have been investigating but are no
 
     /* Execution context */
 
-    struct execution_context {
+    class execution_context {
+     public:
 
       using executor_type = __unspecfied__;
 
-      template <typename ExecutionResource>
-      execution_context(ExecutionResource &&execResource);
+      execution_context(const execution_resource &);
 
       ~execution_context();
 
       const execution_resource &resource() const noexcept;
 
       executor_type executor() noexcept;
 
+      pmr::memory_resource *memory_resource() noexcept;
+
+    };
+
+    /* Affinity query */
+
+    enum class affinity_operation { read, write, copy, move, map };
+    enum class affinity_metric { latency, bandwidth, capacity, power_consumption };
+
+    template <affinity_operation Operation, affinity_metric Metric>
+    class affinity_query {
+     public:
+
+      using native_affinity_type = __unspecified__;
+      using error_type = __unspecified__
+
+      affinity_query(execution_resource &&, execution_resource &&);
+
+      ~affinity_query();
+
+      native_affinity_type native_affinity() const noexcept;
+
+      friend expected<size_t, error_type> operator==(const affinity_query&, const affinity_query&);
+      friend expected<size_t, error_type> operator!=const affinity_query&, const affinity_query&);
+      friend expected<size_t, error_type> operator<(const affinity_query&, const affinity_query&);
+      friend expected<size_t, error_type> operator>(const affinity_query&, const affinity_query&);
+      friend expected<size_t, error_type> operator<=(const affinity_query&, const affinity_query&);
+      friend expected<size_t, error_type> operator>=(const affinity_query&, const affinity_query&);
+
     };
 
     /* This system */
 
     namespace this_system {
-      const execution_resource &resource();
+      std::vector<execution_resource> resources() noexcept;
     }
 
     }  // execution
@@ -167,55 +202,56 @@ There are some additional challenges which we have been investigating but are no
 
 The `execution_resource` class provides an abstraction over a system's hardware capable of memory allocation, execution of light weight exeution agents or both.
 
-### `execution_resource` constructors
+> [*Note:* The `execution_resource` is required to be implemented such that the underlying software abstraction is initialised on when the `execution_resource` is constructed, maintained through reference counting and cleaned up on destruction of the final reference. *--end note*]
 
-	execution_resource() = delete;
+### `execution_resource` constructors
 
+    execution_resource() = delete;
 
-[*Note:* An implementation of `execution_resource` is permitted to provide non-public constructors to allow other objects to construct them. *--end note*]
+> [*Note:* An implementation of `execution_resource` is permitted to provide non-public constructors to allow other objects to construct them. *--end note*]
 
 ### `execution_resource` assignment
 
 The `execution_resource` class is not `CopyConstructible` (C++Std [copyconstructible]).
 
-    execution_resource(const execution_resource &) = delete;
-    execution_resource(execution_resource &&) = delete;
-    execution_resource &operator=(const execution_resource &) = delete;
-    execution_resource &operator=(execution_resource &&) = delete;
+    execution_resource(const execution_resource &);
+    execution_resource(execution_resource &&);
+    execution_resource &operator=(const execution_resource &);
+    execution_resource &operator=(execution_resource &&);
 
 ### `execution_resource` destructor
 
 The `execution_resource` class is not `Destructible` (C++Std [destructible]).
 
-	~execution_resource() = delete;
+    ~execution_resource() = delete;
 
 ### `execution_resource` operations
 
-      size_t concurrency() const noexcept;
+    size_t concurrency() const noexcept;
 
 *Returns:*
 
-      size_t partition_size() const noexcept;
+    size_t partition_size() const noexcept;
 
 *Returns:*
 
-      const execution_resource &partition(size_t i) const noexcept;
+    const execution_resource &partition(size_t) const noexcept;
 
 *Returns:*
 
-      const execution_resource &member_of() const noexcept;
+    const execution_resource &member_of() const noexcept;
 
 *Returns:*
 
-      std::string name() const noexcept;
+    std::string name() const noexcept;
 
 *Returns:*
 
-      bool can_place_memory() const noexcept;
+    bool can_place_memory() const noexcept;
 
 *Returns:*
 
-      bool can_place_agent() const noexcept;
+    bool can_place_agent() const noexcept;
 
 *Returns:*
 
@@ -225,35 +261,83 @@ The `execution_context` class provides an abstraction for managing a number of l
 
 ### `execution_context` member aliases
 
-	using executor_type = __unspecfied__;
+    using executor_type = __unspecfied__;
 
 *Requires:*
 
 ### `execution_context` constructors
 
-	template <typename ExecutionResource>
-	execution_context(ExecutionResource &&execResource);
+    execution_context(const execution_resource &);
 
 ### `execution_context` destructor
-
+    
     ~execution_context();
 
 ### `execution_context` operators
 
-	const execution_resource &resource() const noexcept;
+    const execution_resource &resource() const noexcept;
 
 *Returns:*
 
-	executor_type executor() noexcept;
+    executor_type executor() noexcept;
 
 *Returns:*
 
-## Free functions
+    pmr::memory_resource *memory_resource() noexcept;
+
+*Returns:*
+
+## Class template `affinity_query`
 
-	const this_system::execution_resource &resource();
+The `affinity_query` class template provides an abstraction for a relative affinity value between two `execution_resource`s, derived from a particular `affinity_operation` and `affinity_metric`.
+
+### `affinity_query` member aliases
+
+    using native_affinity_type = __unspecfied__;
+
+*Requires:*
+
+    using error_type = __unspecfied__;
+
+*Requires:*
+
+### `affinity_query` constructors
+
+    affinity_query(const execution_resource &, const execution_resource &);
+
+### `affinity_query` destructor
+
+    ~affinity_query();
+
+### `affinity_query` operators
+
+    native_affinity_type native_affinity() const noexcept;
+
+*Returns:* Unspecified native affinity value.
+
+### `affinity_query` comparisons
+
+    friend expected<size_t> operator==(const affinity_query&, const affinity_query&);
+    friend expected<size_t> operator!=const affinity_query&, const affinity_query&);
+    friend expected<size_t> operator<(const affinity_query&, const affinity_query&);
+    friend expected<size_t> operator>(const affinity_query&, const affinity_query&);
+    friend expected<size_t> operator<=(const affinity_query&, const affinity_query&);
+    friend expected<size_t> operator>=(const affinity_query&, const affinity_query&);
 
 *Returns:*
 
+> [*Note:* The comparison operators rely on the availability of the `expected` class template (see [P0323r4: std::expected][p0323r4]), if this does not become available then an alternative error/value construct will be adopted instead. *--end note*]
+
+## Free functions
+
+	std::vector<execution_resource> resources() noexcept;
+
+*Returns:* A std::vector containing all system level resources.
+
+*Requires:* If `resources().size() > 0`, `resources()[0]` be the `execution_resource` corroponding to the current thread of execution. The value returned by `resources()` be the same at any point after the invocation of `main`.
+
+> [*TODO:* Returning a `std::vector` allows users to potentially manipulate the container of `execution_resource`s after it is returned, we may want to replace this with an alternative type which is more restrictive at a later date. *--end TODO*]
+
 ## Querying a System’s Topology
 
 The first task in allowing C++ applications to leverage memory locality is to provide the ability to query a **system** for its **resource topology** (commonly represented as a tree or graph) and traverse its **execution resources**.
@@ -568,6 +652,7 @@ https://www.open-mpi.org/projects/hwloc/lstopo/
 
 [//]: Links
 
+[p0323r4]: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2017/p0323r4.html
 [hwloc]: https://www.open-mpi.org/projects/hwloc/
 [sycl-1-2-1]: https://www.khronos.org/registry/SYCL/specs/sycl-1.2.1.pdf
 [opencl-2-2]: https://www.khronos.org/registry/OpenCL/specs/opencl-2.2.pdf