Add information about compressed refs and SOA/LOA

- add to memory allocation topic
- link to -Xmx topic for large page guidance
- update -Xmx to add guidance for large pages

Signed-off-by: SueChaplain <[email protected]>

docs/allocator.md

@@ -25,7 +25,7 @@
 
 # Memory allocation
 
-The process of managing memory in the VM is handled by the Allocator and the Garbage Collector (GC). These components operate on an area of memory that is reserved for VM processing called the Java™ heap. 
+The process of managing memory in the VM is handled by the Allocator and the Garbage Collector (GC). These components operate on an area of memory that is reserved for VM processing called the Java™ heap.
 
 See [Garbage collection](gc.md) for more information about the GC.
 
@@ -39,7 +39,7 @@ Every allocation requires a *heap lock* to stop concurrent threads trying to acc
 
 Acquiring a heap lock for every allocation would be an intensive operation with a knock on impact to performance. To get around this problem, small objects are allocated to thread local heaps (TLH).
 
-### Thread local heaps (TLH)
+### Allocation caches
 
 To improve performance, allocation caches are reserved in the heap for different threads. These allocation caches are known as thread local heaps (TLH) and allow each thread to allocate memory from its cache without acquiring the heap lock. A TLH is typically used for small objects of less than 512 bytes (768 bytes on 64-bit VMs) although larger objects can be allocated from the cache if there is sufficient space.
 
@@ -65,5 +65,34 @@ The following options control the requested TLH size:
 Typically, when the maximum TLH size is increased, you should also increase the increment proportionally, so that active threads can
 reach the maximum requested TLH size more quickly.
 
-<!-- ==== END OF TOPIC ==== allocator.md ==== -->
+### SOA and lOA
+
+Some GC policies subdivide areas of the heap for object allocation into the Small Object Area (SOA) and the Large Object Area (LOA).
+
+The allocator initially attempts to allocate objects in the SOA, regardless of size. If the allocation cannot be satisfied the following actions are possible, depending on object size:
+
+- If the object is smaller than 64 KB, an allocation failure occurs, which triggers a GC action.
+- If the object is larger than 64 KB, the allocator attempts to allocate the object in the LOA. If the allocation cannot be satisfied, an allocation failure occurs, which triggers a GC action.
+
+The GC action that is triggered by the allocation failure depends on the GC policy in force.
+
+The overall size of the LOA is calculated when the heap is initialized, and recalculated at the end of each global GC cycle. The GC can expand or shrink the LOA, depending on usage, to avoid allocation failures.
+
+You can control the size of the LOA by using the `-Xloainitial`, `-Xloaminimum`, and `-Xloamaximum` command line options (See [LOA sizing options](xloaminimum.md)). If the LOA is not used, the GC shrinks the LOA after a few cycles, down to the value of `-Xloaminimum`. You can also specify [`-Xnoloa`](xloa.md) to prevent an LOA being created.
+
+An SOA and LOA are used by the OpenJ9 standard GC policies: `gencon`, `optavgpause` and `optthruput`. However, the `balanced` and `metronome` GC policies use a heap configuration that does not include the SOA or LOA. For more information about policies, see [Garbage Collection policies](gc.md).
 
+### Compressed references
+
+On 64-bit systems, the VM can use compressed references to decrease the size of Java objects and make better use of the available space in the Java heap. By storing objects in a 32-bit representation, the object size is identical to a 32-bit object, which occupies a smaller memory footprint. These 4 byte (32-bit) compressed references are converted to 64-bit values at runtime with minimal overhead. Smaller objects enable larger heap sizes that result in less frequent garbage collection and improve memory cache utilization. Overall, the performance of 64-bit applications that store compressed rather than uncompressed 64-bit object references is significantly improved.
+
+Compressed references are used by default when the maximum Java heap size is in the range 0 - 57 GB on AIX&reg;, Linux&reg;, and Windows&reg; systems. The upper limit is also 57 GB on z/OS&reg; systems that have APAR OA49416
+installed (25 GB without APAR OA49416). All GC policies observe these limits except for the [`metronome`](gc.md#metronome-policy) policy, which can only support a heap size of up to 25 GB with compressed references.
+
+When the VM uses compressed references, classes, threads, and monitors are stored in the lowest 4 GB of address space. However, this area of memory is also used by native libraries, the operating system, and for small Java heaps. If you receive native memory `OutOfMemoryError` exceptions when running with compressed references enabled, these errors might result from the lowest 4 GB of address space becoming full. Try specifying a large heap with the [`-Xmx`](xms.md) option, which puts the Java heap into a higher area of address space or using the [`-Xmcrs`](xmcrs.md) option to reserve space in the lowest 4 GB of address space for compressed references.
+
+When setting a Java heap size on IBM Z systems that use large pages, refer to the [`-Xmx`](xms.md) option for information about calculating an appropriate heap size value.
+
+To turn off compressed references, use the [`-Xnocompressedrefs`](xcompressedrefs.md) command-line option.
+
+<!-- ==== END OF TOPIC ==== allocator.md ==== -->

docs/xcompressedrefs.md

@@ -48,7 +48,7 @@ Compressed references are enabled by default when [`-Xmx`](xms.md) ≤ 57 GB.
 
 ## See also
 
-- [Compressed references](https://www.ibm.com/support/knowledgecenter/SSYKE2_8.0.0/com.ibm.java.vm.80.doc/docs/mm_compressed_references.html)
+- [Compressed references](allocator.md#compressed-references)
 
 
 <!-- ==== END OF TOPIC ==== xcompressedrefs.md ==== -->

docs/xms.md

@@ -33,9 +33,7 @@ These Oracle® HotSpot™ options set the initial/minimum Java™ hea
 - If you exceed the limit set by the `-Xmx` option, the OpenJ9 VM generates an `OutofMemoryError`.
 - If you set a value for `-Xms`, the [`-XX:InitialRAMPercentage`](xxinitialrampercentage.md) option is ignored.
 - If you set a value for `-Xmx`, the [`-XX:MaxRAMPercentage`](xxinitialrampercentage.md) option is ignored.
-
-If you are allocating the Java heap with large pages, see also [-Xlp](xlp.md) and
-[More effective heap usage using compressed references](https://www.ibm.com/support/knowledgecenter/SSYKE2_8.0.0/com.ibm.java.vm.80.doc/docs/mm_gc_compressed_refs.html).
+- If you are allocating the Java heap with large pages, see [-Xlp](xlp.md) and [Calculating `-Xmx` for large pages](#calculating-xmx-for-large-pages).
 
 You can also use the [`-Xmo`](xmo.md) option (not supported by the balanced garbage collection policy):  
 If the scavenger is enabled, `-Xms` ≥ `-Xmn` + `-Xmo`  
@@ -51,6 +49,13 @@ If the scavenger is disabled, `-Xms` ≥ `-Xmo`
 See [Using -X command-line options](x_jvm_commands.md) for more information about the `<size>` parameter.  
 See [Default settings for the OpenJ9 VM](openj9_defaults.md) for more about default values.
 
+### Calculating `-Xmx` for large pages
+
+For z/OS&reg; and Linux&reg; on IBM Z&reg;, specifying an object heap size that is a multiple of the page size uses another page of memory. For example, if the page size is 2 GB, setting `-Xmx2G` actually uses 4 GB of memory. To avoid using more memory, make the heap size a little smaller than an integral number of pages by subtracting at least 16 bytes from the value that you set with the `-Xmx` option. The following examples are provided:
+
+- For Linux on IBM Z, you might set `-Xmx=61203283972` for a 57 GB heap (61203283968 bytes minus 16 bytes).
+- For z/OS, you might set `-Xmx=32212254704` for a 30 GB heap (32212254720 bytes minus 16 bytes), or `-Xmx=66571993072` for a 62 GB heap (66571993088 bytes minus 16 bytes).
+
 ## Examples
 
 `-Xms2m -Xmx64m`