diff --git a/distribution/docs/src/main/resources/content/_architectures/_catalogFrameworks/catalog-framework-intro.adoc b/distribution/docs/src/main/resources/content/_architectures/_catalogFrameworks/catalog-framework-intro.adoc index 07163104da8e..1c04c4e7c606 100644 --- a/distribution/docs/src/main/resources/content/_architectures/_catalogFrameworks/catalog-framework-intro.adoc +++ b/distribution/docs/src/main/resources/content/_architectures/_catalogFrameworks/catalog-framework-intro.adoc @@ -37,22 +37,22 @@ The selected <<{managing-prefix}federation_strategy, federation strategy>> calls Then, any Post-query or Post-ingest plug-ins are invoked. Finally, the appropriate response is returned to the calling endpoint. -The Catalog Framework wires all Catalog components together. +The Catalog Framework wires all Catalog components together. -It is responsible for routing Catalog requests and responses to the appropriate target.  +It is responsible for routing Catalog requests and responses to the appropriate target. -<<{integrating-prefix}endpoints,Endpoints>> send Catalog requests to the Catalog Framework. -The Catalog Framework then invokes <<{architecture-prefix}catalog_plugins,Catalog Plugins>>, <<{architecture-prefix}transformers,Transformers>>, and <<{architecture-prefix}resources,Resource Components>> as needed before sending requests to the intended destination, such as one or more <<{managing-prefix}connecting_to_sources,Sources>>.  +<<{integrating-prefix}endpoints,Endpoints>> send Catalog requests to the Catalog Framework. +The Catalog Framework then invokes <<{architecture-prefix}catalog_plugins,Catalog Plugins>>, <<{architecture-prefix}transformers,Transformers>>, and <<{architecture-prefix}resources,Resource Components>> as needed before sending requests to the intended destination, such as one or more <<{managing-prefix}connecting_to_sources,Sources>>. -The Catalog Framework decouples clients from service implementations and provides integration points for Catalog Plugins and convenience methods for Endpoint developers. +The Catalog Framework decouples clients from service implementations and provides integration points for Catalog Plugins and convenience methods for Endpoint developers. == Catalog API Design The Catalog is composed of several components and an API that connects them together. -The ((Catalog API)) is central to ${branding}'s architectural qualities of extensibility and flexibility.  -The Catalog API consists of Java interfaces that define Catalog functionality and specify interactions between components.  +The ((Catalog API)) is central to ${branding}'s architectural qualities of extensibility and flexibility. +The Catalog API consists of Java interfaces that define Catalog functionality and specify interactions between components. These interfaces provide the ability for components to interact without a dependency on a particular underlying implementation, thus allowing the possibility of alternate implementations that can maintain interoperability and share developed components. -As such, new capabilities can be developed independently, in a modular fashion, using the Catalog API interfaces and reused by other ${branding} installations. +As such, new capabilities can be developed independently, in a modular fashion, using the Catalog API interfaces and reused by other ${branding} installations. === Ensuring Compatibility @@ -100,7 +100,7 @@ These examples are for illustrative purposes only and do not necessarily represe The Ingest Service Endpoint, the Catalog Framework, and the Catalog Provider are key components of the Reference Implementation. The Endpoint bundle implements a Web service that allows clients to create, update, and delete metacards. The Endpoint calls the `CatalogFramework` to execute the operations of its specification. -The `CatalogFramework` routes the request through optional `PreIngest` and `PostIngest` Catalog Plugins, which may modify the ingest request/response before/after the Catalog Provider executes the ingest request and provides the response.  +The `CatalogFramework` routes the request through optional `PreIngest` and `PostIngest` Catalog Plugins, which may modify the ingest request/response before/after the Catalog Provider executes the ingest request and provides the response. Note that a `CatalogProvider` must be present for any ingest requests to be successfully processed, otherwise a fault is returned. This process is similar for updating catalog entries, with update requests calling the `update(UpdateRequest)` methods on the Endpoint, `CatalogFramework`, and Catalog Provider. @@ -108,7 +108,7 @@ Similarly, for deletion of catalog entries, the delete requests call the `delete ==== Error Handling -Any ingest attempts that fail inside the Catalog Framework (whether the failure comes from the Catalog Framework itself, pre-ingest plugin failures, or issues with the Catalog Provider) will be logged to a separate log file for ease of error handling. +Any ingest attempts that fail inside the Catalog Framework (whether the failure comes from the Catalog Framework itself, pre-ingest plugin failures, or issues with the Catalog Provider) will be logged to a separate log file for ease of error handling. The file is located at `${home_directory}/data/log/ingest_error.log` and will log the Metacards that fail, their ID and Title name, and the stack trace associated with their failure. By default, successful ingest attempts are not logged. However, that functionality can be achieved by setting the log level of the `ingestLogger` to DEBUG (note that enabling DEBUG can cause a non-trivial performance hit). @@ -158,11 +158,11 @@ log:set The Query Service Endpoint, the Catalog Framework, and the `CatalogProvider` are key components for processing a query request as well. The Endpoint bundle contains a Web service that exposes the interface to query for `Metacards`. -The Endpoint calls the `CatalogFramework` to execute the operations of its specification. +The Endpoint calls the `CatalogFramework` to execute the operations of its specification. The `CatalogFramework` relies on the `CatalogProvider` to execute the actual query. -Optional PreQuery and PostQuery Catalog Plugins may be invoked by the `CatalogFramework` to modify the query request/response prior to the Catalog Provider processing the query request and providing the query response. -If a `CatalogProvider` is not configured and no other remote Sources are configured, a fault will be returned. -It is possible to have only remote Sources configured and no local `CatalogProvider` configured and be able to execute queries to specific remote Sources by specifying the site name(s) in the query request. +Optional PreQuery and PostQuery Catalog Plugins may be invoked by the `CatalogFramework` to modify the query request/response prior to the Catalog Provider processing the query request and providing the query response. +If a `CatalogProvider` is not configured and no other remote Sources are configured, a fault will be returned. +It is possible to have only remote Sources configured and no local `CatalogProvider` configured and be able to execute queries to specific remote Sources by specifying the site name(s) in the query request. ==== Product Caching @@ -171,7 +171,7 @@ If caching is enabled, each time a retrieve product request is received, the Cat If it has, the product is retrieved from the local site and returned to the client, providing a much quicker turnaround because remote product retrieval and network traffic was avoided. If the requested product is not in the cache, the product is retrieved from the Source (local or remote) and cached locally while returning the product to the client. The caching to a local file of the product and the streaming of the product to the client are done simultaneously so that the client does not have to wait for the caching to complete before receiving the product. -If errors are detected during the caching, caching of the product will be abandoned, and the product will be returned to the client.  +If errors are detected during the caching, caching of the product will be abandoned, and the product will be returned to the client. The Catalog Framework attempts to detect any network problems during the product retrieval, e.g., long pauses where no bytes are read implying a network connection was dropped. (The amount of time defined as a "long pause" is configurable, with the default value being five seconds.) @@ -183,4 +183,4 @@ Otherwise, caching is canceled if the user cancels the product download. ==== Product Download Status -As part of the caching of products, the Catalog Framework also posts events to the OSGi notification framework. Information includes when the product download started, whether the download is retrying or failed (after the number of retrieval attempts configured for product caching has been exhausted), and when the download completes. These events are retrieved by the Search UI and presented to the user who initiated the download. +As part of the caching of products, the Catalog Framework also posts events to the OSGi notification framework. Information includes when the product download started, whether the download is retrying or failed (after the number of retrieval attempts configured for product caching has been exhausted), and when the download completes. These events are retrieved by the Search UI and presented to the user who initiated the download. diff --git a/distribution/docs/src/main/resources/content/_architectures/_resources/resource-readers-intro.adoc b/distribution/docs/src/main/resources/content/_architectures/_resources/resource-readers-intro.adoc index b39e766befda..ead097aef0fc 100644 --- a/distribution/docs/src/main/resources/content/_architectures/_resources/resource-readers-intro.adoc +++ b/distribution/docs/src/main/resources/content/_architectures/_resources/resource-readers-intro.adoc @@ -9,7 +9,7 @@ A ((Resource Reader)) retrieves resources associated with metacards via URIs. Each resource reader must know how to interpret the resource's URI and how to interact with the data store to retrieve the resource. -There can be multiple resource readers in a Catalog instance. -The `Catalog Framework` selects the appropriate resource reader based on the scheme of the resource's URI.  +There can be multiple resource readers in a Catalog instance. +The `Catalog Framework` selects the appropriate resource reader based on the scheme of the resource's URI. -In order to make a resource reader available to the Catalog Framework, it must be exported to the OSGi Service Registry as a `${ddf-branding}.catalog.resource.ResourceReader`.  +In order to make a resource reader available to the Catalog Framework, it must be exported to the OSGi Service Registry as a `${ddf-branding}.catalog.resource.ResourceReader`. diff --git a/distribution/docs/src/main/resources/content/_architectures/_resources/url-resource-reader.adoc b/distribution/docs/src/main/resources/content/_architectures/_resources/url-resource-reader.adoc index bc781d57a481..cc84e597be90 100644 --- a/distribution/docs/src/main/resources/content/_architectures/_resources/url-resource-reader.adoc +++ b/distribution/docs/src/main/resources/content/_architectures/_resources/url-resource-reader.adoc @@ -31,11 +31,13 @@ Configuring the URL Resource Reader to retrieve files requires adding Security M After adding permission entries, a system restart is required for them to take effect. ==== +---- grant codeBase "file:/org.apache.tika.core/catalog-core-urlresourcereader" { //# Add permissions here permission java.io.FilePermission "", "read"; permission java.io.FilePermission "", "read"; } +---- Trailing slashes after have no effect on the permissions granted. For example, adding a permission for "${/}test${/}path" and "${/}test${/}path${/}" are equivalent. The recursive forms "${/}test${/}path${/}-", and "${/}test${/}path${/}${/}-" are also equivalent. diff --git a/distribution/docs/src/main/resources/content/_architectures/_securityFramework/realms.adoc b/distribution/docs/src/main/resources/content/_architectures/_securityFramework/realms.adoc index 3a39d6b769d1..0d4c8f4231b1 100644 --- a/distribution/docs/src/main/resources/content/_architectures/_securityFramework/realms.adoc +++ b/distribution/docs/src/main/resources/content/_architectures/_securityFramework/realms.adoc @@ -92,7 +92,7 @@ Does not support the acquisition of credentials. |ddf.security.pdp.realm.AuthZRealm |This realm performs the authorization decision and may or may not delegate out to the external XACML processing engine. It uses the incoming permissions to create a decision. -However, it is possible to extend this realm using the ddf.security.policy.extension.PolicyExtension interface. +However, it is possible to extend this realm using the `ddf.security.policy.extension.PolicyExtension interface`. This interface allows an integrator to add additional policy information to the PDP that can't be covered via its generic matching policies. This approach is often easier to configure for those that are not familiar with XACML. diff --git a/distribution/docs/src/main/resources/content/_architectures/_securityFramework/subject.adoc b/distribution/docs/src/main/resources/content/_architectures/_securityFramework/subject.adoc index d5d9e9d98cc7..d6043b2123fe 100644 --- a/distribution/docs/src/main/resources/content/_architectures/_securityFramework/subject.adoc +++ b/distribution/docs/src/main/resources/content/_architectures/_securityFramework/subject.adoc @@ -16,7 +16,7 @@ The Subject can also be used to perform permission checks to see if the calling This class was made ${branding}-specific because the Shiro interface cannot be added to the Query Request property map. .Implementations of Subject: -[cols="2" options="header"] +[cols="1m,1" options="header"] |=== |Classname diff --git a/distribution/docs/src/main/resources/content/_developing/_devComponents/custom-plugins.adoc b/distribution/docs/src/main/resources/content/_developing/_devComponents/custom-plugins.adoc index fcb6018dc25e..efba63d3af05 100644 --- a/distribution/docs/src/main/resources/content/_developing/_devComponents/custom-plugins.adoc +++ b/distribution/docs/src/main/resources/content/_developing/_devComponents/custom-plugins.adoc @@ -148,9 +148,9 @@ Develop a custom Post-Ingest Plugin. `public class SamplePostIngestPlugin implements ddf.catalog.plugin.PostIngestPlugin` . Implement the required methods. + - * `public CreateResponse process(CreateResponse input) *throws* PluginExecutionException;` - * `public UpdateResponse process(UpdateResponse input) *throws* PluginExecutionException;` - * `public DeleteResponse process(DeleteResponse input) *throws* PluginExecutionException;` + * `public CreateResponse process(CreateResponse input) throws PluginExecutionException;` + * `public UpdateResponse process(UpdateResponse input) throws PluginExecutionException;` + * `public DeleteResponse process(DeleteResponse input) throws PluginExecutionException;` . Import the ${branding} interface packages to the bundle manifest (in addition to any other required packages). + `Import-Package: ddf.catalog,ddf.catalog.plugin` @@ -167,7 +167,7 @@ Develop a custom Pre-Query Plugin `public class SamplePreQueryPlugin implements ddf.catalog.plugin.PreQueryPlugin` . Implement the required method. + -`public QueryRequest process(QueryRequest input) *throws* PluginExecutionException, StopProcessingException;` +`public QueryRequest process(QueryRequest input) throws PluginExecutionException, StopProcessingException;` . Import the ${branding} interface packages to the bundle manifest (in addition to any other required packages). + `Import-Package: ddf.catalog,ddf.catalog.plugin` @@ -182,7 +182,7 @@ Develop a custom Post-Query Plugin `public class SamplePostQueryPlugin implements ddf.catalog.plugin.PostQueryPlugin` . Implement the required method. + -`public QueryResponse process(QueryResponse input) *throws* PluginExecutionException, StopProcessingException;` +`public QueryResponse process(QueryResponse input) throws PluginExecutionException, StopProcessingException;` . Import the ${branding} interface packages to the bundle manifest (in addition to any other required packages). + `Import-Package: ddf.catalog,ddf.catalog.plugin` @@ -195,14 +195,14 @@ Develop a custom Post-Query Plugin Develop a custom Pre-Delivery Plugin. . Create a Java class that implements `PreDeliveryPlugin`. + -`public class SamplePreDeliveryPlugin *implements* ddf.catalog.plugin.PreDeliveryPlugin` +`public class SamplePreDeliveryPlugin implements ddf.catalog.plugin.PreDeliveryPlugin` . Implement the required methods. + -`public Metacard processCreate(Metacard metacard) *throws* PluginExecutionException, StopProcessingException;` -`public Update processUpdateMiss(Update update) *throws* PluginExecutionException, +`public Metacard processCreate(Metacard metacard) throws PluginExecutionException, StopProcessingException;` +`public Update processUpdateMiss(Update update) throws PluginExecutionException, StopProcessingException;` - * `public Update processUpdateHit(Update update) *throws* PluginExecutionException, StopProcessingException;` - * `public Metacard processCreate(Metacard metacard) *throws* PluginExecutionException, StopProcessingException;` + * `public Update processUpdateHit(Update update) throws PluginExecutionException, StopProcessingException;` + * `public Metacard processCreate(Metacard metacard) throws PluginExecutionException, StopProcessingException;` . Import the ${branding} interface packages to the bundle manifest (in addition to any other required packages). + `Import-Package: ddf.catalog,ddf.catalog.plugin,ddf.catalog.operation,ddf.catalog.event` @@ -216,20 +216,20 @@ StopProcessingException;` Develop a custom Pre-Subscription Plugin. . Create a Java class that implements `PreSubscriptionPlugin`. + -`public class SamplePreSubscriptionPlugin *implements* ddf.catalog.plugin.PreSubscriptionPlugin` +`public class SamplePreSubscriptionPlugin implements ddf.catalog.plugin.PreSubscriptionPlugin` . Implement the required method. - * `public Subscription process(Subscription input) *throws* PluginExecutionException, StopProcessingException;` + * `public Subscription process(Subscription input) throws PluginExecutionException, StopProcessingException;` === Implementing Pre-Resource Plugins Develop a custom Pre-Resource Plugin. . Create a Java class that implements `PreResourcePlugin`. -`public class SamplePreResourcePlugin *implements* ddf.catalog.plugin.PreResourcePlugin` +`public class SamplePreResourcePlugin implements ddf.catalog.plugin.PreResourcePlugin` . Implement the required method. + - * `public ResourceRequest process(ResourceRequest input) *throws* PluginExecutionException, StopProcessingException;` + * `public ResourceRequest process(ResourceRequest input) throws PluginExecutionException, StopProcessingException;` . Import the ${branding} interface packages to the bundle manifest (in addition to any other required packages). + `Import-Package: ddf.catalog,ddf.catalog.plugin,ddf.catalog.operation` @@ -246,10 +246,10 @@ Develop a custom Pre-Resource Plugin. Develop a custom Post-Resource Plugin. . Create a Java class that implements `PostResourcePlugin`. + -`public class SamplePostResourcePlugin *implements* ddf.catalog.plugin.PostResourcePlugin` +`public class SamplePostResourcePlugin implements ddf.catalog.plugin.PostResourcePlugin` . Implement the required method. + - * `public ResourceResponse process(ResourceResponse input) *throws* PluginExecutionException, StopProcessingException;` + * `public ResourceResponse process(ResourceResponse input) throws PluginExecutionException, StopProcessingException;` . Import the ${branding} interface packages to the bundle manifest (in addition to any other required packages). + `Import-Package: ddf.catalog,ddf.catalog.plugin,ddf.catalog.operation` @@ -267,14 +267,14 @@ Develop a custom Post-Resource Plugin. Develop a custom Policy Plugin. . Create a Java class that implements `PolicyPlugin`. + -`public class SamplePolicyPlugin *implements* ddf.catalog.plugin.PolicyPlugin` +`public class SamplePolicyPlugin implements ddf.catalog.plugin.PolicyPlugin` . Implement the required methods. + - * `PolicyResponse processPreCreate(Metacard input, Map properties) *throws* StopProcessingException;` - * `PolicyResponse processPreUpdate(Metacard input, Map properties) *throws* StopProcessingException;` - * `PolicyResponse processPreDelete(String attributeName, List attributeValues, Map properties) *throws* StopProcessingException;` - * `PolicyResponse processPreQuery(Query query, Map properties) *throws* StopProcessingException;` - * `PolicyResponse processPostQuery(Result input, Map properties) *throws* StopProcessingException;` + * `PolicyResponse processPreCreate(Metacard input, Map properties) throws StopProcessingException;` + * `PolicyResponse processPreUpdate(Metacard input, Map properties) throws StopProcessingException;` + * `PolicyResponse processPreDelete(String attributeName, List attributeValues, Map properties) throws StopProcessingException;` + * `PolicyResponse processPreQuery(Query query, Map properties) throws StopProcessingException;` + * `PolicyResponse processPostQuery(Result input, Map properties) throws StopProcessingException;` . Import the DDF interface packages to the bundle manifest (in addition to any other required packages). + `Import-Package: ddf.catalog,ddf.catalog.plugin,ddf.catalog.operation` @@ -288,14 +288,14 @@ Develop a custom Policy Plugin. Develop a custom Access Plugin. . Create a Java class that implements `AccessPlugin`. + -`public class SamplePostResourcePlugin *implements* ddf.catalog.plugin.AccessPlugin` +`public class SamplePostResourcePlugin implements ddf.catalog.plugin.AccessPlugin` . Implement the required methods. + - * `CreateRequest processPreCreate(CreateRequest input) *throws* StopProcessingException;` - * `UpdateRequest processPreUpdate(UpdateRequest input) *throws* StopProcessingException;` - * `DeleteRequest processPreDelete(DeleteRequest input) *throws* StopProcessingException;` - * `QueryRequest processPreQuery(QueryRequest input) *throws* StopProcessingException;` - * `QueryResponse processPostQuery(QueryResponse input) *throws* StopProcessingException;` + * `CreateRequest processPreCreate(CreateRequest input) throws StopProcessingException;` + * `UpdateRequest processPreUpdate(UpdateRequest input) throws StopProcessingException;` + * `DeleteRequest processPreDelete(DeleteRequest input) throws StopProcessingException;` + * `QueryRequest processPreQuery(QueryRequest input) throws StopProcessingException;` + * `QueryResponse processPostQuery(QueryResponse input) throws StopProcessingException;` . Import the DDF interface packages to the bundle manifest (in addition to any other required packages). + `Import-Package: ddf.catalog,ddf.catalog.plugin,ddf.catalog.operation` diff --git a/distribution/docs/src/main/resources/content/_developing/_devComponents/custom-sts-claims-handlers.adoc b/distribution/docs/src/main/resources/content/_developing/_devComponents/custom-sts-claims-handlers.adoc index 65f0479fe4c1..6f0f1e9f6b9d 100644 --- a/distribution/docs/src/main/resources/content/_developing/_devComponents/custom-sts-claims-handlers.adoc +++ b/distribution/docs/src/main/resources/content/_developing/_devComponents/custom-sts-claims-handlers.adoc @@ -746,7 +746,7 @@ See the request block below for an example of how these components should be pop * TokenType of http://docs.oasis-open.org/wss/oasis-wss-saml-token-profile-1.1#SAMLV2.0 * KeyType of http://docs.oasis-open.org/ws-sx/ws-trust/200512/PublicKey * X509 Certificate as the Proof of Possession or POP. This needs to be the certificate of the client that will be both requesting the SAML assertion and using the SAML assertion to issue a query -* Claims (optional): Some endpoints may require that the SAML assertion include attributes of the user, such as an authenticated user's role, name identifier, email address, etc. If the SAML assertion needs those attributes, the RequestSecurityToken must specify which ones to include. +* Claims (optional): Some endpoints may require that the SAML assertion include attributes of the user, such as an authenticated user's role, name identifier, email address, etc. If the SAML assertion needs those attributes, the RequestSecurityToken must specify which ones to include. ** UsernameToken: If Claims are required, the RequestSecurityToken security header must contain a UsernameToken element with a username and password. .X.509 PublicKey SAML Security Token Sample Request diff --git a/distribution/docs/src/main/resources/content/_developing/_devComponents/custom-transformers-input.adoc b/distribution/docs/src/main/resources/content/_developing/_devComponents/custom-transformers-input.adoc index 6f2d5cae7e4e..af6bb213a3ce 100644 --- a/distribution/docs/src/main/resources/content/_developing/_devComponents/custom-transformers-input.adoc +++ b/distribution/docs/src/main/resources/content/_developing/_devComponents/custom-transformers-input.adoc @@ -9,7 +9,7 @@ ${branding} supports the creation of custom <<{architecture-prefix}types_of_transformers,input transformers>> for use cases not covered by the included implementations. .Creating a custom input Transformer: -. Create a new Java class that implements ddf.catalog.transform.InputTransformer. + +. Create a new Java class that implements `ddf.catalog.transform.InputTransformer`. + `public class SampleInputTransformer implements ddf.catalog.transform.InputTransformer` . Implement the transform methods. + `public Metacard transform(InputStream input) throws IOException, CatalogTransformerException` + diff --git a/distribution/docs/src/main/resources/content/_developing/_devComponents/managed-service-factories.adoc b/distribution/docs/src/main/resources/content/_developing/_devComponents/managed-service-factories.adoc index 629470564ae3..f9b9ac6d67b4 100644 --- a/distribution/docs/src/main/resources/content/_developing/_devComponents/managed-service-factories.adoc +++ b/distribution/docs/src/main/resources/content/_developing/_devComponents/managed-service-factories.adoc @@ -132,10 +132,10 @@ Because there is a known issue when using OPS4J Pax Exam 4.11.0 and modifying `. * Boolean values will default to `false` if any value other than `true` is provided. * Float values must be representated in the IEEE 754 floating-point "single format" bit layout, preserving Not-a-Number (NaN) values. For example, `F"1093140480"` corresponds to `F"10.5"`. -See the documentation for https://docs.oracle.com/javase/7/docs/api/java/lang/Integer.html#parseInt(java.lang.String)[java.lang.Integer#parseInt(java.lang.String)] and https://docs.oracle.com/javase/7/docs/api/java/lang/Float.html#intBitsToFloat(int)[java.lang.Float#intBitsToFloat(int)] for more details. +See the documentation for https://docs.oracle.com/javase/7/docs/api/java/lang/Integer.html#parseInt(java.lang.String)[`java.lang.Integer#parseInt(java.lang.String)`] and https://docs.oracle.com/javase/7/docs/api/java/lang/Float.html#intBitsToFloat(int)[`java.lang.Float#intBitsToFloat(int)`] for more details. * Double values must be represented in the IEEE 754 floating-point "double format" bit layout, preserving Not-a-Number (NaN) values. For example, `D"4636745974857667812"` corresponds to `D"100.1234"`. -See the documentation for https://docs.oracle.com/javase/7/docs/api/java/lang/Long.html#parseLong(java.lang.String)[java.lang.Long#parseLong(java.lang.String)] and https://docs.oracle.com/javase/7/docs/api/java/lang/Double.html#longBitsToDouble(long)[java.lang.Double#longBitsToDouble] for more details. +See the documentation for https://docs.oracle.com/javase/7/docs/api/java/lang/Long.html#parseLong(java.lang.String)[`java.lang.Long#parseLong(java.lang.String)`] and https://docs.oracle.com/javase/7/docs/api/java/lang/Double.html#longBitsToDouble(long)[`java.lang.Double#longBitsToDouble`] for more details. ==== diff --git a/distribution/docs/src/main/resources/content/_integrating/_eventing/subscriptions.adoc b/distribution/docs/src/main/resources/content/_integrating/_eventing/subscriptions.adoc index a2a733714385..bc439e17d4e5 100644 --- a/distribution/docs/src/main/resources/content/_integrating/_eventing/subscriptions.adoc +++ b/distribution/docs/src/main/resources/content/_integrating/_eventing/subscriptions.adoc @@ -19,9 +19,9 @@ A Subscription itself is a series of events during which various plugins or tran === Evaluation -* When a metacard matching the subscription is created, updated, or deleted in any Source, each Pre-Delivery Plugin will be invoked. +* When a metacard matching the subscription is created, updated, or deleted in any Source, each Pre-Delivery Plugin will be invoked. -* If the delivery is not rejected, the associated Delivery Method callback will be invoked. +* If the delivery is not rejected, the associated Delivery Method callback will be invoked. === Update Evaluation @@ -29,10 +29,10 @@ Notably, the Catalog allows event evaluation on both the previous value (if avai === Durability -Subscription durability is not provided by the Event Processor. +Subscription durability is not provided by the Event Processor. Thus, all subscriptions are transient and will not be recreated in the event of a system restart. -It is the responsibility of Endpoints using subscriptions to persist and re-establish the subscription on startup. -This decision was made for the sake of simplicity, flexibility, and the inability of the Event Processor to recreate a fully-configured Delivery Method without being overly restrictive. +It is the responsibility of Endpoints using subscriptions to persist and re-establish the subscription on startup. +This decision was made for the sake of simplicity, flexibility, and the inability of the Event Processor to recreate a fully-configured Delivery Method without being overly restrictive. [IMPORTANT] ==== @@ -45,57 +45,57 @@ Certain endpoints, however, can persist the subscriptions on their own and recre == Creating a Subscription Currently, the Catalog reference implementation does not contain a subscription endpoint. -Therefore, an endpoint that exposes a web service interface to create, update, and delete subscriptions would provide a client's subscription filtering criteria to be used by Catalog's Event Processor to determine which events are of interest to the client. +Therefore, an endpoint that exposes a web service interface to create, update, and delete subscriptions would provide a client's subscription filtering criteria to be used by Catalog's Event Processor to determine which events are of interest to the client. The endpoint client also provides the callback URL of the event consumer to be called when an event matching the subscription's criteria is found. -This callback to the event consumer is made by a Delivery Method implementation that the client provides when the subscription is created.  -Whenever an event occurs in the Catalog matching the subscription, the Delivery Method implementation will be called by the Event Processor.  -The Delivery Method will, in turn, send the event notification out to the event consumer.  -As part of the subscription creation process, the Catalog verifies that the event consumer at the specified callback URL is available to receive callbacks. +This callback to the event consumer is made by a Delivery Method implementation that the client provides when the subscription is created. +Whenever an event occurs in the Catalog matching the subscription, the Delivery Method implementation will be called by the Event Processor. +The Delivery Method will, in turn, send the event notification out to the event consumer. +As part of the subscription creation process, the Catalog verifies that the event consumer at the specified callback URL is available to receive callbacks. Therefore, the client must ensure the event consumer is running prior to creating the subscription. -The Catalog completes the subscription creation by executing any pre-subscription Catalog Plugins, and then registering the subscription with the OSGi Service Registry. -The Catalog does not persist subscriptions by default. +The Catalog completes the subscription creation by executing any pre-subscription Catalog Plugins, and then registering the subscription with the OSGi Service Registry. +The Catalog does not persist subscriptions by default. === Event Processing and Notification -If an event matches a subscription's criteria, any pre-delivery plugins that are installed are invoked, the subscription's `DeliveryMethod` is retrieved, and its operation corresponding to the type of ingest event is invoked.  +If an event matches a subscription's criteria, any pre-delivery plugins that are installed are invoked, the subscription's `DeliveryMethod` is retrieved, and its operation corresponding to the type of ingest event is invoked. For example, the `DeliveryMethod` `created()` function is called when a metacard is created. The `DeliveryMethod` operations subsequently invoke the corresponding operation in the client's event consumer service, which is specified by the callback URL provided when the `DeliveryMethod` was created. An internal subscription tracker monitors the OSGi registry, looking for subscriptions to be added (or deleted). When it detects a subscription being added, it informs the Event Processor, which sets up the subscription's filtering and is responsible for posting event notifications to the subscriber when events satisfying their criteria are met. -The Standard Event Processor is an implementation of the Event Processor and provides the ability to create/delete subscriptions. -Events are generated by the ${ddf-catalog}Framework as metacards are created/updated/deleted and the Standard Event Processor is called since it is also a Post-Ingest Plugin. -The Standard Event Processor checks each event against each subscription's criteria. +The Standard Event Processor is an implementation of the Event Processor and provides the ability to create/delete subscriptions. +Events are generated by the ${ddf-catalog}Framework as metacards are created/updated/deleted and the Standard Event Processor is called since it is also a Post-Ingest Plugin. +The Standard Event Processor checks each event against each subscription's criteria. -When an event matches a subscription's criteria the Standard Event Processor: +When an event matches a subscription's criteria the Standard Event Processor: -* invokes each pre-delivery plugin on the metacard in the event. -* invokes the `DeliveryMethod` operation corresponding to the type of event being processed, e.g., `created()` operation for the creation of a metacard. +* invokes each pre-delivery plugin on the metacard in the event. +* invokes the `DeliveryMethod` operation corresponding to the type of event being processed, e.g., `created()` operation for the creation of a metacard. .Available Event Processor * <<{architecture-prefix}event_processor,Standard Event Processor>> ==== Using ${branding} Implementation -If applicable, the implementation of `Subscription` that comes with ${branding} should be used. -It is available at `ddf.catalog.event.impl.SubscriptionImpl` and offers a constructor that takes in all of the necessary objects. -Specifically, all that is needed is a `Filter`, `DeliveryMethod`, `Set` of source IDs, and a `boolean` for enterprise. +If applicable, the implementation of `Subscription` that comes with ${branding} should be used. +It is available at `ddf.catalog.event.impl.SubscriptionImpl` and offers a constructor that takes in all of the necessary objects. +Specifically, all that is needed is a `Filter`, `DeliveryMethod`, `Set` of source IDs, and a `boolean` for enterprise. -The following is an example code stub showing how to create a new instance of Subscription using the ${branding} implementation.  +The following is an example code stub showing how to create a new instance of Subscription using the ${branding} implementation. .Creating a Subscription [source,java,linenums] ---- // Create a new filter using an imported FilterBuilder Filter filter = filterBuilder.attribute(Metacard.ANY_TEXT).like().text("*"); -  + // Create a implementation of DeliveryMethod DeliveryMethod deliveryMethod = new MyCustomDeliveryMethod(); -  + // Create a set of source ids // This set is empty as the subscription is not specific to any sources Set sourceIds = new HashSet(); -  + // Set the isEnterprise boolean value // This subscription example should notifications from all sources (not just local) boolean isEnterprise = true; diff --git a/distribution/docs/src/main/resources/content/_managing/_configuring/content-directory-monitor.adoc b/distribution/docs/src/main/resources/content/_managing/_configuring/content-directory-monitor.adoc index b12a30317eb2..faac9a659180 100644 --- a/distribution/docs/src/main/resources/content/_managing/_configuring/content-directory-monitor.adoc +++ b/distribution/docs/src/main/resources/content/_managing/_configuring/content-directory-monitor.adoc @@ -33,8 +33,10 @@ Configuring a CDM requires adding read and write permissions to the directory be After adding permissions, a system restart is required for them to take effect. ==== +---- . permission java.io.FilePermission "", "read"; . permission java.io.FilePermission "${/}-", "read, write"; +---- Trailing slashes after have no effect on the permissions granted. For example, adding a permission for "${/}test${/}path" and "${/}test${/}path${/}" are equivalent. The recursive forms "${/}test${/}path${/}-", and "${/}test${/}path${/}${/}-" are also equivalent. @@ -103,14 +105,20 @@ Regardless of the option, the ${branding} takes each file in a monitored directo * If the original file is renamed, the old metacard is deleted and a new metacard is created. .Parallel Processing -The CDM supports parallel processing of files (up to 8 files processed concurrently). This is configured by setting the number of *Maximum Concurrent Files* in the configuration. A maximum of 8 is imposed to protect system resources. +The CDM supports parallel processing of files (up to 8 files processed concurrently). +This is configured by setting the number of *Maximum Concurrent Files* in the configuration. +A maximum of 8 is imposed to protect system resources. .Read Lock -When the CDM is set up, the directory specified is continuously scanned, and files are locked for processing based on the *ReadLock Time Interval*. This does not apply to the *Monitor in place* processing directive. Files will not be ingested without having a ReadLock that has observed no change in the file size. +When the CDM is set up, the directory specified is continuously scanned, and files are locked for processing based on the *ReadLock Time Interval*. +This does not apply to the *Monitor in place* processing directive. +Files will not be ingested without having a ReadLock that has observed no change in the file size. This is done so that files that are in transit will not be ingested prematurely. The interval should be dependent on the speed of the copy to the directory monitor (ex. network drive vs local disk). -For local files, the default value of 500 milliseconds is recommended. The recommended interval for network drives is 1000 - 2000 milliseconds. If the value provided is less than 100, 100 milliseconds will be used. +For local files, the default value of 500 milliseconds is recommended. The recommended interval for network drives is 1000 - 2000 milliseconds. +If the value provided is less than 100, 100 milliseconds will be used. It is also recommended that the *ReadLock Time Interval* be set to a lower amount of time when the *Maximum Concurrent Files* is set above 1 so that files are -locked in a timely manner and processed as soon as possible. When a higher *ReadLock Time Interval* is set, the time it takes for files to be processed is increased. +locked in a timely manner and processed as soon as possible. +When a higher *ReadLock Time Interval* is set, the time it takes for files to be processed is increased. .Attribute Overrides The CDM supports setting metacard attributes directly when ${branding} ingests a file. Custom overrides are entered in the form: @@ -147,8 +155,10 @@ To override attributes that are not part of the metacard type, <<{developing-pre .Blacklist The CDM blacklist uses the "bad.files" and "bad.file.extensions" properties from the custom.system.properties file in "etc/" in order to prevent -malicious or unwanted data from being ingested into DDF. While the CDM automatically omits hidden files, this is particularly useful when -an operating system automatically generates files that should not be ingested. One such example of this is "thumbs.db" in Windows. +malicious or unwanted data from being ingested into DDF. +While the CDM automatically omits hidden files, this is particularly useful when +an operating system automatically generates files that should not be ingested. +One such example of this is "thumbs.db" in Windows. This file type and any temporary files are included in the blacklist. .Errors diff --git a/distribution/docs/src/main/resources/content/_managing/_running/catalog-commands.adoc b/distribution/docs/src/main/resources/content/_managing/_running/catalog-commands.adoc index 4328410d6ea5..a1031ee2d852 100644 --- a/distribution/docs/src/main/resources/content/_managing/_running/catalog-commands.adoc +++ b/distribution/docs/src/main/resources/content/_managing/_running/catalog-commands.adoc @@ -47,7 +47,7 @@ Provides a list of environment variables. |Provides the various fields of a metacard for inspection. |catalog:latest -|Retrieves the latest records from the Catalog based on the Core.METACARD_MODIFIED date. +|Retrieves the latest records from the Catalog based on the `Core.METACARD_MODIFIED` date. |catalog:range |Searches by the given range arguments (exclusively). diff --git a/distribution/docs/src/main/resources/content/_managing/_running/validator-plugins.adoc b/distribution/docs/src/main/resources/content/_managing/_running/validator-plugins.adoc index 82a5abff2b81..5ad20cfd0949 100644 --- a/distribution/docs/src/main/resources/content/_managing/_running/validator-plugins.adoc +++ b/distribution/docs/src/main/resources/content/_managing/_running/validator-plugins.adoc @@ -34,10 +34,10 @@ validating that they are in the past. * *((ISO3 Country Code Validator))*: Validates an attribute's value(s) against the ISO_3166-1 Alpha3 country codes. * *((Pattern Validator))*: Validates an attribute's value(s) against a regular expression. * *((Required Attributes Metacard Validator))*: Validates that a metacard contains certain attributes. -- ID: *ddf.catalog.validation.impl.validator.RequiredAttributesMetacardValidator* +- ID: `ddf.catalog.validation.impl.validator.RequiredAttributesMetacardValidator` * *((Duplication Validator))*: Validates metacard against the local catalog for duplicates based on configurable attributes. -- ID: *org.codice.ddf.validator.metacard.duplication.DuplicationValidator* +- ID: `org.codice.ddf.validator.metacard.duplication.DuplicationValidator` * *((Relationship Validator))*: Validates values that an attribute *must have*, *can only have*, and/or *can't have*. -- ID: *ddf.catalog.validation.impl.validator.RelationshipValidator* +- ID: `ddf.catalog.validation.impl.validator.RelationshipValidator` * *((Metacard WKT Validator))*: Validates a location metacard attribute (WKT string) against valid geometric shapes. -- ID: *org.codice.ddf.validator.metacard.wkt.MetacardWktValidator* +- ID: `org.codice.ddf.validator.metacard.wkt.MetacardWktValidator` diff --git a/distribution/docs/src/main/resources/content/_reference/_tables/ContentDirectoryMonitor.adoc b/distribution/docs/src/main/resources/content/_reference/_tables/ContentDirectoryMonitor.adoc index ed6b35c578a2..45f0b97725dc 100644 --- a/distribution/docs/src/main/resources/content/_reference/_tables/ContentDirectoryMonitor.adoc +++ b/distribution/docs/src/main/resources/content/_reference/_tables/ContentDirectoryMonitor.adoc @@ -33,7 +33,7 @@ |ReadLock Time Interval |readLockIntervalMilliseconds |Integer -|Specifies the time to wait (in milliseconds) before acquiring a lock on a file in the monitored directory. This interval is used for sleeping between attempts to acquire the read lock on a file to be ingested. The default value of 100 milliseconds is recommended. +|Specifies the time to wait (in milliseconds) before acquiring a lock on a file in the monitored directory. This interval is used for sleeping between attempts to acquire the read lock on a file to be ingested. The default value of 100 milliseconds is recommended. |100 |true diff --git a/distribution/docs/src/main/resources/content/_reference/_tables/Wfs_v2_0_0_Connected_Source.adoc b/distribution/docs/src/main/resources/content/_reference/_tables/Wfs_v2_0_0_Connected_Source.adoc index 8799bf0d9cfc..ffa0b448af2d 100644 --- a/distribution/docs/src/main/resources/content/_reference/_tables/Wfs_v2_0_0_Connected_Source.adoc +++ b/distribution/docs/src/main/resources/content/_reference/_tables/Wfs_v2_0_0_Connected_Source.adoc @@ -47,7 +47,7 @@ |Disable Sorting |disableSorting |Boolean -|When selected, the system will not specify sort criteria with the query. This should only be used if the remote source is unable to handle sorting even when the capabilities states 'ImplementsSorting' is supported. +|When selected, the system will not specify sort criteria with the query. This should only be used if the remote source is unable to handle sorting even when the capabilities states 'ImplementsSorting' is supported. |false |true diff --git a/distribution/docs/src/main/resources/content/_reference/_tables/Wfs_v2_0_0_Federated_Source.adoc b/distribution/docs/src/main/resources/content/_reference/_tables/Wfs_v2_0_0_Federated_Source.adoc index fdada134ac0f..54105f7a252a 100644 --- a/distribution/docs/src/main/resources/content/_reference/_tables/Wfs_v2_0_0_Federated_Source.adoc +++ b/distribution/docs/src/main/resources/content/_reference/_tables/Wfs_v2_0_0_Federated_Source.adoc @@ -54,7 +54,7 @@ |Disable Sorting |disableSorting |Boolean -|When selected, the system will not specify sort criteria with the query. This should only be used if the remote source is unable to handle sorting even when the capabilities states 'ImplementsSorting' is supported. +|When selected, the system will not specify sort criteria with the query. This should only be used if the remote source is unable to handle sorting even when the capabilities states 'ImplementsSorting' is supported. |false |true