Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HBASE-26666 Add native TLS encryption support to RPC server/client #4125

Closed
wants to merge 2 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.ClassRule;
import org.junit.Ignore;
import org.junit.Test;
import org.junit.experimental.categories.Category;
import org.slf4j.Logger;
Expand Down Expand Up @@ -68,6 +69,7 @@ public static void tearDown() throws IOException {
Closeables.close(CLIENT, true);
}

@Ignore
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there a new reason to ignore this test with this change?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test doesn't make sense anymore, because one of the main changes in this patch is to avoid running HBase methods running on the event loop. Previously I removed the test which caused unexpected issues with the CI and as @busbey suggested, I just Ignore this test for now and remove it in a later patch.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

were you planning to remove this test now?

@Test
public void testPrivateMethodExecutedInEventLoop() throws IllegalAccessException {
assertThrows(AssertionError.class, () -> {
Expand Down
10 changes: 10 additions & 0 deletions hbase-common/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -152,6 +152,16 @@
<artifactId>kerb-simplekdc</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.bouncycastle</groupId>
<artifactId>bcprov-jdk15on</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.bouncycastle</groupId>
<artifactId>bcpkix-jdk15on</artifactId>
<scope>test</scope>
</dependency>
</dependencies>

<build>
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.exceptions;

import org.apache.yetus.audience.InterfaceAudience;

/**
* This file has been copied from the Apache ZooKeeper project.
* @see <a href=
* "https://github.com/apache/zookeeper/blob/c74658d398cdc1d207aa296cb6e20de00faec03e/zookeeper-server/src/main/java/org/apache/zookeeper/common/X509Exception.java">Base
* revision</a>
*/
@SuppressWarnings("serial")
@InterfaceAudience.Private
public class X509Exception extends Exception {

public X509Exception(String message) {
super(message);
}

public X509Exception(Throwable cause) {
super(cause);
}

public X509Exception(String message, Throwable cause) {
super(message, cause);
}

public static class KeyManagerException extends X509Exception {

public KeyManagerException(String message) {
super(message);
}

public KeyManagerException(Throwable cause) {
super(cause);
}

}

public static class TrustManagerException extends X509Exception {

public TrustManagerException(String message) {
super(message);
}

public TrustManagerException(Throwable cause) {
super(cause);
}

}

public static class SSLContextException extends X509Exception {

public SSLContextException(String message) {
super(message);
}

public SSLContextException(Throwable cause) {
super(cause);
}

public SSLContextException(String message, Throwable cause) {
super(message, cause);
}

}

}
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.io.crypto.tls;

import org.apache.yetus.audience.InterfaceAudience;

/**
* This enum represents the file type of a KeyStore or TrustStore. Currently, JKS (Java keystore),
* PEM, PKCS12, and BCFKS types are supported.
* <p/>
* This file has been copied from the Apache ZooKeeper project.
* @see <a href=
* "https://github.com/apache/zookeeper/blob/c74658d398cdc1d207aa296cb6e20de00faec03e/zookeeper-server/src/main/java/org/apache/zookeeper/common/KeyStoreFileType.java">Base
* revision</a>
*/
@InterfaceAudience.Private
public enum KeyStoreFileType {
JKS(".jks"),
PEM(".pem"),
PKCS12(".p12"),
BCFKS(".bcfks");

private final String defaultFileExtension;

KeyStoreFileType(String defaultFileExtension) {
this.defaultFileExtension = defaultFileExtension;
}

/**
* The property string that specifies that a key store or trust store should use this store file
* type.
*/
public String getPropertyValue() {
return this.name();
}

/**
* The file extension that is associated with this file type.
*/
public String getDefaultFileExtension() {
return defaultFileExtension;
}

/**
* Converts a property value to a StoreFileType enum. If the property value is <code>null</code>
* or an empty string, returns <code>null</code>.
* @param propertyValue the property value.
* @return the KeyStoreFileType, or <code>null</code> if <code>propertyValue</code> is
* <code>null</code> or empty.
* @throws IllegalArgumentException if <code>propertyValue</code> is not one of "JKS", "PEM",
* "BCFKS", "PKCS12", or empty/null.
*/
public static KeyStoreFileType fromPropertyValue(String propertyValue) {
if (propertyValue == null || propertyValue.length() == 0) {
return null;
}
return KeyStoreFileType.valueOf(propertyValue.toUpperCase());
}

/**
* Detects the type of KeyStore / TrustStore file from the file extension. If the file name ends
* with ".jks", returns <code>StoreFileType.JKS</code>. If the file name ends with ".pem", returns
* <code>StoreFileType.PEM</code>. If the file name ends with ".p12", returns
* <code>StoreFileType.PKCS12</code>. If the file name ends with ".bckfs", returns
* <code>StoreFileType.BCKFS</code>. Otherwise, throws an IllegalArgumentException.
* @param filename the filename of the key store or trust store file.
* @return a KeyStoreFileType.
* @throws IllegalArgumentException if the filename does not end with ".jks", ".pem", "p12" or
* "bcfks".
*/
public static KeyStoreFileType fromFilename(String filename) {
int i = filename.lastIndexOf('.');
if (i >= 0) {
String extension = filename.substring(i);
for (KeyStoreFileType storeFileType : KeyStoreFileType.values()) {
if (storeFileType.getDefaultFileExtension().equals(extension)) {
return storeFileType;
}
}
}
throw new IllegalArgumentException(
"Unable to auto-detect store file type from file name: " + filename);
}

/**
* If <code>propertyValue</code> is not null or empty, returns the result of
* <code>KeyStoreFileType.fromPropertyValue(propertyValue)</code>. Else, returns the result of
* <code>KeyStoreFileType.fromFileName(filename)</code>.
* @param propertyValue property value describing the KeyStoreFileType, or null/empty to
* auto-detect the type from the file name.
* @param filename file name of the key store file. The file extension is used to auto-detect
* the KeyStoreFileType when <code>propertyValue</code> is null or empty.
* @return a KeyStoreFileType.
* @throws IllegalArgumentException if <code>propertyValue</code> is not one of "JKS", "PEM",
* "PKCS12", "BCFKS", or empty/null.
* @throws IllegalArgumentException if <code>propertyValue</code>is empty or null and the type
* could not be determined from the file name.
*/
public static KeyStoreFileType fromPropertyValueOrFileName(String propertyValue,
String filename) {
KeyStoreFileType result = KeyStoreFileType.fromPropertyValue(propertyValue);
if (result == null) {
result = KeyStoreFileType.fromFilename(filename);
}
return result;
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.io.crypto.tls;

import static java.util.Objects.requireNonNull;
import static org.apache.hadoop.hbase.io.crypto.tls.X509Util.CONFIG_PREFIX;

import java.util.Arrays;
import java.util.Collections;
import java.util.List;
import javax.net.ssl.SSLContext;
import org.apache.hadoop.conf.Configuration;
import org.apache.yetus.audience.InterfaceAudience;

import org.apache.hbase.thirdparty.io.netty.handler.ssl.ClientAuth;
import org.apache.hbase.thirdparty.io.netty.handler.ssl.IdentityCipherSuiteFilter;
import org.apache.hbase.thirdparty.io.netty.handler.ssl.JdkSslContext;
import org.apache.hbase.thirdparty.io.netty.handler.ssl.SslContext;

/**
* This file has been copied from the Apache ZooKeeper project.
* @see <a href=
* "https://github.com/apache/zookeeper/blob/c74658d398cdc1d207aa296cb6e20de00faec03e/zookeeper-server/src/main/java/org/apache/zookeeper/common/SSLContextAndOptions.java">Base
* revision</a>
*/
@InterfaceAudience.Private
public class SSLContextAndOptions {
private static final String TLS_ENABLED_PROTOCOLS = CONFIG_PREFIX + "enabledProtocols";
private static final String TLS_CIPHER_SUITES = CONFIG_PREFIX + "ciphersuites";

private final String[] enabledProtocols;
private final List<String> cipherSuitesAsList;
private final SSLContext sslContext;

/**
* Note: constructor is intentionally package-private, only the X509Util class should be creating
* instances of this class.
* @param config The HBase configuration
* @param sslContext The SSLContext.
*/
SSLContextAndOptions(final Configuration config, final SSLContext sslContext) {
this.sslContext = requireNonNull(sslContext);
this.enabledProtocols = getEnabledProtocols(requireNonNull(config), sslContext);
String[] ciphers = getCipherSuites(config);
this.cipherSuitesAsList = Collections.unmodifiableList(Arrays.asList(ciphers));
}

public SSLContext getSSLContext() {
return sslContext;
}

public SslContext createNettyJdkSslContext(SSLContext sslContext, boolean isClientSocket) {
return new JdkSslContext(sslContext, isClientSocket, cipherSuitesAsList,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use SslContextBuilder instead of hard coded JdkSslContext here.

SslContextBuilder will try to use open ssl if possible, where the performance is much better than the jdk one.

IdentityCipherSuiteFilter.INSTANCE, null, ClientAuth.NONE, enabledProtocols, false);
}

private String[] getEnabledProtocols(final Configuration config, final SSLContext sslContext) {
String enabledProtocolsInput = config.get(TLS_ENABLED_PROTOCOLS);
if (enabledProtocolsInput == null) {
return new String[] { sslContext.getProtocol() };
}
return enabledProtocolsInput.split(",");
}

private String[] getCipherSuites(final Configuration config) {
String cipherSuitesInput = config.get(TLS_CIPHER_SUITES);
if (cipherSuitesInput == null) {
return X509Util.getDefaultCipherSuites();
} else {
return cipherSuitesInput.split(",");
}
}
}
Loading