Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HBASE-23095 Reuse FileStatus in StoreFileInfo #674

Merged
merged 4 commits into from
Oct 8, 2019
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,8 @@ public class StoreFileInfo {
// timestamp on when the file was created, is 0 and ignored for reference or link files
private long createdTimestamp;

private long size;

/**
* Create a Store File Info
* @param conf the {@link Configuration} to use
Expand All @@ -109,6 +111,11 @@ public class StoreFileInfo {
*/
public StoreFileInfo(final Configuration conf, final FileSystem fs, final Path initialPath)
throws IOException {
this(conf, fs, initialPath, null, null);
}

private StoreFileInfo(final Configuration conf, final FileSystem fs, final Path initialPath,
final Long createdTimestamp, final Long size) throws IOException {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about just passing a FileStatus in?

Copy link
Contributor Author

@karthikhw karthikhw Sep 30, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I initially thought the same Duo like directly passing "FileStatus" in constructor but what I feel that "Path" is also in same constructor and even that Path can be constructed from FileStatus. So that looks to me not right. :) Another important is Path can not be removed from constructor because many callers pass only Path object, not FileStatus.

I am fine Duo if you still want to change?
public StoreFileInfo(final Configuration conf, final FileSystem fs, final Path initialPath, final FileStatus filestatus)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is fine, as the FileStatus could be null. And maybe you could add a @nullable annotation to indicate this.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Passing the individual params Ts and size looks cleaner IMO. Anyways explained why selected that path.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But this requires extra boxing/unboxing? And what if we need more parameters in the future...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any way is ok. As this was not in a hot path, even that Autoboxing thing was also ok. That 2 params passing looked more clean from the calling stand point. Anyway not so big concern. This looks clear too.

Copy link
Contributor Author

@karthikhw karthikhw Oct 1, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @anoopsjohn for the confirmation.

@Apache9 Duo, I already did new commit for your suggestions. Please confirm which one you would like to keep the old or new change?
6f4baae

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Its ok.. Keep it as the way u have pushed as of now.. Its ok only.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure @anoopsjohn. Thanks again!!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Appreciate Duo @Apache9 if you get some time for review :)

assert fs != null;
assert initialPath != null;
assert conf != null;
Expand Down Expand Up @@ -136,7 +143,14 @@ public StoreFileInfo(final Configuration conf, final FileSystem fs, final Path i
" reference to " + referencePath);
} else if (isHFile(p)) {
// HFile
this.createdTimestamp = fs.getFileStatus(initialPath).getModificationTime();
if (createdTimestamp != null && size != null) {
this.createdTimestamp = createdTimestamp;
this.size = size;
} else {
FileStatus fileStatus = fs.getFileStatus(initialPath);
this.createdTimestamp = fileStatus.getModificationTime();
this.size = fileStatus.getLen();
}
this.reference = null;
this.link = null;
} else {
Expand All @@ -152,7 +166,7 @@ public StoreFileInfo(final Configuration conf, final FileSystem fs, final Path i
*/
public StoreFileInfo(final Configuration conf, final FileSystem fs, final FileStatus fileStatus)
throws IOException {
this(conf, fs, fileStatus.getPath());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, this is the problem...

this(conf, fs, fileStatus.getPath(), fileStatus.getModificationTime(), fileStatus.getLen());
}

/**
Expand Down Expand Up @@ -207,6 +221,14 @@ public StoreFileInfo(final Configuration conf, final FileSystem fs, final FileSt
this.link = link;
}

/**
* Size of the Hfile
* @return size
*/
public long getSize() {
return size;
}

/**
* Sets the region coprocessor env.
* @param coprocessorHost
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,11 @@ public void storeFile(final SnapshotRegionManifest.Builder region,
if (storeFile.isReference()) {
sfManifest.setReference(storeFile.getReference().convert());
}
sfManifest.setFileSize(storeFile.getReferencedFileStatus(rootFs).getLen());
if (!storeFile.isReference() && !storeFile.isLink()) {
sfManifest.setFileSize(storeFile.getSize());
} else {
sfManifest.setFileSize(storeFile.getReferencedFileStatus(rootFs).getLen());
}
family.addStoreFiles(sfManifest.build());
}
}
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
* http://www.apache.org/licenses/LICENSE-2.0
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.hadoop.hbase.snapshot;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hbase.HBaseClassTestRule;
import org.apache.hadoop.hbase.HBaseTestingUtility;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.*;
import org.apache.hadoop.hbase.master.snapshot.SnapshotManager;
import org.apache.hadoop.hbase.regionserver.HRegionFileSystem;
import org.apache.hadoop.hbase.regionserver.StoreFileInfo;
import org.apache.hadoop.hbase.testclassification.MasterTests;
import org.apache.hadoop.hbase.testclassification.MediumTests;
import org.apache.hadoop.hbase.util.FSUtils;
import org.junit.*;

import java.io.IOException;
import java.util.*;

import org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDataManifest;
import org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotDescription;
import org.apache.hadoop.hbase.shaded.protobuf.generated.SnapshotProtos.SnapshotRegionManifest;
import org.junit.experimental.categories.Category;

/**
* Validate if storefile length match
* both snapshop manifest and filesystem.
*/
@Category({ MasterTests.class, MediumTests.class })
public class TestSnapshotStoreFileSize {

@ClassRule
public static final HBaseClassTestRule CLASS_RULE =
HBaseClassTestRule.forClass(TestSnapshotStoreFileSize.class);

private static final HBaseTestingUtility UTIL = new HBaseTestingUtility();
private static final TableName TABLE_NAME = TableName.valueOf("t1");
private static final String SNAPSHOT_NAME = "s1";
private static final String FAMILY_NAME = "cf";
private static Configuration conf;
private Admin admin;
private FileSystem fs;

@BeforeClass
public static void setup() throws Exception {
conf = UTIL.getConfiguration();
conf.setBoolean(SnapshotManager.HBASE_SNAPSHOT_ENABLED, true);
UTIL.startMiniCluster(1);
}

@AfterClass
public static void teardown() throws Exception {
UTIL.shutdownMiniCluster();
}

@Test
public void testIsStoreFileSizeMatchFilesystemAndManifest() throws IOException {
admin = UTIL.getAdmin();
fs = UTIL.getTestFileSystem();
UTIL.createTable(TABLE_NAME, FAMILY_NAME.getBytes());
Table table = admin.getConnection().getTable(TABLE_NAME);
UTIL.loadRandomRows(table, FAMILY_NAME.getBytes(), 3, 1000);
admin.snapshot(SNAPSHOT_NAME, TABLE_NAME);

Map<String, Long> storeFileInfoFromManifest = new HashMap<String, Long>();
Map<String, Long> storeFileInfoFromFS = new HashMap<String, Long>();
String storeFileName = "";
long storeFilesize = 0L;
Path snapshotDir = SnapshotDescriptionUtils
.getCompletedSnapshotDir(SNAPSHOT_NAME, UTIL.getDefaultRootDirPath());
SnapshotDescription snapshotDesc = SnapshotDescriptionUtils.readSnapshotInfo(fs, snapshotDir);
SnapshotManifest snaphotManifest = SnapshotManifest.open(conf, fs, snapshotDir, snapshotDesc);
List<SnapshotRegionManifest> regionManifest = snaphotManifest.getRegionManifests();
for (int i = 0; i < regionManifest.size(); i++) {
SnapshotRegionManifest.FamilyFiles family = regionManifest.get(i).getFamilyFiles(0);
List<SnapshotRegionManifest.StoreFile> storeFiles = family.getStoreFilesList();
for (int j = 0; j < storeFiles.size(); j++) {
storeFileName = storeFiles.get(j).getName();
storeFilesize = storeFiles.get(j).getFileSize();
storeFileInfoFromManifest.put(storeFileName, storeFilesize);
}
}
List<RegionInfo> regionsInfo = admin.getRegions(TABLE_NAME);
Path path = FSUtils.getTableDir(UTIL.getDefaultRootDirPath(), TABLE_NAME);
for (RegionInfo regionInfo : regionsInfo) {
HRegionFileSystem hRegionFileSystem =
HRegionFileSystem.openRegionFromFileSystem(conf, fs, path, regionInfo, true);
Collection<StoreFileInfo> storeFilesFS = hRegionFileSystem.getStoreFiles(FAMILY_NAME);
Iterator<StoreFileInfo> sfIterator = storeFilesFS.iterator();
while (sfIterator.hasNext()) {
StoreFileInfo sfi = sfIterator.next();
FileStatus[] fileStatus = FSUtils.listStatus(fs, sfi.getPath());
storeFileName = fileStatus[0].getPath().getName();
storeFilesize = fileStatus[0].getLen();
storeFileInfoFromFS.put(storeFileName, storeFilesize);
}
}
Assert.assertEquals(storeFileInfoFromManifest, storeFileInfoFromFS);
}
}