Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Persistence redesign #4589

Merged
merged 89 commits into from
Oct 4, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
89 commits
Select commit Hold shift + click to select a range
8aff1f8
Refactor: Move FileUtil
chimp1984 Oct 1, 2020
2350710
Refactor: Move CorruptedDatabaseFilesHandler
chimp1984 Oct 1, 2020
cda9d9e
Refactor: Move other 2 classes to file package
chimp1984 Oct 1, 2020
802dcad
Refactor: Rename CorruptedDatabaseFilesHandler
chimp1984 Oct 1, 2020
d27e4dc
Refactor: Rename fields and methods, return early
chimp1984 Oct 1, 2020
6f6836f
Refactor: Add removeAndBackupFile method
chimp1984 Oct 1, 2020
85b33bc
Remove awaitTermination
chimp1984 Oct 1, 2020
92c9c3c
Add getDefaultStorageFileName method
chimp1984 Oct 1, 2020
937c006
Add PersistenceManager class
chimp1984 Oct 1, 2020
fd0cc63
Refactor: use FileUtil.removeAndBackupFile instead of FileManager.rem…
chimp1984 Oct 1, 2020
c245026
Remove Storage and FileManager from logback
chimp1984 Oct 1, 2020
93f1108
Add PersistenceManager.flushAllDataToDisk to shutdown to write all pe…
chimp1984 Oct 1, 2020
4f3ded6
Use PersistenceManager instead of Storage for import/export
chimp1984 Oct 1, 2020
35d9338
Remove UserThreadMappedPersistableList and use PersistableList instead
chimp1984 Oct 1, 2020
d3be8dd
Remove UserThreadMappedPersistableEnvelope and use PersistableEnvelop…
chimp1984 Oct 1, 2020
3a348d5
Remove ThreadedPersistableEnvelope and use PersistableEnvelope instead
chimp1984 Oct 1, 2020
2e55251
Use PersistenceManager for StoreService (not yet supported for subcla…
chimp1984 Oct 1, 2020
92b0ea9
Use PersistenceManager for MapStoreService (not yet supported for sub…
chimp1984 Oct 1, 2020
5a1cd0e
Use PersistableNetworkPayloadStore for AppendOnlyDataStoreService
chimp1984 Oct 1, 2020
e2f0f61
Update PersistableNetworkPayloadStore
chimp1984 Oct 1, 2020
0682b99
Update ProtectedDataStoreService
chimp1984 Oct 1, 2020
9494439
Use PersistenceManager in HistoricalDataStoreService
chimp1984 Oct 1, 2020
827775c
Add missing import
chimp1984 Oct 1, 2020
34d8909
Add final at ResourceDataStoreService
chimp1984 Oct 1, 2020
a4acdda
Apply generics and super class methods
chimp1984 Oct 1, 2020
dc77b5c
Use PersistenceManager
chimp1984 Oct 1, 2020
d9a7aac
Update SignedWitness classes
chimp1984 Oct 1, 2020
8781c80
Update AccountAgeWitnessStorage classes
chimp1984 Oct 1, 2020
e6211ef
Update TempProposalStore and ProposalStore classes
chimp1984 Oct 1, 2020
9190f17
Update TradeStatistics2Storage classes
chimp1984 Oct 1, 2020
074ed67
Remove lombok Delegate from PersistableList
chimp1984 Oct 1, 2020
80e16a0
Use PersistenceManager
chimp1984 Oct 1, 2020
308c970
Use PersistenceManager
chimp1984 Oct 1, 2020
878f645
Use PersistenceManager
chimp1984 Oct 1, 2020
5d6cbf5
Refactor: Move classes to new package
chimp1984 Oct 1, 2020
0eb99f7
Use PersistenceManager
chimp1984 Oct 1, 2020
8f38de3
Use PersistableList as base class
chimp1984 Oct 1, 2020
158b7a9
Remove storage from dispute. We do persistence from outside now.
chimp1984 Oct 1, 2020
8a6b2a7
Use PersistenceManager. Make disputeList final
chimp1984 Oct 1, 2020
2d91c26
Remove storage and persist calls. Remove unused methods
chimp1984 Oct 1, 2020
1e91b09
Remove persist method
chimp1984 Oct 1, 2020
63fe1c9
Remove storage
chimp1984 Oct 1, 2020
4f18577
Extend PersistableList, remvoe storage
chimp1984 Oct 1, 2020
f6be9b0
Use PersistenceManager
chimp1984 Oct 2, 2020
2582f58
Use PersistenceManager
chimp1984 Oct 2, 2020
f4165dc
Use PersistenceManager
chimp1984 Oct 2, 2020
c3c869b
Use PersistenceManager
chimp1984 Oct 2, 2020
d4f460b
Use PersistenceManager
chimp1984 Oct 2, 2020
406afa9
Use PersistenceManager. Add setAll method
chimp1984 Oct 2, 2020
fae1cda
Use PersistenceManager
chimp1984 Oct 2, 2020
986f1bf
Use PersistenceManager. We do not persist inside openOffer anymore bu…
chimp1984 Oct 2, 2020
27c2cb8
Use PersistenceManager
chimp1984 Oct 2, 2020
e0ea742
Use PersistenceManager
chimp1984 Oct 2, 2020
e825613
Use PersistenceManager
chimp1984 Oct 2, 2020
fac4200
Remove unused fields, apply new params
chimp1984 Oct 2, 2020
486e6f4
Use PersistenceManager
chimp1984 Oct 2, 2020
9da13c3
Refactor: Rename persist to requestPersistence
chimp1984 Oct 2, 2020
482bc56
Use PersistenceManager
chimp1984 Oct 2, 2020
fe4c21b
Use PersistenceManager
chimp1984 Oct 2, 2020
197e871
Remove FileManager ans Storage. All is replaces not by PersistenceMan…
chimp1984 Oct 2, 2020
42e10f2
Apply generics
chimp1984 Oct 2, 2020
a797413
Remove persist method
chimp1984 Oct 2, 2020
7b4611e
Add requestPersistence call
chimp1984 Oct 2, 2020
b901f35
Add missing persist calls. Add generics. Remove persist method. Add f…
chimp1984 Oct 2, 2020
253c6c3
Remove unneeded param. Add requestPersistence calls. Cleanup removeTr…
chimp1984 Oct 2, 2020
fa0d261
Fix remaining issues, cleanups, add persit calls, remove unused code,...
chimp1984 Oct 2, 2020
270e89a
Fix wrong params
chimp1984 Oct 2, 2020
ccfda5b
Cleanups, add generics, add final, remove unused code/comments/todos,
chimp1984 Oct 2, 2020
5f47ed2
Use decrementAndGet instead of getAndDecrement
chimp1984 Oct 2, 2020
e242c54
Refactor: Rename getTradesAsObservableList to getObservableList
chimp1984 Oct 2, 2020
e88a810
Revert exit code to 1 as it was before. Not sure if there have been s…
chimp1984 Oct 2, 2020
c789650
Move methods to base class
chimp1984 Oct 2, 2020
32238f2
Rename getClosedTradables and getFailedTrades to getObservableList
chimp1984 Oct 2, 2020
b712445
Remove duplicate startApplication call. Remove checkCryptoSetup call …
chimp1984 Oct 2, 2020
232fa8d
Execute result handler on userThread
chimp1984 Oct 2, 2020
10e9b05
Rename writeCompleted to onWriteCompleted
chimp1984 Oct 2, 2020
18f64d5
Persist all Priority.HIGH stores at shutdown to be more safe that we …
chimp1984 Oct 2, 2020
137b5e8
Enforce that clients set the priority. Remove the initialize methods …
chimp1984 Oct 2, 2020
e9db7c6
Rename Priority to Source. This should make it more clear what is the…
chimp1984 Oct 2, 2020
c9cd58a
Fix rename mistakes
chimp1984 Oct 2, 2020
6693a03
Fix test class
chimp1984 Oct 2, 2020
a612405
Update common/src/main/java/bisq/common/persistence/PersistenceManage…
chimp1984 Oct 4, 2020
3b31dc9
Fix typo
chimp1984 Oct 4, 2020
0cc0cfc
Use runAfter instead of runPeriodically
chimp1984 Oct 4, 2020
f7c2464
Fix typo
chimp1984 Oct 4, 2020
f9f33aa
Add PersistableListAsObservable
chimp1984 Oct 4, 2020
d18b243
Do not use persistNow for snapshots but requestPersistence
chimp1984 Oct 4, 2020
c797317
Fix typo
chimp1984 Oct 4, 2020
e179ee4
Use Collection for constructor which is called from the fromProto met…
chimp1984 Oct 4, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion common/src/main/java/bisq/common/crypto/KeyStorage.java
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
package bisq.common.crypto;

import bisq.common.config.Config;
import bisq.common.storage.FileUtil;
import bisq.common.file.FileUtil;

import com.google.inject.Inject;

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
* along with Bisq. If not, see <http://www.gnu.org/licenses/>.
*/

package bisq.common.storage;
package bisq.common.file;

import javax.inject.Inject;
import javax.inject.Singleton;
Expand All @@ -28,29 +28,29 @@

@Slf4j
@Singleton
public class CorruptedDatabaseFilesHandler {
private final List<String> corruptedDatabaseFiles = new ArrayList<>();
public class CorruptedStorageFileHandler {
private final List<String> files = new ArrayList<>();

@Inject
public CorruptedDatabaseFilesHandler() {
public CorruptedStorageFileHandler() {
}

public void onFileCorrupted(String fileName) {
corruptedDatabaseFiles.add(fileName);
public void addFile(String fileName) {
files.add(fileName);
}

public Optional<List<String>> getCorruptedDatabaseFiles() {
if (!corruptedDatabaseFiles.isEmpty()) {
if (corruptedDatabaseFiles.size() == 1 && corruptedDatabaseFiles.get(0).equals("ViewPathAsString")) {
log.debug("We detected incompatible data base file for Navigation. " +
"That is a minor issue happening with refactoring of UI classes " +
"and we don't display a warning popup to the user.");
return Optional.empty();
} else {
return Optional.of(corruptedDatabaseFiles);
}
} else {
public Optional<List<String>> getFiles() {
if (files.isEmpty()) {
return Optional.empty();
}

if (files.size() == 1 && files.get(0).equals("ViewPathAsString")) {
log.debug("We detected incompatible data base file for Navigation. " +
"That is a minor issue happening with refactoring of UI classes " +
"and we don't display a warning popup to the user.");
return Optional.empty();
}

return Optional.of(files);
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
* along with Bisq. If not, see <http://www.gnu.org/licenses/>.
*/

package bisq.common.storage;
package bisq.common.file;

import bisq.common.util.Utilities;

Expand All @@ -37,14 +37,12 @@
import java.util.Date;
import java.util.List;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import lombok.extern.slf4j.Slf4j;

import javax.annotation.Nullable;

@Slf4j
public class FileUtil {
private static final Logger log = LoggerFactory.getLogger(FileUtil.class);

public static void rollingBackup(File dir, String fileName, int numMaxBackupFiles) {
if (dir.exists()) {
File backupDir = new File(Paths.get(dir.getAbsolutePath(), "backup").toString());
Expand Down Expand Up @@ -203,11 +201,24 @@ public static void copyDirectory(File source, File destination) throws IOExcepti
FileUtils.copyDirectory(source, destination);
}

static File createNewFile(Path path) throws IOException {
public static File createNewFile(Path path) throws IOException {
File file = path.toFile();
if (!file.createNewFile()) {
throw new IOException("There already exists a file with path: " + path);
}
return file;
}

public static void removeAndBackupFile(File dbDir, File storageFile, String fileName, String backupFolderName)
throws IOException {
File corruptedBackupDir = new File(Paths.get(dbDir.getAbsolutePath(), backupFolderName).toString());
if (!corruptedBackupDir.exists() && !corruptedBackupDir.mkdir()) {
log.warn("make dir failed");
}

File corruptedFile = new File(Paths.get(dbDir.getAbsolutePath(), backupFolderName, fileName).toString());
if (storageFile.exists()) {
renameFile(storageFile, corruptedFile);
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,8 @@
* along with Bisq. If not, see <http://www.gnu.org/licenses/>.
*/

package bisq.common.storage;
package bisq.common.file;

import bisq.common.UserThread;
import bisq.common.util.Utilities;

import java.nio.file.Paths;
Expand All @@ -26,13 +25,12 @@
import java.io.PrintWriter;

import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;

import lombok.extern.slf4j.Slf4j;

@Slf4j
public class JsonFileManager {
private final ThreadPoolExecutor executor = Utilities.getThreadPoolExecutor("saveToDiscExecutor", 5, 50, 60);
private final ThreadPoolExecutor executor = Utilities.getThreadPoolExecutor("JsonFileManagerExecutor", 5, 50, 60);
private final File dir;


Expand All @@ -47,18 +45,12 @@ public JsonFileManager(File dir) {
if (!dir.mkdir())
log.warn("make dir failed");

Runtime.getRuntime().addShutdownHook(new Thread(() -> {
UserThread.execute(JsonFileManager.this::shutDown);
}, "WriteOnlyFileManager.ShutDownHook"));
Runtime.getRuntime().addShutdownHook(new Thread(JsonFileManager.this::shutDown,
"JsonFileManager.ShutDownHook"));
}

public void shutDown() {
executor.shutdown();
try {
executor.awaitTermination(5, TimeUnit.SECONDS);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
}

public void writeToDisc(String json, String fileName) {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
* along with Bisq. If not, see <http://www.gnu.org/licenses/>.
*/

package bisq.common.storage;
package bisq.common.file;

public class ResourceNotFoundException extends Exception {
public ResourceNotFoundException(String path) {
Expand Down
Loading