Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added parallel mode to utilise power of all CPUs #67

Merged
merged 21 commits into from
Mar 15, 2021
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 0 additions & 2 deletions Leanify.vcxproj
Original file line number Diff line number Diff line change
Expand Up @@ -171,7 +171,6 @@
<CompileAsManaged>false</CompileAsManaged>
<CompileAsWinRT>false</CompileAsWinRT>
<StringPooling>true</StringPooling>
<ExceptionHandling>false</ExceptionHandling>
<EnableEnhancedInstructionSet>StreamingSIMDExtensions2</EnableEnhancedInstructionSet>
<FloatingPointModel>Fast</FloatingPointModel>
<RuntimeTypeInfo>false</RuntimeTypeInfo>
Expand Down Expand Up @@ -209,7 +208,6 @@
<CompileAsManaged>false</CompileAsManaged>
<CompileAsWinRT>false</CompileAsWinRT>
<StringPooling>true</StringPooling>
<ExceptionHandling>false</ExceptionHandling>
<EnableEnhancedInstructionSet>StreamingSIMDExtensions2</EnableEnhancedInstructionSet>
<FloatingPointModel>Fast</FloatingPointModel>
<RuntimeTypeInfo>false</RuntimeTypeInfo>
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -231,7 +231,7 @@ Usage: leanify [options] paths
-f, --fastmode Fast mode, no recompression.
-q, --quiet No output to stdout.
-v, --verbose Verbose output.
-c, --concurrent Distribute all tasks to all CPUs.
-p, --parallel Distribute all tasks to all CPUs.
--keep-exif Do not remove Exif.
```

Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
40 changes: 21 additions & 19 deletions main.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
#include "leanify.h"
#include "version.h"

#include "taskflow/taskflow.hpp"
#include "lib/taskflow/taskflow.hpp"
doterax marked this conversation as resolved.
Show resolved Hide resolved

using std::cerr;
using std::cout;
Expand Down Expand Up @@ -48,14 +48,11 @@ int ProcessFile(const wchar_t* file_path) {
WideCharToMultiByte(CP_ACP, 0, file_path, -1, mbs, sizeof(mbs) - 1, nullptr, nullptr);
doterax marked this conversation as resolved.
Show resolved Hide resolved
string filename(mbs);
#else
// written like this in order to be callback function of ftw()
int ProcessFile(const char* file_path, const struct stat* sb = nullptr, int typeflag = FTW_F) {
if (typeflag != FTW_F)
return 0;
int ProcessFile(const char* file_path) {
string filename(file_path);
#endif // _WIN32

if (!concurrent_processing)
if (!parallel_processing)
cout << "Processing: " << filename << endl;
File input_file(file_path);

Expand All @@ -65,8 +62,8 @@ int ProcessFile(const char* file_path, const struct stat* sb = nullptr, int type
size_t new_size = LeanifyFile(input_file.GetFilePionter(), original_size, 0, filename);

std::string log;
if (concurrent_processing)
log = "Processing: " + filename + "\n";
if (parallel_processing)
log = "Processed: " + filename + "\n";

log +=
BuildSize(original_size) +
Expand Down Expand Up @@ -108,7 +105,7 @@ void PrintInfo() {
" -f, --fastmode Fast mode, no recompression.\n"
" -q, --quiet No output to stdout.\n"
" -v, --verbose Verbose output.\n"
" -c, --concurrent Distribute all tasks to all CPUs.\n"
" -p, --parallel Distribute all tasks to all CPUs.\n"
" --keep-exif Do not remove Exif.\n"
" --keep-icc Do not remove ICC profile.\n"
"\n"
Expand Down Expand Up @@ -204,8 +201,8 @@ int main(int argc, char** argv) {
cout.clear();
is_verbose = true;
break;
case 'c':
concurrent_processing = true;
case 'p':
parallel_processing = true;
break;
case '-':
if (STRCMP(argv[i] + j + 1, "fastmode") == 0) {
Expand All @@ -223,9 +220,9 @@ int main(int argc, char** argv) {
} else if (STRCMP(argv[i] + j + 1, "verbose") == 0) {
j += 6;
argv[i][j + 1] = 'v';
} else if (STRCMP(argv[i] + j + 1, "concurrent") == 0) {
j += 9;
argv[i][j + 1] = 'c';
} else if (STRCMP(argv[i] + j + 1, "parallel") == 0) {
j += 7;
argv[i][j + 1] = 'p';
} else if (STRCMP(argv[i] + j + 1, "keep-exif") == 0) {
j += 9;
Jpeg::keep_exif_ = true;
Expand Down Expand Up @@ -263,6 +260,12 @@ int main(int argc, char** argv) {
i += num_optargs;
}

if (parallel_processing && is_verbose) {
cerr << "Verbose logs not supported in parallel mode." << endl;
PrintInfo();
return 1;
}

if (i == argc) {
cerr << "No file path provided." << endl;
PrintInfo();
Expand All @@ -277,12 +280,11 @@ int main(int argc, char** argv) {
TraversePath(argv[i], EnqueueProcessFileTask);
} while (++i < argc);

size_t concurrent_tasks = std::thread::hardware_concurrency();

if (!concurrent_processing)
concurrent_tasks = 1;
size_t parallel_tasks = 1;
if (parallel_processing)
parallel_tasks = std::thread::hardware_concurrency();

tf::Executor executor(concurrent_tasks);
tf::Executor executor(parallel_tasks);
executor.run(taskflow).wait();

PauseIfNotTerminal();
Expand Down
2 changes: 1 addition & 1 deletion main.h
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ int iterations;
// file inside zip that is inside another zip: depth 3
int depth;
int max_depth;
bool concurrent_processing = false;
bool parallel_processing = false;


#endif // MAIN_H_