Skip to content

Commit

Permalink
update README.md and help message
Browse files Browse the repository at this point in the history
  • Loading branch information
BWbwchen committed Dec 11, 2021
1 parent a823c17 commit 1d3cb2c
Show file tree
Hide file tree
Showing 2 changed files with 97 additions and 22 deletions.
112 changes: 93 additions & 19 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,28 +7,14 @@
This is an easy-to-use Map Reduce Go framework inspired by [2021 6.824 lab1](http://nil.csail.mit.edu/6.824/2021/labs/lab-mr.html).

## Feature
- Multiple workers on single machine right now.
- Multiple workers goroutine in a program on a single machine.
- Multiple workers process in separate program on a single machine.
- Fault tolerance.
- Easy to parallel your code with just Map and Reduce function.

## Usage

## Library Usage - Your own map and reduce function
Here's a simply example for word count program.

main.go
```golang
package main

import (
mp "github.com/BWbwchen/MapReduce"
)

func main() {
mp.StartSingleMachineJob(mp.ParseArg())
}
```

And write down your own Map and Reduce function.

wc.go
```golang
package main
Expand Down Expand Up @@ -56,16 +42,104 @@ func Reduce(key string, values []string, ctx worker.MrContext) {
}
```

### Usage - 1 program with master, worker goroutine

main.go
```golang
package main

import (
mp "github.com/BWbwchen/MapReduce"
)

func main() {
mp.StartSingleMachineJob(mp.ParseArg())
}
```

Run with :
```
# Compile plugin
go build -race -buildmode=plugin -o wc.so wc.go
go run -race main.go -i 'txt/*' -p 'wc.so' -r 1 -w 8
# Word count
go run -race main.go -i 'input/files' -p 'wc.so' -r 1 -w 8
```

Output file name is `mr-out-0.txt`

More example can be found in the [`mrapps/`](mrapps/) folder, and we will add more example in the future.

## Usage - Master program, and worker program (Isolate master and workers)

master.go
```golang
package main

import (
mp "github.com/BWbwchen/MapReduce"
)

func main() {
mp.StartMaster(mp.ParseArg())
}
```

worker.go
```golang
package main

import (
mp "github.com/BWbwchen/MapReduce"
)

func main() {
mp.StartWorker(mp.ParseArg())
}
```

Run with :
```
# Compile plugin
go build -race -buildmode=plugin -o wc.so wc.go
# Word count
go run -race cmd/master.go -i 'txt/*' -p 'cmd/wc.so' -r 1 -w 8 &
sleep 1
go run -race cmd/worker.go -i 'txt/*' -p 'cmd/wc.so' -r 1 -w 1 &
go run -race cmd/worker.go -i 'txt/*' -p 'cmd/wc.so' -r 1 -w 2 &
go run -race cmd/worker.go -i 'txt/*' -p 'cmd/wc.so' -r 1 -w 3 &
go run -race cmd/worker.go -i 'txt/*' -p 'cmd/wc.so' -r 1 -w 4 &
go run -race cmd/worker.go -i 'txt/*' -p 'cmd/wc.so' -r 1 -w 5 &
go run -race cmd/worker.go -i 'txt/*' -p 'cmd/wc.so' -r 1 -w 6 &
go run -race cmd/worker.go -i 'txt/*' -p 'cmd/wc.so' -r 1 -w 7 &
go run -race cmd/worker.go -i 'txt/*' -p 'cmd/wc.so' -r 1 -w 8
```



## Help
```
MapReudce is an easy-to-use Map Reduce Go parallel-computing framework inspired by 2021 6.824 lab1.
It supports multiple workers threads on a single machine and multiple processes on a single machine right now.
Usage:
mapreduce [flags]
Flags:
-h, --help help for mapreduce
-m, --inRAM Whether write the intermediate file in RAM (default true)
-i, --input strings Input files
-p, --plugin string Plugin .so file
--port int Port number (default 10000)
-r, --reduce int Number of Reducers (default 1)
-w, --worker int Number of Workers(for master node)
ID of worker(for worker node) (default 4)
```

## Contributions
Pull requests are always welcome!


<sup>
Made by Bo-Wei Chen. All code is
Expand Down
7 changes: 4 additions & 3 deletions util.go
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,9 @@ func ParseArg() ([]string, string, int, int, bool) {
var port int64
var rootCmd = &cobra.Command{
Use: "mapreduce",
Short: "mapreduce is a easy-to-use parallel framework",
Long: `mapreduce is a easy-to-use parallel framework`,
Short: "MapReduce is an easy-to-use parallel framework by Bo-Wei Chen(BWbwchen)",
Long: `MapReudce is an easy-to-use Map Reduce Go parallel-computing framework inspired by 2021 6.824 lab1.
It supports multiple workers threads on a single machine and multiple processes on a single machine right now.`,
Run: func(cmd *cobra.Command, args []string) {
tempFiles := []string{}

Expand Down Expand Up @@ -53,7 +54,7 @@ func ParseArg() ([]string, string, int, int, bool) {
rootCmd.PersistentFlags().StringVarP(&plugin, "plugin", "p", "", "Plugin .so file")
rootCmd.MarkPersistentFlagRequired("plugin")
rootCmd.PersistentFlags().Int64VarP(&nReducer, "reduce", "r", 1, "Number of Reducers")
rootCmd.PersistentFlags().Int64VarP(&nWorker, "worker", "w", 4, "Number of Workers")
rootCmd.PersistentFlags().Int64VarP(&nWorker, "worker", "w", 4, "Number of Workers(for master node)\nID of worker(for worker node)")
rootCmd.PersistentFlags().Int64Var(&port, "port", 10000, "Port number")
rootCmd.PersistentFlags().BoolVarP(&inRAM, "inRAM", "m", true, "Whether write the intermediate file in RAM")

Expand Down

0 comments on commit 1d3cb2c

Please sign in to comment.