Skip to content

Commit

Permalink
feat(proxy): fallback to a global proxy list (#1388)
Browse files Browse the repository at this point in the history
  • Loading branch information
Doridian authored Dec 13, 2020
1 parent 445689e commit be1953b
Show file tree
Hide file tree
Showing 3 changed files with 36 additions and 6 deletions.
2 changes: 1 addition & 1 deletion docs/reference/application.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
| `WEB_PORT` | Starts a webserver to be able to control the bot while it is running. Setting this value starts this service. |

???+ info
You can also have a list of proxies that are rotated while searching stores. Proxies can be read from a file named `STORENAME.proxies` in the format of `socks5://username:password@ip`; one per line. In this case, there is no need to use the `PROXY_*` environments.
There is more information on proxy settings in the [Proxy documentation](proxy.md).

???+ tip
- You can also have a list of proxies that are rotated while searching stores. Proxies can be read from a file named `STORENAME.proxies` in the format of `socks5://username:password@ip`; one per line.
Expand Down
20 changes: 20 additions & 0 deletions docs/reference/proxy.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# Proxy

## Filename

Proxy configuration can be set either per store in a file called `storename.proxies` or globally in `global.proxies` in the streetmerchant root directory.

If both exist, the store specific file will take precedence.

## Format

The format is one proxy per line with the following structure:
`protocol://[user:password@]ip[:port]`

Supported protocols are `http` and `socks5`.

Valid examples include:
- `socks5://1.2.3.4:3180`
- `socks5://abcd:[email protected]:5678`
- `http://1.2.3.4:80`
- `http://abcd:[email protected]:8080`
20 changes: 15 additions & 5 deletions src/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -147,6 +147,14 @@ function envOrNumberMax(
return number ?? 0;
}

function loadProxyList(filename: string) {
return readFileSync(`${filename}.proxies`)
.toString()
.trim()
.split('\n')
.map((x) => x.trim());
}

const browser = {
isHeadless: envOrBoolean(process.env.HEADLESS),
isIncognito: envOrBoolean(process.env.INCOGNITO, false),
Expand Down Expand Up @@ -384,13 +392,15 @@ const store = {

let proxyList;
try {
proxyList = readFileSync(`${name}.proxies`)
.toString()
.trim()
.split('\n')
.map((x) => x.trim());
proxyList = loadProxyList(name);
} catch {}

if (!proxyList) {
try {
proxyList = loadProxyList('global');
} catch {}
}

return {
maxPageSleep: envOrNumberMax(
minPageSleep,
Expand Down

0 comments on commit be1953b

Please sign in to comment.