ʻO Mobiletto kahi ʻāpana abstraction storage JavaScript, me ka hoʻopunipuni ʻaoʻao o ka mea kūʻai aku.
- [No ke aha ʻo Mobiletto?](#No ke aha-Mobiletto?)
- [Hoʻomaka wikiwiki](#Hoʻomaka wikiwiki)
- Mobiletto CLI
- Source
- Hoʻokomo
- Kōkoʻo a me ke kālā
- [Hoʻohana kumu](#Hoʻohana kumu)
- Metadata
- [Kaila hoʻokomo ʻokoʻa](#Kaila-hoʻokomo ʻokoʻa)
- Ke hoʻopaʻa ʻia
- Mirroring
- Hoʻopili paʻapū
- [Hoʻololi kī](#Hoʻololi kī)
- Ke kaʻa hoʻokele
- Logging
Ua unuhi ʻia kēia palapala README.md, ma o hokeylization, i loko kākoʻo ʻia kēlā me kēia ʻōlelo e ka Unuhi Google!
Ua maopopo iaʻu ʻaʻole ia i hemolele, akā manaʻo wau ʻoi aku ka maikaʻi ma mua o ka ʻole!
🇸🇦 ʻAlapia 🇧🇩 Bengali 🇩🇪 Kelemānia 🇺🇸 Pelekania 🇪🇸 Paniolo 🇫🇷 Palani 🇹🇩 Hausa 🇮🇳 Hindi 🇮🇩 Indonesia 🇮🇹 Italia 🇯🇵 Kepani 🇰🇷 Korean 🇮🇳 Marathi 🇵🇱 Polani 🇧🇷 Pukiki 🇷🇺 Lūkia 🇰🇪 Swahili 🇵🇭 Tagalog 🇹🇷 Turkish 🇵🇰 Urdu 🇻🇳 Vietnamese 🇨🇳 Pākē
ʻO kēia unuhi kikoʻī o ka [README] kumu (https://github.com/cobbzilla/mobiletto/blob/master/README.md) hewa paha -- * mahalo nui ʻia nā hoʻoponopono!* E ʻoluʻolu e hoʻouna i noi huki ma GitHub, a inā ʻaʻole ʻoluʻolu ʻoe e hana i kēlā, wehe i kahi pilikia
Ke hana ʻoe i kahi pilikia GitHub hou e pili ana i kahi unuhi, e ʻoluʻolu e hana:
- hoʻokomo i ka URL o ka ʻaoʻao (kope/paʻi mai ka ʻaoʻao ʻaoʻao pūnaewele)
- hoʻokomo i ka kikokikona pololei i hewa (kope/paʻi mai ka polokalamu kele pūnaewele)
- ʻoluʻolu e wehewehe i ka hewa -- hewa ka unuhi? Ua haki paha ke ʻano o ke ʻano?
- ʻoluʻolu e hāʻawi i ka manaʻo o kahi unuhi ʻoi aku ka maikaʻi, a i ʻole pehea e hoʻopili pono ʻia ai ka kikokikona
- Mahalo!
Loaʻa nā API like ʻole i nā mea hoʻolako mālama kapua. ʻO ka poʻe e hoʻoikaika nei no ka "S3 compatibility" loaʻa nā ʻano idiosyncratic.
Ke koho ʻoe i kahi mea kūʻai waihona no kāu polokalamu, inā ʻoe e helu pololei i kā lākou API, kāu app i kēia manawa ke hilinaʻi nei i kēlā lawelawe. Ke hele nei ka manawa a hōʻiliʻili nā code, lilo nā mea kūʻai aku mahuahua hiki ole. Welina mai i ka honua leʻaleʻa o ka mea kūʻai lock-in!
Ua hoʻolālā ʻia ʻo Mobiletto e hoʻoponopono i kēia pilikia. Ma ka hoʻopili ʻana i kāu polokalamu i ka mobiletto's API, hiki iā ʻoe ke maʻalahi e hoʻololi i nā mea hoʻolako waihona a ʻike e ʻano like ke ʻano o ka papa waiho o kāu app.
Hoʻāʻo ʻia nā mea hoʻokele a pau no ka ʻano like me 60+ mau hoʻāʻo no kēlā me kēia kaʻa. Hoʻāʻo mākou i nā mea hoʻokele āpau me kēlā me kēia hui ʻana o:
- Hoʻopili: hoʻohana ʻia a paʻa ʻole
- Redis cache: hoʻohana ʻia a paʻa ʻole
Hāʻawi kēia ala iā mākou i ka maluhia o ka noʻonoʻo e hana like ana ʻo mobiletto me ka nānā ʻole i ka mea hoʻokele āu e hoʻohana ai, a me ka nānā ʻole inā hiki iā ʻoe ke hoʻokomo i ka cache a/a i ʻole ka hoʻopili ʻana.
Nā mea hoʻokele mālama ʻo Mobiletto i kēia manawa:
s3
: Amazon S3b2
: Backblaze B2local
: ʻōnaehana waihona kūloko
Hoʻokipa maikaʻi ʻia nā hāʻawi e kākoʻo i nā mea hoʻolako mālama kapua!
Hoʻohana ʻia ʻo Mobiletto ma ke ʻano he waihona e nā code JavaScript ʻē aʻe.
No ka hana ʻana me mobiletto ma ka laina kauoha, e hoʻohana iā mobiletto-cli
Ke hoʻāʻo nei au e lilo i mea hoʻomohala polokalamu open source. Ua hana au i loko ka ʻoihana polokalamu no nā makahiki he nui, ua hoʻomaka wau i nā hui kūleʻa a kūʻai aku iā lākou i nā hui lehulehu. I kēia mau lā ua nalowale kaʻu hana, a ʻaʻohe aʻu hana ʻē aʻe i laina
No laila e hoʻāʻo wau e kākau i nā polokalamu kōkua a ʻike inā pono ia
Inā hauʻoli ʻoe i ka hoʻohana ʻana i kēia polokalamu, e mahalo nui wau no ka liʻiliʻi loa koʻikoʻi o kēlā me kēia mahina ma o Patreon
Mahalo!
E hoʻouka me ka hoʻohana ʻana i ka npm
a i ʻole yarn
. Makemake paha ʻoe i ka mana lite
ʻaʻole i hoʻokomo i nā mea āpau
i unuhi ʻia nā faila README:
npm install mobiletto-lite
yarn add mobiletto-lite
Inā makemake ʻoe i nā faila README i kēlā me kēia ʻōlelo, e hoʻokomo i ka mana piha:
npm install mobiletto
yarn add mobiletto
He laʻana pōkole e hoʻohana ana i ka mea hoʻokele mobiletto s3
.
E holo like ana kēia code inā ʻo ka mea hoʻokele b2
a i ʻole local
.
const storage = require('mobiletto')
const bucket = await storage.connect('s3', aws_key, aws_secret, {bucket: 'bk'})
// list objects: returns array of metadata objects
const listing = await bucket.list()
const dirList = await bucket.list('some/dir/')
const everything = await bucket.list('', {recursive: true})
// write an entire file
let bytesWritten = await bucket.writeFile('some/path', someBufferOfData)
// write a file from a stream/generator
bytesWritten = await bucket.write('some/path', streamOrGenerator)
// read an entire file
// returns null if an exception would otherwise be thrown
const bufferOrNull = await bucket.safeReadFile('some/path')
// stream-read a file, passing data to callback
const bytesRead = await bucket.read('some/path', (chunk) => { ...do something with chunk... } )
// remove a file, returns the path removed
let removed = await bucket.remove('some/path') // removed is a string
// remove a directory, returns array of paths removed
removed = await bucket.remove('some/directory', {recursive: true}) // removed is now an array!
ʻO kahi laʻana nui loa, e hōʻike ana i ka hapa nui o nā hiʻohiʻona i hāʻawi ʻia:
const { mobiletto } = require('mobiletto')
// General usage
const api = await mobiletto(driverName, key, secret, opts)
// To use 'local' driver:
// * key: base directory
// * secret: ignored, can be null
// * opts object:
// * readOnly: optional, never change anything on the filesystem; default is false
// * fileMode: optional, permissions used when creating new files, default is 0600. can be string or integer
// * dirMode: optional, permissions used when creating new directories, default is 0700. can be string or integer
const local = await mobiletto('local', '/home/ubuntu/tmp', null, {fileMode: 0o0600, dirMode: '0700'})
// To use 's3' driver:
// * key: AWS Access Key ID
// * secret: AWS Secret Key
// * opts object:
// * readOnly: optional, never change anything on the bucket; default is false
// * bucket: required, name of the S3 bucket
// * region: optional, the AWS region to communicate with, default is us-east-1
// * prefix: optional, all read/writes within the S3 bucket will be under this prefix
// * delimiter: optional, directory delimiter, default is '/' (note: always '/' when encryption is enabled)
const s3 = await mobiletto('s3', aws_key, aws_secret, {bucket: 'bk', region: 'us-east-1'})
// To use 'b2' driver:
// * key: Backblaze Key ID
// * secret: Backblaze Application Key
// * opts object:
// * readOnly: optional, never change anything on the bucket; default is false
// * bucket: required, the ID (**not the name**) of the B2 bucket
// * prefix: optional, all read/writes within the B2 bucket will be under this prefix
// * delimiter: optional, directory delimiter, default is '/' (note: always '/' when encryption is enabled)
// * partSize: optional, large files will be split into chunks of this size when uploading
const b3 = await mobiletto('b2', b2_key_id, b2_app_key, {bucket: 'bk', partSize: 10000000})
// List files
api.list() // --> returns an array of metadata objects
// List files recursively
api.list({ recursive: true })
// List files in a directory
const path = 'some/path'
api.list(path)
api.list(path, { recursive: true }) // also supports recursive flag
// Visit files in a directory -- visitor function must be async
api.list(path, { visitor: myAsyncFunc })
api.list(path, { visitor: myAsyncFunc, recursive: true })
// The `list` method throws MobilettoNotFoundError if the path does not exist
// When you call `safeList` on a non-existent path, it returns an empty array
api.safeList('/path/that/does/not/exist') // returns []
// Read metadata for a file
api.metadata(path) // returns metadata object
// The `metadata` method throws MobilettoNotFoundError if the path does not exist
// When you call `safeMetadata` on a non-existent path, it returns null
api.safeMetadata('/tmp/does_not_exist') // returns null
// Read a file
// Provide a callback that writes the data someplace
const callback = (chunk) => { ... write chunk somewhere ... }
api.read(path, callback) // returns count of bytes read
// Read an entire file at once
const data = await api.readFile(path) // returns a byte Buffer of the file contents
// Read an entire file at once
// returns null if an exception would otherwise be thrown
const bufferOrNull = await bucket.safeReadFile('some/path')
// Write a file
// Provide a generator function that yields chunks of data
const generator = function* () {
while ( ... more-data-to-return ... ) {
data = ... load-data ...
yield data
}
}
local.api(path, generator) // returns count of bytes written
// Write an entire file at once (convenience method)
await api.writeFile(path, bufferOrString) // returns count of bytes written
// Delete a file
// Quiet param is optional (default false), when set errors will not be thrown if the path does not exist
// Always returns a value or throws an error.
// Return value may be a single string of the file removed, or an array of all files removed (driver-dependent)
const quiet = true
api.remove(path, {quiet}) // returns single path removed
// Recursively delete a directory and do it quietly (do not report errors)
const recursive = true
const quiet = true
api.remove(path, {recursive, quiet}) // returns array of paths removed
Hoʻihoʻi ke kauoha metadata
i ka metadata e pili ana i hoʻokahi waihona waihona waihona.
Pēlā nō, ʻo ka waiwai hoʻihoʻi mai ke kauoha list
he ʻano o nā mea metadata.
Penei ke ano o kekahi mea metadata:
{
"name": "fully/qualified/path/to/file",
"type": "entry-type",
"size": size-in-bytes,
"ctime": creation-time-epoch-millis,
"mtime": modification-time-epoch-millis
}
Hiki ke ʻano ʻano ʻano type
file
, dir
, link
, a i ʻole special
.
Ma muli o ke ʻano o ka mea hoʻokele, ʻaʻole hiki i kahi kauoha list
ke hoʻihoʻi i nā kahua āpau. ʻO ka name
a me type
pono e noho mau. E hoʻihoʻi mai kahi kauoha metadata
i nā waiwai āpau i loaʻa.
Hoʻokomo i ka module piha piha a hoʻohana i ka hana connect
:
const storage = require('mobiletto')
const opts = {bucket: 'bk', region: 'us-east-1'}
const s3 = await storage.connect('s3', aws_key, aws_secret, opts)
const objectData = await s3.readFile('some/path')
Hana maikaʻi ʻo Mobiletto me kahi cache redis .
E hoʻāʻo ʻo Mobiletto e hoʻopili i kahi ʻano redis ma 127.0.0.1:6379
Hiki iā ʻoe ke hoʻololi i kekahi o kēia:
- E hoʻonohonoho i ka
MOBILETTO_REDIS_HOST
env var, mobiletto hoʻohui ma ʻaneʻi ma kahi o ka localhost - E hoʻonoho i ka
MOBILETTO_REDIS_PORT
env var, e hoʻohana ʻia kēia awa
E mālama ʻo Mobiletto i kāna mau kī redis me ka prefix _mobiletto__
. Hiki iā ʻoe ke hoʻololi i kēia
ma ka hoʻonohonoho ʻana i ka var env MOBILETTO_REDIS_PREFIX
.
Hiki iā ʻoe ke hoʻonohonoho i ka hoʻopaʻa ʻana i kēlā me kēia pilina me ka mea opts.redisConfig
':
const redisConfig = {
enabled: true, // optional, default is true. if false other props are ignored
host: '127.0.0.1',
port: 6379,
prefix: '_mobiletto__'
}
const opts = { redisConfig, bucket: 'bk', region: 'us-east-1' }
const s3 = await storage.connect('s3', aws_key, aws_secret, opts)
No ka hoʻopau: e hoʻokomo i ka enabled: false
i kāu mea opts.redisConfig
ke hoʻokumu ʻoe i kāu pilina.
E like me ka mea i kūkākūkā ʻia ma lalo nei, ʻo ka hoʻopau ʻana i ka cache e loaʻa i kahi hopena maikaʻi ʻole i ka hana a loaʻa i nā noi hou aku i ka waihona āu e pono ai.
** Waihona i hoʻopaʻa ʻia**: ʻoi aku ka lohi o ka heluhelu/kākau ʻana i kahi waihona i hoʻopili ʻia ma mua o ka maʻamau, akā ʻo ka hoʻokele ʻana a puni nā papa kuhikuhi (kahi e hana ai kekahi mau mea) ʻoi aku ka pipiʻi. Ke hoʻohana nei i kahi cache redis e hāʻawi iā ʻoe i kahi hoʻoikaika nui.
Paʻa ka cache paʻamau, akā ʻaʻole hana maikaʻi inā he nui kāu mau hana kākau/wehe. Hoʻopau ka hana kākau a wehe paha i ka cache holoʻokoʻa, e hōʻoia ana e ʻike ʻia nā heluhelu ma hope nā hoʻololi hou.
Inā ʻoe e hoʻohana nei i kahi mea hana CLI e like me mobiletto-cli,
Makemake ʻoe e hoʻohana ʻia ka cache redis, no ka mea e mau ana ia ma nā ʻōlelo ʻana o ke kauoha mo
.
// Copy a local filesystem mobiletto to S3
s3.mirror(local)
// Mirror a local subdirectory from one mobiletto to an S3 mobiletto, with it's own subdirectory
local.mirror(s3, 'some/local-folder', 'some/s3-folder')
Hoʻohana ke kauoha mirror
i hoʻokahi kope o nā faila a pau mai kekahi mobiletto a i kekahi.
ʻAʻole ia e holo i kekahi kaʻina hana e mālama i ke aniani i ka manawa. E holo hou i ke kauoha mirror
e hoʻonohonoho i nā faila i nalowale.
ʻO ka waiwai hoʻihoʻi mai mirror
' he mea maʻalahi me nā helu helu no ka nui o nā faila i kūleʻa
aniani a me ka nui o nā faila i hewa:
{
success: count-of-files-mirrored,
errors: count-of-files-with-errors
}
'Ōlelo Aʻo: ʻO ka hoʻohālikelike ʻana i nā pūʻulu ʻikepili nui hiki ke hoʻopau i ka manawa a me ka bandwidth-nui
Me ka mirror
call semantics hiki ke huikau i kekahi manawa ke maopopo ʻo wai ka
mea heluhelu a ʻo wai ka mea kākau. E noʻonoʻo ʻoe e like me ka ʻōlelo kuhikuhi: ka "lima hema mobiletto"
ʻo ia ka mea i hāʻawi ʻia i (ka ʻikepili aniani i kākau ʻia), a me ka "mobiletto lima ʻākau" (ka
ʻO ka hoʻopaʻapaʻa i ke mirror
') ʻo ia ka waiwai i hāʻawi ʻia (heluhelu ʻia ka ʻike aniani).
E ho'ā i ka hoʻopunipuni ʻaoʻao o nā mea kūʻai aku:
// Pass encryption parameters
const encryption = {
// key is required, must be >= 16 chars
key: randomstring.generate(128),
// optional, the default is to derive IV from key
// when set, IV must be >= 16 chars
iv: randomstring.generate(128),
// optional, the default is aes-256-cbc
algo: 'aes-256-cbc'
}
const api = await mobiletto(driverName, key, secret, opts, encryption)
// Subsequent write operations will encrypt data (client side) when writing
// Subsequent read operations will decrypt data (client side) when reading
He aha ka hana? Hoʻokaʻawale ʻia kahi papa kuhikuhi "hoʻokomo" (dirent) (hoʻopili ʻia) i nā faila i loko directory (aka the dirent directory).
- Heluhelu ke kauoha
list
i nā faila hoʻokomo papa kuhikuhi, wehe i kēlā me kēia ala i helu ʻia; a laila hoʻihoʻi i ka metadata no kēlā me kēia faila - ʻOi aku ka maikaʻi ʻole o nā kauoha
list
, ʻoi aku hoʻi no nā papa kuhikuhi me ka nui o nā faila - ʻO ke kauoha
write
e kākau i nā faila i loko o ka papa kuhikuhi kuhikuhi o kēlā me kēia makua, me ka recursively; a laila kākau i ka faila - E loaʻa i nā kauoha
write
O(N) kākau, me ka N = hohonu i ka hierarchy papa kuhikuhi. - Na ke kauoha
remove
e wehe i ka faila dirent e pili ana, a me kona makua ina nele, me ka recursively; a laila wehe i ka faila - ʻO nā kauoha
remove
e hoʻihoʻi hou e loaʻa iā O(N) heluhelu a hiki i ka nui o nā holoi ʻana, me ka N = hohonu i ka hierarchy papa kuhikuhi. - Hiki ke pipiʻi i nā kauoha
remove
ma nā ʻōnaehana waihona nui a hohonu
E hoʻomanaʻo, ʻoiai me ka hoʻopili ʻana i ka ʻaoʻao o ka mea kūʻai aku, he ʻenemi me ka ʻike piha i kāu ʻaoʻao kikowaena i hoʻopili ʻia. waihona, ʻoiai me ka ʻole o ke kī, hiki ke ʻike i ka huina o nā papa kuhikuhi a me ka nui o nā faila i kēlā me kēia, a me i kekahi hana, e ʻike i kekahi a i ʻole ka ʻōnaehana holoʻokoʻa o ka hierarchy papa kuhikuhi. E hoʻomaopopo: E hoʻohana i kahi hale palahalaha no ka palekana maikaʻi. ʻAʻole ʻike ka ʻenemi i nā inoa o nā papa kuhikuhi/faila ke ʻole lākou i ʻike i kāu hoʻopunipuni kī a i ʻole ua haki ʻia ka hoʻopunipuni. Pau nā pili a pau ia manawa!
Hiki ke lohi nā hana ma ka waihona i hoʻopili ʻia. Hiki ke lohi loa nā papa inoa recursive a me ka wehe ʻana. He kōkua nui ka hoʻokahe ʻana ma o redis, akā e hoʻomanaʻo e hoʻoheheʻe ʻia ka cache ma luna o nā mea kākau a wehe paha.
E hana i mobiletto me kāu kī hou, a laila e hoʻohālike i ka ʻikepili kahiko i loko:
const storage = require('mobiletto')
const oldEncryption = { key: .... }
const oldStorage = await storage.connect('s3', aws_key, aws_secret, {bucket: 'bk', region: 'us-east-1'}, oldEncryption)
const newEncryption = { key: .... }
const newStorage = await storage.connect('s3', aws_key, aws_secret, {bucket: 'zz', region: 'us-east-1'}, newEncryption)
newStorage.mirror(oldStorage) // if oldStorage is very large, this may take a looooooong time...
ʻO ka mea hoʻokele ka waihona JS e hoʻokuʻu aku i kahi hana 'storageClient' me kēia pūlima:
function storageClient (key, secret, opts)
key
: he kaula, kāu kī API (no ka mea hoʻokelelocal
ʻo ia ka papa kuhikuhi kumu)secret
: he kaula, kāu mea huna API (hiki ke waiho ʻia no ka mea hoʻokelelocal
)opts
: he mea, aia nā waiwai i kēlā me kēia mea hoʻokele:- No
local
, nāfileMode
a medirMode
e ho'oholo i ka hana 'ana i nā faila a me nā papa kuhikuhi hou. - No
s3
, pono ka waiwaibucket
. ʻO nā waiwai koho:region
: the S3 region, default is us-east-1prefix
: a prefix to prepend to all S3 paths, default is the empty stringdelimiter
: the directory delimiter, default is '/'
Pono ka mea i hoʻihoʻi ʻia e ka storageClient i kēia mau hana:
// Test the driver before using, ensure proper configuration
async testConfig ()
// List files in path (or from base-directory)
// If recursive is true, list recursively
// If visitor is defined, it will be an async function. await the visitor function on each file found
// Otherwise, perform the listing and return an array of objects
async list (path, recursive = false, visitor = null) // path may be omitted
// Read metadata for a path
async metadata (path)
// Read a file
// callback receives a chunk of data. endCallback is called at end-of-stream
async read (path, callback, endCallback = null)
// Write a file
// driver must be able to handle a generator or a stream
async write (path, generatorOrReadableStream)
// Remove a file, or recursively delete a directory
// returns a string of a single path removed, or an array of multiple paths removed
async remove (path, recursive = false, quiet = false)
Hoʻohana ʻo Mobiletto i ka waihona logging winston.
Loaʻa i nā moʻolelo ** nā ala faila a me nā memo hewa, akā ʻaʻole ** loaʻa nā kī, nā mea huna, a i ʻole kekahi ʻike hoʻonohonoho pili.
E hoʻohana i ka MOBILETTO_LOG_LEVEL
hoʻololi kaiapuni no ka hoʻonohonoho ʻana i ka pae log, me ka hoʻohana ʻana i kekahi
o nā pae npm
i wehewehe ʻia ma https://www.npmjs.com/package/winston#logging-levels
ʻO ka pae paʻamau ʻo error
. ʻO ka pae haʻihaʻi ʻoi loa ʻo silly
, ʻoiai he mobiletto i kēia manawa
ʻaʻole i hoʻopaʻa inoa ma nā pae ma lalo o debug
MOBILETTO_LOG_LEVEL=silly # maximum logs!
Ma ka maʻamau, kākau ka mea logger i ka console. No ka hoʻouna ʻana i nā moʻolelo i kahi faila, hoʻonohonoho i ka MOBILETTO_LOG_FILE
hoʻololi kaiapuni. Ke hoʻopaʻa inoa i kahi faila, ʻaʻole e kākau hou ʻia nā lāʻau i ka console.
MOBILETTO_LOG_FILE=/var/my_mobiletto_log
E hoʻopau i ka hoʻopaʻa inoa ʻana:
MOBILETTO_LOG_FILE=/dev/null