New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kvsdir_* API should provide atomicity in namespace traversal #64

Closed

garlick opened this issue Oct 16, 2014 · 0 comments

Labels

Member

garlick commented Oct 16, 2014

Walking the KVS namespace requires multiple kvs_get_*() or kvsdir_get_* calls, and each call looks up a key name from the root, thus values may be retrieved from multiple "versions" of the cache.

The hash tree organization should allow us to look up values by hash and maintain atomicity within a recursive key walk. A simple rework of the kvsdir_t / kvsdir_get_*() implementation could allow this to work without exposing hashes directly to API users.

The text was updated successfully, but these errors were encountered:

garlick added the enhancement label

garlick mentioned this issue

API: overhaul KVS interfaces #76

Closed

garlick mentioned this issue

Integrate Job Status and Control (JSC) interface #205

Merged

This was referenced Jul 1, 2015

wreck job state changes cannot be reliably collected by JSC #234

Closed

Synchronizing producers and consumers via the KVS #206

Closed

garlick mentioned this issue

kvs object store needs garbage collection #258

Closed

This was referenced Sep 10, 2016

kvs_watch is inefficient and doesn't scale well #803

Closed

improve kvs watch efficiency #810

Closed

garlick added a commit to garlick/flux-core that referenced this issue


          modules/kvs: add kvs_getat()

b11f886

Add a new function kvs_getat() which is like kvs_get() but has
a new "treeobj" argument that specifies the specific root
(or other directory) "snapshot" that the key will be looked
up in.

Partially fixes flux-framework#64

garlick added a commit to garlick/flux-core that referenced this issue


          modules/kvs: rework kvsdir_get_*() functions

4ddcef7

If a kvsdir_t was created with kvs_get_dirat(), then the
convenience functions kvsdir_get_*() operate "at" (relative to)
the original treeobj.

This means a recursive walk using kvsdir_get_dir() will
follow a consistent snapshot, unaffected by changes being
made concurrently to the KVS namespace.

Fixes flux-framework#64

garlick added a commit to garlick/flux-core that referenced this issue


          modules/kvs: add kvs_getat()

Add a new function kvs_getat() which is like kvs_get() but has
a new "treeobj" argument that specifies the specific root
(or other directory) "snapshot" that the key will be looked
up in.

Partially fixes flux-framework#64

garlick added a commit to garlick/flux-core that referenced this issue


          modules/kvs: rework kvsdir_get_*() functions

5a17908

If a kvsdir_t was created with kvs_get_dirat(), then the
convenience functions kvsdir_get_*() operate "at" (relative to)
the original treeobj.

This means a recursive walk using kvsdir_get_dir() will
follow a consistent snapshot, unaffected by changes being
made concurrently to the KVS namespace.

Fixes flux-framework#64

garlick added a commit to garlick/flux-core that referenced this issue


          modules/kvs: add kvs_getat()

3ca9aeb

Add a new function kvs_getat() which is like kvs_get() but has
a new "treeobj" argument that specifies the specific root
(or other directory) "snapshot" that the key will be looked
up in.

Partially fixes flux-framework#64

garlick added a commit to garlick/flux-core that referenced this issue


          modules/kvs: rework kvsdir_get_*() functions

e680a3e

If a kvsdir_t was created with kvs_get_dirat(), then the
convenience functions kvsdir_get_*() operate "at" (relative to)
the original treeobj.

This means a recursive walk using kvsdir_get_dir() will
follow a consistent snapshot, unaffected by changes being
made concurrently to the KVS namespace.

Fixes flux-framework#64

garlick mentioned this issue

add functions to allow stable KVS namespace traversal #820

Closed

garlick added a commit to garlick/flux-core that referenced this issue


          modules/kvs: rework kvsdir_get_*() functions

ecf6ffb

If a kvsdir_t was created with kvs_get_dirat(), then the
convenience functions kvsdir_get_*() operate "at" (relative to)
the original treeobj.

This means a recursive walk using kvsdir_get_dir() will
follow a consistent snapshot, unaffected by changes being
made concurrently to the KVS namespace.

Fixes flux-framework#64

garlick added a commit to garlick/flux-core that referenced this issue


          modules/kvs: add kvs_getat()

a9eb862

Add a new function kvs_getat() which is like kvs_get() but has
a new "treeobj" argument that specifies the specific root
(or other directory) "snapshot" that the key will be looked
up in.

Partially fixes flux-framework#64

garlick added a commit to garlick/flux-core that referenced this issue


          modules/kvs: add kvs_get_dirat, kvs_get_symlinkat

a63fe88

Add kvs_get_dirat() and kvs_get_symlinkat(), which are
like kvs_get_dir() and kvs_get_symlink() but lookup 'key'
starting at 'treeobj'.

These functions are sufficient to be able to walk a KVS
snapshot recursively.  To further facilitate a recursive
snapshot walk, if a kvsdir_t has the FLUX_KVSDIR_SNAPSHOT
flag (added by kvs_get_dirat, inherited across kvsdir_get_dir)
then the convenience functions kvsdir_get_*() operate "at"
(relative to) the original treeobj and kvsidr_put_*() functions
fail with errno set to EROFS.

A recursive walk using kvsdir_get_dir() will follow a consistent
snapshot, unaffected by changes being made concurrently to the
KVS namespace, which fixes flux-framework#64.

Caching the number of objects in the directory seems
like probably a non-useful optimization.  Drop it.

garlick added a commit to garlick/flux-core that referenced this issue


          modules/kvs: add kvs_getat()

f5c3808

Add a new function kvs_getat() which is like kvs_get() but has
a new "treeobj" argument that specifies the specific root
(or other directory) "snapshot" that the key will be looked
up in.

Partially fixes flux-framework#64

garlick added a commit to garlick/flux-core that referenced this issue


          modules/kvs: add kvs_get_dirat, kvs_get_symlinkat

d29bba8

Add kvs_get_dirat() and kvs_get_symlinkat(), which are
like kvs_get_dir() and kvs_get_symlink() but lookup 'key'
starting at 'treeobj'.

These functions are sufficient to be able to walk a KVS
snapshot recursively.  To further facilitate a recursive
snapshot walk, if a kvsdir_t has the FLUX_KVSDIR_SNAPSHOT
flag (added by kvs_get_dirat, inherited across kvsdir_get_dir)
then the convenience functions kvsdir_get_*() operate "at"
(relative to) the original treeobj and kvsidr_put_*() functions
fail with errno set to EROFS.

A recursive walk using kvsdir_get_dir() will follow a consistent
snapshot, unaffected by changes being made concurrently to the
KVS namespace, which fixes flux-framework#64.

Caching the number of objects in the directory seems
like probably a non-useful optimization.  Drop it.

garlick added a commit to garlick/flux-core that referenced this issue


          modules/kvs: add kvs_getat()

0de887d

Add a new function kvs_getat() which is like kvs_get() but has
a new "treeobj" argument that specifies the specific root
(or other directory) "snapshot" that the key will be looked
up in.

Partially fixes flux-framework#64

garlick added a commit to garlick/flux-core that referenced this issue


          modules/kvs: add kvs_get_dirat()

16c18f0

Add kvs_get_dirat() which is like kvs_get_dir() but looks
up 'key' starting at 'treeobj'.

If a kvsdir_t was obtained with kvs_get_dirat(), then the
convenience functions kvsdir_get_*() operate "at" (relative to)
the original treeobj and kvsidr_put_*() functions fail with
errno set to EROFS.

A recursive walk using kvsdir_get_dir() will follow a consistent
snapshot, unaffected by changes being made concurrently to the
KVS namespace, which fixes flux-framework#64.

garlick added a commit to garlick/flux-core that referenced this issue


          modules/kvs: add kvs_get_dirat()

de48dc2

Add kvs_get_dirat(), which is like kvs_get_dir() but looks
up key relative to a snapshot reference.

The snapshot reference is stored in the returned kvsdir_t object,
and is then used in subsequent kvsdir_get_*() operations on that
object.

It is thus possible to recursively walk a snapshot of a
section of the namespace by starting with kvsdir_get_dirat().

Fixes flux-framework#64.

garlick added a commit to garlick/flux-core that referenced this issue


          modules/kvs: add kvs_getat()

98651b8

Add a new function kvs_getat() which is like kvs_get() but has
a new "treeobj" argument that specifies the specific root
(or other directory) "snapshot" that the key will be looked
up in.

Partially fixes flux-framework#64

garlick added a commit to garlick/flux-core that referenced this issue


          modules/kvs: add kvs_get_dirat()

d6298d5

Add kvs_get_dirat(), which is like kvs_get_dir() but looks
up key relative to a snapshot reference.

The snapshot reference is stored in the returned kvsdir_t object,
and is then used in subsequent kvsdir_get_*() operations on that
object.  It is thus possible to recursively walk a snapshot of a
section of the namespace by starting with kvs_get_dirat() and
looking up subdirectories with kvsdir_get_dir().

Fixes flux-framework#64.

garlick added a commit to garlick/flux-core that referenced this issue


          modules/kvs: add kvs_get_dirat()

8edd95e

Add kvs_get_dirat(), which is like kvs_get_dir() but looks
up key relative to a snapshot reference.

The snapshot reference is stored in the returned kvsdir_t object,
and is then used in subsequent kvsdir_get_*() operations on that
object.  It is thus possible to recursively walk a snapshot of a
section of the namespace by starting with kvs_get_dirat() and
looking up subdirectories with kvsdir_get_dir().

Fixes flux-framework#64.

garlick mentioned this issue

kvs: add kvs_getat() and related functions #824

Merged

garlick added a commit to garlick/flux-core that referenced this issue


          modules/kvs: add kvs_getat()

89438b5

Add a new function kvs_getat() which is like kvs_get() but has
a new "treeobj" argument that specifies the specific root
(or other directory) "snapshot" that the key will be looked
up in.

Partially fixes flux-framework#64

garlick added a commit to garlick/flux-core that referenced this issue


          modules/kvs: add kvs_get_dirat()

b8da795

Add kvs_get_dirat(), which is like kvs_get_dir() but looks
up key relative to a snapshot reference.

The snapshot reference is stored in the returned kvsdir_t object,
and is then used in subsequent kvsdir_get_*() operations on that
object.  It is thus possible to recursively walk a snapshot of a
section of the namespace by starting with kvs_get_dirat() and
looking up subdirectories with kvsdir_get_dir().

Fixes flux-framework#64.

garlick added a commit to garlick/flux-core that referenced this issue


          modules/kvs: add kvs_get_dirat()

8ba169f

Add kvs_get_dirat(), which is like kvs_get_dir() but looks
up key relative to a snapshot reference.

The snapshot reference is stored in the returned kvsdir_t object,
and is then used in subsequent kvsdir_get_*() operations on that
object.  It is thus possible to recursively walk a snapshot of a
section of the namespace by starting with kvs_get_dirat() and
looking up subdirectories with kvsdir_get_dir().

Fixes flux-framework#64.

garlick added a commit to garlick/flux-core that referenced this issue


          modules/kvs: add kvs_get_dirat()

fc905ee

Add kvs_get_dirat(), which is like kvs_get_dir() but looks
up key relative to a snapshot reference.

The snapshot reference is stored in the returned kvsdir_t object,
and is then used in subsequent kvsdir_get_*() operations on that
object.  It is thus possible to recursively walk a snapshot of a
section of the namespace by starting with kvs_get_dirat() and
looking up subdirectories with kvsdir_get_dir().

Fixes flux-framework#64.

grondo closed this as completed in

166dc00

garlick added a commit to garlick/flux-core that referenced this issue


          build: require python-yaml >= 3.11.0

1447bfe

Add a configure check for PyYAML.  Version 3.11 was released
in 2014 and not much has changed since then, so it seems like
a good minimum version to start with.

From https://pyyaml.org/wiki/PyYAML:

3.13 (2018-07-05)
* Rebuild wheels using latest Cython for Python 3.7 support.

3.12 (2016-08-28)
* Wheel packages for Windows binaries.
* Adding an implicit resolver to a derived loader should not affect
  the base loader (fixes issue flux-framework#57).
* Uniform representation for OrderedDict across different versions
  of Python (fixes issue flux-framework#61).
* Fixed comparison to None warning (closes issue flux-framework#64).

3.11 (2014-03-26)
* Source and binary distributions are rebuilt against the latest
  versions of Cython and LibYAML.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment