Cache attributes

[alimanfoo]

This PR adds support for caching user attributes, enabled by default. Resolves #218.

TODO:

• document attributes class (resolves DOC: Attributes class #204)
• release notes

[alimanfoo]

cc @jhamman, @mrocklin

[mrocklin]

As a general note, in the future for remote datasets we might want to minimize multiple touches of self.store. I can imagine wanting to change things like this to code like the following instead:

try:
    data = self.store[self.key[
except KeyError:
    d = {}
else:
    d = json.loads(text_type(data))

This is just a general comment though, not something that should necessarily be done here. I'll bring it up again if we find this a challenge when profiling.

[alimanfoo]

Good suggestion, asking for permission is a hard habit to break :-)

[jhamman]

@alimanfoo - is it possible or could it be possible to load all the attributes of an array/store at once? Caching is great but accessing each attribute individually will still be pretty costly.

[alimanfoo]

You can get all the attributes of an array or group in one call with o.attrs.asdict(). Btw all attributes for a single array or group are stored together in a single JSON resource, so with this caching PR all attributes of an array or group will be cached as soon as the first one is accessed, so performance should be same as manually calling asdict(). Hope that makes sense.

On Tue, 26 Dec 2017 at 20:55, Joe Hamman ***@***.***> wrote: @alimanfoo <https://github.com/alimanfoo> - is it possible or could it be possible to load all the attributes of an array/store at once? Caching is great but accessing each attribute individually will still be pretty costly. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/alimanfoo/zarr/pull/220#issuecomment-354012284>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAq8Qu76IULssdeTZuu1CNZjdVH2bICLks5tEV1BgaJpZM4RLy6r> .

-- Alistair Miles Head of Epidemiological Informatics Centre for Genomics and Global Health <http://cggh.org> Big Data Institute Building Old Road Campus Roosevelt Drive Oxford OX3 7LF United Kingdom Phone: +44 (0)1865 743596 Email: alimanfoo@googlemail.com Web: http://a <http://purl.org/net/aliman>limanfoo.github.io/ Twitter: https://twitter.com/alimanfoo

[jakirkham]

Is there a way to toggle caching on an existing object? Also is there a way to clear the cache?

[alimanfoo]

Btw I've also now exposed a put() method on the Attributes class in the public API, which should be even more efficient than update() in the case where you know you want to initially set or completely replace all attributes. put() requires only a single write operation (overwrites whatever .zattrs is there) whereas update() requires a read and a write operation (retrieves .zattrs then applies update before writing).

[alimanfoo]

@jakirkham you can turn caching off with o.attrs.cache = False. Also I've added o.attrs.refresh() to force reloading the cached attributes from the store.

[jakirkham]

Thanks @alimanfoo. That sounds great.