-
Notifications
You must be signed in to change notification settings - Fork 52
Custom Hierarchies #1432
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: dev
Are you sure you want to change the base?
Custom Hierarchies #1432
Conversation
6c7f23a to
c692dc7
Compare
c8a68a5 to
6c87958
Compare
86d8a73 to
399e6cd
Compare
|
comment removed, updated version in comments below |
8c28fab to
605bd55
Compare
bef9c6b to
b4779a3
Compare
|
For the meshesPath (equivalently for particlesPath), I have now implemented a prototype that does the following: A path
The underlying rule: Full paths are denoted by a leading slash and are based on the data path ( Remark: The shorthand notation achieves backwards compatibility with old openPMD files |
|
One nontrivial design question is how to deal with the traditional openPMD hierarchy, especially with the paths This suggests to me that in the extended openPMD standard with custom hierarchies these paths should be treated as "nothing special". Rather, they become the canonical, but not mandatory layout/organization of a simple openPMD dataset. Two somewhat tricky consequences from this point of view: 1. There might be more than 1 meshes paths in the same group Imo, the best solution is to consider
(Note: Since
|
b0d370e to
4873e21
Compare
53f968c to
ba10099
Compare
ba10099 to
d86fa69
Compare
31c7a25 to
1d47d17
Compare
443147c to
fd7a443
Compare
033ca9f to
6da1081
Compare
dd48459 to
106a28a
Compare
6d5e1cb to
cfe8299
Compare
cfe8299 to
5812de6
Compare
5812de6 to
931619b
Compare
ed8f82c to
f28de4f
Compare
|
What's the status/schedule here? I assume that this is because the attributes in "groups" are not parsed by default. and here's what's in the file Any short/long term recommendations? |
ffb510c to
20f2cd5
Compare
| IOHandler()->enqueue(IOTask(this, pList)); | ||
| std::string version = s.openPMD(); | ||
| bool hasMeshes = false; | ||
| bool hasParticles = false; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
TODO: unused
20f2cd5 to
ce5704d
Compare
03ba4bc to
1032573
Compare
1032573 to
cbe5863
Compare
0c9f7ee to
2f620f9
Compare
1f16ac3 to
7134070
Compare
7134070 to
ea58ad1
Compare
bf9f44c to
08c421d
Compare
JSON backend: Fail when trying to open non-existing groups Insert CustomHierarchy class to Iteration Help older compilers deal with this Add vector variants of meshes/particlesPath Move meshes and particles over to CustomHierarchies class Move dirtyRecursive to CustomHierarchy Move Iteration reading logic to CustomHierarchy Move Iteration flushing logic to CustomHierarchy class Support for custom datasets Treat "meshes"/"particles" as normal subgroups Introduction of iteration["meshes"].asContainerOf<Mesh>() as a more explicit variant for iteration.meshes. Regex-based list of meshes/particlesPaths More extended testing Fix Python bindings without adding new functionality yet Overload resolution Add simple Python bindings and an example Replace Regexes with Globbing TODO: Since meshes/particles can no longer be directly addressed with this, maybe adapt the class hierarchy to disallow mixed groups that contain meshes, particles, groups and datasets at the same time. Only maybe though.. Move .meshes and .particles back to Iteration class The have their own meaning now and are no longer just carefully maintained for backwards compatibility. Instead, they are supposed to serve as a shortcut to all openPMD data found further down the hierarchy. Some fixes in read error handling More symmetric design for container types Don't write unitSI in custom datasets Discouraged support for custom datasets inside the particlesPath Fix after rebase: dirtyRecursive Fixes to the dirty/dirtyRecursive logic [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Some cleanup in CustomHierarchies class Use polymorphism for meshes/particlesPath in Python Remove hasMeshes / hasParticles logic Sort dirty files This is a workaround only, only one file should be dirty in this test. Formatting [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Fixes after rebase
08c421d to
2b708a2
Compare
The openPMD standard works by defining "what must be there", but does not impose restrictions as to "what must not be there". By this principle, openPMD is an extensible standard.
So far, standard extensions relied mostly on defining additional metadata in terms of attributes, e.g. for storing the name of the employed field solver for the ED-PIC extension. Custom hierarchies and custom n-dimensional datasets ("heavy" data in comparison to lightweight metadata) have not been employed so far despite the theoretical possibility to do so, granted by the openPMD standard. The major hindrance to such data organization has been the lacking support at the level of the openPMD-api, i.e. the implementation of the standard.
As the first part of this PR, the openPMD-api now supports writing custom-defined hierarchies and datasets within the basepath, i.e. within Iterations. This change is entirely independent from the standard as it makes use of the already existing liberty within the standard's conception as explained in the introduction.
This alone finds useful applications already:
Building on top of this, the other logical component of this PR consists in the support of this standard extension. While the PR as described so far brings custom hierarchies and datasets to the openPMD-api in a way that is transparent to the standard itself, the purpose of this next standard extension is to now make the standard aware of these hierarchies by embedding openPMD markup within them.
The schematic idea behind this is pictured below:

With this, the data organization can step back into openPMD markup from anywhere within a custom-defined hierarchy. This further extends the use of this PR to:
TODO
SharedAttributableDatapattern to better implement variable-based encoding (whereseries.iterationsandseries.iterations[0]are the same backend objects)Iteration::mesheswithIteration::mesh("subdir/E")andIteration::allMeshes() -> std::map<std::string, Mesh>, similarIteration::species("subdir/e")andIteration::allSpecies() -> std::map<std::string, ParticleSpecies>. But should it bespecies("subdir/particles/e")orspecies("subdir/e")?Attributable::openAsCustomHierarchy()?Diff: https://github.com/franzpoeschel/openPMD-api/compare/topic-remove-scalar-component..topic-custom-hierarchies