Skip to content

Commit 2b708a2

Browse files
franzpoeschelfranzpoeschel
committed
Custom hierarchies (squashed commit)
JSON backend: Fail when trying to open non-existing groups Insert CustomHierarchy class to Iteration Help older compilers deal with this Add vector variants of meshes/particlesPath Move meshes and particles over to CustomHierarchies class Move dirtyRecursive to CustomHierarchy Move Iteration reading logic to CustomHierarchy Move Iteration flushing logic to CustomHierarchy class Support for custom datasets Treat "meshes"/"particles" as normal subgroups Introduction of iteration["meshes"].asContainerOf<Mesh>() as a more explicit variant for iteration.meshes. Regex-based list of meshes/particlesPaths More extended testing Fix Python bindings without adding new functionality yet Overload resolution Add simple Python bindings and an example Replace Regexes with Globbing TODO: Since meshes/particles can no longer be directly addressed with this, maybe adapt the class hierarchy to disallow mixed groups that contain meshes, particles, groups and datasets at the same time. Only maybe though.. Move .meshes and .particles back to Iteration class The have their own meaning now and are no longer just carefully maintained for backwards compatibility. Instead, they are supposed to serve as a shortcut to all openPMD data found further down the hierarchy. Some fixes in read error handling More symmetric design for container types Don't write unitSI in custom datasets Discouraged support for custom datasets inside the particlesPath Fix after rebase: dirtyRecursive Fixes to the dirty/dirtyRecursive logic [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Some cleanup in CustomHierarchies class Use polymorphism for meshes/particlesPath in Python Remove hasMeshes / hasParticles logic Sort dirty files This is a workaround only, only one file should be dirty in this test. Formatting [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Fixes after rebase
1 parent ec449af commit 2b708a2

34 files changed

+1883
-333
lines changed

‎CMakeLists.txt‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -391,6 +391,7 @@ include(${openPMD_SOURCE_DIR}/cmake/dependencies/pybind11.cmake)
391391
set(CORE_SOURCE
392392
src/config.cpp
393393
src/ChunkInfo.cpp
394+
src/CustomHierarchy.cpp
394395
src/Dataset.cpp
395396
src/Datatype.cpp
396397
src/Error.cpp
@@ -575,6 +576,7 @@ if(openPMD_HAVE_PYTHON)
575576
src/binding/python/Attributable.cpp
576577
src/binding/python/BaseRecordComponent.cpp
577578
src/binding/python/ChunkInfo.cpp
579+
src/binding/python/CustomHierarchy.cpp
578580
src/binding/python/Dataset.cpp
579581
src/binding/python/Datatype.cpp
580582
src/binding/python/Error.cpp
@@ -743,6 +745,7 @@ set(openPMD_PYTHON_EXAMPLE_NAMES
743745
11_particle_dataframe
744746
12_span_write
745747
13_write_dynamic_configuration
748+
14_custom_hierarchy
746749
15_compression
747750
)
748751

‎examples/14_custom_hierarchy.py‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
import numpy as np
2+
import openpmd_api as io
3+
4+
5+
def main():
6+
if "bp" in io.file_extensions:
7+
filename = "../samples/custom_hierarchy.bp"
8+
else:
9+
filename = "../samples/custom_hierarchy.json"
10+
s = io.Series(filename, io.Access.create)
11+
it = s.write_iterations()[100]
12+
13+
# write openPMD part
14+
temp = it.meshes["temperature"]
15+
temp.axis_labels = ["x", "y"]
16+
temp.unit_dimension = {io.Unit_Dimension.T: 1}
17+
temp.position = [0.5, 0.5]
18+
temp.grid_spacing = [1, 1]
19+
temp.grid_global_offset = [0, 0]
20+
temp.reset_dataset(io.Dataset(np.dtype("double"), [5, 5]))
21+
temp[()] = np.zeros((5, 5))
22+
23+
# write NeXus part
24+
nxentry = it["Scan"]
25+
nxentry.set_attribute("NX_class", "NXentry")
26+
nxentry.set_attribute("default", "data")
27+
28+
data = nxentry["data"]
29+
data.set_attribute("NX_class", "NXdata")
30+
data.set_attribute("signal", "counts")
31+
data.set_attribute("axes", ["two_theta"])
32+
data.set_attribute("two_theta_indices", [0])
33+
34+
counts = data.as_container_of_datasets()["counts"]
35+
counts.set_attribute("units", "counts")
36+
counts.set_attribute("long_name", "photodiode counts")
37+
counts.reset_dataset(io.Dataset(np.dtype("int"), [15]))
38+
counts[()] = np.zeros(15, dtype=np.dtype("int"))
39+
40+
two_theta = data.as_container_of_datasets()["two_theta"]
41+
two_theta.set_attribute("units", "degrees")
42+
two_theta.set_attribute("long_name", "two_theta (degrees)")
43+
two_theta.reset_dataset(io.Dataset(np.dtype("double"), [15]))
44+
two_theta[()] = np.zeros(15)
45+
46+
47+
if __name__ == "__main__":
48+
main()

‎include/openPMD/CustomHierarchy.hpp‎

Lines changed: 256 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,256 @@
1+
/* Copyright 2023 Franz Poeschel
2+
*
3+
* This file is part of openPMD-api.
4+
*
5+
* openPMD-api is free software: you can redistribute it and/or modify
6+
* it under the terms of of either the GNU General Public License or
7+
* the GNU Lesser General Public License as published by
8+
* the Free Software Foundation, either version 3 of the License, or
9+
* (at your option) any later version.
10+
*
11+
* openPMD-api is distributed in the hope that it will be useful,
12+
* but WITHOUT ANY WARRANTY; without even the implied warranty of
13+
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
14+
* GNU General Public License and the GNU Lesser General Public License
15+
* for more details.
16+
*
17+
* You should have received a copy of the GNU General Public License
18+
* and the GNU Lesser General Public License along with openPMD-api.
19+
* If not, see <http://www.gnu.org/licenses/>.
20+
*/
21+
#pragma once
22+
23+
#include "openPMD/IO/AbstractIOHandler.hpp"
24+
#include "openPMD/Mesh.hpp"
25+
#include "openPMD/ParticleSpecies.hpp"
26+
#include "openPMD/RecordComponent.hpp"
27+
#include "openPMD/backend/Container.hpp"
28+
29+
#include <memory>
30+
#include <regex>
31+
#include <set>
32+
#include <stdexcept>
33+
#include <string>
34+
#include <type_traits>
35+
#include <utility>
36+
#include <vector>
37+
38+
namespace openPMD
39+
{
40+
class CustomHierarchy;
41+
namespace internal
42+
{
43+
enum class ContainedType
44+
{
45+
Group,
46+
Mesh,
47+
Particle
48+
};
49+
struct MeshesParticlesPath
50+
{
51+
std::regex meshRegex;
52+
std::set<std::string> collectNewMeshesPaths;
53+
std::regex particleRegex;
54+
std::set<std::string> collectNewParticlesPaths;
55+
56+
/*
57+
* These values decide which path will be returned upon use of the
58+
* shorthand notation s.iterations[0].meshes or .particles.
59+
*
60+
*/
61+
std::string m_defaultMeshesPath = "meshes";
62+
std::string m_defaultParticlesPath = "particles";
63+
64+
explicit MeshesParticlesPath() = default;
65+
MeshesParticlesPath(
66+
std::vector<std::string> const &meshes,
67+
std::vector<std::string> const &particles);
68+
MeshesParticlesPath(Series const &);
69+
70+
[[nodiscard]] ContainedType
71+
determineType(std::vector<std::string> const &path) const;
72+
[[nodiscard]] bool
73+
isParticleContainer(std::vector<std::string> const &path) const;
74+
[[nodiscard]] bool
75+
isMeshContainer(std::vector<std::string> const &path) const;
76+
};
77+
78+
struct CustomHierarchyData
79+
: ContainerData<CustomHierarchy>
80+
, ContainerData<RecordComponent>
81+
, ContainerData<Mesh>
82+
, ContainerData<ParticleSpecies>
83+
{
84+
explicit CustomHierarchyData();
85+
86+
void syncAttributables();
87+
88+
#if 0
89+
inline Container<CustomHierarchy> customHierarchiesWrapped()
90+
{
91+
Container<CustomHierarchy> res;
92+
res.setData(
93+
{static_cast<ContainerData<CustomHierarchy> *>(this),
94+
[](auto const *) {}});
95+
return res;
96+
}
97+
#endif
98+
inline Container<RecordComponent> embeddedDatasetsWrapped()
99+
{
100+
Container<RecordComponent> res;
101+
res.setData(
102+
{static_cast<ContainerData<RecordComponent> *>(this),
103+
[](auto const *) {}});
104+
return res;
105+
}
106+
inline Container<Mesh> embeddedMeshesWrapped()
107+
{
108+
Container<Mesh> res;
109+
res.setData(
110+
{static_cast<ContainerData<Mesh> *>(this),
111+
[](auto const *) {}});
112+
return res;
113+
}
114+
115+
inline Container<ParticleSpecies> embeddedParticlesWrapped()
116+
{
117+
Container<ParticleSpecies> res;
118+
res.setData(
119+
{static_cast<ContainerData<ParticleSpecies> *>(this),
120+
[](auto const *) {}});
121+
return res;
122+
}
123+
124+
#if 0
125+
inline Container<CustomHierarchy>::InternalContainer &
126+
customHierarchiesInternal()
127+
{
128+
return static_cast<ContainerData<CustomHierarchy> *>(this)
129+
->m_container;
130+
}
131+
#endif
132+
inline Container<RecordComponent>::InternalContainer &
133+
embeddedDatasetsInternal()
134+
{
135+
return static_cast<ContainerData<RecordComponent> *>(this)
136+
->m_container;
137+
}
138+
inline Container<Mesh>::InternalContainer &embeddedMeshesInternal()
139+
{
140+
return static_cast<ContainerData<Mesh> *>(this)->m_container;
141+
}
142+
143+
inline Container<ParticleSpecies>::InternalContainer &
144+
embeddedParticlesInternal()
145+
{
146+
return static_cast<ContainerData<ParticleSpecies> *>(this)
147+
->m_container;
148+
}
149+
};
150+
} // namespace internal
151+
152+
template <typename MappedType>
153+
class ConversibleContainer : public Container<MappedType>
154+
{
155+
template <typename>
156+
friend class ConversibleContainer;
157+
158+
protected:
159+
using Container_t = Container<MappedType>;
160+
using Data_t = internal::CustomHierarchyData;
161+
static_assert(
162+
std::is_base_of_v<typename Container_t::ContainerData, Data_t>);
163+
164+
ConversibleContainer(Attributable::NoInit)
165+
: Container_t(Attributable::NoInit{})
166+
{}
167+
168+
std::shared_ptr<Data_t> m_customHierarchyData;
169+
170+
[[nodiscard]] Data_t &get()
171+
{
172+
return *m_customHierarchyData;
173+
}
174+
[[nodiscard]] Data_t const &get() const
175+
{
176+
return *m_customHierarchyData;
177+
}
178+
179+
inline void setData(std::shared_ptr<Data_t> data)
180+
{
181+
m_customHierarchyData = data;
182+
Container_t::setData(std::move(data));
183+
}
184+
185+
public:
186+
template <typename TargetType>
187+
auto asContainerOf() -> ConversibleContainer<TargetType>
188+
{
189+
if constexpr (
190+
std::is_same_v<TargetType, CustomHierarchy> ||
191+
std::is_same_v<TargetType, Mesh> ||
192+
std::is_same_v<TargetType, ParticleSpecies> ||
193+
std::is_same_v<TargetType, RecordComponent>)
194+
{
195+
ConversibleContainer<TargetType> res(Attributable::NoInit{});
196+
res.setData(m_customHierarchyData);
197+
return res;
198+
}
199+
else
200+
{
201+
static_assert(
202+
auxiliary::dependent_false_v<TargetType>,
203+
"[CustomHierarchy::asContainerOf] Type parameter must be "
204+
"one of: CustomHierarchy, RecordComponent, Mesh, "
205+
"ParticleSpecies.");
206+
}
207+
}
208+
};
209+
210+
class CustomHierarchy : public ConversibleContainer<CustomHierarchy>
211+
{
212+
friend class Iteration;
213+
friend class Container<CustomHierarchy>;
214+
215+
private:
216+
using Container_t = Container<CustomHierarchy>;
217+
using Parent_t = ConversibleContainer<CustomHierarchy>;
218+
using Data_t = typename Parent_t::Data_t;
219+
220+
using EraseStaleMeshes = internal::EraseStaleEntries<Container<Mesh>>;
221+
using EraseStaleParticles =
222+
internal::EraseStaleEntries<Container<ParticleSpecies>>;
223+
void readNonscalarMesh(EraseStaleMeshes &map, std::string const &name);
224+
void readScalarMesh(EraseStaleMeshes &map, std::string const &name);
225+
void readParticleSpecies(EraseStaleParticles &map, std::string const &name);
226+
227+
protected:
228+
CustomHierarchy();
229+
CustomHierarchy(NoInit);
230+
231+
void read(internal::MeshesParticlesPath const &);
232+
void read(
233+
internal::MeshesParticlesPath const &,
234+
std::vector<std::string> &currentPath);
235+
236+
void flush_internal(
237+
internal::FlushParams const &,
238+
internal::MeshesParticlesPath &,
239+
std::vector<std::string> currentPath);
240+
void flush(std::string const &path, internal::FlushParams const &) override;
241+
242+
/**
243+
* @brief Link with parent.
244+
*
245+
* @param w The Writable representing the parent.
246+
*/
247+
void linkHierarchy(Writable &w) override;
248+
249+
public:
250+
CustomHierarchy(CustomHierarchy const &other) = default;
251+
CustomHierarchy(CustomHierarchy &&other) = default;
252+
253+
CustomHierarchy &operator=(CustomHierarchy const &) = default;
254+
CustomHierarchy &operator=(CustomHierarchy &&) = default;
255+
};
256+
} // namespace openPMD

‎include/openPMD/IO/JSON/JSONIOHandlerImpl.hpp‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -399,7 +399,8 @@ class JSONIOHandlerImpl : public AbstractIOHandlerImpl
399399

400400
// make sure that the given path exists in proper form in
401401
// the passed json value
402-
static void ensurePath(nlohmann::json *json, std::string const &path);
402+
static void
403+
ensurePath(nlohmann::json *json, std::string const &path, Access);
403404

404405
// In order not to insert the same file name into the data structures
405406
// with a new pointer (e.g. when reopening), search for a possibly

0 commit comments

Comments
 (0)