Redesign GeoAPI

[FObermaier]

If I understood @DGuidi correctly, GeoAPI started off by trying to implement some early OGC - GeoAPI specification.

Due to lack of resources it never got that far and instead it ended up as a mix of early GeoAPI (CoordinateSystem namespace ) along with a set of interfaces extracted from early NetTopologySuite classes (matching JTS 1.7, iirc).

Now there is the Support Spatial Datatypes issue around. In the course of the discussion someone suggested to use NetTopologySuite/GeoAPI to achive that, and currently it seems at least to be investigated as an option.

I propose the following:

• Create a core GeoAPI library that contains all necessary interfaces to describe and persist geometries
• Create an additional GeoAPI library that covers all the spatial operations on those core GeoAPI geometries.

Are there other ideas around?

[DGuidi]

I'd like to have a "core api" that is considered as standard (if used by EF.Core) and that GeoAPI extends.
Not sure it's a viable option in short times.

[airbreather]

I suggest we start by enumerating the actual use cases that the redesigned GeoAPI should support (i.e., the different candidate implementations of the framework, perhaps NTS + EFCore (+ others?)), as well as any other requirements we have (esp. re: JTS similarity), and then letting those drive the design.

I agree that a proper redesign would be an expensive effort.

[xivk]

I agree we can do this as long as we don't change the purpose of GeoAPI to a library that contains entities for Entity Framework, I, and I'm guessing many others, never use ER-mappers.

If we can accomodate the usecase without:

• Adding a new dependency.
• Adding a dependency on a certain version the .NET framework that will block us in the future.
• Adding attributes to all the classes or something similar.

There's also the question of the need for a seperate GeoAPI library if we do this, why not just use NTS. The other way around is also valid, it looks to me like if we do this in GeoAPI there won't be much need for NTS anymore because we will have geometries/features and their operations here. All we would need to do then is port over the IO portions.

Perhaps we need to think about a strategy for NTS/GeoAPI going forward as @airbreather suggests?

I also think chances are small that GeoAPI get's adopted as part of this spatial data types support. Everyone seems to wait for Microsoft to fix things for them. Not many people are really open-source minded, meaning up for contribution/helping.

Also something I learned from this issue, we need better documentation and also some governance rules for this project so people trust it more. Perhaps we should consider becoming part of the .NET foundation. That would put more trust in the project longer term.

[DGuidi]

as long as we don't change the purpose of GeoAPI to a library that contains entities for Entity Framework

Absolutely agree with this.
GeoAPI now is coupled with NTS implementation
EF.Core maybe can define it's own interface, and both (EF interface and GeoAPI) can share a simpler, cleaner and common interface.

I also think chances are small that GeoAPI get's adopted as part of this spatial data types support.

Yep, sadly I agree too.

we need better documentation and also some governance rules for this project so people trust it more

Personally, I need more time for this project but, again, I agree

[airbreather]

EF.Core maybe can define it's own interface, and both (EF interface and GeoAPI) can share a simpler, cleaner and common interface.

This sounds similar to what @FObermaier originally opened with. Is this something that anyone actually needs or even wants? What applications would actually use this simpler "common" interface when they still need to take a dependency on either EF or GeoAPI anyway? If nobody uses it, then it's wasted.

I admit that I haven't thought about this too much, but at the moment, I think that at most, the consumers of an API like that would be some intermediate library that only needs to store spatial types as properties on some classes and/or method parameters on some interfaces that have no implementations. That doesn't sound compelling to me. A perfectly reasonable alternative would be to store WKB or WKT.

If we want to go the "greenfield" route, I think we absolutely need to make sure that we don't clutter up the "core" interfaces. Unlike with classes, API design for interfaces should try to minimize the number of members, because all consumers need to implement all members.

We should also try to keep an eye to being able to efficiently extract the description out of these interface types. These instances can be massive, and having to choose between millions of GetX / GetY virtual calls or downcasting to get a more efficient code path is annoying.

interface IGeometry
{
    // this or maybe return a C# 7 tuple, or even a ValueEnvelope struct.
    // IMO, it should *not* force anyone to heap-allocate.
    void GetExtents(out double minX, out double maxX, out double minY, out double maxY);

    // everything below this line in IGeometry is a rough idea: maybe none of it makes sense.
    int WKBLengthInBytes { get; }
    void WriteWKB(Span<byte> span);

    Task WriteWKBAsync(Stream stream, CancellationToken cancellationToken = default);

    Task WriteWKTAsync(TextWriter writer, CancellationToken cancellationToken = default);
}

interface IPoint : IGeometry
{
    double X { get; }
    double Y { get; }
}

[StructLayout(LayoutKind.Sequential)]
struct ValueCoordinate
{
    // yes, these are public fields.  it matters.
    double X;
    double Y;
}

interface ICoordinateSequence
{
    int Count { get; }
    double GetX(int index);
    double GetY(int index);
    void CopyTo(int sourceIndex, Span<ValueCoordinate> destination);

    // see comment in IGeometry's similar method.
    void GetExtents(out double minX, out double maxX, out double minY, out double maxY);
}

interface ILineString : IGeometry
{
    ICoordinateSequence CoordinateSequence { get; }
}

// and so on, along the same lines, trying to keep everything to a bare minimum.

[airbreather]

See also some discussion on dotnet/corefx#12034.

[airbreather]

Also, wouldn't EF Core still need the things like Union, Intersection, etc., in order to be able to do their expression tree magic anyway?

[veikkoeeva]

I came via https://github.com/dotnet/corefx/issues/12034 to NetTopology and noticed this discussion. If I may, I could have some design food for mind.

Some premises so you can see how I approach this problem:

1. I know very little about GIS applications, but I want to learn more. Especially because of EU INSPIRE and other related reasons.
2. Indeed, class and interface designs can get bloated and there perhaps are ways to avoid that.
3. One should avoid hard dependencies to persistence and serialization technologies. This seem to be very common for what I see after taking a bit of time to tour around .NET GIS (and other language) libraries.
4. NET Standard 2.0 compatible GIS library (considering WebAssembly) is sorely, very sorely needed especially now that the [EU INSPIRE])(https://inspire.ec.europa.eu/) is starting to kick in by force. It's not just GIS, but obligation to provide BIM data (and associated IoT) and this could be even bigger loss for .NET than Java on server side.

I'll start with a thought with respect to the proposed IGeometry design. It seems to be more and more common design is to provide a minimal set of concrete types with a minimum of functionality that makes sense and provide the rest as extension methods. For the example design sketch, this could perhaps mean that there is:

public static class WkbGeometryOrSomesuch
{
     public static ValueCoordinate DoWkbThing(this ValueCoordinate coordinate, int parameter);
}

This way whatever operations there are for ValueCoordinate, they could grouped by a theme and exposed as needed from extra packages and namespaces. A design where functions operate only on input parameters and all the effects are via return values is also on the easier side to test (e.g. with tools such as FsCheck or IntellTest and BenchMarkDotnet).

Similarily for the streaming the design could be along the lines of

    /// <summary>
    /// A canonical interface for a deserializer.
    /// </summary>
    public interface IDeserializer
    {
        /// <summary>
        /// Can this provider stream data.
        /// </summary>
        bool CanStream { get; }

        /// <summary>
        /// Deserializes the given data.
        /// </summary>
        /// <param name="data">The data to be serialized.</param>
        /// <param name="type">The type of the data.</param>
        /// <returns>The deserialized object.</returns>
        object Deserialize(object data, Type type);

        /// <summary>
        /// Deserializes the given data from a stream.
        /// </summary>
        /// <param name="dataStream">The stream from which to serialize.</param>
        /// <param name="type">The type of the data.</param>
        /// <returns>The deserialized object.</returns>
        object Deserialize(Stream dataStream, Type type);

        /// <summary>
        /// Deserializes the given data from a text stream.
        /// </summary>
        /// <param name="reader">The text stream from which to serialize.</param>
        /// <param name="type">The type of the data.</param>
        /// <returns>The deserialized object.</returns>
        object Deserialize(TextReader reader, Type type);
}

public class DefaultJsonSerializer: ISerializer
{
        /// <summary>
        /// The serializer this uses for reference purposes.
        /// </summary>
        public JsonSerializer Serializer { get; }

        /// <summary>
        /// The settings this serializer uses for reference purposes.
        /// </summary>
        public JsonSerializerSettings Settings { get; }

        /// <summary>
        /// <see cref="IStorageSerializer.CanStream"/>
        /// </summary>
        public bool CanStream { get; } = true;

        /// <summary>
        /// Constructs this serializer from the given parameters.
        /// </summary>
        /// <param name="settings">The settings which was used to construct a JSON serializer.</param>
        public DefaultJsonSerializer(JsonSerializerSettings settings)
        {
            if(settings == null)
            {
                throw new ArgumentNullException(nameof(settings));
            }

            Serializer = JsonSerializer.Create(settings);
            Settings = settings;
        }


        /// <summary>
        /// <see cref="ISerializer.Serialize(Stream, object)"/>
        /// </summary>
        public object Serialize(Stream dataStream, object data)
        {
            using(var streamWriter = new StreamWriter(dataStream))
            {
                using(JsonTextWriter writer = new JsonTextWriter(streamWriter))
                {
                    Serializer.Serialize(writer, data);
                    writer.Flush();
                }
            }

            return dataStream;
        }


        /// <summary>
        /// <see cref="ISerializer.Serialize(object)"/>
        /// </summary>
        public object Serialize(object data)
        {
            return JsonConvert.SerializeObject(data, Settings);
        }


        /// <summary>
        /// <see cref="ISerializer.Serialize(TextWriter, object)"/>.
        /// </summary>
        public object Serialize(TextWriter writer, object data)
        {
            using(JsonTextWriter jsonWriter = new JsonTextWriter(writer))
            {
                Serializer.Serialize(jsonWriter, data);
                jsonWriter.Flush();
            }

            return writer;
        }
}

This could provide also Task interfaces (and the design could be further refined). The gist of the idea is to allow a system where one can fit multiple concrete implementations of one case (such as JSON.NET or Jil, say) or even completely different ways of (de)serialiazing and even chaining them together. In fact, these are from Orleans and there are more (de)serializer implementations here.

Orleans is a good vehicle to go around with concrete implementations, ideas and perhaps collaboration ideas. It looks to be it could very suitable caching and even operational GIS application backend and it has been certainly discussed as such in the past, which I'll get back to in a minute too.

Orleans is an example of a system which has multiple persistence backends without having a hard dependency on none. As is discussed here and in the cross-referenced CoreFX thread and one could wonder that why take a hard dependency on any concrete implementation if it can be avoid. As it turns out, there can be other benefits too, I'll get to them in a minute too.

In Orleans, the main abstraction for persistence is IStorage. This has allowed the community to rally around Orleans and provide multipe backends taking advantage of any technology, organization or deployment properties needed for the case and consume it via a common abstraction. As a concrete implementation out of the many, I'll take ADO.NET.

The way ADO.NET abstraction works is that it sends both the data and the script that executes it to the database. The ADO.NET implementation only expects certain parameter names with certain types are present and provides a default table layout and queries to operate on them. Orleans loads the queries by key into memory and they can be changed, even on the fly, as well as the table layout. The main script (for SQL Server) can be seen here, and specifically for persistence here.

Working like this, Orleans could save GIS types, say, as JSON by being completely oblivious they are a special type. But if needed, the script SQLServer-Persistence.sql could be modified so that an extra column is added to Storage table, or perhaps another table, and the queries would be changes so that they include a conditional clause for checking certain types and reroute the INSERT and UPDATE queries accordingly. This might be important also in such a way that working like this allows one to divide hot and cold data perhaps to different tables and even different filegroups (and schemas), even dynamically, even when the database doesn't support for partitions. This idea could work also for non-relational databases and allow for the implementer a custom way to do application-specif scan-logic (say, to Azure Table Storage prefix scans on geohashes).

I think GeoAPI, and NetTopology, could use some of these ideas to build a backend agnostic abstraction. Even more intriguing, a Nuget package that provides GeoGrains, i.e. divide the world into a grid and one piece of the grid is one square. Augmented the ADO.NET backend script if needed. I won't go deeper into Orleans other than that note that one can control the lifetime of grains and it looks to me it could make an excellent GIS backend without much effort (looks like so) that would be like Redis, but with more options to work on geographics types and associated data both in storage and in memory.

Finally, I joined your Gitter channel if you have something to ask. I'll tag @galvesribeiro, @ashkan-saeedi-mazdeh and @sergeybykov from Orleans world, just in case.

(Further implementation ideas without bloating the interface, see at IRelationalStorage, its implementation RelationlStorage and some extension methods at RelationalStorageExtensions).

[airbreather]

I think the place to start for a redesign is to identify who the intended audience is, what they need from the library, and why the existing GeoAPI / NetTopologySuite is insufficient. Personally, I'm not interested in spending any of my time trying to help start a new(ish) project without knowing at least those things.

The title of the linked CoreFX issue refers to a need for "SQL-Independent geography code". The most comprehensive problem statement I can find in there sounds pretty specific to EF Core. I work on an app that doesn't use EF Core; in fact, I've never seriously worked with any ORM at all, so the whole post reads like a foreign language to me.

@veikkoeeva I'm trying to understand what you mean when you mention Java... are you saying that there's an existing Java-based solution that you feel does a much better job than anything in the .NET ecosystem? If so, can you please point to it so we can have that as a common point of reference?

I'm sure the detail from your comment will be useful at some point, but right now, without knowing the key pieces of information that I opened with, it just looks like a bunch of disconnected thoughts.

• A distributed backend-agnostic design with no hard dependencies on anything sounds great from an architectural perspective, but computationally intensive work with long parallelism-limiting dependency chains is not uncommon. Whenever you run into those, you want your CPU to be working on solving the problem, not slogging through layers of buzzword-induced overhead.
  • Not that JTS / NTS are paragons of efficiency, mind you...
• If it turns out that we're actually more interested in powering native Android / iOS apps that do the computations on-the-fly on low-end devices in areas with poor internet access, then we might need to rethink a few of those ideas.
• Similarly, if my app is running on a bandwidth-constrained computing cluster, then I'm interested in pushing around inputs, outputs, and intermediate results using as few bytes as possible, and I'm very likely to want to trade away that "backend-agnostic" property in exchange.

I'm not sure if those last two applications can (or should) use the same libraries.

[airbreather]

(I'd mis-clicked "Close and Comment"... sorry about that, it wasn't my intention to close it.)

[veikkoeeva]

I think the place to start for a redesign is to identify who the intended audience is, what they need from the library, and why the existing GeoAPI / NetTopologySuite is insufficient. Personally, I'm not interested in spending any of my time trying to help start a new(ish) project without knowing at least those things.

I concur.

The title of the linked CoreFX issue refers to a need for "SQL-Independent geography code". The most comprehensive problem statement I can find in there sounds pretty specific to EF Core. I work on an app that doesn't use EF Core; in fact, I've never seriously worked with any ORM at all, so the whole post reads like a foreign language to me.

It could be I have expressed myself poorly as I just came from the referenced CoreFX thread and checked that this started with ideas, some of which being database agnostic -- and then later some more concrete design ideas.

Create a core GeoAPI library that contains all necessary interfaces to describe and persist geometries
and other discussion of spatial data types.

Basically I would like to observe (+1 if you will) what others had already also observed: dependency on database or EF specific types would be problematic. It appears there isn't a commonly useable "type library" and so if there's one being designed, offered some design ideas to the specific problem of "crowded too much things in the API" and "up and down casting".

Also one other thing is that one might want to use a backend that is different from the ones available or one would like the change the data layout to suit specific needs. It appeas that if the design would be possible as POCO classes, then any backend can support them in whaever form, I feel like.

For instance, the POCO classes could be saved in database native geographic form, or perhaps in JSON or maybe do some geohashing like things. What comes to geohashing, maybe then use something like Cassandra, but the libraries would still work. Here it shows I'm juggling with ideas, trying to learn and seeing if there are ideas or some other assistance I could help with. I don't know, I don't really know GIS standards.

I'm sure the detail from your comment will be useful at some point, but right now, without knowing the key pieces of information that I opened with, it just looks like a bunch of disconnected thoughts.

Sure. I tried to just offer some ideas and conflated the rough sketches design you had there with other in various levels. Such that it's possible to detach Stream from Geometry. But this goes a bit deeper.

A distributed backend-agnostic design with no hard dependencies on anything sounds great from an architectural perspective, but computationally intensive work with long parallelism-limiting dependency chains is not uncommon.

Which one reason I think Orleans could work as a smart caching layer. It can support this kind of a work-load. It occurred to me there might be other synergies being had or learned.

Whenever you run into those, you want your CPU to be working on solving the problem, not slogging through layers of buzzword-induced overhead.

This was not my intention. Your note JTS / NTS hits the chord here, I'm new to this stuff, so there's a lot of "buzzwords" that take time to decipher. Even in the project I'm currently involved with there is MapProxy, a database product (I let the vendor go unnamed) and plenty of performance problems, naturally, on the backend but also on GeoServer (hence MapProxy) and also on the front-end. Some self-induced harm, some just due to plenty of processing.

In any event, I'm on the channel and try to be available for further chat if it matters.

If it turns out that we're actually more interested in powering native Android / iOS apps that do the computations on-the-fly on low-end devices in areas with poor internet access, then we might need to rethink a few of those ideas.

I may have understood the purpose of this library wrongly. I'm not sure now if this is general purpose library or something geared towards such edge computing or perhaps a primary, real reference you can currently use to aid the design (which I believe is good way to go about design).

Similarly, if my app is running on a bandwidth-constrained computing cluster, then I'm interested in pushing around inputs, outputs, and intermediate results using as few bytes as possible, and I'm very likely to want to trade away that "backend-agnostic" property in exchange.

I'm not clear what you mean here. If this comes from a though that back-end agnostic design would mean inefficient processing, I was thinking just the opposite with the limited experience I currently have on what I'm going through at (day) job. I understand there is a trade-off on how much scaleable computing power there can be on database (e.g. scaling horizontally, vertically, license-wise, using features), what could be done and scaled on a server farm and what perhaps on edge devices. In the current project (day job) I'm involved in some of the problems come from problems to get data laid out and queried efficiently and it's somehow tied on how the libraries work against the database. It's a difficult to me to get into specific details since this isn't what I'm working on and then NDAs.

But the design ideas from Orleans was that on the contrary, it could be possible to have even deployment-specific and time-dependent tuning and the libraries could still work.

I'm trying to understand what you mean when you mention Java... are you saying that there's an existing Java-based solution that you feel does a much better job than anything in the .NET ecosystem? If so, can you please point to it so we can have that as a common point of reference?

It appears there isn't anything like GeoServer in the .NET world. In this current day job project I'm involved in it's GeoServer, MapProxy etc., which has had a consequence on staffing the project and which other technology has been chosen. Looking at the channels (internal to some companies) where I'm currently involve in, the common perception seem to be Java based toolign is the way to go in GIS projects on the server and partially elsewhere too -- or JavaScript. Some of the desktop apps have embedded browsers to have more tooling and libraries to use with geodata. Now that INSPIRE is starting, besides IoT / Big Data projects, there's an uptick on GIS systems and mostly the discussions I've seen seem to take granted it's Java on the backend due to libraries and tooling. Hence I started also looking around, it looks like there isn't a good story on .NET and hence I happened to find this library too.