data attribute filter accepts date-time

[flinden68]

Thema / Theme

Objecten API

Omschrijving / Description

Momenteel kunnen we voor bijvoorbeeld LTE alleen maar een date of numeric waarde gebruiken.
Maar in ons object hebben we te maken met date-time en dan faalt het LTE filter als je de huidige datum gebruikt om alle objecten op wilt halen voor vandaag en ouder.

Toegevoegde waarde / Added value

Hierdoor kan je beter filteren op basis van een date-time

Aanvullende opmerkingen / Additional context

No response

[alextreme]

JSON doesn't have a native datetime format so this will be quite complicated, we would recommend to store the date separately if it is necessary to filter on it

[sdegroot]

@alextreme what do you mean there is no native date time format? If you mean that dates are serialised either as numeric timestamps or string values (RFC 3339 / ISO 8601) then yes.

However, JSON Schema very much allows to enforce those standards upon a property using the "format" attribute.
See https://json-schema.org/draft/2020-12/draft-bhutton-json-schema-validation-00#rfc.section.7.3

Unfortunately, the validator used in objects-api/objecttypes-api does not adhere to the standards and simply ignores the format field. I discussed this several months ago with @joeribekker but haven't heard back about this.

@flinden68 is this what you meant?

[flinden68]

@sdegroot yes, because we pass on the date also a date format as string, should be that hard also accept a date-time formatted string

[alextreme]

@sdegroot https://json-schema.org/understanding-json-schema/reference/string#format

The format keyword allows for basic semantic identification of certain kinds of string values that are commonly used. For example, because JSON doesn't have a "DateTime" type, dates need to be encoded as strings. format allows the schema author to indicate that the string value should be interpreted as a date. By default, format is just an annotation and does not effect validation.

Optionally, validator implementations can provide a configuration option to enable format to function as an assertion rather than just an annotation. That means that validation will fail if, for example, a value with a date format isn't in a form that can be parsed as a date. This can allow values to be constrained beyond what the other tools in JSON Schema, including Regular Expressions can do.

[sdegroot]

Ah, so it is optional. I would highly suggest enabling this as it will increase data quality -a lot- :)

[JanBrek]

@alextreme I agree with @sdegroot. It would help a lot for connecting applications as they would get errors while unmarshalling the json objects.
We currently have an issue where an object has a wrong date '000' even though the json schema has the date-time format. This causes the search function for that object type to break (One item in the list is invalid, thus the whole list is unable to unmarshall resulting in a broken query/call.
If the objecten API would honor the dateTime format this error would be caught while trying to store the object.
Must we make a separate (taiga?) issue to request this?

[alextreme]

@JanBrek please indeed create a Taiga issue with the details of the object and json schema involved, combined with the API request which failed. I'll then pick this up with Team Bron to triage further

[flinden68]

@JanBrek @alextreme an example could be /api/v2/objects?page=1&pageSize=10&type=https://example.com/api/v1/objecttypes/78731088-430f-49fd-9a4c-80ddd42ded28&data_attrs=identificatie__type__exact__bsn,identificatie__value__exact__999990755,publicatiedatum__lte__2025-01-10T12:00:00.000

Nice to would be where the Zone is included

/api/v2/objects?page=1&pageSize=10&type=https://example.com/api/v1/objecttypes/78731088-430f-49fd-9a4c-80ddd42ded28&data_attrs=identificatie__type__exact__bsn,identificatie__value__exact__999990755,publicatiedatum__lte__2025-01-10T12:00:00.000Z

[alextreme]

Discussed with Team Bron and Dimpact
Two changes are necessary:

• An ability to enable datetime format validation for the data in an object. This could be added overall to the Objects API (so all datetime-strings are validated when creating/modifying an objects) or we could add the ability to specify this per objecttype in the ObjectType API. We assume the first (with an envvar/setting by default this would be off to avoid a breaking change) would be sufficient
• Then we need to investigate using data_attr comparison filters together with datetimes, as per the example of @flinden68.

Estimated at a week of development time, for @PeterVanBragt to approve