front: add debug interface to access logged data

[eckter]

Disclamer: the code was mostly LLM generated at first, but I did carefully review everything. And it's just for parts I'm not familiar with, and just the code, this description is written normally.

(for the record, I kind of regret doing it that way for the front-end changes, but it's hard to go back and rewrite it from scratch now)

I recommend reviewing per file and not per commit, as I didn't manage to distribute the review comments over their matching commits. I did try to fix the history and the result was mostly correct but still left a few commits in an invalid state, it would take quite a lot of time to entirely fix it.

---

Change summary:

• New endpoint in editoast + core to access s3 data for a given requests
  • Requires admin access
• Add a new front-end page at /stdcm/debug, with:
  • A space-time chart, with the new train and requirements from other trains and work schedules (if a path is found)
  • A speed chart (if a path is found)
  • A map with conflicts encountered during the search
• In stdcm page, when debug mode is enabled, each request tab has a link towards its debug page. It replaces the previous speed and time charts, which were too slow and had a chance to kill the browser on large timetables.

Minor changes to existing code:

• Speed distance diagram has a new property to specify layers that are initially displayed (we always want max speed to be shown here)
• Core generates a conflict list even on successful searches
• A few minor changes to editoast openapi schema (to avoid duplicated type names)

Not included (yet) but could be added later on:

• Engineering allowance data, it was part of the space time chart in the original python script
• Train paths on the map: could include the path toward each listed conflict (on hover?), and the full path if successful
• Removing original time and speed charts from the result tab results in lots of unused code, this isn't removed yet

---

How to test the happy path:

• Run the stack normally
  • With telemetry enabled: ./osrd-compose telemetry ...
  • Editoast needs to be recompiled
  • Core doesn't need to be, but if not recompiled the conflict map would only show up on unsuccessful searches
• Run an stdcm request
• Then there's two ways to visit the debug page:
  1. Get the PDF and look for an ID at the bottom right, visit http://localhost:4000/stdcm/debug and enter the ID
  2. Activate debug mode and look for the Link to debug page with more simulation details link at the bottom. It's discreet.

How to test the not happy paths:

• Run editoast with no or incorrect s3 configuration
• Enter IDs that don't exist
• Try to access the page without having admin rights (the page is accessible but unlisted, the endpoint would return an error)

No automated test has been included, it's a POC / MVP and this version is intended for internal use by the OSRD team. Front-end / E2E tests are a bit tedious to setup and would be overkill here (in my opinion at least).

---

The new page looks like this. It's not fancy but it does its job

Accessing /stdcm/debug directly looks like this:

[github-actions]

⚠️ API changes

This Pull Request introduces some changes in the API:

• please own it: notify or even prepare dedicated PR(s) to consumer projects

[bougue-pe]

Major QoL improvement ❤️

[woshilapin]

At first, I had really no opinion (between connecting editoast to S3, or letting only core connect to S3). After the few discussions, here and there, and the proposition of implementation using editoast as only a proxy (but with a weird hack about the routing key), I’m more inclined with the solution connecting editoast and the S3.

However, there is still the architecture risk of having 2 different services accessing the S3 concurrently. Today, it’s not entirely a problem since editoast don’t write to the S3, only core does it. But it would be nice if we could enforce that editoast only has read-only access to core’s bucket. Looking quickly on the web, it seems there are some options with bucket policies, but I am not entirely sure how hard will it be (and how generic, and/or AWS specific it will be).

[Khoyo]

However, there is still the architecture risk of having 2 different services accessing the S3 concurrently

I'm not completely sure what you mean? Both editoast and core have multiple instances, so the "badly handled concurrent write" risk is already a thing? (Note: at least for AWS, all PUT are atomic with strong read-after write consistency.)

Amazon S3 delivers strong read-after-write consistency automatically, without changes to performance or availability, without sacrificing regional isolation for applications, and at no additional cost. After a successful write of a new object or an overwrite of an existing object, any subsequent read request immediately receives the latest version of the object. S3 also provides strong consistency for list operations, so after a write, you can immediately perform a listing of the objects in a bucket with any changes reflected.

[eckter]

I rebased and squashed the commits. I didn't really manage to distribute the fixups added during with the reviews over their relevant commits, I spend a couple hours on that and still ended up with invalid states on intermediate commits.

As for the s3 connected to core or editoast, I removed the last commit and leave it as it was originally: editoast is connected to the object storage when the env variables are set and it fetches the data. If editoast is missing the s3 config or it hasn't started yet, editoast starts normally, and the endpoint returns an error when accessed.

[flomonster]

Thanks for this PR, I reviewed editoast part only.

• I did not find a button to go to /stdcm/debug is it expected to craft url?
• What is the usual size of output_simulation_data.json. Do we expect ram usage spike?

[flomonster]

@eckter I tried to test, this feature and I could not find the trace id. Probably because tracing is not enabled for one of my services, do we have to enable something? If so it could be nice to change the default config in docker-compose (can be done in follow-up PR).

[eckter]

Probably because tracing is not enabled for one of my services, do we have to enable something?

Ah yes, osrd-compose needs the telemetry flag. (to be fair, that has been removed from the default behavior during the development of this PR)

Note: I couldn't test it again after the rebase for an unrelated issue I mentioned on matrix, it's not working on dev either on my local setup. So there could also be some other new issue I couldn't see on the branch. I'm working on it.

(edit: all good)

[eckter]

Oh I didn't see this comment

• I did not find a button to go to /stdcm/debug is it expected to craft url?

It's missing when the front doesn't know the trace_id, it would be missing from the pdf as well. That's likely because of the telemetry flag mentioned above

• What is the usual size of output_simulation_data.json. Do we expect ram usage spike?

It depends a lot on the path, it technically scales with path_length * travel_time.

Of all the 66 simulations I've run locally with telemetry, the largest one is 13MB, second largest 6MB, average 1MB, median 0.3MB

[SharglutDev]

Great job ! Didn't have time to test it but looks like enough people did.

All the comments are very nitty and rather there to stick more to the rest of the front codebase but feel free to ignore what you want. You can close the comment as pleased.

[flomonster]

Oh I didn't see this comment

• I did not find a button to go to /stdcm/debug is it expected to craft url?

Okay

It's missing when the front doesn't know the trace_id, it would be missing from the pdf as well. That's likely because of the telemetry flag mentioned above

• What is the usual size of output_simulation_data.json. Do we expect ram usage spike?

It depends a lot on the path, it technically scales with path_length * travel_time.

Of all the 66 simulations I've run locally with telemetry, the largest one is 13MB, second largest 6MB, average 1MB, median 0.3MB

Good to know, it's not that heavy.

[SharglutDev]

Thanks for fixing the comments ✅

[flomonster]

Tested and everything works fine.

[Erashin]

Core LGTM.