Leverage markdown and fixtures to write intentions

[ELC]

Since intentions are plain text, it would be great to have a plain text file intentions.md in the same folder as the test to have the intentions, one per line.

I managed to do this with the following snippet:

intentions.md file:

SRP: All the code in {module} should follow Single Responsability Principle.
OCP: All the code in {module} should follow Open Close Principle.
LSP: All the code in {module} should follow Liskov Substitution Principle.
ISP: All the code in {module} should follow Interface Segregation Principle.
DIP: All the code in {module} should follow Dependency Inversion Principle.

conftest.py file:

def extract_intentions(file_path: Path) -> tuple[list[str], list[str]]:
    intentions_file = file_path.parent / "intentions.md"

    if not intentions_file.exists():
        msg = "Intention files does not exists"
        raise ValueError(msg)

    intentions_raw = intentions_file.read_text(encoding="utf-8").split("\n")
    intention_names, _, intentions = list[list[str]](
        zip(
            *(intention_raw.partition(":") for intention_raw in intentions_raw),
            strict=False,
        )
    )

    clean_intentions_names = [
        intention_name
        for raw_intention_name in intention_names
        if (intention_name := raw_intention_name.strip())
    ]

    clean_intentions = [
        intention
        for raw_intention in intentions
        if (intention := raw_intention.strip())
    ]

    if not clean_intentions:
        msg = "Intention file is empty"
        raise ValueError(msg)

    return clean_intentions_names, clean_intentions


@pytest.fixture
def intentions(request: pytest.FixtureRequest) -> tuple[list[str], list[str]]:
    return extract_intentions(Path(request.node.location[0]))

This may be specific to pytest but having the library automatically register convient fixtures like this would improve the developer experience

[kdunee]

Thanks for the suggestion!

If I understand correctly, you're proposing loading intention assertions from a separate intentions.md file instead of directly embedding them in Python test code. This would involve parsing the markdown file and using pytest fixtures to make these intentions available in tests.

It's an interesting approach for sure. My initial thought was to keep things simple and just define intentions as Python constants directly in your test modules.

For instance, you could structure your test like this:

import intentguard as ig

SRP_ASSERTION = "All the code in {module} should follow Single Responsibility Principle"
OCP_ASSERTION = "All the code in {module} should follow Open Close Principle"
LSP_ASSERTION = "All the code in {module} should follow Liskov Substitution Principle"
ISP_ASSERTION = "All the code in {module} should follow Interface Segregation Principle"
DIP_ASSERTION = "All the code in {module} should follow Dependency Inversion Principle"


def test_solid_principles():
    guard = ig.IntentGuard()
    my_module = ... # your module here

    guard.assert_code(SRP_ASSERTION, {"module": my_module})
    guard.assert_code(OCP_ASSERTION, {"module": my_module})
    guard.assert_code(LSP_ASSERTION, {"module": my_module})
    guard.assert_code(ISP_ASSERTION, {"module": my_module})
    guard.assert_code(DIP_ASSERTION, {"module": my_module})

This way, intentions are still clearly separated as constants at the top, without needing to parse files or introduce fixtures.

Could you elaborate a bit more on why you think the markdown file and fixtures approach would be beneficial compared to this simpler Python constant method? Perhaps I'm missing something!

Thanks again for your input!

[ELC]

Inpiration comes on how Github Copilot works on VS Code where all the different prompts are defined in a .github/copilot-instructions.md.

The benefit of this approach is that the actual Python code is very much the same every time, something like:

from types import ModuleType

import intentguard as ig

INTENTION: str = ...

def test_solid_principles(guard: ig.IntentGuard, module: ModuleType):
    guard.assert_code(INTENTION, {"module": module})

If the only thing that would change between tests are the modules to evaluate and the intention to evaluate against, I believe it is aligned with the pytest philosophy to abstract those away. Not necessarily as a markdown file but as fixtures at least.

My first refactor looked like this:

from types import ModuleType

import intentguard as ig
import pytest

LIST_OF_INTENTIONS: list[str] = [...]

@pytest.fixture(params=LIST_OF_INTENTIONS)
def intention(request: pytest.FixtureRequest) -> str:
    return request.param

def test_solid_principles(guard: ig.IntentGuard, module: ModuleType, intention: str):
    guard.assert_code(intention, {"module": module})

The rationale behind going this route is multiple fold:

1. I want tests to be atomic, following single responsibility, therefore, I want to test at most one intention per test, not multiple ones in a single test. Why? Because...
   1. When a test fails, I want a concise assertion error.
   2. When a test fails, I don't want other test to not run due to that error.
   3. When a test fails, I want to know just by the title in the Testing pane what it is and why it failed.
   4. When a test fails, I want to isolate the root cause fast.
   5. When adding more tests, I don't want to modify existing passing tests, basically open-closed principle. 2. Generating tests dynamically by leveraging either pytest.fixture(params=) or pytest.mark.parametrize seems more idiomatic than just repeating the assert_code block.
2. Having a single, parametrized tests resembles how other tools like Hypothesis work.

There is a fair bit of personal preference on how I like to structure tests, but I believe there justification is there for those who agree.

Moving this parametrized fixture to a markdown adds one more step to this approach which is separating the responsabilities between definition and execution, similar to the Arrange-Act-Assert idea. Arrange of intentions should happen separately (i.e. parametrized fixture or markdown file) than the actual execution (act) of the test. By enabling a markdown file, I'm forcing that the intentions are pure plain text and prevent unexpected manipulations at runtime.

Pull requests that only adds/removes intentions will only modify markdown files instead of code, which I believe would be easier to spot.

Since this is dealing with LLMs, I'm not worried about the performance hit of I/O and parsing a markdown file, the LLM will always be the bottleneck, even if it is not run locally and it is through a rest API (e.g. OpenAI), parsing a local file is neglectable time compared to network latencies.

This is indeed opinionated but I would like to hear your thoughts.

[ELC]

As a reference this is my current setup:

One test file that runs over all the values from the fixtures.
One markdown files with the intentions
One fixture in the conftest that parses the markdown
One fixture in the global conftest that retrieves all the modules of the app
The extract_intentions implementation is from the first comment.

The main added value is how it is displayed in the VS Code Testing pane, I can run any of my desired intentions for any module in a single click without modifying the tests. If I want to add an intention, I just add a line to the markdown file and refresh the pytest discovery process.

In this way I would group my intentions by topic, e.g. SOLID would be one group. And create a folder with a markdown file, a test_intention.py file and a conftest.py where I parse the markdown.

I tried to get rid of the conftest.py part to automatically detect the neighbour file but no luck there yet.