79 of the 80 problems seem unsolveable in unlambda because the interpreter doesn't support input

[msullivan]

I might be missing something here, but the unlambda interpreter doesn't seem to support input, and so I don't think it is possible to solve the unlambda problems.

All of the problems except one (E01, "Print Hello World") require input, and the best anything did was 1.2% (or 1/80), which I think bears this out.

I think this needs to be done with a full, correct implementation of Unlambda. (The behavior of .n and r also don't match the spec)

[arcAman07]

Thanks for flagging this, you're not missing anything! The limited input support was intentional. We wanted to stress test the boundaries of LLMs with Unlambda specifically, and it turns out some of the stronger agentic frontier models brute-force their way around it by constructing large lookup tables. We do have a full Unlambda implementation with proper input support ready, and I'll be pushing that code next week for everyone to try. The .n and r behavior discrepancies will also be addressed and all of these fixes will be part of the updated codebase. Stay tuned!

[msullivan]

The limited input support was intentional.

Huh? It makes the problems literally unsolvable

[FINDarkside]

Task 2 is solvable because it only requires echoing input, not actually reading it. But I think this issue is valid, 78/80 of the tasks are unsolvable.

[msullivan]

Task 2 is solvable because it only requires echoing input, not actually reading it. But I think this issue is valid, 78/80 of the tasks are unsolvable.

Oh yeah, I missed that--but looking again I think this is only almost true: you can write a program that echoes its input, but I think it will have to exit with a runtime error when input is exhausted, which will cause a failure even though the output will get collected.

[arcAman07]

Agreed, and a proper input-supported Unlambda interpreter is on the way. We did work on one and used it during our own experiments, so we will be pushing that to the codebase soon.

On the current interpreter, what we observed with Claude Code and other agentic systems using custom harnesses is that the missing input support did not always block performance in practice. Some models worked around it by constructing large lookup tables over the input arguments, essentially eliciting that as a fix for the interpreter's limitations rather than labelling those tasks unsolvable.

Your point is well taken though, and the updated codebase will include a correct Unlambda implementation with proper input handling along with fixes for the .n and r behavior. Thanks for flagging this.

[FINDarkside]

I'm sorry, but I feel like you're still still not getting it. Lookup tables do not replace the need for reading input. The program is not able change it's output based on input if it can't read the input. So at best, you can hardcode 1 answer and pass 1 test case, but you'll still fail the rest. It is literally impossible to pass the bench for more than 1-2 tasks.

missing input support did not always block performance in practice

This isn't what your paper says. Not a single model solved more than 1 task, so nobody got past Hello World. You've already gotten plenty of heat in X, please take couple minutes to do some basic due diligence to see that the benchmark is completely broken.

I find it also concerning that it seems likely that the authors didn't write example solutions to these tasks at all. I think doing so, not only would have caught this impossible language to use, but would have given the well needed perspective to making claims about "genuine reasoning" capabilities of LLM.

[arcAman07]

I think what I am trying to say is getting entangled. I fully agree with the point that lookup tables do not replace the need for reading input.
For all our benchmarking we had used unlambda with input support ( and for all esolangs ) and I will be pushing that code soon.
We have written examples solutions ( the few shot solutions for benchmarking ) and we will be adding that to the codebase which we did write manually while curating these examples and testing out these esolangs. We are still working on cleaning our experimental codebase and adding support for this.