[BUG] Reconcile differences between `at/iat` and `loc/iloc`, respectively

[Matt711]

Just an FYI that at/iat are not exactly equivalent to loc/iloc in pandas (the format only expects scalars), but first approximation equivalence is OK

So adding the right restrictions to at/iat would be enough?

It should be modulo bugs. In pandas there's a different inheritance hierarchy for at vs loc that mainly separates the scalar restriction

at/iat base class: https://github.com/pandas-dev/pandas/blob/9b82f9635de6d4765e1c25904e6d268c793d905a/pandas/core/indexing.py#L2520
iloc/loc base class: https://github.com/pandas-dev/pandas/blob/9b82f9635de6d4765e1c25904e6d268c793d905a/pandas/core/indexing.py#L756

Originally posted by @mroeschke in #20529 (comment)