For anyone interested in exploring this idea without having to go full on assembly warrior, give Batari Basic a try. It's actually quite well done and it exposes the limited resources of the VCS in an easy to understand way.

Your basic program is compiled down to 6502 machine language and it all runs at basically native speed.

I found it extremely productive. One can knock out a simple game concept in an hour.

And yes, the sharp limits really does boil down to less is more. You won't find yourself looking through tons of options and or making one of a very large number of possible choices either.