I tried with an fpga, but it is difficult to get the needed features and performance at a reasonable price.
I also tried with a RP2040, and this seems to be easier, although the performance is not quite there yet.
Have you looked at the Propeller chips from Parallax?
The P1 can do retro grade graphics from very solid TV graphics through higher resolution VGA.
The P2 can drive any display, had 640x480 HDMI capability built in, and analog video of all kinds is very well supported.
It would work in both of the classic ways for you. One way is to have it read the actual video RAM when the CPU allows that, and the other would be for it to have its own video RAM and the CPU communicates to it in some fashion.
The P1 can do retro grade graphics from very solid TV graphics through higher resolution VGA.
The P2 can drive any display, had 640x480 HDMI capability built in, and analog video of all kinds is very well supported.
It would work in both of the classic ways for you. One way is to have it read the actual video RAM when the CPU allows that, and the other would be for it to have its own video RAM and the CPU communicates to it in some fashion.