Why we don't use 'ever'?

It sounds like bad grammar to me. I'm not even sure what you mean by "things that we had ever done". "Ever" basically means at all times, like "I'm ever hoping to hit the lottery", or for emphasis, like "I'm ever so sorry." Though the emphasis usage is mostly considered Old English.