I've had my Crucial M4 boot drive going now for six months or so without trim, and it still gets the same speeds during tests as before I cloned my OS on it. I only use 98GB out of the 256GB available, and frequently drag large files on the desktop before relocating them to other drives, but other than that I never "fill it up" or anything.
That said, I haven't had any issues, either. Apps boot quick, nothing hangs or gets glitchy... nothing. (Knock on wood, right?) If something were to start screwing up on my M4, I'd try it then. Until then, I subscribe to "if it ain't broke, don't fix it."
Read speed is not affected by poor garbage collection, only write speeds, and garbage collects when files are frequently deleted. You don't need to fill the drive up at all. Moving files from one location to another on the same drive is not what you think: the file remains physically in the same place on the drive (i.e., it takes up the same memory cells as before), but it has it's reference changed in the file table as to which directory it's under.
I guarantee you that your write speeds are not what they once were.