One of the great rimfire gurus (Can't remember who) shot 2" blocks 10,000 times w/o cleaning. No change in accuracy.
On the other hand, with some 22 lever actions (and presumably other action types such as a Mossberg 430 semi-auto) such as Mossberg Palominos, too frequent cleaning, followed by improper storage can damage the stock. Storage in an upright position allows excess oil to drain into the rear of the action and soften the stock. This can result in a split. I store my lever actions muzzle down. I bought a Palomino for a good price on Gunbroker with a large chip missing. We tried to replace it (a friend is a good woodworker and has repaired quality shotgun stocks). He brought it back, along with a replacement piece from a local gunsmith friend and stated it couldn't be glued. I figured what the heck, stripped it down, and it took a couple of weeks of degreasers and acetone, but we got it to stick (file for future reference, friends and neighbors!) but it did and my friend did a great job on reshaping it. You can hardly tell it was broken!
Gotta brag:
After repair @ 20 yards. Weaver C-4.
Also, using the wrong brush can accelerate wear at the muzzle (e.g. hard stainless steel rather than brass), but I would imagine that might be more of an issue with classic (and older) guns. I imagine so could steel rods if slightly too large, etc.
Overall, except for the oil drainage problem, I have never had any problem in 50+ years of shooting.
Good luck!