Also, if you limit it to a certain number per day and run the contest for x many days, you'll also be able to easily spot non-compliant posts because the numbers will be off if you run some basic aggregate testing on the entries. For example, just looking at who's posted in this thread, I can see that MrBoss has more posts than other posters (the next highest is Celestiiaa--this post doesn't count because I'm still writing it). If the rules limited the number of entries, say, to 1 per person, you'd know that MrBoss and Celestiiaa had posted beyond their allowed number.
Sometimes the extra posts are necessary for clarifying the rules or something, sometimes they are chit-chat between contestants (and sometimes chit-chat posts are labelled as such). I suppose there are some contests in which the vendor doesn't care if the randomly-selected winning post is a correct entry, but I think most vendors will generally check a randomly-selected post to make sure it adheres to the general rules (numbered correctly, posts the proper theme, etc.) and re-select if they pick something that doesn't match.
As far as enforcement, most of it is on the honor system, as best as I can tell. Often, other contestants will point out bad entries either directly (hey, you've already posted five times today!) or indirectly (The rules said we can only post twice per day. Have they changed?)--either way, the offending poster usually rescinds their entry. I've seen cases in which contestants will point out that the winning post did not adhere to the rules, but for the most part, people pretty much accept the chosen winner(s) if people appear to be playing by the rules.