This is what I've been saying all along. Use thumbnails or links.
IF your having to download the whole video (without clicking "play") than it sounds like a setting somewhere that could be fixed since this is NOT how most people see youtube (or other) videos embedded. Sure it has to pull/download the video thumbnail image but I'm not sure that's much more bandwidth intensive than an equally sized picture.
Here's a test, roughly time how long this page with photo takes to load: http://www.e-cigarette-forum.com/forum/sandbox/293611-test-photo-load-time.html#post6002095
compared to this page with the same image but this time as a video thumbnail: http://www.e-cigarette-forum.com/forum/sandbox/293613-test-video-load-time.html#post6002111
For ME they the photo loads in 2-3 seconds, while the video page takes 3-4 (I know, not scientifically accurate but an embedded video DOES take me a second or so longer to load since it has to also load the embedded video player, and maybe youtube is just slow sometimes?).