I had a moment of madness and put together something fancy. 
The php code could be just:
$message = preg_replace_callback('~<a href="(http[s]?://twitter.com/\w+/status/\d+)" (.*?)>\1</a>~',
function($matches) {
$matches[2] = str_replace('class="', 'class="twitter_embed ', $matches[2]);
return '<a href="' . $matches[1] . '" ' . $matches[2] . '>' . $matches[1] . '</a>';
},
$message
);
static $load_js = true;
if ($load_is === true)
{
loadJavascriptFile('twitter_embed.jquery.js');
addInlineJavascript('
$(document).ready(function() {
$(".twitter_embed").lazyload();
});');
$load_js = false;
}
put in a function and attached to the integrate_post_parsebbc hook.
Then there is a bit of javascript that is a modified version of https://github.com/tuupola/jquery_lazyload (and is attached here).
The two will work together to embed the twitter post only when scrolling the message appears in the viewport.