That will be in your theme.js (in themes/your theme/scripts)
Look for "// Smooth scroll to top." and "// Smooth scroll to bottom." In those areas you will see something like
$("html,body").animate({scrollTop: 0}, 1200);
that 1200 is milliseconds so 1.2 seconds ... lower the number to increase the speed, increase the number to slow it down.