I seem to have found the right or a right place.
In the index.template.php of the default theme I have included the long code here:
// Show all the relative links, such as help, search, contents, and the like.
echo '
<link rel="shortcut icon" sizes="196x196" href="' . $context['favicon'] . '" />
<link rel="help" href="', $scripturl, '?action=help" />
<link rel="contents" href="', $scripturl, '" />', ($context['allow_search'] ? '
<link rel="search" href="' . $scripturl . '?action=search" />' : '');
There I inserted the code between the third and the fourth line.
The new favicons and icons are now displayed everywhere where they should appear.