l' onclick va nel tag <a>,
prova cambiando
codice:
<a href="../instances/<?php echo $family."/".$instance;?>.tt"> onclick="popup(this.href);return false">
in
codice:
<a href="../instances/<?php echo $family."/".$instance;?>.tt" onclick="popup(this.href);return false">