The proper way of doing it would be to use DOMDocument to parse your HTML code. Then iterate recursively over children. Skip nodes that have tagName equal to "a". Then analyse the textNodes and if they are not part of node, then replace the textNode with a node and put the textNode value in it.
Finally use saveHTML to get back html string.
manual about loading html: https://www.php.net/manual/en/domdocument.loadhtml.php
Stack Overflow ticket about iterating over childNodes: Loop over DOMDocument
Here is another quick version for your specific case:
<?php $input = "http://www.google.com is a great website. Visit <a href='http://www.google.com' >http://google.com</a>"; $output = preg_replace_callback("/(^|[^\"'>])(https?:\/\/[^ \n\r]+)/s",function($in){ $url = $in[2]; return "<a rel=\"nofollow\" href=\"$url\">$url</a>"; }, $input);
As you can see, we are using trick regex to look for http/https links that do not seem to be part of a tag. Note, that this will not work for cases like <b>https://google.com</b>. If you need more advanced solution, you should either go with DOMDocument or alternatively you can check the text before the each occurrence of the https? marker.
And here is an even better version, as suggested by @mickmackusa:
<?php $input = "http://www.google.com is a great website. Visit <a href='http://www.google.com' >http://google.com</a>"; $output = preg_replace_callback("~<a[^>]+>.*?</a>(*SKIP)(*FAIL)|(https?:\/\/\S+)~i",function($match){ return sprintf("<a rel=\"nofollow\" href=\"%s\">%s</a>", $match[1], $match[1]); }, $input);
it uses ultra magic regular expression syntax with (*SKIP)(*FAIL) and a bit more elegant sprintf for building the replacement value.
preg_replace_callback