Check whether a url exists or not in Shopify

How can I validate a Shopify store’s URL? Given a URL how can I know whether it is a valid URL or 404 page not found? I’m using PHP. I’ve tried using PHP get_headers().

<?php
$getheadersvalidurlresponse= get_headers('https://test072519.myshopify.com/products/test-product1'); // VALID URL
print_r($getheadersvalidurlresponse);

$getheadersinvalidurlresponse= get_headers('https://test072519.myshopify.com/products/test-product1451'); // INVALID URL
print_r($getheadersinvalidurlresponse); 
?>

But for both valid and invalid URLs, I got the same response.

Array
(
    [0] => HTTP/1.1 403 Forbidden
    [1] => Date: Wed, 08 Jul 2020 13:27:52 GMT
    [2] => Content-Type: text/html
    [3] => Connection: close
   ..............
)

I’m expecting 200 OK status code for valid URL and 404 for invalid URL.

Can anyone please help to check whether given shopify URL is valid or not using PHP?

Thanks in advance.

Answer

This happens because Shopify differentiates between bot requests and actual genuine requests to avoid denial of service attack up to a certain point. To overcome this problem, you will have to specify the user-agent header to mimic a browser request for an appropriate HTTP response.

As an improvement, you can make a HEAD request instead of a GET request(as get_headers() uses GET request by default, as mentioned in the examples) because here we are only concerned about response metadata and not response body.

Snippet:

<?php

$opts = array(
  'http'=>array(
    'method'=> "HEAD",
    'header'=> "User-agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36" 
  )
);


$headers1 = get_headers('https://test072519.myshopify.com/products/test-product1',0,stream_context_create($opts));
$headers2 = get_headers('https://test072519.myshopify.com/products/test-product1451',0,stream_context_create($opts));
echo "<pre>";
print_r($headers1);
print_r($headers2);