//Web Scrape user comments by php curl and store in the mysql. Is it poossible?? <?php $url = 'https://www.flipkart.com/samsung-galaxy-on5-gold-8-gb/product-reviews/itmedhx3uy3qsfks?pid=MOBECCA5FHQD43KA'; $ch = curl_init(); curl_setopt($ch, CURLOPT_URL,$url); curl_setopt($ch, CURLOPT_USERAGENT, 'Opera/9.23 (Windows NT 5.1; U; en)'); curl_setopt($ch, CURLOPT_RETURNTRANSFER,1); curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE); curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2); $result = curl_exec($ch); //echo $result; preg_match_all ('/<div class="_3DCdKt">([^`]*?)</div>/', $result, $matches); echo sizeof($matches); echo sprintf(print_r($matches, true)); ?>
Error: get blank page of flipkart.
Any help much appreciated.
Thanks In Advance
Advertisement
Answer
please read this https://stackoverflow.com/a/1732454/1067003 and come back. read it? good, now here’s a decent alternative: DOMDocument. now you’ve solved your first problem, trying to parse HTML with regex.
your second problem, is that this website does not serve the reviews themselves in the html, but rather fetches the reviews via javascript. the obvious course of action then is to start digging into the internal api the javascript use to fetch the comments. do that, and you’ll probably find the url https://www.flipkart.com/api/3/page/dynamic/product-reviews
, where the javascript is fetching the comments from. however, replicating the the javascript request with curl, you’ll probably get a 403 Forbidden error, until you figure out that you must fake the custom x-user-agent
header to match whatever you put in CURLOPT_USERAGENT. doing that, you’ll probably end up with something like this:
<?php declare(strict_types = 1); $url = 'https://www.flipkart.com/samsung-galaxy-on5-gold-8-gb/product-reviews/itmedhx3uy3qsfks?pid=MOBECCA5FHQD43KA'; parse_str ( parse_url ( $url, PHP_URL_QUERY ), $productID ); $productID = $productID ['pid']; $ch = curl_init ( '' ); $useragent = 'Opera/9.23 (Windows NT 5.1; U; en)'; curl_setopt_array ( $ch, array ( CURLOPT_USERAGENT => $useragent, CURLOPT_URL => 'https://www.flipkart.com/api/3/page/dynamic/product-reviews', CURLOPT_POST => true, CURLOPT_HTTPHEADER => array ( 'Content-Type: application/json', 'x-user-agent: ' . $useragent . ' FKUA/website/41/website/Desktop' ), CURLOPT_POSTFIELDS => json_encode ( array ( 'requestContext' => array ( 'productId' => $productID ) ) ), CURLOPT_RETURNTRANSFER => true ) ); $json = curl_exec ( $ch ); $parsed = json_decode ( $json, true ); $comments = $parsed ['RESPONSE'] ['data'] ['product_review_page_default_1'] ['data']; foreach ( $comments as $comment ) { $comment = $comment ['value'] ['text']; var_dump ( $comment ); }
which outputs:
string(157) "I have bought it in ₹6,291.00 Very good product in this price. I have updated it to Marshmallow. Only drawback is there is no Notification LED in the set." string(422) "More than Worth to buy . Simply super With 1.5GB RAM Processor is Excellent RAM is No hanging up & No heating Replaceble battery Wonderful clicks with Excellent back and front camera 4G is Awesome Awesome touchscreen Battery is backup also great Excellent UI Additional features 1. Power saving .. 2. Data restrict. Both are excellent and useful to batter backup Final verdict, I will recommend this phone to my buddies" string(189) "I Bought this phone yesterday.. Positive Points 1.Camera Quality is Good 2.HD Display 3.Slim & Good Design 4. 4G Supports 5.Latest Android Version Negative Points 1. Less Internal Storage" string(261) "Under 9k this buy is good you get a handset with good display and battery backup And it is good for day to day usage your cant play heavy games on it due to 1 gb Ram For Android User this is the cheapest and the best phone which Samsung can offer So go for it" string(2515) "This post is long:) I will not be repeating the features which others have already liked (e.g. the display, camera, gaming speed, storage etc...) All these are good and work well as expected from Samsung phones. The phone does not look cheap from any angle. Some have even confused this with the Galaxy A5, so this says a lot! First, with our Indian Air force issuing some statement in TOI about Chinese phones snooping in on our data, i was reluctant to buy any of these phones mentioned below. Yu, MI, Huawei, Lenovo, Gionee and Micromax (yes, they import these from China)) This phone ticks all the right boxes with Samsung reliability. Second, glad to see this phone being manufactured in India as part of 'Make in India' campaign. Way to go Samsung! The South Korean manufacturer and in general the South Korean industry has grown into a superpower. (LG, Samsung, Hyundai are all shining examples of craftsmanship and quality) The Gold color looks classy. Cover her up with a matching authentic gold flip color from Samsung and you have a phone which turns heads. What more! you get a Samsung S5 design of last year, which is good, considering the bezel lines, chrome touch and minimalist look to this phone. Display is great and does not make you feel that you lose out on the Amoled display. So full marks here. Outdoor visibility can be tuned manually. Speaker quality is good and for those who are complaining, there is an 'Extra volume' option when answering calls. There is also a 'Sound Adapt' feature to tune the phone to your hearing needs. The phone also comes with a 'Smart Manager' to take care of RAM, storage, battery and device security. For enhanced security, you can register your mobile on samsung find my mobile and enable remote tracking on. This will help you to locate your device using GPS. Though i'm yet to use this feature (hopefully not:), the remote tracking feature can also wipe or lock your phone remotely, in case of theft. The usb charger seems to be fragile, so you need to be careful while plugging in. Have also discovered the FM Radio service which is absent in now-a-days phone. This is other than the 'MixRadio' feature which does online streaming. So like old days, you can listen to your favourite FM channel on the go without a 3G or 4G connection. Wow! i now can listen to my favourite songs and can record these too on my device. So go for it. It is definitely worth every rupiah (penny) PS: if you found this review useful, give me a thumbs up" string(1124) "Hello friends, I am writing this review after 2 days of usage. I always compare/read reviews before buying any products. Pros: *It arrives in style via ekart. *Flipkart packing is very nice and protective. *Weight of phone including battery is quiet light. *Look and feel is very much attractive, my colleagues compared with their expensive samsung phones, this phone also looks of same segment. *Fine details on edge looks posh, it gets broader at the bottom, overall it is great. *UI and 5.1 lollipop is simple and very organised. *Music setting has a feature called Adapt sound, it enable every individual to fine tune the amount of sound for both ears, so that it suites you best without hurting your eardrums. The earphone which come is bad for music lovers, try getting your best brand's headphone, I use sony xb400 (old one,that time price was aprx 2K) it bounces nice. ;) * Apart from this every feature is nicely blended to cooperate daily life activities. *There is no heating issue yet. *This phone is definitely value for money product. Cons: *No notification lights. *No themes *Less default wallpapers" string(77) "Everthing os good and fine...thank u flipkart...mobile is so good and nice..." string(818) "Initial Review: What I liked: + SAR Value of 0.551 W/Kg. + 5" large screen looks great inspite of TFT display. + Excellent Camera - Front and Back. + Decent performance, no lag noticed so far. + Android 5.1 + Good build quality with premium finish. + No bloatware. What I didn't like: - Average Speaker. Wish it was louder, crisper and in the front. - No Notification light. Why Samsung why? - No Ambient Light Sensor. - A little expensive considering the Redmi 2 is available for Rs. 5999. If you are big on music, the speaker quality will disappoint you. No excuse for not adding notification light and better quality speaker for the price. Redmi 2 is a superior VFM phone, but since I as buying it for my father - the low SAR value was a big selling point including the big screen and the Samsung brand value." string(249) "Can give good competition to other phone makers. Amazing fast, good features, and the biggest brand name "SAMSUNG". Prefer buying Samsung over any other Chinese phones, because at this price it can beat any other phone and woo factor also included." string(203) "Good product...delivered on 21.10.16 in 2 days. Camera, battery backup everything is ok. No heating,hanging problem found. 550 mb RAM was accessible. Even after upgrading to MARSHMALLOW no problem found."
EDIT: didnt paste the whole code first time around x.x