Handling rate limits when using an API in Laravel

In my Laravel application I am using the HubSpot API extensively to perform various actions. I have read in the documentation that you can make 150 requests per each 10 second period.

To monitor this HubSpot provide the following Headers when making any API call.

"X-HubSpot-RateLimit-Daily" => array:1 [▶]
"X-HubSpot-RateLimit-Daily-Remaining" => array:1 [▶]
"X-HubSpot-RateLimit-Interval-Milliseconds" => array:1 [▶]
"X-HubSpot-RateLimit-Max" => array:1 [▶]
"X-HubSpot-RateLimit-Remaining" => array:1 [▶]
"X-HubSpot-RateLimit-Secondly" => array:1 [▶]
"X-HubSpot-RateLimit-Secondly-Remaining" => array:1 [▶]

In my application I am making use of Laravel’s Http Client, which is basically just a wrapper for Guzzle.

In order to adhere to the rate limits would I literally just have to wrap an if statement around every request?

Here’s an example:

$endpoint = ‘https://api.hubapi.com/crm/v3/owners/’;

$response = Http::get($endpoint, [
    'limit' => 100,
    'hapikey' => config('hubspot.api_key'),
]);

In this case the $response would contain the Headers but would there be a way to effectively use them, as surely I would only know what the rates were once I’d made the API call?

I ask as I have to pull down 1,000 + deals and then update some records, but this would definately go over the API limit. For reference, here is the command I wrote.

<?php

namespace AppConsoleCommands;

use AppEventsDealImportedFromHubspot;
use AppHubspotPipelineHubspot;
use AppModelsDeal;
use AppModelsDealStage;
use IlluminateConsoleCommand;
use IlluminateSupportFacadesHttp;

class ImportHubspotDeals extends Command
{
    /**
     * The name and signature of the console command.
     *
     * @var string
     */
    protected $signature = 'import:hubspot-deals
        {--force : Whether we should force the command}
    ';

    /**
     * The console command description.
     *
     * @var string
     */
    protected $description = 'Import Deal objects from the HubSpot API in bulk.';

    /**
     * An array to store imported Deals
     *
     * @var array
     */
    private $importedDeals = [];

    /**
     * Create a new command instance.
     *
     * @return void
     */
    public function __construct()
    {
        parent::__construct();
    }

    /**
     * Execute the console command.
     *
     * @return int
     */
    public function handle()
    {
        $this->line('Importing Pipelines & Deal Stages from HubSpot API...');

        PipelineHubspot::import();

        $this->line('Importing Deals from HubSpot API...');

        $this->getDealsFromHubspot();

        $this->line('Found ' . count($this->importedDeals) . ' Deals to import');

        if ($this->option('force')) {
            $this->doImport();
        } else {
            if ($this->confirm('Do you want to import these deals? (yes|no)', false)) {
                $this->doImport();
            } else {
                $this->line('Process aborted');
            }
        }
    }

    /**
     * Grab Deals from Hubspot by calling the Deals API and looping through the paginated data
     *
     * @param int    $limit: the number of deals per page
     * @param string $next:  the link to the next page of results
     */
    private function getDealsFromHubspot(?int $limit = 100, string $next = null)
    {
        $endpoint = 'https://api.hubapi.com/crm/v3/objects/deals';

        $properties = [
            'limit' => $limit,
            'properties' => implode(',', Deal::HUBSPOT_DEAL_PROPERTIES),
            'hapikey' => config('hubspot.api_key'),
            'associations' => 'engagements',
        ];

        // If there's another page, append the after parameter.
        if ($next) {
            $properties['after'] = $next;
        }

        $response = Http::get($endpoint, $properties);

        if ($response->successful()) {
            $data = $response->json();

            // If there are results, get them.
            if (isset($data['results'])) {
                foreach ($data['results'] as $hubspotDeal) {
                    $this->importedDeals[] = $hubspotDeal['properties'];
                }
            }

            // If there's paginate we need to call the function on itself
            if (isset($data['paging']['next']['link'])) {
                $this->getDealsFromHubspot(null, $data['paging']['next']['after']);
            }
        }

        $response->json();
    }

    /**
     * Pull the Deal data in order to create a Deal model.
     *
     * @param array $data
     */
    private function syncDeal(array $data)
    {
        $excludedDealStages = DealStage::excludeFromDealImport()->pluck('hubspot_id');

        if ($excludedDealStages->contains($data['dealstage'])) {
            return false;
        }

        $deal = Deal::updateOrCreate([
            'hubspot_id' => $data['hs_object_id'],
        ], [
            'name' => $data['dealname'],
            'deal_stage_id' => $data['dealstage'],
            'hubspot_owner_id' => $data['hubspot_owner_id'] ?? null,
        ]);

        event(new DealImportedFromHubspot($deal));

        return $deal;
    }

    /**
     * Create and increment a nice progress bar as we import deals.
     */
    private function doImport()
    {
        $bar = $this->output->createProgressBar(count($this->importedDeals));

        $bar->start();

        foreach ($this->importedDeals as $deal) {
            $this->syncDeal($deal);

            $bar->advance();
        }

        $bar->finish();

        $this->newLine(2);

        $this->line('Successfully imported ' . count($this->importedDeals) . ' Deals from HubSpot.');
    }
}

Building on this event(new DealImportedFromHubspot($deal)); also makes an API call back to HubSpot to add the URL of the portal it had just been pulled into.

In this situation I’m thinking I either need to treat the deal importing as its own job, or add in some kind of rate limiter.

Would it be bad practise just to use sleep(10) to get around the rate limiting?

Answer

Sounds like a job for a Queue.

You can define your own rate limiter on the Queue, but the correct solution is probably to extend ShouldQueue and run $this->fail() when you get a response saying your request has been throttled.