Create a process to dynamically listen, work or monitor queues in laravel

I’m dead stuck in a problem where I have to dynamically create queues and those queues should start to listen to all the active jobs assigned to them. “Using general queue” is not feasible in my case because the task execution delay gets increased exponentially due to long jobs chain in such generic queues.

Let’s assume that there are following jobs (A, B, C, D, E, F, …..)

let X be the only single or sequential queue. So the job dispatch for queue X goes like this:

Queue X = A, B, C, D, E, F, ……

This case blocks other jobs that could be simultaneously processed by other queues. So what I want to do is, dynamically create certain queues for each job that falls in a category. For these jobs, queues are generated (say X, Y, Z,…) such that:

Queue X = A, D, F

Queue Y = B, E

Queue Z = C
….and so on.

Is there any way to listen all those queues having queue names being dispatched?

Love to know.

Answer

Dynamic queues is not very good idea. It is inefficient to keep up with the queues or process runner CLIs as this architecture will cramp up memory so fast and is hard to implement too.

What you can do is set a fixed number of queues for this process and schedule the tasks in these using round robin algorithm. A good approach can be as follows:

• Set the number of queues in the env variable(e.g. QUEUES=5) so that you can increase it as the process increases).
• Create an array of queue names after getting the value from .env (e.g. env("QUEUES")).
• On the code where you dispatch a job using YourWorker::dispatch($data)->onQueue($queueName) in a loop.
• Now, create a supervisor and run those queue workers from it using php artisan queue:listen --queue <queue-name>. Go through this doc for more info about how to run a process.