a PHPpunit test fail randomly

I have a Quiz class. This class load 10 questions from a database depending on the level and the type of the quiz Object: level 0 load the ten first, level 1 load the next ten and so on.

So in my test i create in a test database 30 questions. Then i create quiz object with different level and i check that the first question in the quiz steps array match what i expect.

This test “quiz_contain_steps_depending_on_type_and_level()” failed randomly at least once every 5 launches.

This is the QuizTest class

<?php


namespace AppTestsQuiz;

use AppQuizQuestion;
use AppQuizQuiz;
use AppQuizQuizQuestionRepositoryManager;
use AppQuizQuizStep;
use AppQuizQuizType;
use DoctrineORMEntityManagerInterface;
use DoctrinePersistenceObjectRepository;
use FakerFactory;
use FakerGenerator;
use ReflectionException;
use SymfonyBundleFrameworkBundleTestKernelTestCase;
use SymfonyComponentConfigDefinitionExceptionException;


class QuizTest extends KernelTestCase
{
    use QuestionLoremTrait;
    use PrivatePropertyValueTestTrait;

    private Generator $faker;
    private ?EntityManagerInterface $em;
    private ObjectRepository $questionRepo;
    private QuizQuestionRepositoryManager $quizQuestionManager;

    protected function setUp(): void
    {
        $kernel = self::bootKernel();
        $this->faker = Factory::create();
        $this->em = $kernel->getContainer()->get('doctrine')->getManager();
        $this->em->getConnection()->beginTransaction();

        $this->questionRepo = $kernel->getContainer()->get('doctrine')->getRepository(Question::class);
        $this->quizQuestionManager = new QuizQuestionRepositoryManager($this->questionRepo);
    }

    protected function tearDown(): void
    {
        parent::tearDown();
        $this->em->getConnection()->rollBack();
        $this->em->close();
        $this->em = null;
    }

    /**
     * @test
     * @dataProvider provideQuizDataAndFirstQuestionExpectedIndex
     * @param array $quizData
     * @param int $firstQuestionExpectedIndex
     * @throws ReflectionException
     * @throws Exception
     */
    public function quiz_contain_steps_depending_on_type_and_level(array $quizData, int $firstQuestionExpectedIndex)
    {
        //We have questions in db
        $questions = [];

        for ($q = 1; $q <= 30; $q++) {
            $question = $this->persistLoremQuestion($this->faker, $this->em);
            $questions[] = $question;
        }
        $this->em->flush();


        //When we create Quiz instance $quiz
        $quiz = new Quiz($this->quizQuestionManager,quizData:  $quizData);

        //When we look at this $quiz steps property
        $quizSteps = $quiz->getSteps();
        /** @var QuizStep $firstStep */
        $firstStep = $quizSteps[0];

        //We expect
        $this->assertNotEmpty($quizSteps);
        $this->assertCount(10, $quizSteps);

        //We expect if quiz is type normal and level variable questions depends of level:
        $this->assertEquals($firstStep->getQuestion(), $questions[$firstQuestionExpectedIndex]);

    }

    public function provideQuizDataAndFirstQuestionExpectedIndex(): array
    {
        return [
            [[], 0],
            [['type' => QuizType::NORMAL, 'level' => '1'], 10],
            [['type' => QuizType::NORMAL, 'level' => '2'], 20]
        ];
    }
}

This is the Trait who generate fake question

<?php

namespace AppTestsQuiz;

use AppQuizQuestion;
use DoctrineORMEntityManagerInterface;
use Exception;
use FakerGenerator;

Trait QuestionLoremTrait{

    /**
     * This function persist a aleatory generated question, you must flush after
     * @param Generator $faker
     * @param EntityManagerInterface $em
     * @return Question
     * @throws Exception
     */
    public function persistLoremQuestion(Generator $faker, EntityManagerInterface $em): Question
    {
        $nbrOfProps = random_int(2,4);
        $answerPosition = random_int(0, $nbrOfProps - 1);
        $props = [];

        for ($i = 0; $i < $nbrOfProps; $i++){
            $props[$i] = $faker->sentence ;
        }

        $question = new Question();

        $question
            ->setSharedId(random_int(1, 2147483647))
            ->setInfo($faker->paragraph(3))
            ->setStatement($faker->sentence ."?")
            ->setProps($props)
            ->setAnswerPosition($answerPosition)
        ;

        $em->persist($question);

        return $question;
    }
}

This is my Quiz class:

<?php


namespace AppQuiz;


use SymfonyComponentConfigDefinitionExceptionException;

class Quiz
{
    /**
     * Quiz constructor.
     * @param QuizQuestionManagerInterface $quizQuestionManager
     * @param array $quizData
     * This array of key->value represent quiz properties.
     * Valid keys are 'step','level','type'.
     * You must use QuizType constant as type value
     * @param string $type
     * @param int $level
     * @param int $currentStep
     * @param array $steps
     */
    public function __construct(
        private QuizQuestionManagerInterface $quizQuestionManager,
        private string $type = QuizType::FAST,
        private int $level = 0,
        private array $quizData = [],
        private int $currentStep = 0,
        private array $steps = [])
    {

        if ($quizData != []) {
            $this->hydrate($quizData);
        }
        $this->setSteps();
    }


    private function hydrate(array $quizData)
    {
        foreach ($quizData as $key => $value) {
            $method = 'set' . ucfirst($key);

            // If the matching setter exists
            if (method_exists($this, $method) && $method != 'setQuestions') {
                // One calls the setter.
                $this->$method($value);
            }
        }
    }

    public function getCurrentStep(): int
    {
        return $this->currentStep;
    }

    public function getLevel(): int
    {
        return $this->level;
    }

    public function getType(): string
    {
        return $this->type;
    }

    public function getSteps(): array
    {
        return $this->steps;
    }

    private function setCurrentStep($value): void
    {
        $this->currentStep = $value;
    }

    private function setLevel(int $level): void
    {
        $this->level = $level;
    }

    private function setType($type): void
    {
        if (!QuizType::exist($type)) {
            throw new Exception("This quiz type didn't exist, you must use QuizType constante to define type", 400);
        }
        $this->type = $type;
    }

    private function setSteps()
    {
        $this->steps = [];
        $questions = $this->quizQuestionManager->getQuestions($this->type, $this->level);
        foreach ($questions as $question) {
            $this->steps[] = new QuizStep(question: $question);
        }
    }
}

This is the Question class:

<?php


namespace AppQuiz;

use AppRepositoryQuestionRepository;
use DoctrineORMMapping as ORM;
use SymfonyComponentValidatorConstraints as Assert;

/**
 * @ORMEntity(repositoryClass=QuestionRepository::class)
 */
class Question
{
    /**
     * @ORMId
     * @ORMGeneratedValue
     * @ORMColumn(type="integer")
     */
    private ?int $id;

    /**
     * @ORMColumn(type="integer")
     */
    private ?int $sharedId;

    /**
     * @ORMColumn(type="string", length=1000, nullable=true)
     * @AssertLength(max=1000)
     */
    private ?string $info;

    /**
     * @ORMColumn(type="string", length=255, nullable=true)
     */
    private ?string $statement;

    /**
     * @ORMColumn(type="array")
     */
    private array $props = [];

    /**
     * @ORMColumn(type="integer")
     */
    private ?int $answerPosition;

    public function getId(): ?int
    {
        return $this->id;
    }

    public function getSharedId(): ?int
    {
        return $this->sharedId;
    }

    public function setSharedId(int $sharedId): self
    {
        $this->sharedId = $sharedId;

        return $this;
    }

    public function getInfo(): ?string
    {
        return $this->info;
    }

    public function setInfo(?string $info): self
    {
        $this->info = $info;

        return $this;
    }

    public function getStatement(): ?string
    {
        return $this->statement;
    }

    public function setStatement(?string $statement): self
    {
        $this->statement = $statement;

        return $this;
    }

    public function getProps(): ?array
    {
        return $this->props;
    }

    public function setProps(array $props): self
    {
        $this->props = $props;

        return $this;
    }

    public function getAnswerPosition(): ?int
    {
        return $this->answerPosition;
    }

    public function setAnswerPosition(int $answerPosition): self
    {
        $this->answerPosition = $answerPosition;

        return $this;
    }
}

If anyone understands this behavior. I thank him in advance for helping me sleep better 🙂

Answer

Thanks to @AlessandroChitolina comments.

The set of questions created in my test was not always recorded in the same order by my in my database.

So instead of testing the expected question from my starting $questions array, i retrieve the questions from the database in a new $dbQuestions array. That solve my problème.

This is the new test:

/**
     * @test
     * @dataProvider provideQuizDataAndFirstQuestionExpectedIndex
     * @param array $quizData
     * @param int $firstQuestionExpectedIndex
     * @throws Exception
     */
    public function quiz_contain_steps_depending_on_type_and_level(array $quizData, int $firstQuestionExpectedIndex)
    {
        //We have questions in db
        $questions = [];

        for ($q = 1; $q <= 30; $q++) {
            $question = $this->persistLoremQuestion($this->faker, $this->em);
            $questions[] = $question;
        }
        $this->em->flush();

        $dbQuestions = $this->questionRepo->findAll();

        //When we create Quiz instance $quiz
        $quiz = new Quiz($this->quizQuestionManager,quizData:  $quizData);

        //When we look at this $quiz steps property
        $quizSteps = $quiz->getSteps();
        /** @var QuizStep $firstStep */
        $firstStep = $quizSteps[0];

        //We expect
        $this->assertNotEmpty($quizSteps);
        $this->assertCount(10, $quizSteps);

        //We expect if quiz is type normal and level variable questions depends of level:
        $this->assertEquals($firstStep->getQuestion(), $dbQuestions[$firstQuestionExpectedIndex]);
    }