Use cases

Case scenario 1: Bisecting permanent issue

  • We have a job failing
  • There are jobs that were coalesced between the last good and the first bad job
  • We need to backfill between good revision and the bad revision

This has been completed by the with –backfill.

Case scenario 2: Bisecting intermittent issue

  • We have an intermittent job
  • We want to determine when it started happening
  • It is not only a matter of coalescing but also a matter of frequency
  • We want to give a range of changesets and bisect until spotting culprit

NOTE: We trigger more than one job compared to case scenario 1

The script helps with triggering multiple times the same jobs. The script helps with tracking filed intermittent oranges in bugzilla.

This use case requires implementing the following milestones:

Case scenario 3: Retrigger an intermittent job on a changeset until hit

  • This is more of an optimization.
  • The intent is to hit the orange with extra debugging information.
  • We’re not bisecting in here.
  • We can trigger batches (e.g. 5 at a time)

Not in scope at the moment.

This could be done with a modification of where we monitor the jobs running until one of them fails.

This use case requires implementing the following milestones:

and this issue:

Case scenario 4: Bisecting Talos

  • We have a performance regression
  • We want to determine when it started showing up
  • Given a revision that _failed_
  • Re-trigger that revision N times and all revisions prior to it until the last data point + 1 more

We can do this manually by using –back-revisions::

python scripts/ \
    --buildername "Rev5 MacOSX Yosemite 10.10 fx-team talos dromaeojs" \
    --rev e16054134e12 --back-revisions 10

We currently don’t have the ability to determine the previous baseline and trigger everything up to the last known data point close to the baseline.

Case scenario 5: After uplift we need a new baseline for release branches

  • We need several data points to establish a baseline
  • After an uplift we need to generate a new baseline
  • Once there is a baseline we can determine regression

This can already be accomplished like this::

python --repo-name try --times 2 --rev 921277f744d9

Case scenario 6: New test validation

  • New test validation
  • Re-triggering to determine indeterminacy
  • Single revision
  • All platforms running test

NOTE: Review this thread on how to determine on which job and which platforms do we run a specific test.

Not in scope at the moment.

Case scenario 7: Fill in a changeset

  • We know that a changeset is missing jobs
  • We want to add all missing jobs

This would need grabbing the list of builders for such repo, determine if there has been a successful job for each builder and trigger it if not. We might need to filter out some periodic builders (i.e. a repo can have pgo periodic jobs).

If we want to watch that revision until it is completely filled we will need to accomplish the following milestones:

Case scenario 8: Developer needs to add missing platforms/jobs for a Try push

  • The developer pushes to try specifying only a subset of all jobs
  • The developer realizes that it needs more jobs to run on that push
  • The developer uses mozci to not have to push again to try with the right syntax

This has been filed as issue 109

Case scenario 9: We generate data to build a dynamic TryChooser UI

  • TryChooser UI is always out of date
  • mozci can generate the data we need to create an up-to-date TryChooser UI

See for an example on how to generate the data needed.