[{"data":1,"prerenderedAt":327},["ShallowReactive",2],{"navigation_docs":3,"-learn-running-in-ci":93,"-learn-running-in-ci-surround":322},[4,41,46,89],{"title":5,"icon":6,"path":7,"stem":8,"children":9,"page":40},"Learn","i-lucide-book-open","\u002Flearn","1.learn",[10,15,20,25,30,35],{"title":11,"path":12,"stem":13,"icon":14},"What visual regression testing is","\u002Flearn\u002Fwhat-is-visual-regression-testing","1.learn\u002F1.what-is-visual-regression-testing","i-lucide-eye",{"title":16,"path":17,"stem":18,"icon":19},"How image diffing works","\u002Flearn\u002Fhow-image-diffing-works","1.learn\u002F2.how-image-diffing-works","i-lucide-diff",{"title":21,"path":22,"stem":23,"icon":24},"Why screenshots flake","\u002Flearn\u002Fwhy-screenshots-flake","1.learn\u002F3.why-screenshots-flake","i-lucide-shuffle",{"title":26,"path":27,"stem":28,"icon":29},"Baselines and approval","\u002Flearn\u002Fbaselines-and-approval","1.learn\u002F4.baselines-and-approval","i-lucide-check-check",{"title":31,"path":32,"stem":33,"icon":34},"Running in CI","\u002Flearn\u002Frunning-in-ci","1.learn\u002F5.running-in-ci","i-lucide-workflow",{"title":36,"path":37,"stem":38,"icon":39},"Choosing a tool","\u002Flearn\u002Fchoosing-a-tool","1.learn\u002F6.choosing-a-tool","i-lucide-list-checks",false,{"title":42,"path":43,"stem":44,"icon":45},"Compare the tools","\u002Fcompare","2.compare","i-lucide-scale",{"title":47,"path":48,"stem":49,"children":50,"icon":53},"Manifest","\u002Fmanifest","3.manifest\u002F1.index",[51,54,59,64,69,74,79,84],{"title":52,"path":48,"stem":49,"icon":53},"The run manifest","i-lucide-file-json",{"title":55,"path":56,"stem":57,"icon":58},"The format","\u002Fmanifest\u002Fformat","3.manifest\u002F2.format","i-lucide-file-code",{"title":60,"path":61,"stem":62,"icon":63},"Reading manifests","\u002Fmanifest\u002Freading","3.manifest\u002F3.reading","i-lucide-book-open-check",{"title":65,"path":66,"stem":67,"icon":68},"Merging the manifests of a run","\u002Fmanifest\u002Fmerging","3.manifest\u002F4.merging","i-lucide-merge",{"title":70,"path":71,"stem":72,"icon":73},"Writing manifests","\u002Fmanifest\u002Fwriting","3.manifest\u002F5.writing","i-lucide-pencil-line",{"title":75,"path":76,"stem":77,"icon":78},"Converting to and from other formats","\u002Fmanifest\u002Fconverting","3.manifest\u002F6.converting","i-lucide-arrow-left-right",{"title":80,"path":81,"stem":82,"icon":83},"Reference","\u002Fmanifest\u002Freference","3.manifest\u002F7.reference","i-lucide-list",{"title":85,"path":86,"stem":87,"icon":88},"Schema reference","\u002Fmanifest\u002Fschema","3.manifest\u002F8.schema","i-lucide-braces",{"title":90,"path":91,"stem":92,"icon":14},"Viewer","\u002Fviewer","4.viewer",{"id":94,"title":31,"body":95,"description":315,"extension":316,"links":317,"meta":318,"navigation":319,"path":32,"seo":320,"stem":33,"__hash__":321},"docs\u002F1.learn\u002F5.running-in-ci.md",{"type":96,"value":97,"toc":304},"minimark",[98,102,107,120,124,144,151,155,178,182,185,217,231,235,249,252,256,274,278],[99,100,101],"p",{},"Visual tests in CI differ from other tests in two ways: they produce files that a human must see, and they are sensitive to the machine they run on. Both shape the pipeline.",[103,104,106],"h2",{"id":105},"run-in-the-pinned-environment","Run in the pinned environment",[99,108,109,110,114,115,119],{},"The CI job should run the comparison in the same container image that produced the baselines (",[111,112,113],"a",{"href":22},"why","). In GitHub Actions that is a ",[116,117,118],"code",{},"container:"," on the job or a Docker step; the same applies elsewhere. Pin the image by tag or digest and update baselines when you update it.",[103,121,123],{"id":122},"keep-the-images","Keep the images",[99,125,126,127,131,132,135,136,139,140,143],{},"A failed comparison leaves actual and diff images on the runner's disk, which disappears with the job. Upload them as an ",[128,129,130],"strong",{},"artifact"," so a reviewer can see them: ",[116,133,134],{},"actions\u002Fupload-artifact"," on GitHub, ",[116,137,138],{},"artifacts:"," in GitLab, the equivalent elsewhere. Upload on failure at least; uploading always is simpler and lets you inspect passing runs. Include the baseline too, so the three images travel together, and include the ",[111,141,142],{"href":48},"manifest"," if the tool writes one, so the artifact describes itself.",[99,145,146,147,150],{},"A good artifact is self-contained: unpack it anywhere and a viewer (the ",[111,148,149],{"href":91},"one on this site",", Playwright's HTML report, BackstopJS's report) can show the run.",[103,152,154],{"id":153},"shard-and-merge","Shard and merge",[99,156,157,158,161,162,165,166,169,170,173,174,177],{},"Large suites run across several machines, and each machine produces its own images and results. The reporting step then needs to ",[128,159,160],{},"merge"," them. Playwright's ",[116,163,164],{},"--shard"," with ",[116,167,168],{},"merge-reports",", Vitest's sharding with its blob reporter, Cypress parallelisation through a dashboard service all handle their own results; a visual review step on top of them has to read every shard's output. This is the case the manifest was designed for: one file per writer, named ",[116,171,172],{},"visual-regression-manifest.\u003Clabel>.json",", found with one glob across all artifacts and merged with ",[116,175,176],{},"mergeManifests",", with a re-run attempt winning over the earlier one.",[103,179,181],{"id":180},"report-on-the-pull-request","Report on the pull request",[99,183,184],{},"Failing the job is necessary and not sufficient. The reviewer needs to see the diffs without leaving the PR. Options, cheapest first:",[186,187,188,199,205,211],"ol",{},[189,190,191,194,195,198],"li",{},[128,192,193],{},"A job summary"," (",[116,196,197],{},"$GITHUB_STEP_SUMMARY"," on GitHub) with the failed screenshot names and the artifact link.",[189,200,201,204],{},[128,202,203],{},"A PR comment"," with the counts and, where the images are reachable, the baseline, actual and diff inline. Images in comments need a URL, which means uploading them somewhere public or using the artifact's download link.",[189,206,207,210],{},[128,208,209],{},"A check run"," with annotations per failed screenshot, and a button that approves. This is what the hosted services and GitHub Apps provide.",[189,212,213,216],{},[128,214,215],{},"The service's UI",", linked from a status check, for tools with service-side baselines.",[99,218,219,220,223,224,230],{},"If your tool writes CTRF or a manifest (which ",[111,221,222],{"href":76},"converts to CTRF","), the existing ",[111,225,229],{"href":226,"rel":227},"https:\u002F\u002Fgithub.com\u002Fctrf-io",[228],"nofollow","CTRF GitHub Actions"," post summaries and comments without any custom code.",[103,232,234],{"id":233},"approve-from-ci","Approve from CI",[99,236,237,238,241,242,244,245,248],{},"With repository baselines, the approval has to end as a commit. Two patterns work well: a workflow triggered by a PR comment (",[116,239,240],{},"\u002Fapprove-visuals",") or a label, which downloads the artifact, copies the approved actual images over their baselines and pushes to the PR branch; or an app that does the same from a button on the check. Both need the paths of the images relative to the repository, which is what ",[116,243,176],{}," computes as ",[116,246,247],{},"repoPaths",", and both must refuse paths that escape the repository.",[99,250,251],{},"Make sure the approval commit does not trigger an infinite loop (skip CI on it, or let it run and expect it to pass), and that the token used can push to forks only if you intend that.",[103,253,255],{"id":254},"keep-it-fast-and-cheap","Keep it fast and cheap",[257,258,259,262,265,268,271],"ul",{},[189,260,261],{},"Run visual tests on PRs, not on every push to every branch, unless they are fast.",[189,263,264],{},"Screenshot components rather than whole pages where you can: smaller images, faster diffs, more precise failures.",[189,266,267],{},"Cache the browser binaries and the renderer image.",[189,269,270],{},"Compare locally in the runner before uploading anything; upload only the failures' images when the artifact gets big.",[189,272,273],{},"For hosted services billed per screenshot, screenshot once per PR, not once per push, and skip the run when only backend code changed.",[103,275,277],{"id":276},"further-reading","Further reading",[257,279,280,288,296],{},[189,281,282,287],{},[111,283,286],{"href":284,"rel":285},"https:\u002F\u002Fdocs.github.com\u002Fen\u002Factions\u002Fwriting-workflows\u002Fchoosing-what-your-workflow-does\u002Fstoring-and-sharing-data-from-a-workflow",[228],"Storing and sharing data from a workflow"," in the GitHub Actions docs.",[189,289,290,295],{},[111,291,294],{"href":292,"rel":293},"https:\u002F\u002Fplaywright.dev\u002Fdocs\u002Ftest-sharding",[228],"Playwright sharding"," and merging reports.",[189,297,298,303],{},[111,299,302],{"href":300,"rel":301},"https:\u002F\u002Fctrf.io",[228],"CTRF",", the Common Test Report Format, and its GitHub Actions.",{"title":305,"searchDepth":306,"depth":306,"links":307},"",2,[308,309,310,311,312,313,314],{"id":105,"depth":306,"text":106},{"id":122,"depth":306,"text":123},{"id":153,"depth":306,"text":154},{"id":180,"depth":306,"text":181},{"id":233,"depth":306,"text":234},{"id":254,"depth":306,"text":255},{"id":276,"depth":306,"text":277},"Artifacts, sharding, merging results from several machines, reporting on the pull request, and keeping the bill down.","md",null,{},{"icon":34},{"title":31,"description":315},"2odZ2940M6uq7I_b4g8FwLgNcqzKrABlCZo8ydPNfb8",[323,325],{"title":26,"path":27,"stem":28,"description":324,"icon":29,"children":-1},"Where baselines live, who approves a change, how to keep history, and why the review step is the part of the workflow most teams get wrong.",{"title":36,"path":37,"stem":38,"description":326,"icon":39,"children":-1},"The four questions that narrow the field, the trade-offs between local tools and hosted services, and what to look for so you can change your mind later.",1791993875572]