Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(e2e): add logging for failing test #5742

Closed
wants to merge 9 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions .github/workflows/e2e.yml
Original file line number Diff line number Diff line change
Expand Up @@ -115,8 +115,8 @@ jobs:
# Be sure to update all instances in this file and `pr-cleanup.yml` if updated
project: [chromium, firefox]
# Add more shards here if needed
shardIndex: [1, 2]
shardTotal: [2]
shardIndex: [1, 2, 3]
shardTotal: [3]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
Expand Down Expand Up @@ -214,6 +214,14 @@ jobs:
name: playwright-report
path: blob-report
retention-days: 30

# Upload tracing reports
- uses: actions/upload-artifact@v3
if: always()
with:
name: ${{ matrix.project }}-test-report
path: test/e2e/results

merge-reports:
if: always()
needs: [playwright-test]
Expand Down
71 changes: 52 additions & 19 deletions test/e2e/tests/inputs/text.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
* Makeshift code to reproduce a specific bug.
*/
import {expect} from '@playwright/test'
import {createClient} from '@sanity/client'
import {test} from '@sanity/test'

const kanji = `
Expand All @@ -12,62 +13,94 @@
住ゅなぜ日16語約セヤチ任政崎ソオユ枠体ぞン古91一専泉給12関モリレネ解透ぴゃラぼ転地す球北ドざう記番重投ぼづ。期ゃ更緒リだすし夫内オ代他られくド潤刊本クヘフ伊一ウムニヘ感週け出入ば勇起ょ関図ぜ覧説めわぶ室訪おがト強車傾町コ本喰杜椿榎ほれた。暮る生的更芸窓どさはむ近問ラ入必ラニス療心コウ怒応りめけひ載総ア北吾ヌイヘ主最ニ余記エツヤ州5念稼め化浮ヌリ済毎養ぜぼ。
`.trim()

const sanityClient = createClient({
projectId: process.env.SANITY_E2E_PROJECT_ID,
dataset: process.env.SANITY_E2E_DATASET,
token: process.env.SANITY_E2E_SESSION_TOKEN,
useCdn: false,
apiVersion: '2024-02-14',
})

test.describe('inputs: text', () => {
test.slow() // Because of waiting for mutations, remote values etc

test('correctly applies kanji edits', async ({page, sanityClient, createDraftDocument}) => {
test('correctly applies kanji edits', async ({page, createDraftDocument}) => {
const documentId = await createDraftDocument('/test/content/input-ci;textsTest')

function getRemoteValue() {
return sanityClient
.getDocument(`drafts.${documentId}`)
.then((doc) => (doc ? doc.simple : null))
async function getRemoteValue() {
try {
const docs = await sanityClient.fetch(`*[_type == $type] { _id, _type, simple }`, {
type: 'textsTest',
})

// eslint-disable-next-line no-console
console.log('docs', JSON.stringify(docs, null, 2))
const doc = await sanityClient.getDocument(`drafts.${documentId}`)

// eslint-disable-next-line no-console
console.log('doc', `drafts.${documentId}`, doc)
return doc ? doc.simple : null
} catch (err) {
console.error('Error fetching remote value', err)
return null
}
}

await page.waitForSelector('data-testid=field-simple', {timeout: 30000})
const field = page.getByTestId('field-simple').getByRole('textbox')

// Enter initial text and wait for the mutate call to be sent
const response = page.waitForResponse(/mutate/)
const response1 = page.waitForResponse(async (res) => {
// eslint-disable-next-line no-console
console.log(res.url(), await res.json())
return res.url().includes('mutate') && res.status() === 200
})
await field.fill(kanji)
await response
// Enter initial text and wait for the mutate call to be sent
await response1

// Expect the document to now have the base value
let currentExpectedValue = kanji
expect(await field.inputValue()).toBe(currentExpectedValue)
expect(await getRemoteValue()).toBe(currentExpectedValue)
await expect(field).toHaveValue(currentExpectedValue)
const remoteValue1 = await getRemoteValue()
expect(remoteValue1).toBe(currentExpectedValue)

Check failure on line 65 in test/e2e/tests/inputs/text.spec.ts

View workflow job for this annotation

GitHub Actions / playwright-test (chromium, 3, 3)

[chromium] › inputs/text.spec.ts:27:7 › inputs: text › correctly applies kanji edits

1) [chromium] › inputs/text.spec.ts:27:7 › inputs: text › correctly applies kanji edits ────────── Error: expect(received).toBe(expected) // Object.is equality Expected: "速ヒマヤレ誌相ルなあね日諸せ変評ホ真攻同潔ク作先た員勝どそ際接レゅ自17浅ッ実情スヤ籍認ス重力務鳥の。8平はートご多乗12青國暮整ル通国うれけこ能新ロコラハ元横ミ休探ミソ梓批ざょにね薬展むい本隣ば禁抗ワアミ部真えくト提知週むすほ。査ル人形ルおじつ政謙減セヲモ読見れレぞえ録精てざ定第ぐゆとス務接産ヤ写馬エモス聞氏サヘマ有午ごね客岡ヘロ修彩枝雨父のけリド。· 住ゅなぜ日16語約セヤチ任政崎ソオユ枠体ぞン古91一専泉給12関モリレネ解透ぴゃラぼ転地す球北ドざう記番重投ぼづ。期ゃ更緒リだすし夫内オ代他られくド潤刊本クヘフ伊一ウムニヘ感週け出入ば勇起ょ関図ぜ覧説めわぶ室訪おがト強車傾町コ本喰杜椿榎ほれた。暮る生的更芸窓どさはむ近問ラ入必ラニス療心コウ怒応りめけひ載総ア北吾ヌイヘ主最ニ余記エツヤ州5念稼め化浮ヌリ済毎養ぜぼ。" Received: null 63 | await expect(field).toHaveValue(currentExpectedValue) 64 | const remoteValue1 = await getRemoteValue() > 65 | expect(remoteValue1).toBe(currentExpectedValue) | ^ 66 | 67 | // Edit the value to start with "Paragraph 1: " 68 | const p1Prefix = 'Paragraph 1: ' at /home/runner/work/sanity/sanity/test/e2e/tests/inputs/text.spec.ts:65:26

Check failure on line 65 in test/e2e/tests/inputs/text.spec.ts

View workflow job for this annotation

GitHub Actions / playwright-test (chromium, 3, 3)

[chromium] › inputs/text.spec.ts:27:7 › inputs: text › correctly applies kanji edits

1) [chromium] › inputs/text.spec.ts:27:7 › inputs: text › correctly applies kanji edits ────────── Retry #1 ─────────────────────────────────────────────────────────────────────────────────────── Error: expect(received).toBe(expected) // Object.is equality Expected: "速ヒマヤレ誌相ルなあね日諸せ変評ホ真攻同潔ク作先た員勝どそ際接レゅ自17浅ッ実情スヤ籍認ス重力務鳥の。8平はートご多乗12青國暮整ル通国うれけこ能新ロコラハ元横ミ休探ミソ梓批ざょにね薬展むい本隣ば禁抗ワアミ部真えくト提知週むすほ。査ル人形ルおじつ政謙減セヲモ読見れレぞえ録精てざ定第ぐゆとス務接産ヤ写馬エモス聞氏サヘマ有午ごね客岡ヘロ修彩枝雨父のけリド。· 住ゅなぜ日16語約セヤチ任政崎ソオユ枠体ぞン古91一専泉給12関モリレネ解透ぴゃラぼ転地す球北ドざう記番重投ぼづ。期ゃ更緒リだすし夫内オ代他られくド潤刊本クヘフ伊一ウムニヘ感週け出入ば勇起ょ関図ぜ覧説めわぶ室訪おがト強車傾町コ本喰杜椿榎ほれた。暮る生的更芸窓どさはむ近問ラ入必ラニス療心コウ怒応りめけひ載総ア北吾ヌイヘ主最ニ余記エツヤ州5念稼め化浮ヌリ済毎養ぜぼ。" Received: null 63 | await expect(field).toHaveValue(currentExpectedValue) 64 | const remoteValue1 = await getRemoteValue() > 65 | expect(remoteValue1).toBe(currentExpectedValue) | ^ 66 | 67 | // Edit the value to start with "Paragraph 1: " 68 | const p1Prefix = 'Paragraph 1: ' at /home/runner/work/sanity/sanity/test/e2e/tests/inputs/text.spec.ts:65:26

Check failure on line 65 in test/e2e/tests/inputs/text.spec.ts

View workflow job for this annotation

GitHub Actions / playwright-test (firefox, 3, 3)

[firefox] › inputs/text.spec.ts:27:7 › inputs: text › correctly applies kanji edits

1) [firefox] › inputs/text.spec.ts:27:7 › inputs: text › correctly applies kanji edits ─────────── Error: expect(received).toBe(expected) // Object.is equality Expected: "速ヒマヤレ誌相ルなあね日諸せ変評ホ真攻同潔ク作先た員勝どそ際接レゅ自17浅ッ実情スヤ籍認ス重力務鳥の。8平はートご多乗12青國暮整ル通国うれけこ能新ロコラハ元横ミ休探ミソ梓批ざょにね薬展むい本隣ば禁抗ワアミ部真えくト提知週むすほ。査ル人形ルおじつ政謙減セヲモ読見れレぞえ録精てざ定第ぐゆとス務接産ヤ写馬エモス聞氏サヘマ有午ごね客岡ヘロ修彩枝雨父のけリド。· 住ゅなぜ日16語約セヤチ任政崎ソオユ枠体ぞン古91一専泉給12関モリレネ解透ぴゃラぼ転地す球北ドざう記番重投ぼづ。期ゃ更緒リだすし夫内オ代他られくド潤刊本クヘフ伊一ウムニヘ感週け出入ば勇起ょ関図ぜ覧説めわぶ室訪おがト強車傾町コ本喰杜椿榎ほれた。暮る生的更芸窓どさはむ近問ラ入必ラニス療心コウ怒応りめけひ載総ア北吾ヌイヘ主最ニ余記エツヤ州5念稼め化浮ヌリ済毎養ぜぼ。" Received: null 63 | await expect(field).toHaveValue(currentExpectedValue) 64 | const remoteValue1 = await getRemoteValue() > 65 | expect(remoteValue1).toBe(currentExpectedValue) | ^ 66 | 67 | // Edit the value to start with "Paragraph 1: " 68 | const p1Prefix = 'Paragraph 1: ' at /home/runner/work/sanity/sanity/test/e2e/tests/inputs/text.spec.ts:65:26

Check failure on line 65 in test/e2e/tests/inputs/text.spec.ts

View workflow job for this annotation

GitHub Actions / playwright-test (firefox, 3, 3)

[firefox] › inputs/text.spec.ts:27:7 › inputs: text › correctly applies kanji edits

1) [firefox] › inputs/text.spec.ts:27:7 › inputs: text › correctly applies kanji edits ─────────── Retry #1 ─────────────────────────────────────────────────────────────────────────────────────── Error: expect(received).toBe(expected) // Object.is equality Expected: "速ヒマヤレ誌相ルなあね日諸せ変評ホ真攻同潔ク作先た員勝どそ際接レゅ自17浅ッ実情スヤ籍認ス重力務鳥の。8平はートご多乗12青國暮整ル通国うれけこ能新ロコラハ元横ミ休探ミソ梓批ざょにね薬展むい本隣ば禁抗ワアミ部真えくト提知週むすほ。査ル人形ルおじつ政謙減セヲモ読見れレぞえ録精てざ定第ぐゆとス務接産ヤ写馬エモス聞氏サヘマ有午ごね客岡ヘロ修彩枝雨父のけリド。· 住ゅなぜ日16語約セヤチ任政崎ソオユ枠体ぞン古91一専泉給12関モリレネ解透ぴゃラぼ転地す球北ドざう記番重投ぼづ。期ゃ更緒リだすし夫内オ代他られくド潤刊本クヘフ伊一ウムニヘ感週け出入ば勇起ょ関図ぜ覧説めわぶ室訪おがト強車傾町コ本喰杜椿榎ほれた。暮る生的更芸窓どさはむ近問ラ入必ラニス療心コウ怒応りめけひ載総ア北吾ヌイヘ主最ニ余記エツヤ州5念稼め化浮ヌリ済毎養ぜぼ。" Received: null 63 | await expect(field).toHaveValue(currentExpectedValue) 64 | const remoteValue1 = await getRemoteValue() > 65 | expect(remoteValue1).toBe(currentExpectedValue) | ^ 66 | 67 | // Edit the value to start with "Paragraph 1: " 68 | const p1Prefix = 'Paragraph 1: ' at /home/runner/work/sanity/sanity/test/e2e/tests/inputs/text.spec.ts:65:26

// Edit the value to start with "Paragraph 1: "
const p1Prefix = 'Paragraph 1: '
let nextExpectedValue = `${p1Prefix}${kanji}`
const response2 = page.waitForResponse(/mutate/)
await field.fill(nextExpectedValue)
await page.waitForTimeout(1000) // Hack, we need to wait for the mutation to be received
await response2

// Expect both the browser input and the document to now have the updated value
currentExpectedValue = `${p1Prefix}${kanji}`
expect(await field.inputValue()).toBe(currentExpectedValue)
expect(await getRemoteValue()).toBe(currentExpectedValue)
await expect(field).toHaveValue(currentExpectedValue)
const remoteValue2 = await getRemoteValue()
expect(remoteValue2).toBe(currentExpectedValue)

// Now move to the end of the paragraph and add a suffix
const p1Suffix = ' (end of paragraph 1)'
nextExpectedValue = currentExpectedValue.replace(/\n\n/, `${p1Suffix}\n\n`)
const response3 = page.waitForResponse(/mutate/)
await field.fill(nextExpectedValue)
await page.waitForTimeout(1000) // Hack, we need to wait for the mutation to be received
await response3

// Expect both the browser input and the document to now have the updated value
currentExpectedValue = nextExpectedValue
expect(await field.inputValue()).toBe(currentExpectedValue)
expect(await getRemoteValue()).toBe(currentExpectedValue)
await expect(field).toHaveValue(currentExpectedValue)
const remoteValue3 = await getRemoteValue()
expect(remoteValue3).toBe(currentExpectedValue)

// Move to the end of the field and add a final suffix
const p2Suffix = `. EOL.`
nextExpectedValue = `${currentExpectedValue}${p2Suffix}`
const response4 = page.waitForResponse(/mutate/)
await field.fill(nextExpectedValue)
await page.waitForTimeout(1000) // Hack, we need to wait for the mutation to be received
await response4

// Expect both the browser input and the document to now have the updated value
currentExpectedValue = nextExpectedValue
expect(await field.inputValue()).toBe(currentExpectedValue)
expect(await getRemoteValue()).toBe(currentExpectedValue)
await expect(field).toHaveValue(currentExpectedValue)
const remoteValue4 = await getRemoteValue()
expect(remoteValue4).toBe(currentExpectedValue)
})
})
Loading