!5261odin/ci-untested-gaps → develop2a7912e028

Run the never-executed L1 daemon tests in CI, and link the errata probes to issues

AI-generatedEverything on this page was written by an AI agent from the diff: the summaries, the narration, the diagrams, the flags and the glossary. It can be wrong or incomplete, so check it against the code before you rely on it.

3commits
4files
123lines changed
111lines to read
4tests now run
0chain code lines

Where the diff is

Read these2 files · 111 lines

.gitlab-ci.yml+96 −0Makefile+15 −0

Tests1 files · 9 lines

x/thorchain/handler_errata_tx_ordering_test.go+5 −4
1pull Zebra + dogecoind2copy zebrad out3start daemons in dind4wait for init blocks5make test-utxo-daemons

4-minute explainer

What changed

Four tests never ran

Every job skipped them: each needs a real daemon.

  • TestV5Zebra · official zebrad
  • TestV6Zebra · official zebrad
  • TestDogeStalledBroadcastIsNotDuplicated · fresh dogecoind
  • TestZecIsolatedFunding · fresh Zebra

New job: test-utxo-daemons

Fresh mocknet daemons in dind, owned by the job.

pullcopy zebradstart in dindwait 1000 / 400make

golang:1.26.10 (trixie) · zebrad needs its glibc

Fails on any skip

--require-test and --max-skips 0: unset variable fails the run.

vars set4 run · 4 passexit 0a var unsetskip > max 0make exits 2

The ZEBRA_ trap

zebrad reads ZEBRA_* env as config; renamed to ZEC_IMAGE.

ZEBRA_IMAGEfield `image`?zebrad exitsZEC_IMAGE ✓

Errata probes name issues

Skip messages link #3298 and #3431, not a closed draft MR.

QueuedErrataCancelsUnexecutedSwap → #3298OrphanedOutboundPermutationsConverge → #3431

still opt-in: RUN_KNOWN_ERRATA_ISSUES=1

Look here first

Slow daemons fail without retryReadiness timeout exits 1, not 75, so no auto-retry; workflow on_job_failure: all then cancels the whole MR pipeline..gitlab-ci.yml:565 ↗verify
556+ -d '{"jsonrpc":"1.0","id":1,"method":"getblockcount","params":[]}' |
557+ python3 -c 'import json, sys; print(json.load(sys.stdin)["result"])' 2>/dev/null || echo 0
558+ }
559+ for i in $(seq 1 150); do
560+ doge=$(height --user thorchain:password http://docker:18332/)
561+ zec=$(height http://docker:18232/)
562+ [ "$doge" -ge 1000 ] && [ "$zec" -gt 400 ] && break
563+ if [ "$i" = 150 ]; then
564+ echo "daemons not ready after 5 minutes: dogecoin=$doge zcash=$zec"
565+ exit 1
566+ fi
567+ sleep 2
568+ done
569+ echo "dogecoin height $doge, zcash height $zec"
570+ - |
571+ export ZEC_DAEMON=/usr/local/bin/zebrad
572+ export DOGE_SIM_INTEGRATION_RPC_URL=http://thorchain:password@docker:18332
573+ export ZEC_SIM_INTEGRATION_RPC_URL=http://docker:18232
574+ make test-utxo-daemons
575+after_script:
576+ - |
577+ if [ "$CI_JOB_STATUS" != "success" ]; then
578+ mkdir -p logs
Fail-closed: no skips allowed--max-skips 0 and --require-test turn an unset daemon variable into a failure; check both invocations carry them.Makefile:345 ↗verify
336336
337+# Tests that need a real L1 daemon. Each daemon must be fresh and used by no
338+# simulation, because the tests spend from the mocknet master address. Set
339+# ZEC_DAEMON to an official zebrad binary, and DOGE_SIM_INTEGRATION_RPC_URL and
340+# ZEC_SIM_INTEGRATION_RPC_URL to the mocknet dogecoind and Zebra RPC endpoints.
341+# A skipped test fails the run, so an unset variable cannot pass unnoticed.
342+.PHONY: test-utxo-daemons
343+test-utxo-daemons:
344+ @python3 scripts/test-quality-run.py --tags mocknet --fresh --run '^TestV[56]Zebra$$' \
345+ --require-test TestV5Zebra --require-test TestV6Zebra --max-skips 0 \
346+ --report-dir test-quality-reports/utxo-daemons-zebra ./bifrost/pkg/chainclients/utxo/zcash
347+ @python3 scripts/test-quality-run.py --module test/simulation --tags mocknet --fresh \
348+ --run '^(TestDogeStalledBroadcastIsNotDuplicated|TestZecIsolatedFunding)$$' \
349+ --require-test TestDogeStalledBroadcastIsNotDuplicated --require-test TestZecIsolatedFunding --max-skips 0 \
350+ --report-dir test-quality-reports/utxo-daemons-simulation ./pkg/utxo
351+
337352 # ------------------------------ Performance Tests ------------------------------
338353
339354 # Run comprehensive benchmarks with profiling (recommended)
340355 test-performance:
341356 @echo "Running comprehensive performance benchmarks with profiling..."
342357 @./test/performance/run_quote_benchmarks.sh
343358
Any ZEBRA_* variable breaks zebradzebrad inherits the job env and maps ZEBRA_<FIELD> to config; a project-level ZEBRA_* CI variable would break it too..gitlab-ci.yml:514 ↗careful
505+rules:
506+ - if: '$CI_MERGE_REQUEST_EVENT_TYPE == "merge_train" && $CI_MERGE_REQUEST_TARGET_BRANCH_SHA == $CI_MERGE_REQUEST_DIFF_BASE_SHA'
507+ when: never
508+ - *rules
509+stage: test
510+needs: []
511+timeout: 30m
512+variables:
513+ PKGS: python3 docker-cli
514+ # Not ZEBRA_*: zebrad reads ZEBRA_<FIELD> env vars as config overrides, and
515+ # the signing tests' zebrad inherits the job environment, so ZEBRA_IMAGE
516+ # makes it reject the unknown config field `image`.
517+ ZEC_IMAGE: zfnd/zebra:6.4.2
518+ DOGE_IMAGE: registry.gitlab.com/thorchain/devops/node-launcher:dogecoin-daemon-1.14.9
519+services:
520+ - docker:28.5.2-dind
521+before_script:
522+ - !reference [.pkg-install, before_script]
523+ - !reference [.wait-for-dind, before_script]
524+ - !reference [.go-private-auth, before_script]
525+ - !reference [.stable-mtimes, before_script]
526+ - !reference [.dind-registries, before_script]
527+script:
Thresholds copied from the entrypoints1000 and 400 mirror DOGE_INITIAL_BLOCKS and zcash INIT_BLOCKS; changing an entrypoint without this stalls the job..gitlab-ci.yml:562 ↗careful
553+ - |
554+ height() {
555+ curl -s --max-time 5 "$@" -H 'Content-Type: application/json' \
556+ -d '{"jsonrpc":"1.0","id":1,"method":"getblockcount","params":[]}' |
557+ python3 -c 'import json, sys; print(json.load(sys.stdin)["result"])' 2>/dev/null || echo 0
558+ }
559+ for i in $(seq 1 150); do
560+ doge=$(height --user thorchain:password http://docker:18332/)
561+ zec=$(height http://docker:18232/)
562+ [ "$doge" -ge 1000 ] && [ "$zec" -gt 400 ] && break
563+ if [ "$i" = 150 ]; then
564+ echo "daemons not ready after 5 minutes: dogecoin=$doge zcash=$zec"
565+ exit 1
566+ fi
567+ sleep 2
568+ done
569+ echo "dogecoin height $doge, zcash height $zec"
570+ - |
571+ export ZEC_DAEMON=/usr/local/bin/zebrad
572+ export DOGE_SIM_INTEGRATION_RPC_URL=http://thorchain:password@docker:18332
573+ export ZEC_SIM_INTEGRATION_RPC_URL=http://docker:18232
574+ make test-utxo-daemons
575+after_script:
zebrad binary runs on the Go imageNeeds trixie's glibc; if the default image leaves trixie, zebrad --version at line 542 fails first..gitlab-ci.yml:540 ↗careful
531+ if [ -n "${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX:-}" ] &&
532+ docker pull -q "$CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX/$ZEC_IMAGE"; then
533+ ZEC_IMAGE="$CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX/$ZEC_IMAGE"
534+ else
535+ docker pull -q "$ZEC_IMAGE" || exit 75
536+ fi
537+ docker pull -q "$DOGE_IMAGE" || exit 75
538+- |
539+ zebra=$(docker create "$ZEC_IMAGE")
540+ docker cp "$zebra:/usr/local/bin/zebrad" /usr/local/bin/zebrad
541+ docker rm "$zebra"
542+ zebrad --version
543+# The same images, entrypoints and settings as the mocknet compose services.
544+- |
545+ mocknet="$CI_PROJECT_DIR/build/docker/mocknet"
546+ docker run -d --name dogecoin --user root -p 18332:18332 -v "$mocknet:/mocknet" \
547+ -e BLOCK_TIME=1 -e DOGE_MASTER_ADDR=mnaioCtEGdw6bd6rWJ13Mbre1kN5rPa2Mo \
548+ --entrypoint /mocknet/dogecoin-entrypoint.sh "$DOGE_IMAGE"
549+ docker run -d --name zcash -p 18232:18232 -e BLOCK_TIME=1 \
550+ -v "$mocknet/zcash/zebrad.toml:/etc/zebrad.toml" -v "$mocknet/zcash/entrypoint.sh:/etc/entrypoint.sh" \
551+ --entrypoint /etc/entrypoint.sh "$ZEC_IMAGE"
552+# Wait for the initial blocks each entrypoint mines to the master address.
553+- |
Proxy pull bypasses DOCKER_MIRROR probedocker-cli lacks buildx, so .dind-registries unsets DOCKER_MIRROR; the job tries the proxy itself, then Docker Hub, exit 75..gitlab-ci.yml:532 ↗look
523+ - !reference [.wait-for-dind, before_script]
524+ - !reference [.go-private-auth, before_script]
525+ - !reference [.stable-mtimes, before_script]
526+ - !reference [.dind-registries, before_script]
527+script:
528+ # .dind-registries probes the mirror with buildx, which docker-cli lacks,
529+ # so try the dependency proxy with a plain pull before Docker Hub.
530+ - |
531+ if [ -n "${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX:-}" ] &&
532+ docker pull -q "$CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX/$ZEC_IMAGE"; then
533+ ZEC_IMAGE="$CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX/$ZEC_IMAGE"
534+ else
535+ docker pull -q "$ZEC_IMAGE" || exit 75
536+ fi
537+ docker pull -q "$DOGE_IMAGE" || exit 75
538+ - |
539+ zebra=$(docker create "$ZEC_IMAGE")
540+ docker cp "$zebra:/usr/local/bin/zebrad" /usr/local/bin/zebrad
541+ docker rm "$zebra"
542+ zebrad --version
543+ # The same images, entrypoints and settings as the mocknet compose services.
544+ - |
545+ mocknet="$CI_PROJECT_DIR/build/docker/mocknet"
Second run skipped if Zebra failsmake stops at the first failing recipe line, so the simulation tests and their JUnit report are missing then.Makefile:347 ↗look
338+# simulation, because the tests spend from the mocknet master address. Set
339+# ZEC_DAEMON to an official zebrad binary, and DOGE_SIM_INTEGRATION_RPC_URL and
340+# ZEC_SIM_INTEGRATION_RPC_URL to the mocknet dogecoind and Zebra RPC endpoints.
341+# A skipped test fails the run, so an unset variable cannot pass unnoticed.
342+.PHONY: test-utxo-daemons
343+test-utxo-daemons:
344+ @python3 scripts/test-quality-run.py --tags mocknet --fresh --run '^TestV[56]Zebra$$' \
345+ --require-test TestV5Zebra --require-test TestV6Zebra --max-skips 0 \
346+ --report-dir test-quality-reports/utxo-daemons-zebra ./bifrost/pkg/chainclients/utxo/zcash
347+ @python3 scripts/test-quality-run.py --module test/simulation --tags mocknet --fresh \
348+ --run '^(TestDogeStalledBroadcastIsNotDuplicated|TestZecIsolatedFunding)$$' \
349+ --require-test TestDogeStalledBroadcastIsNotDuplicated --require-test TestZecIsolatedFunding --max-skips 0 \
350+ --report-dir test-quality-reports/utxo-daemons-simulation ./pkg/utxo
351+
337352 # ------------------------------ Performance Tests ------------------------------
338353
339354 # Run comprehensive benchmarks with profiling (recommended)
340355 test-performance:
341356 @echo "Running comprehensive performance benchmarks with profiling..."
342357 @./test/performance/run_quote_benchmarks.sh
343358
344359 # Quick benchmark comparison (Queue Disabled vs Enabled)
345360 test-performance-quick:
Skip links a confidential issue#3431 is confidential, so non-members get a 404 from the skip message. Issue contents not verifiable offline.handler_errata_tx_ordering_test.go:30 ↗look
2223
2324 // Keep the two known-defect probes runnable without weakening their safety
2425 // assertions. The notes beside each probe explain the unresolved accounting
2526 // behavior. All other ordering tests run normally. Remove each opt-in guard
2627 // once the underlying fix makes the original safety assertions pass in normal CI.
27−func requireKnownErrataIssue(c *C, reason string) {
28+func requireKnownErrataIssue(c *C, reason string, issue int) {
2829 if os.Getenv("RUN_KNOWN_ERRATA_ISSUES") != "1" {
29− c.Skip(reason + "; set RUN_KNOWN_ERRATA_ISSUES=1 to reproduce; follow-up tracking: https://gitlab.com/thorchain/thornode/-/merge_requests/4983#focused-mr-series-expected---may-change")
30+ c.Skip(fmt.Sprintf("%s; set RUN_KNOWN_ERRATA_ISSUES=1 to reproduce; tracked in https://gitlab.com/thorchain/thornode/-/issues/%d", reason, issue))
3031 }
3132 }
3233
3334 type errataOrderingFixture struct {
3435 ctx cosmos.Context
3536 mgr *Mgrs
3637 eb *ebifrost.EnshrinedBifrost
3738 nodes NodeAccounts
3839 keys []secp256k1.PrivKey
3940 tx common.ObservedTx
4041 inbound bool
4142 }
4243

Ask an AI about this change

Chat assistantClaude, ChatGPT, … (reads a link)

Coding agentClaude Code, Cursor, … (reads the repo)

The context file is context.md. Use Copy with context for an AI that can't open links; it pastes the whole file in.

Glossary

Dependency proxy
GitLab's group-level cache of Docker Hub images, addressed via CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX. GitLab docs ↗
dind
Docker-in-Docker: a docker:28.5.2-dind service container the job's docker CLI talks to, reachable as host docker.
Errata
A THORChain tx that reverses an observed inbound that vanished from the external chain, e.g. after a reorg. how-bifrost-works.md ↗ handler_errata_tx.go ↗
Exit 75
EX_TEMPFAIL. The repo's default retry policy retries a job on exit 75, used to flag infrastructure failures. .gitlab-ci.yml ↗
mocknet
THORNode's local docker-compose network with regtest L1 daemons. The job reuses its images, entrypoints and settings. docker-compose.yml ↗
RUN_KNOWN_ERRATA_ISSUES
Opt-in switch for two errata probes that still fail on develop; unset, they skip with their issue link. ordering test ↗
test-quality-run.py
The repo's go test wrapper: writes JUnit and a summary, and fails on missing required tests or too many skips. scripts/test-quality-run.py ↗
trixie
Debian 13. golang:1.26.10 is built on it, and zebrad 6.4.2 needs its glibc.
zebrad / Zebra
The Zcash Foundation's Zcash node. Mocknet runs zfnd/zebra:6.4.2; the job copies its zebrad binary out for the v5/v6 signing tests. zcash entrypoint ↗ Zebra book ↗