01HQP784GW2SPDBN9GAWF20MJ3: test-live-repair

BasicConfig {
    output_rules: [
        "/tmp/*.txt",
        "/tmp/*.log",
        "/tmp/core.*",
        "/tmp/dsc/*.txt",
        "/tmp/dsc.tar",
    ],
    rust_toolchain: None,
    target: Some(
        "helios-2.0",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HQP78E03M7TVGWH25PMRNTXN

Artefacts:

/tmp/crutest-replace.log (1.67MiB) [rendered]
/tmp/dsc.log (1.94KiB) [rendered]
/tmp/dsc/downstairs-8810.txt (137.03KiB)
/tmp/dsc/downstairs-8820.txt (93.88KiB)
/tmp/dsc/downstairs-8830.txt (112.95KiB)
/tmp/dsc/downstairs-8840.txt (113.45KiB)

Output:

SEQ	GLOBAL TIME	DETAILS
1	2024-02-27T21:40:23.327Z	job dependencies complete; ready to run (waiting for 16 m 7 s)
2	2024-02-27T21:41:29.390Z	job assigned to worker 01HQP86AVBAYHQYRB9F2T2N7Y7 (queued for 1 m 6 s)
3	2024-02-27T21:41:30.145Z	downloading input: /input/build/work/bins/crucible-downstairs.gz
4	2024-02-27T21:41:57.336Z	downloaded input: /input/build/work/bins/crucible-downstairs.gz
5	2024-02-27T21:41:57.336Z	downloading input: /input/build/work/bins/crucible-hammer.gz
6	2024-02-27T21:42:34.780Z	downloaded input: /input/build/work/bins/crucible-hammer.gz
7	2024-02-27T21:42:34.780Z	downloading input: /input/build/work/bins/crutest.gz
8	2024-02-27T21:43:02.301Z	downloaded input: /input/build/work/bins/crutest.gz
9	2024-02-27T21:43:02.301Z	downloading input: /input/build/work/bins/dsc.gz
10	2024-02-27T21:43:18.907Z	downloaded input: /input/build/work/bins/dsc.gz
11	2024-02-27T21:43:18.913Z	downloading input: /input/build/work/scripts/perf-downstairs-tick.d
12	2024-02-27T21:43:18.935Z	downloaded input: /input/build/work/scripts/perf-downstairs-tick.d
13	2024-02-27T21:43:18.935Z	downloading input: /input/build/work/scripts/test_ds.sh
14	2024-02-27T21:43:18.953Z	downloaded input: /input/build/work/scripts/test_ds.sh
15	2024-02-27T21:43:18.953Z	downloading input: /input/build/work/scripts/test_live_repair.sh
16	2024-02-27T21:43:19.005Z	downloaded input: /input/build/work/scripts/test_live_repair.sh
17	2024-02-27T21:43:19.006Z	downloading input: /input/build/work/scripts/test_repair.sh
18	2024-02-27T21:43:19.041Z	downloaded input: /input/build/work/scripts/test_repair.sh
19	2024-02-27T21:43:19.042Z	downloading input: /input/build/work/scripts/test_replay.sh
20	2024-02-27T21:43:19.080Z	downloaded input: /input/build/work/scripts/test_replay.sh
21	2024-02-27T21:43:19.080Z	downloading input: /input/build/work/scripts/test_up.sh
22	2024-02-27T21:43:19.099Z	downloaded input: /input/build/work/scripts/test_up.sh
23	2024-02-27T21:43:19.099Z	downloading input: /input/build/work/scripts/upstairs_info.d
24	2024-02-27T21:43:19.583Z	downloaded input: /input/build/work/scripts/upstairs_info.d
25	2024-02-27T21:43:19.596Z	downloading input: /input/build/tmp/cargo-test-out.log
26	2024-02-27T21:43:56.137Z	downloaded input: /input/build/tmp/cargo-test-out.log

27	2024-02-27T21:43:56.137Z	starting task 0: "setup"
28	2024-02-27T21:43:56.352Z	++ uname -s
29	2024-02-27T21:43:56.362Z	+ kern=SunOS
30	2024-02-27T21:43:56.362Z	+ case "$kern" in
31	2024-02-27T21:43:56.362Z	+ groupadd -g 12345 build
32	2024-02-27T21:43:56.363Z	+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
33	2024-02-27T21:43:58.378Z	+ zfs create -o mountpoint=/work rpool/work
34	2024-02-27T21:43:58.531Z	++ awk '$2 == "/home" { print $3 }' /etc/mnttab
35	2024-02-27T21:43:58.957Z	+ home_fs=zfs
36	2024-02-27T21:43:58.957Z	+ [[ zfs == autofs ]]
37	2024-02-27T21:43:58.957Z	+ mkdir -p /home/build
38	2024-02-27T21:43:58.957Z	+ chown build:build /home/build /work
39	2024-02-27T21:43:59.572Z	+ chmod 0700 /home/build /work
40	2024-02-27T21:43:59.602Z	process exited: duration 3430 ms, exit code 0

41	2024-02-27T21:43:59.682Z	starting task 1: "authentication"
42	2024-02-27T21:44:00.150Z	process exited: duration 488 ms, exit code 0

43	2024-02-27T21:44:00.171Z	starting task 2: "build"
44	2024-02-27T21:44:00.628Z	+ banner cores
45	2024-02-27T21:44:00.629Z
46	2024-02-27T21:44:00.629Z	#### #### ##### ###### ####
47	2024-02-27T21:44:00.629Z	# # # # # # # #
48	2024-02-27T21:44:00.629Z	# # # # # ##### ####
49	2024-02-27T21:44:00.629Z	# # # ##### # #
50	2024-02-27T21:44:00.629Z	# # # # # # # # #
51	2024-02-27T21:44:00.629Z	#### #### # # ###### ####
52	2024-02-27T21:44:00.629Z
53	2024-02-27T21:44:00.629Z	+ pfexec coreadm -i /tmp/core.%f.%p -g /tmp/core.%f.%p -e global -e log -e proc-setid -e global-setid
54	2024-02-27T21:44:00.630Z	+ pfexec chmod +x /input/build/work/scripts/perf-downstairs-tick.d /input/build/work/scripts/test_ds.sh /input/build/work/scripts/test_live_repair.sh /input/build/work/scripts/test_repair.sh /input/build/work/scripts/test_replay.sh /input/build/work/scripts/test_up.sh /input/build/work/scripts/upstairs_info.d
55	2024-02-27T21:44:00.630Z	+ echo 'input bins dir contains:'
56	2024-02-27T21:44:00.630Z	input bins dir contains:
57	2024-02-27T21:44:00.630Z	+ ls -ltr /input/build/work/bins
58	2024-02-27T21:44:00.630Z	total 740604
59	2024-02-27T21:44:00.630Z	-rw-r--r-- 1 root root 109635226 Feb 27 21:41 crucible-downstairs.gz
60	2024-02-27T21:44:00.630Z	-rw-r--r-- 1 root root 91526609 Feb 27 21:42 crucible-hammer.gz
61	2024-02-27T21:44:00.631Z	-rw-r--r-- 1 root root 105659381 Feb 27 21:43 crutest.gz
62	2024-02-27T21:44:00.631Z	-rw-r--r-- 1 root root 72121864 Feb 27 21:43 dsc.gz
63	2024-02-27T21:44:00.631Z	+ echo 'input script dir contains:'
64	2024-02-27T21:44:00.631Z	input script dir contains:
65	2024-02-27T21:44:00.632Z	+ ls -ltr /input/build/work/scripts
66	2024-02-27T21:44:00.632Z	total 39
67	2024-02-27T21:44:00.632Z	-rwxr-xr-x 1 root root 3056 Feb 27 21:43 perf-downstairs-tick.d
68	2024-02-27T21:44:00.632Z	-rwxr-xr-x 1 root root 1738 Feb 27 21:43 test_ds.sh
69	2024-02-27T21:44:00.632Z	-rwxr-xr-x 1 root root 3746 Feb 27 21:43 test_live_repair.sh
70	2024-02-27T21:44:00.633Z	-rwxr-xr-x 1 root root 6416 Feb 27 21:43 test_repair.sh
71	2024-02-27T21:44:00.633Z	-rwxr-xr-x 1 root root 2969 Feb 27 21:43 test_replay.sh
72	2024-02-27T21:44:00.633Z	-rwxr-xr-x 1 root root 9465 Feb 27 21:43 test_up.sh
73	2024-02-27T21:44:00.633Z	-rwxr-xr-x 1 root root 2824 Feb 27 21:43 upstairs_info.d
74	2024-02-27T21:44:00.634Z	+ banner Unpack
75	2024-02-27T21:44:00.634Z	# #
76	2024-02-27T21:44:00.634Z	# # # # ##### ## #### # #
77	2024-02-27T21:44:00.634Z	# # ## # # # # # # # # #
78	2024-02-27T21:44:00.634Z	# # # # # # # # # # ####
79	2024-02-27T21:44:00.634Z	# # # # # ##### ###### # # #
80	2024-02-27T21:44:00.634Z	# # # ## # # # # # # #
81	2024-02-27T21:44:00.635Z	##### # # # # # #### # #
82	2024-02-27T21:44:00.635Z
83	2024-02-27T21:44:00.635Z	+ mkdir -p /var/tmp/bins
84	2024-02-27T21:44:00.635Z	+ for t in "$input/bins/"*.gz
85	2024-02-27T21:44:00.635Z	++ basename /input/build/work/bins/crucible-downstairs.gz
86	2024-02-27T21:44:00.635Z	+ b=crucible-downstairs.gz
87	2024-02-27T21:44:00.636Z	+ b=crucible-downstairs
88	2024-02-27T21:44:00.636Z	+ gunzip
89	2024-02-27T21:44:02.803Z	+ chmod +x /var/tmp/bins/crucible-downstairs
90	2024-02-27T21:44:02.816Z	+ for t in "$input/bins/"*.gz
91	2024-02-27T21:44:02.816Z	++ basename /input/build/work/bins/crucible-hammer.gz
92	2024-02-27T21:44:02.816Z	+ b=crucible-hammer.gz
93	2024-02-27T21:44:02.816Z	+ b=crucible-hammer
94	2024-02-27T21:44:02.816Z	+ gunzip
95	2024-02-27T21:44:04.900Z	+ chmod +x /var/tmp/bins/crucible-hammer
96	2024-02-27T21:44:04.906Z	+ for t in "$input/bins/"*.gz
97	2024-02-27T21:44:04.906Z	++ basename /input/build/work/bins/crutest.gz
98	2024-02-27T21:44:04.906Z	+ b=crutest.gz
99	2024-02-27T21:44:04.906Z	+ b=crutest
100	2024-02-27T21:44:04.907Z	+ gunzip
101	2024-02-27T21:44:07.351Z	+ chmod +x /var/tmp/bins/crutest
102	2024-02-27T21:44:07.361Z	+ for t in "$input/bins/"*.gz
103	2024-02-27T21:44:07.361Z	++ basename /input/build/work/bins/dsc.gz
104	2024-02-27T21:44:07.361Z	+ b=dsc.gz
105	2024-02-27T21:44:07.361Z	+ b=dsc
106	2024-02-27T21:44:07.361Z	+ gunzip
107	2024-02-27T21:44:08.964Z	+ chmod +x /var/tmp/bins/dsc
108	2024-02-27T21:44:08.990Z	+ export BINDIR=/var/tmp/bins
109	2024-02-27T21:44:08.990Z	+ BINDIR=/var/tmp/bins
110	2024-02-27T21:44:08.990Z	+ echo 'BINDIR is /var/tmp/bins'
111	2024-02-27T21:44:08.990Z	BINDIR is /var/tmp/bins
112	2024-02-27T21:44:08.990Z	+ echo 'bindir contains:'
113	2024-02-27T21:44:08.990Z	bindir contains:
114	2024-02-27T21:44:08.991Z	+ ls -ltr /var/tmp/bins
115	2024-02-27T21:44:08.991Z	total 771659
116	2024-02-27T21:44:08.991Z	-rwxr-xr-x 1 build build 494825320 Feb 27 21:44 crucible-downstairs
117	2024-02-27T21:44:08.991Z	-rwxr-xr-x 1 build build 403710952 Feb 27 21:44 crucible-hammer
118	2024-02-27T21:44:08.991Z	-rwxr-xr-x 1 build build 466360072 Feb 27 21:44 crutest
119	2024-02-27T21:44:08.991Z	-rwxr-xr-x 1 build build 318272688 Feb 27 21:44 dsc
120	2024-02-27T21:44:08.991Z	+ banner CreateDS
121	2024-02-27T21:44:08.991Z	##### ###### #####
122	2024-02-27T21:44:08.991Z	# # ##### ###### ## ##### ###### # # # #
123	2024-02-27T21:44:08.991Z	# # # # # # # # # # #
124	2024-02-27T21:44:08.992Z	# # # ##### # # # ##### # # #####
125	2024-02-27T21:44:08.992Z	# ##### # ###### # # # # #
126	2024-02-27T21:44:08.992Z	# # # # # # # # # # # # #
127	2024-02-27T21:44:08.992Z	##### # # ###### # # # ###### ###### #####
128	2024-02-27T21:44:08.992Z
129	2024-02-27T21:44:08.992Z	+ echo /var/tmp/bins/dsc create --ds-bin /var/tmp/bins/crucible-downstairs --region-count 4 --cleanup
130	2024-02-27T21:44:08.992Z	/var/tmp/bins/dsc create --ds-bin /var/tmp/bins/crucible-downstairs --region-count 4 --cleanup
131	2024-02-27T21:44:08.993Z	+ /var/tmp/bins/dsc create --ds-bin /var/tmp/bins/crucible-downstairs --region-count 4 --cleanup
132	2024-02-27T21:44:09.012Z	Creating region directory at: /var/tmp/dsc/region
133	2024-02-27T21:44:09.012Z	Creating dsc directory at: /tmp/dsc
134	2024-02-27T21:44:09.463Z	Downstairs region 0 created at /var/tmp/dsc/region/8810 in 0.07886992
135	2024-02-27T21:44:09.463Z	Downstairs region 1 created at /var/tmp/dsc/region/8820 in 0.024365803
136	2024-02-27T21:44:09.464Z	Downstairs region 2 created at /var/tmp/dsc/region/8830 in 0.024049832
137	2024-02-27T21:44:09.464Z	Downstairs region 3 created at /var/tmp/dsc/region/8840 in 0.027382668
138	2024-02-27T21:44:09.464Z	Region set with 4 regions was created
139	2024-02-27T21:44:09.464Z	+ banner StartDS
140	2024-02-27T21:44:09.464Z	##### ###### #####
141	2024-02-27T21:44:09.464Z	# # ##### ## ##### ##### # # # #
142	2024-02-27T21:44:09.464Z	# # # # # # # # # #
143	2024-02-27T21:44:09.464Z	##### # # # # # # # # #####
144	2024-02-27T21:44:09.465Z	# # ###### ##### # # # #
145	2024-02-27T21:44:09.465Z	# # # # # # # # # # # #
146	2024-02-27T21:44:09.465Z	##### # # # # # # ###### #####
147	2024-02-27T21:44:09.465Z
148	2024-02-27T21:44:09.465Z	+ sleep 5
149	2024-02-27T21:44:09.465Z	+ /var/tmp/bins/dsc start --ds-bin /var/tmp/bins/crucible-downstairs --region-count 4
150	2024-02-27T21:44:14.157Z	++ pgrep dsc
151	2024-02-27T21:44:14.166Z	+ dsc_pid=1133
152	2024-02-27T21:44:14.167Z	+ [[ 1133 -eq 0 ]]
153	2024-02-27T21:44:14.167Z	+ ps -p 1133
154	2024-02-27T21:44:14.167Z	PID TTY TIME CMD
155	2024-02-27T21:44:14.167Z	1133 ? 0:00 dsc
156	2024-02-27T21:44:14.167Z	+ echo 'Found dsc running, continue tests'
157	2024-02-27T21:44:14.168Z	Found dsc running, continue tests
158	2024-02-27T21:44:14.168Z	+ banner LR
159	2024-02-27T21:44:14.168Z	# ######
160	2024-02-27T21:44:14.168Z	# # #
161	2024-02-27T21:44:14.168Z	# # #
162	2024-02-27T21:44:14.168Z	# ######
163	2024-02-27T21:44:14.169Z	# # #
164	2024-02-27T21:44:14.169Z	# # #
165	2024-02-27T21:44:14.169Z	####### # #
166	2024-02-27T21:44:14.169Z
167	2024-02-27T21:44:14.169Z	+ ptime -m /var/tmp/bins/crutest replace -t 127.0.0.1:8810 -t 127.0.0.1:8820 -t 127.0.0.1:8830 --replacement 127.0.0.1:8840 -g 1 -c 10 --stable
168	2024-02-27T21:45:30.707Z	+ banner StopDSC
169	2024-02-27T21:45:30.731Z	##### ###### ##### #####
170	2024-02-27T21:45:30.731Z	# # ##### #### ##### # # # # # #
171	2024-02-27T21:45:30.731Z	# # # # # # # # # #
172	2024-02-27T21:45:30.731Z	##### # # # # # # # ##### #
173	2024-02-27T21:45:30.732Z	# # # # ##### # # # #
174	2024-02-27T21:45:30.732Z	# # # # # # # # # # # #
175	2024-02-27T21:45:30.732Z	##### # #### # ###### ##### #####
176	2024-02-27T21:45:30.732Z
177	2024-02-27T21:45:30.733Z	+ /var/tmp/bins/dsc cmd shutdown
178	2024-02-27T21:45:30.762Z	+ banner WaitStop
179	2024-02-27T21:45:30.772Z	# # #####
180	2024-02-27T21:45:30.772Z	# # # ## # ##### # # ##### #### #####
181	2024-02-27T21:45:30.772Z	# # # # # # # # # # # # #
182	2024-02-27T21:45:30.773Z	# # # # # # # ##### # # # # #
183	2024-02-27T21:45:30.773Z	# # # ###### # # # # # # #####
184	2024-02-27T21:45:30.773Z	# # # # # # # # # # # # #
185	2024-02-27T21:45:30.773Z	## ## # # # # ##### # #### #
186	2024-02-27T21:45:30.773Z
187	2024-02-27T21:45:30.773Z	+ wait 1133
188	2024-02-27T21:45:31.765Z	+ _exit_trap
189	2024-02-27T21:45:31.765Z	+ local status=0
190	2024-02-27T21:45:31.765Z	+ [[ 0 -eq 0 ]]
191	2024-02-27T21:45:31.765Z	+ exit 0
192	2024-02-27T21:45:31.775Z	process exited: duration 91586 ms, exit code 0

193	2024-02-27T21:45:31.819Z	found 6 output files
194	2024-02-27T21:45:31.819Z	uploading: /tmp/crutest-replace.log (1753733 bytes)
195	2024-02-27T21:45:33.426Z	uploaded: /tmp/crutest-replace.log
196	2024-02-27T21:45:33.432Z	uploading: /tmp/dsc.log (1991 bytes)
197	2024-02-27T21:45:34.445Z	uploaded: /tmp/dsc.log
198	2024-02-27T21:45:34.445Z	uploading: /tmp/dsc/downstairs-8810.txt (140323 bytes)
199	2024-02-27T21:45:35.532Z	uploaded: /tmp/dsc/downstairs-8810.txt
200	2024-02-27T21:45:35.532Z	uploading: /tmp/dsc/downstairs-8820.txt (96137 bytes)
201	2024-02-27T21:45:36.597Z	uploaded: /tmp/dsc/downstairs-8820.txt
202	2024-02-27T21:45:36.601Z	uploading: /tmp/dsc/downstairs-8830.txt (115665 bytes)
203	2024-02-27T21:45:37.625Z	uploaded: /tmp/dsc/downstairs-8830.txt
204	2024-02-27T21:45:37.629Z	uploading: /tmp/dsc/downstairs-8840.txt (116172 bytes)
205	2024-02-27T21:45:38.656Z	uploaded: /tmp/dsc/downstairs-8840.txt

01HQP784GW2SPDBN9GAWF20MJ3: test-live-repair

Buildomat Job: 01HQP78E03M7TVGWH25PMRNTXN

Tags:

Artefacts:

Output: