01KKCDN0NP8BEDTNWCXZ98KST1: test-linux

BasicConfig {
    output_rules: [],
    rust_toolchain: None,
    target: Some(
        "ubuntu-22.04",
    ),
    access_repos: [],
    publish: [],
    skip_clone: false,
}

Buildomat Job: 01KKCDNJW20734V4KCSZXHWXKK

Tags:

Output:

SEQ GLOBAL TIME DETAILS
12026-03-10T17:46:13.026Zjob assigned to worker 01KKCDNSCP933P7QQDTPJ9ETHD [factory aws, i-0e42d401a553a53c6] (queued for 29 s)
 
22026-03-10T17:46:13.925Zstarting task 0: "setup"
32026-03-10T17:46:13.928Z++ uname -s
42026-03-10T17:46:13.928Z+ kern=Linux
52026-03-10T17:46:13.928Z+ build_user=build
62026-03-10T17:46:13.928Z+ build_uid=12345
72026-03-10T17:46:13.928Z+ work_dir=/work
82026-03-10T17:46:13.928Z+ input_dir=/input
92026-03-10T17:46:13.928Z+ [[ 0 == 12345 ]]
102026-03-10T17:46:13.928Z+ case "$kern" in
112026-03-10T17:46:13.928Z+ apt-get -y update
122026-03-10T17:46:13.983ZHit:1 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy InRelease
132026-03-10T17:46:13.983ZGet:2 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB]
142026-03-10T17:46:13.989ZGet:3 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB]
152026-03-10T17:46:14.035ZGet:4 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/universe amd64 Packages [14.1 MB]
162026-03-10T17:46:14.118ZGet:5 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/universe Translation-en [5,652 kB]
172026-03-10T17:46:14.137ZGet:6 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB]
182026-03-10T17:46:14.148ZGet:7 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/universe amd64 c-n-f Metadata [286 kB]
192026-03-10T17:46:14.155ZGet:8 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/multiverse amd64 Packages [217 kB]
202026-03-10T17:46:14.155ZGet:9 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/multiverse Translation-en [112 kB]
212026-03-10T17:46:14.155ZGet:10 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/multiverse amd64 c-n-f Metadata [8,372 B]
222026-03-10T17:46:14.155ZGet:11 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [3,285 kB]
232026-03-10T17:46:14.163ZGet:12 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [499 kB]
242026-03-10T17:46:14.166ZGet:13 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [19.2 kB]
252026-03-10T17:46:14.166ZGet:14 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [5,414 kB]
262026-03-10T17:46:14.185ZGet:15 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [1,028 kB]
272026-03-10T17:46:14.187ZGet:16 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [676 B]
282026-03-10T17:46:14.190ZGet:17 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,256 kB]
292026-03-10T17:46:14.193ZGet:18 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [314 kB]
302026-03-10T17:46:14.196ZGet:19 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [30.4 kB]
312026-03-10T17:46:14.196ZGet:20 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [59.0 kB]
322026-03-10T17:46:14.196ZGet:21 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [13.5 kB]
332026-03-10T17:46:14.196ZGet:22 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [612 B]
342026-03-10T17:46:14.199ZGet:23 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [69.4 kB]
352026-03-10T17:46:14.199ZGet:24 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.5 kB]
362026-03-10T17:46:14.202ZGet:25 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [412 B]
372026-03-10T17:46:14.202ZGet:26 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-backports/restricted amd64 c-n-f Metadata [116 B]
382026-03-10T17:46:14.205ZGet:27 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [31.7 kB]
392026-03-10T17:46:14.205ZGet:28 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.9 kB]
402026-03-10T17:46:14.208ZGet:29 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B]
412026-03-10T17:46:14.208ZGet:30 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-backports/multiverse amd64 c-n-f Metadata [116 B]
422026-03-10T17:46:14.533ZGet:31 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [3,017 kB]
432026-03-10T17:46:15.149ZGet:32 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [430 kB]
442026-03-10T17:46:15.156ZGet:33 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [14.1 kB]
452026-03-10T17:46:15.160ZGet:34 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [5,238 kB]
462026-03-10T17:46:15.401ZGet:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [998 kB]
472026-03-10T17:46:15.418ZGet:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [680 B]
482026-03-10T17:46:15.421ZGet:37 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [1,018 kB]
492026-03-10T17:46:15.482ZGet:38 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [225 kB]
502026-03-10T17:46:15.485ZGet:39 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [22.7 kB]
512026-03-10T17:46:15.489ZGet:40 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [51.9 kB]
522026-03-10T17:46:15.489ZGet:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [10.6 kB]
532026-03-10T17:46:15.489ZGet:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [388 B]
542026-03-10T17:46:20.817ZFetched 43.8 MB in 3s (15.0 MB/s)
552026-03-10T17:46:21.309ZReading package lists...
562026-03-10T17:46:21.322Z+ apt-get -y install sysvbanner build-essential
572026-03-10T17:46:21.343ZReading package lists...
582026-03-10T17:46:21.459ZBuilding dependency tree...
592026-03-10T17:46:21.462ZReading state information...
602026-03-10T17:46:21.562ZThe following additional packages will be installed:
612026-03-10T17:46:21.562Z bzip2 cpp cpp-11 dpkg-dev fakeroot fontconfig-config fonts-dejavu-core g++
622026-03-10T17:46:21.562Z g++-11 gcc gcc-11 gcc-11-base gcc-12-base libalgorithm-diff-perl
632026-03-10T17:46:21.562Z libalgorithm-diff-xs-perl libalgorithm-merge-perl libasan6 libatomic1
642026-03-10T17:46:21.565Z libc-dev-bin libc-devtools libc6 libc6-dev libcc1-0 libcrypt-dev libdeflate0
652026-03-10T17:46:21.565Z libdpkg-perl libfakeroot libfile-fcntllock-perl libfontconfig1 libgcc-11-dev
662026-03-10T17:46:21.565Z libgcc-s1 libgd3 libgomp1 libisl23 libitm1 libjbig0 libjpeg-turbo8 libjpeg8
672026-03-10T17:46:21.565Z liblsan0 libmpc3 libnsl-dev libquadmath0 libstdc++-11-dev libstdc++6
682026-03-10T17:46:21.565Z libtiff5 libtirpc-common libtirpc-dev libtirpc3 libtsan0 libubsan1 libwebp7
692026-03-10T17:46:21.565Z libxpm4 linux-libc-dev lto-disabled-list make manpages-dev rpcsvc-proto
702026-03-10T17:46:21.565ZSuggested packages:
712026-03-10T17:46:21.565Z bzip2-doc cpp-doc gcc-11-locales debian-keyring g++-multilib g++-11-multilib
722026-03-10T17:46:21.565Z gcc-11-doc gcc-multilib autoconf automake libtool flex bison gdb gcc-doc
732026-03-10T17:46:21.565Z gcc-11-multilib glibc-doc bzr libgd-tools libstdc++-11-doc make-doc
742026-03-10T17:46:21.565ZRecommended packages:
752026-03-10T17:46:21.565Z libnss-nis libnss-nisplus
762026-03-10T17:46:21.612ZThe following NEW packages will be installed:
772026-03-10T17:46:21.612Z build-essential bzip2 cpp cpp-11 dpkg-dev fakeroot fontconfig-config
782026-03-10T17:46:21.612Z fonts-dejavu-core g++ g++-11 gcc gcc-11 gcc-11-base libalgorithm-diff-perl
792026-03-10T17:46:21.616Z libalgorithm-diff-xs-perl libalgorithm-merge-perl libasan6 libatomic1
802026-03-10T17:46:21.616Z libc-dev-bin libc-devtools libc6-dev libcc1-0 libcrypt-dev libdeflate0
812026-03-10T17:46:21.616Z libdpkg-perl libfakeroot libfile-fcntllock-perl libfontconfig1 libgcc-11-dev
822026-03-10T17:46:21.616Z libgd3 libgomp1 libisl23 libitm1 libjbig0 libjpeg-turbo8 libjpeg8 liblsan0
832026-03-10T17:46:21.616Z libmpc3 libnsl-dev libquadmath0 libstdc++-11-dev libtiff5 libtirpc-dev
842026-03-10T17:46:21.616Z libtsan0 libubsan1 libwebp7 libxpm4 linux-libc-dev lto-disabled-list make
852026-03-10T17:46:21.616Z manpages-dev rpcsvc-proto sysvbanner
862026-03-10T17:46:21.616ZThe following packages will be upgraded:
872026-03-10T17:46:21.616Z gcc-12-base libc6 libgcc-s1 libstdc++6 libtirpc-common libtirpc3
882026-03-10T17:46:21.629Z6 upgraded, 53 newly installed, 0 to remove and 301 not upgraded.
892026-03-10T17:46:21.652ZNeed to get 68.3 MB of archives.
902026-03-10T17:46:21.652ZAfter this operation, 208 MB of additional disk space will be used.
912026-03-10T17:46:21.652ZGet:1 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 gcc-12-base amd64 12.3.0-1ubuntu1~22.04.3 [216 kB]
922026-03-10T17:46:21.663ZGet:2 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libstdc++6 amd64 12.3.0-1ubuntu1~22.04.3 [699 kB]
932026-03-10T17:46:21.680ZGet:3 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libgcc-s1 amd64 12.3.0-1ubuntu1~22.04.3 [53.9 kB]
942026-03-10T17:46:21.683ZGet:4 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libc6 amd64 2.35-0ubuntu3.13 [3,235 kB]
952026-03-10T17:46:21.716ZGet:5 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libtirpc-common all 1.3.2-2ubuntu0.1 [7,766 B]
962026-03-10T17:46:21.719ZGet:6 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libtirpc3 amd64 1.3.2-2ubuntu0.1 [82.3 kB]
972026-03-10T17:46:21.719ZGet:7 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libc-dev-bin amd64 2.35-0ubuntu3.13 [20.3 kB]
982026-03-10T17:46:21.722ZGet:8 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-libc-dev amd64 5.15.0-171.181 [1,316 kB]
992026-03-10T17:46:21.753ZGet:9 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 libcrypt-dev amd64 1:4.4.27-1 [112 kB]
1002026-03-10T17:46:21.757ZGet:10 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 rpcsvc-proto amd64 1.4.2-0ubuntu6 [68.5 kB]
1012026-03-10T17:46:21.760ZGet:11 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libtirpc-dev amd64 1.3.2-2ubuntu0.1 [192 kB]
1022026-03-10T17:46:21.764ZGet:12 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 libnsl-dev amd64 1.3.0-2build2 [71.3 kB]
1032026-03-10T17:46:21.767ZGet:13 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libc6-dev amd64 2.35-0ubuntu3.13 [2,101 kB]
1042026-03-10T17:46:21.819ZGet:14 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 gcc-11-base amd64 11.4.0-1ubuntu1~22.04.3 [216 kB]
1052026-03-10T17:46:21.827ZGet:15 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 libisl23 amd64 0.24-2build1 [727 kB]
1062026-03-10T17:46:21.845ZGet:16 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 libmpc3 amd64 1.2.1-2build1 [46.9 kB]
1072026-03-10T17:46:21.849ZGet:17 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 cpp-11 amd64 11.4.0-1ubuntu1~22.04.3 [10.0 MB]
1082026-03-10T17:46:21.930ZGet:18 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 cpp amd64 4:11.2.0-1ubuntu1 [27.7 kB]
1092026-03-10T17:46:21.933ZGet:19 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libcc1-0 amd64 12.3.0-1ubuntu1~22.04.3 [48.3 kB]
1102026-03-10T17:46:21.933ZGet:20 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libgomp1 amd64 12.3.0-1ubuntu1~22.04.3 [127 kB]
1112026-03-10T17:46:21.937ZGet:21 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libitm1 amd64 12.3.0-1ubuntu1~22.04.3 [30.2 kB]
1122026-03-10T17:46:21.937ZGet:22 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libatomic1 amd64 12.3.0-1ubuntu1~22.04.3 [10.5 kB]
1132026-03-10T17:46:21.941ZGet:23 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libasan6 amd64 11.4.0-1ubuntu1~22.04.3 [2,283 kB]
1142026-03-10T17:46:21.989ZGet:24 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 liblsan0 amd64 12.3.0-1ubuntu1~22.04.3 [1,069 kB]
1152026-03-10T17:46:22.001ZGet:25 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libtsan0 amd64 11.4.0-1ubuntu1~22.04.3 [2,260 kB]
1162026-03-10T17:46:22.064ZGet:26 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libubsan1 amd64 12.3.0-1ubuntu1~22.04.3 [976 kB]
1172026-03-10T17:46:22.080ZGet:27 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libquadmath0 amd64 12.3.0-1ubuntu1~22.04.3 [154 kB]
1182026-03-10T17:46:22.083ZGet:28 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libgcc-11-dev amd64 11.4.0-1ubuntu1~22.04.3 [2,517 kB]
1192026-03-10T17:46:22.113ZGet:29 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 gcc-11 amd64 11.4.0-1ubuntu1~22.04.3 [20.1 MB]
1202026-03-10T17:46:22.363ZGet:30 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 gcc amd64 4:11.2.0-1ubuntu1 [5,112 B]
1212026-03-10T17:46:22.363ZGet:31 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libstdc++-11-dev amd64 11.4.0-1ubuntu1~22.04.3 [2,101 kB]
1222026-03-10T17:46:22.386ZGet:32 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 g++-11 amd64 11.4.0-1ubuntu1~22.04.3 [11.4 MB]
1232026-03-10T17:46:22.537ZGet:33 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 g++ amd64 4:11.2.0-1ubuntu1 [1,412 B]
1242026-03-10T17:46:22.540ZGet:34 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 make amd64 4.3-4.1build1 [180 kB]
1252026-03-10T17:46:22.540ZGet:35 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libdpkg-perl all 1.21.1ubuntu2.6 [237 kB]
1262026-03-10T17:46:22.544ZGet:36 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 bzip2 amd64 1.0.8-5build1 [34.8 kB]
1272026-03-10T17:46:22.547ZGet:37 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 lto-disabled-list all 24 [12.5 kB]
1282026-03-10T17:46:22.550ZGet:38 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 dpkg-dev all 1.21.1ubuntu2.6 [922 kB]
1292026-03-10T17:46:22.559ZGet:39 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 build-essential amd64 12.9ubuntu3 [4,744 B]
1302026-03-10T17:46:22.562ZGet:40 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 libfakeroot amd64 1.28-1ubuntu1 [31.5 kB]
1312026-03-10T17:46:22.565ZGet:41 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 fakeroot amd64 1.28-1ubuntu1 [60.4 kB]
1322026-03-10T17:46:22.568ZGet:42 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 fonts-dejavu-core all 2.37-2build1 [1,041 kB]
1332026-03-10T17:46:22.589ZGet:43 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 fontconfig-config all 2.13.1-4.2ubuntu5 [29.1 kB]
1342026-03-10T17:46:22.592ZGet:44 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 libalgorithm-diff-perl all 1.201-1 [41.8 kB]
1352026-03-10T17:46:22.595ZGet:45 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 libalgorithm-diff-xs-perl amd64 0.04-6build3 [11.9 kB]
1362026-03-10T17:46:22.598ZGet:46 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 libalgorithm-merge-perl all 0.08-3 [12.0 kB]
1372026-03-10T17:46:22.598ZGet:47 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 libfontconfig1 amd64 2.13.1-4.2ubuntu5 [131 kB]
1382026-03-10T17:46:22.601ZGet:48 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 libjpeg-turbo8 amd64 2.1.2-0ubuntu1 [134 kB]
1392026-03-10T17:46:22.606ZGet:49 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 libjpeg8 amd64 8c-2ubuntu10 [2,264 B]
1402026-03-10T17:46:22.609ZGet:50 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 libdeflate0 amd64 1.10-2 [70.9 kB]
1412026-03-10T17:46:22.611ZGet:51 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libjbig0 amd64 2.1-3.1ubuntu0.22.04.1 [29.2 kB]
1422026-03-10T17:46:22.614ZGet:52 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libwebp7 amd64 1.2.2-2ubuntu0.22.04.2 [206 kB]
1432026-03-10T17:46:22.621ZGet:53 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libtiff5 amd64 4.3.0-6ubuntu0.12 [185 kB]
1442026-03-10T17:46:22.627ZGet:54 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libxpm4 amd64 1:3.5.12-1ubuntu0.22.04.2 [36.7 kB]
1452026-03-10T17:46:22.630ZGet:55 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libgd3 amd64 2.3.0-2ubuntu2.3 [129 kB]
1462026-03-10T17:46:22.633ZGet:56 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libc-devtools amd64 2.35-0ubuntu3.13 [29.0 kB]
1472026-03-10T17:46:22.637ZGet:57 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 libfile-fcntllock-perl amd64 0.22-3build7 [33.9 kB]
1482026-03-10T17:46:22.640ZGet:58 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 manpages-dev all 5.10-1ubuntu1 [2,309 kB]
1492026-03-10T17:46:22.680ZGet:59 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/universe amd64 sysvbanner amd64 1:1.0-17fakesync1 [7,112 B]
1502026-03-10T17:46:22.785Zdebconf: unable to initialize frontend: Dialog
1512026-03-10T17:46:22.785Zdebconf: (TERM is not set, so the dialog frontend is not usable.)
1522026-03-10T17:46:22.785Zdebconf: falling back to frontend: Readline
1532026-03-10T17:46:22.796Zdebconf: unable to initialize frontend: Readline
1542026-03-10T17:46:22.796Zdebconf: (This frontend requires a controlling tty.)
1552026-03-10T17:46:22.797Zdebconf: falling back to frontend: Teletype
1562026-03-10T17:46:22.800Zdpkg-preconfigure: unable to re-open stdin:
1572026-03-10T17:46:22.813ZFetched 68.3 MB in 1s (65.7 MB/s)
1582026-03-10T17:46:23.083Z(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 63625 files and directories currently installed.)
1592026-03-10T17:46:23.089ZPreparing to unpack .../gcc-12-base_12.3.0-1ubuntu1~22.04.3_amd64.deb ...
1602026-03-10T17:46:23.107ZUnpacking gcc-12-base:amd64 (12.3.0-1ubuntu1~22.04.3) over (12-20220319-1ubuntu1) ...
1612026-03-10T17:46:23.163ZSetting up gcc-12-base:amd64 (12.3.0-1ubuntu1~22.04.3) ...
1622026-03-10T17:46:23.239Z(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 63625 files and directories currently installed.)
1632026-03-10T17:46:23.243ZPreparing to unpack .../libstdc++6_12.3.0-1ubuntu1~22.04.3_amd64.deb ...
1642026-03-10T17:46:23.285ZUnpacking libstdc++6:amd64 (12.3.0-1ubuntu1~22.04.3) over (12-20220319-1ubuntu1) ...
1652026-03-10T17:46:23.351ZSetting up libstdc++6:amd64 (12.3.0-1ubuntu1~22.04.3) ...
1662026-03-10T17:46:23.422Z(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 63625 files and directories currently installed.)
1672026-03-10T17:46:23.426ZPreparing to unpack .../libgcc-s1_12.3.0-1ubuntu1~22.04.3_amd64.deb ...
1682026-03-10T17:46:23.448ZUnpacking libgcc-s1:amd64 (12.3.0-1ubuntu1~22.04.3) over (12-20220319-1ubuntu1) ...
1692026-03-10T17:46:23.506ZSetting up libgcc-s1:amd64 (12.3.0-1ubuntu1~22.04.3) ...
1702026-03-10T17:46:23.594Z(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 63625 files and directories currently installed.)
1712026-03-10T17:46:23.597ZPreparing to unpack .../libc6_2.35-0ubuntu3.13_amd64.deb ...
1722026-03-10T17:46:23.655Zdebconf: unable to initialize frontend: Dialog
1732026-03-10T17:46:23.655Zdebconf: (TERM is not set, so the dialog frontend is not usable.)
1742026-03-10T17:46:23.655Zdebconf: falling back to frontend: Readline
1752026-03-10T17:46:23.689ZUnpacking libc6:amd64 (2.35-0ubuntu3.13) over (2.35-0ubuntu3) ...
1762026-03-10T17:46:23.793ZSetting up libc6:amd64 (2.35-0ubuntu3.13) ...
1772026-03-10T17:46:23.858Zdebconf: unable to initialize frontend: Dialog
1782026-03-10T17:46:23.858Zdebconf: (TERM is not set, so the dialog frontend is not usable.)
1792026-03-10T17:46:23.858Zdebconf: falling back to frontend: Readline
1802026-03-10T17:46:24.072Z(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 63625 files and directories currently installed.)
1812026-03-10T17:46:24.075ZPreparing to unpack .../libtirpc-common_1.3.2-2ubuntu0.1_all.deb ...
1822026-03-10T17:46:24.095ZUnpacking libtirpc-common (1.3.2-2ubuntu0.1) over (1.3.2-2build1) ...
1832026-03-10T17:46:24.150ZSetting up libtirpc-common (1.3.2-2ubuntu0.1) ...
1842026-03-10T17:46:24.226Z(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 63625 files and directories currently installed.)
1852026-03-10T17:46:24.229ZPreparing to unpack .../libtirpc3_1.3.2-2ubuntu0.1_amd64.deb ...
1862026-03-10T17:46:24.246ZUnpacking libtirpc3:amd64 (1.3.2-2ubuntu0.1) over (1.3.2-2build1) ...
1872026-03-10T17:46:24.299ZSetting up libtirpc3:amd64 (1.3.2-2ubuntu0.1) ...
1882026-03-10T17:46:24.350ZSelecting previously unselected package libc-dev-bin.
1892026-03-10T17:46:24.357Z(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 63625 files and directories currently installed.)
1902026-03-10T17:46:24.360ZPreparing to unpack .../00-libc-dev-bin_2.35-0ubuntu3.13_amd64.deb ...
1912026-03-10T17:46:24.367ZUnpacking libc-dev-bin (2.35-0ubuntu3.13) ...
1922026-03-10T17:46:24.411ZSelecting previously unselected package linux-libc-dev:amd64.
1932026-03-10T17:46:24.414ZPreparing to unpack .../01-linux-libc-dev_5.15.0-171.181_amd64.deb ...
1942026-03-10T17:46:24.417ZUnpacking linux-libc-dev:amd64 (5.15.0-171.181) ...
1952026-03-10T17:46:24.548ZSelecting previously unselected package libcrypt-dev:amd64.
1962026-03-10T17:46:24.550ZPreparing to unpack .../02-libcrypt-dev_1%3a4.4.27-1_amd64.deb ...
1972026-03-10T17:46:24.555ZUnpacking libcrypt-dev:amd64 (1:4.4.27-1) ...
1982026-03-10T17:46:24.602ZSelecting previously unselected package rpcsvc-proto.
1992026-03-10T17:46:24.607ZPreparing to unpack .../03-rpcsvc-proto_1.4.2-0ubuntu6_amd64.deb ...
2002026-03-10T17:46:24.612ZUnpacking rpcsvc-proto (1.4.2-0ubuntu6) ...
2012026-03-10T17:46:24.654ZSelecting previously unselected package libtirpc-dev:amd64.
2022026-03-10T17:46:24.657ZPreparing to unpack .../04-libtirpc-dev_1.3.2-2ubuntu0.1_amd64.deb ...
2032026-03-10T17:46:24.661ZUnpacking libtirpc-dev:amd64 (1.3.2-2ubuntu0.1) ...
2042026-03-10T17:46:24.707ZSelecting previously unselected package libnsl-dev:amd64.
2052026-03-10T17:46:24.710ZPreparing to unpack .../05-libnsl-dev_1.3.0-2build2_amd64.deb ...
2062026-03-10T17:46:24.713ZUnpacking libnsl-dev:amd64 (1.3.0-2build2) ...
2072026-03-10T17:46:24.755ZSelecting previously unselected package libc6-dev:amd64.
2082026-03-10T17:46:24.758ZPreparing to unpack .../06-libc6-dev_2.35-0ubuntu3.13_amd64.deb ...
2092026-03-10T17:46:24.765ZUnpacking libc6-dev:amd64 (2.35-0ubuntu3.13) ...
2102026-03-10T17:46:24.860ZSelecting previously unselected package gcc-11-base:amd64.
2112026-03-10T17:46:24.864ZPreparing to unpack .../07-gcc-11-base_11.4.0-1ubuntu1~22.04.3_amd64.deb ...
2122026-03-10T17:46:24.869ZUnpacking gcc-11-base:amd64 (11.4.0-1ubuntu1~22.04.3) ...
2132026-03-10T17:46:24.926ZSelecting previously unselected package libisl23:amd64.
2142026-03-10T17:46:24.929ZPreparing to unpack .../08-libisl23_0.24-2build1_amd64.deb ...
2152026-03-10T17:46:24.934ZUnpacking libisl23:amd64 (0.24-2build1) ...
2162026-03-10T17:46:24.996ZSelecting previously unselected package libmpc3:amd64.
2172026-03-10T17:46:24.999ZPreparing to unpack .../09-libmpc3_1.2.1-2build1_amd64.deb ...
2182026-03-10T17:46:25.004ZUnpacking libmpc3:amd64 (1.2.1-2build1) ...
2192026-03-10T17:46:25.047ZSelecting previously unselected package cpp-11.
2202026-03-10T17:46:25.050ZPreparing to unpack .../10-cpp-11_11.4.0-1ubuntu1~22.04.3_amd64.deb ...
2212026-03-10T17:46:25.056ZUnpacking cpp-11 (11.4.0-1ubuntu1~22.04.3) ...
2222026-03-10T17:46:25.163ZSelecting previously unselected package cpp.
2232026-03-10T17:46:25.166ZPreparing to unpack .../11-cpp_4%3a11.2.0-1ubuntu1_amd64.deb ...
2242026-03-10T17:46:25.169ZUnpacking cpp (4:11.2.0-1ubuntu1) ...
2252026-03-10T17:46:25.189ZSelecting previously unselected package libcc1-0:amd64.
2262026-03-10T17:46:25.193ZPreparing to unpack .../12-libcc1-0_12.3.0-1ubuntu1~22.04.3_amd64.deb ...
2272026-03-10T17:46:25.193ZUnpacking libcc1-0:amd64 (12.3.0-1ubuntu1~22.04.3) ...
2282026-03-10T17:46:25.216ZSelecting previously unselected package libgomp1:amd64.
2292026-03-10T17:46:25.219ZPreparing to unpack .../13-libgomp1_12.3.0-1ubuntu1~22.04.3_amd64.deb ...
2302026-03-10T17:46:25.222ZUnpacking libgomp1:amd64 (12.3.0-1ubuntu1~22.04.3) ...
2312026-03-10T17:46:25.271ZSelecting previously unselected package libitm1:amd64.
2322026-03-10T17:46:25.274ZPreparing to unpack .../14-libitm1_12.3.0-1ubuntu1~22.04.3_amd64.deb ...
2332026-03-10T17:46:25.274ZUnpacking libitm1:amd64 (12.3.0-1ubuntu1~22.04.3) ...
2342026-03-10T17:46:25.298ZSelecting previously unselected package libatomic1:amd64.
2352026-03-10T17:46:25.303ZPreparing to unpack .../15-libatomic1_12.3.0-1ubuntu1~22.04.3_amd64.deb ...
2362026-03-10T17:46:25.305ZUnpacking libatomic1:amd64 (12.3.0-1ubuntu1~22.04.3) ...
2372026-03-10T17:46:25.358ZSelecting previously unselected package libasan6:amd64.
2382026-03-10T17:46:25.361ZPreparing to unpack .../16-libasan6_11.4.0-1ubuntu1~22.04.3_amd64.deb ...
2392026-03-10T17:46:25.365ZUnpacking libasan6:amd64 (11.4.0-1ubuntu1~22.04.3) ...
2402026-03-10T17:46:25.442ZSelecting previously unselected package liblsan0:amd64.
2412026-03-10T17:46:25.446ZPreparing to unpack .../17-liblsan0_12.3.0-1ubuntu1~22.04.3_amd64.deb ...
2422026-03-10T17:46:25.451ZUnpacking liblsan0:amd64 (12.3.0-1ubuntu1~22.04.3) ...
2432026-03-10T17:46:25.517ZSelecting previously unselected package libtsan0:amd64.
2442026-03-10T17:46:25.520ZPreparing to unpack .../18-libtsan0_11.4.0-1ubuntu1~22.04.3_amd64.deb ...
2452026-03-10T17:46:25.524ZUnpacking libtsan0:amd64 (11.4.0-1ubuntu1~22.04.3) ...
2462026-03-10T17:46:25.598ZSelecting previously unselected package libubsan1:amd64.
2472026-03-10T17:46:25.601ZPreparing to unpack .../19-libubsan1_12.3.0-1ubuntu1~22.04.3_amd64.deb ...
2482026-03-10T17:46:25.605ZUnpacking libubsan1:amd64 (12.3.0-1ubuntu1~22.04.3) ...
2492026-03-10T17:46:25.663ZSelecting previously unselected package libquadmath0:amd64.
2502026-03-10T17:46:25.666ZPreparing to unpack .../20-libquadmath0_12.3.0-1ubuntu1~22.04.3_amd64.deb ...
2512026-03-10T17:46:25.673ZUnpacking libquadmath0:amd64 (12.3.0-1ubuntu1~22.04.3) ...
2522026-03-10T17:46:25.718ZSelecting previously unselected package libgcc-11-dev:amd64.
2532026-03-10T17:46:25.723ZPreparing to unpack .../21-libgcc-11-dev_11.4.0-1ubuntu1~22.04.3_amd64.deb ...
2542026-03-10T17:46:25.728ZUnpacking libgcc-11-dev:amd64 (11.4.0-1ubuntu1~22.04.3) ...
2552026-03-10T17:46:25.802ZSelecting previously unselected package gcc-11.
2562026-03-10T17:46:25.805ZPreparing to unpack .../22-gcc-11_11.4.0-1ubuntu1~22.04.3_amd64.deb ...
2572026-03-10T17:46:25.811ZUnpacking gcc-11 (11.4.0-1ubuntu1~22.04.3) ...
2582026-03-10T17:46:25.989ZSelecting previously unselected package gcc.
2592026-03-10T17:46:25.993ZPreparing to unpack .../23-gcc_4%3a11.2.0-1ubuntu1_amd64.deb ...
2602026-03-10T17:46:25.997ZUnpacking gcc (4:11.2.0-1ubuntu1) ...
2612026-03-10T17:46:26.038ZSelecting previously unselected package libstdc++-11-dev:amd64.
2622026-03-10T17:46:26.041ZPreparing to unpack .../24-libstdc++-11-dev_11.4.0-1ubuntu1~22.04.3_amd64.deb ...
2632026-03-10T17:46:26.047ZUnpacking libstdc++-11-dev:amd64 (11.4.0-1ubuntu1~22.04.3) ...
2642026-03-10T17:46:26.159ZSelecting previously unselected package g++-11.
2652026-03-10T17:46:26.162ZPreparing to unpack .../25-g++-11_11.4.0-1ubuntu1~22.04.3_amd64.deb ...
2662026-03-10T17:46:26.169ZUnpacking g++-11 (11.4.0-1ubuntu1~22.04.3) ...
2672026-03-10T17:46:26.293ZSelecting previously unselected package g++.
2682026-03-10T17:46:26.296ZPreparing to unpack .../26-g++_4%3a11.2.0-1ubuntu1_amd64.deb ...
2692026-03-10T17:46:26.302ZUnpacking g++ (4:11.2.0-1ubuntu1) ...
2702026-03-10T17:46:26.337ZSelecting previously unselected package make.
2712026-03-10T17:46:26.340ZPreparing to unpack .../27-make_4.3-4.1build1_amd64.deb ...
2722026-03-10T17:46:26.346ZUnpacking make (4.3-4.1build1) ...
2732026-03-10T17:46:26.386ZSelecting previously unselected package libdpkg-perl.
2742026-03-10T17:46:26.389ZPreparing to unpack .../28-libdpkg-perl_1.21.1ubuntu2.6_all.deb ...
2752026-03-10T17:46:26.394ZUnpacking libdpkg-perl (1.21.1ubuntu2.6) ...
2762026-03-10T17:46:26.442ZSelecting previously unselected package bzip2.
2772026-03-10T17:46:26.445ZPreparing to unpack .../29-bzip2_1.0.8-5build1_amd64.deb ...
2782026-03-10T17:46:26.451ZUnpacking bzip2 (1.0.8-5build1) ...
2792026-03-10T17:46:26.489ZSelecting previously unselected package lto-disabled-list.
2802026-03-10T17:46:26.492ZPreparing to unpack .../30-lto-disabled-list_24_all.deb ...
2812026-03-10T17:46:26.498ZUnpacking lto-disabled-list (24) ...
2822026-03-10T17:46:26.542ZSelecting previously unselected package dpkg-dev.
2832026-03-10T17:46:26.544ZPreparing to unpack .../31-dpkg-dev_1.21.1ubuntu2.6_all.deb ...
2842026-03-10T17:46:26.549ZUnpacking dpkg-dev (1.21.1ubuntu2.6) ...
2852026-03-10T17:46:26.633ZSelecting previously unselected package build-essential.
2862026-03-10T17:46:26.641ZPreparing to unpack .../32-build-essential_12.9ubuntu3_amd64.deb ...
2872026-03-10T17:46:26.641ZUnpacking build-essential (12.9ubuntu3) ...
2882026-03-10T17:46:26.678ZSelecting previously unselected package libfakeroot:amd64.
2892026-03-10T17:46:26.681ZPreparing to unpack .../33-libfakeroot_1.28-1ubuntu1_amd64.deb ...
2902026-03-10T17:46:26.686ZUnpacking libfakeroot:amd64 (1.28-1ubuntu1) ...
2912026-03-10T17:46:26.737ZSelecting previously unselected package fakeroot.
2922026-03-10T17:46:26.741ZPreparing to unpack .../34-fakeroot_1.28-1ubuntu1_amd64.deb ...
2932026-03-10T17:46:26.744ZUnpacking fakeroot (1.28-1ubuntu1) ...
2942026-03-10T17:46:26.786ZSelecting previously unselected package fonts-dejavu-core.
2952026-03-10T17:46:26.794ZPreparing to unpack .../35-fonts-dejavu-core_2.37-2build1_all.deb ...
2962026-03-10T17:46:26.797ZUnpacking fonts-dejavu-core (2.37-2build1) ...
2972026-03-10T17:46:26.897ZSelecting previously unselected package fontconfig-config.
2982026-03-10T17:46:26.901ZPreparing to unpack .../36-fontconfig-config_2.13.1-4.2ubuntu5_all.deb ...
2992026-03-10T17:46:26.905ZUnpacking fontconfig-config (2.13.1-4.2ubuntu5) ...
3002026-03-10T17:46:26.945ZSelecting previously unselected package libalgorithm-diff-perl.
3012026-03-10T17:46:26.950ZPreparing to unpack .../37-libalgorithm-diff-perl_1.201-1_all.deb ...
3022026-03-10T17:46:26.954ZUnpacking libalgorithm-diff-perl (1.201-1) ...
3032026-03-10T17:46:27.007ZSelecting previously unselected package libalgorithm-diff-xs-perl.
3042026-03-10T17:46:27.012ZPreparing to unpack .../38-libalgorithm-diff-xs-perl_0.04-6build3_amd64.deb ...
3052026-03-10T17:46:27.012ZUnpacking libalgorithm-diff-xs-perl (0.04-6build3) ...
3062026-03-10T17:46:27.051ZSelecting previously unselected package libalgorithm-merge-perl.
3072026-03-10T17:46:27.054ZPreparing to unpack .../39-libalgorithm-merge-perl_0.08-3_all.deb ...
3082026-03-10T17:46:27.058ZUnpacking libalgorithm-merge-perl (0.08-3) ...
3092026-03-10T17:46:27.111ZSelecting previously unselected package libfontconfig1:amd64.
3102026-03-10T17:46:27.114ZPreparing to unpack .../40-libfontconfig1_2.13.1-4.2ubuntu5_amd64.deb ...
3112026-03-10T17:46:27.118ZUnpacking libfontconfig1:amd64 (2.13.1-4.2ubuntu5) ...
3122026-03-10T17:46:27.170ZSelecting previously unselected package libjpeg-turbo8:amd64.
3132026-03-10T17:46:27.173ZPreparing to unpack .../41-libjpeg-turbo8_2.1.2-0ubuntu1_amd64.deb ...
3142026-03-10T17:46:27.178ZUnpacking libjpeg-turbo8:amd64 (2.1.2-0ubuntu1) ...
3152026-03-10T17:46:27.223ZSelecting previously unselected package libjpeg8:amd64.
3162026-03-10T17:46:27.226ZPreparing to unpack .../42-libjpeg8_8c-2ubuntu10_amd64.deb ...
3172026-03-10T17:46:27.232ZUnpacking libjpeg8:amd64 (8c-2ubuntu10) ...
3182026-03-10T17:46:27.282ZSelecting previously unselected package libdeflate0:amd64.
3192026-03-10T17:46:27.285ZPreparing to unpack .../43-libdeflate0_1.10-2_amd64.deb ...
3202026-03-10T17:46:27.289ZUnpacking libdeflate0:amd64 (1.10-2) ...
3212026-03-10T17:46:27.344ZSelecting previously unselected package libjbig0:amd64.
3222026-03-10T17:46:27.347ZPreparing to unpack .../44-libjbig0_2.1-3.1ubuntu0.22.04.1_amd64.deb ...
3232026-03-10T17:46:27.352ZUnpacking libjbig0:amd64 (2.1-3.1ubuntu0.22.04.1) ...
3242026-03-10T17:46:27.405ZSelecting previously unselected package libwebp7:amd64.
3252026-03-10T17:46:27.408ZPreparing to unpack .../45-libwebp7_1.2.2-2ubuntu0.22.04.2_amd64.deb ...
3262026-03-10T17:46:27.417ZUnpacking libwebp7:amd64 (1.2.2-2ubuntu0.22.04.2) ...
3272026-03-10T17:46:27.468ZSelecting previously unselected package libtiff5:amd64.
3282026-03-10T17:46:27.472ZPreparing to unpack .../46-libtiff5_4.3.0-6ubuntu0.12_amd64.deb ...
3292026-03-10T17:46:27.478ZUnpacking libtiff5:amd64 (4.3.0-6ubuntu0.12) ...
3302026-03-10T17:46:27.529ZSelecting previously unselected package libxpm4:amd64.
3312026-03-10T17:46:27.533ZPreparing to unpack .../47-libxpm4_1%3a3.5.12-1ubuntu0.22.04.2_amd64.deb ...
3322026-03-10T17:46:27.536ZUnpacking libxpm4:amd64 (1:3.5.12-1ubuntu0.22.04.2) ...
3332026-03-10T17:46:27.589ZSelecting previously unselected package libgd3:amd64.
3342026-03-10T17:46:27.592ZPreparing to unpack .../48-libgd3_2.3.0-2ubuntu2.3_amd64.deb ...
3352026-03-10T17:46:27.596ZUnpacking libgd3:amd64 (2.3.0-2ubuntu2.3) ...
3362026-03-10T17:46:27.639ZSelecting previously unselected package libc-devtools.
3372026-03-10T17:46:27.642ZPreparing to unpack .../49-libc-devtools_2.35-0ubuntu3.13_amd64.deb ...
3382026-03-10T17:46:27.646ZUnpacking libc-devtools (2.35-0ubuntu3.13) ...
3392026-03-10T17:46:27.687ZSelecting previously unselected package libfile-fcntllock-perl.
3402026-03-10T17:46:27.691ZPreparing to unpack .../50-libfile-fcntllock-perl_0.22-3build7_amd64.deb ...
3412026-03-10T17:46:27.694ZUnpacking libfile-fcntllock-perl (0.22-3build7) ...
3422026-03-10T17:46:27.732ZSelecting previously unselected package manpages-dev.
3432026-03-10T17:46:27.735ZPreparing to unpack .../51-manpages-dev_5.10-1ubuntu1_all.deb ...
3442026-03-10T17:46:27.740ZUnpacking manpages-dev (5.10-1ubuntu1) ...
3452026-03-10T17:46:27.923ZSelecting previously unselected package sysvbanner.
3462026-03-10T17:46:27.926ZPreparing to unpack .../52-sysvbanner_1%3a1.0-17fakesync1_amd64.deb ...
3472026-03-10T17:46:27.933ZUnpacking sysvbanner (1:1.0-17fakesync1) ...
3482026-03-10T17:46:27.984ZSetting up gcc-11-base:amd64 (11.4.0-1ubuntu1~22.04.3) ...
3492026-03-10T17:46:28.003ZSetting up manpages-dev (5.10-1ubuntu1) ...
3502026-03-10T17:46:28.024ZSetting up lto-disabled-list (24) ...
3512026-03-10T17:46:28.042ZSetting up libxpm4:amd64 (1:3.5.12-1ubuntu0.22.04.2) ...
3522026-03-10T17:46:28.076ZSetting up libfile-fcntllock-perl (0.22-3build7) ...
3532026-03-10T17:46:28.081ZSetting up libalgorithm-diff-perl (1.201-1) ...
3542026-03-10T17:46:28.100ZSetting up libdeflate0:amd64 (1.10-2) ...
3552026-03-10T17:46:28.121ZSetting up linux-libc-dev:amd64 (5.15.0-171.181) ...
3562026-03-10T17:46:28.142ZSetting up libgomp1:amd64 (12.3.0-1ubuntu1~22.04.3) ...
3572026-03-10T17:46:28.162ZSetting up bzip2 (1.0.8-5build1) ...
3582026-03-10T17:46:28.181ZSetting up libjbig0:amd64 (2.1-3.1ubuntu0.22.04.1) ...
3592026-03-10T17:46:28.200ZSetting up libfakeroot:amd64 (1.28-1ubuntu1) ...
3602026-03-10T17:46:28.225ZSetting up libasan6:amd64 (11.4.0-1ubuntu1~22.04.3) ...
3612026-03-10T17:46:28.242ZSetting up fakeroot (1.28-1ubuntu1) ...
3622026-03-10T17:46:28.271Zupdate-alternatives: using /usr/bin/fakeroot-sysv to provide /usr/bin/fakeroot (fakeroot) in auto mode
3632026-03-10T17:46:28.291ZSetting up libtirpc-dev:amd64 (1.3.2-2ubuntu0.1) ...
3642026-03-10T17:46:28.309ZSetting up rpcsvc-proto (1.4.2-0ubuntu6) ...
3652026-03-10T17:46:28.329ZSetting up make (4.3-4.1build1) ...
3662026-03-10T17:46:28.347ZSetting up libquadmath0:amd64 (12.3.0-1ubuntu1~22.04.3) ...
3672026-03-10T17:46:28.369ZSetting up libmpc3:amd64 (1.2.1-2build1) ...
3682026-03-10T17:46:28.388ZSetting up libatomic1:amd64 (12.3.0-1ubuntu1~22.04.3) ...
3692026-03-10T17:46:28.408ZSetting up fonts-dejavu-core (2.37-2build1) ...
3702026-03-10T17:46:28.506ZSetting up libjpeg-turbo8:amd64 (2.1.2-0ubuntu1) ...
3712026-03-10T17:46:28.524ZSetting up libdpkg-perl (1.21.1ubuntu2.6) ...
3722026-03-10T17:46:28.545ZSetting up libwebp7:amd64 (1.2.2-2ubuntu0.22.04.2) ...
3732026-03-10T17:46:28.564ZSetting up libubsan1:amd64 (12.3.0-1ubuntu1~22.04.3) ...
3742026-03-10T17:46:28.584ZSetting up libnsl-dev:amd64 (1.3.0-2build2) ...
3752026-03-10T17:46:28.602ZSetting up libcrypt-dev:amd64 (1:4.4.27-1) ...
3762026-03-10T17:46:28.624ZSetting up sysvbanner (1:1.0-17fakesync1) ...
3772026-03-10T17:46:28.641ZSetting up libisl23:amd64 (0.24-2build1) ...
3782026-03-10T17:46:28.663ZSetting up libc-dev-bin (2.35-0ubuntu3.13) ...
3792026-03-10T17:46:28.682ZSetting up libalgorithm-diff-xs-perl (0.04-6build3) ...
3802026-03-10T17:46:28.702ZSetting up libcc1-0:amd64 (12.3.0-1ubuntu1~22.04.3) ...
3812026-03-10T17:46:28.721ZSetting up liblsan0:amd64 (12.3.0-1ubuntu1~22.04.3) ...
3822026-03-10T17:46:28.740ZSetting up libitm1:amd64 (12.3.0-1ubuntu1~22.04.3) ...
3832026-03-10T17:46:28.761ZSetting up libalgorithm-merge-perl (0.08-3) ...
3842026-03-10T17:46:28.782ZSetting up libtsan0:amd64 (11.4.0-1ubuntu1~22.04.3) ...
3852026-03-10T17:46:28.800ZSetting up libjpeg8:amd64 (8c-2ubuntu10) ...
3862026-03-10T17:46:28.818ZSetting up cpp-11 (11.4.0-1ubuntu1~22.04.3) ...
3872026-03-10T17:46:28.839ZSetting up fontconfig-config (2.13.1-4.2ubuntu5) ...
3882026-03-10T17:46:29.115ZSetting up dpkg-dev (1.21.1ubuntu2.6) ...
3892026-03-10T17:46:29.147ZSetting up libgcc-11-dev:amd64 (11.4.0-1ubuntu1~22.04.3) ...
3902026-03-10T17:46:29.166ZSetting up gcc-11 (11.4.0-1ubuntu1~22.04.3) ...
3912026-03-10T17:46:29.185ZSetting up cpp (4:11.2.0-1ubuntu1) ...
3922026-03-10T17:46:29.212ZSetting up libc6-dev:amd64 (2.35-0ubuntu3.13) ...
3932026-03-10T17:46:29.230ZSetting up libtiff5:amd64 (4.3.0-6ubuntu0.12) ...
3942026-03-10T17:46:29.249ZSetting up libfontconfig1:amd64 (2.13.1-4.2ubuntu5) ...
3952026-03-10T17:46:29.266ZSetting up gcc (4:11.2.0-1ubuntu1) ...
3962026-03-10T17:46:29.304ZSetting up libgd3:amd64 (2.3.0-2ubuntu2.3) ...
3972026-03-10T17:46:29.323ZSetting up libstdc++-11-dev:amd64 (11.4.0-1ubuntu1~22.04.3) ...
3982026-03-10T17:46:29.345ZSetting up libc-devtools (2.35-0ubuntu3.13) ...
3992026-03-10T17:46:29.362ZSetting up g++-11 (11.4.0-1ubuntu1~22.04.3) ...
4002026-03-10T17:46:29.382ZSetting up g++ (4:11.2.0-1ubuntu1) ...
4012026-03-10T17:46:29.405Zupdate-alternatives: using /usr/bin/g++ to provide /usr/bin/c++ (c++) in auto mode
4022026-03-10T17:46:29.418ZSetting up build-essential (12.9ubuntu3) ...
4032026-03-10T17:46:29.438ZProcessing triggers for man-db (2.10.2-1) ...
4042026-03-10T17:46:29.834ZProcessing triggers for libc-bin (2.35-0ubuntu3) ...
4052026-03-10T17:46:30.225Z
4062026-03-10T17:46:30.225ZRunning kernel seems to be up-to-date.
4072026-03-10T17:46:30.225Z
4082026-03-10T17:46:30.225ZServices to be restarted:
4092026-03-10T17:46:30.228Z systemctl restart acpid.service
4102026-03-10T17:46:30.231Z systemctl restart buildomat-agent.service
4112026-03-10T17:46:30.231Z systemctl restart chrony.service
4122026-03-10T17:46:30.233Z systemctl restart cron.service
4132026-03-10T17:46:30.237Z systemctl restart irqbalance.service
4142026-03-10T17:46:30.240Z systemctl restart multipathd.service
4152026-03-10T17:46:30.244Z systemctl restart packagekit.service
4162026-03-10T17:46:30.246Z systemctl restart polkit.service
4172026-03-10T17:46:30.249Z systemctl restart rsyslog.service
4182026-03-10T17:46:30.249Z systemctl restart serial-getty@ttyS0.service
4192026-03-10T17:46:30.252Z systemctl restart snapd.service
4202026-03-10T17:46:30.252Z systemctl restart ssh.service
4212026-03-10T17:46:30.255Z systemctl restart systemd-fsckd.service
4222026-03-10T17:46:30.255Z systemctl restart systemd-hostnamed.service
4232026-03-10T17:46:30.258Z systemctl restart systemd-journald.service
4242026-03-10T17:46:30.261Z systemctl restart systemd-networkd.service
4252026-03-10T17:46:30.264Z systemctl restart systemd-resolved.service
4262026-03-10T17:46:30.267Z systemctl restart systemd-timedated.service
4272026-03-10T17:46:30.267Z systemctl restart systemd-udevd.service
4282026-03-10T17:46:30.270Z systemctl restart udisks2.service
4292026-03-10T17:46:30.270Z
4302026-03-10T17:46:30.270ZService restarts being deferred:
4312026-03-10T17:46:30.270Z systemctl restart ModemManager.service
4322026-03-10T17:46:30.270Z /etc/needrestart/restart.d/dbus.service
4332026-03-10T17:46:30.270Z systemctl restart getty@tty1.service
4342026-03-10T17:46:30.270Z systemctl restart networkd-dispatcher.service
4352026-03-10T17:46:30.270Z systemctl restart systemd-logind.service
4362026-03-10T17:46:30.270Z systemctl restart unattended-upgrades.service
4372026-03-10T17:46:30.270Z
4382026-03-10T17:46:30.270ZNo containers need to be restarted.
4392026-03-10T17:46:30.270Z
4402026-03-10T17:46:30.270ZNo user sessions are running outdated binaries.
4412026-03-10T17:46:30.270Z
4422026-03-10T17:46:30.270ZNo VM guests are running outdated hypervisor (qemu) binaries on this host.
4432026-03-10T17:46:32.529Z+ groupadd -g 12345 build
4442026-03-10T17:46:32.554Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build build
4452026-03-10T17:46:32.594Z+ echo 'build ALL=(ALL:ALL) NOPASSWD:ALL'
4462026-03-10T17:46:32.597Z+ chmod 0440 /etc/sudoers.d/build
4472026-03-10T17:46:32.597Z+ cat
4482026-03-10T17:46:32.597Z+ chmod 0755 /bin/pfexec
4492026-03-10T17:46:32.597Z+ cat
4502026-03-10T17:46:32.597Z+ chmod 0755 /bin/ptime
4512026-03-10T17:46:32.597Z+ [[ ! -L /bin ]]
4522026-03-10T17:46:32.597Z+ mkdir -p /work
4532026-03-10T17:46:32.597Z+ mkdir -p /home/build
4542026-03-10T17:46:32.597Z+ chown build:build /home/build /work
4552026-03-10T17:46:32.597Z+ chmod 0700 /home/build /work
4562026-03-10T17:46:32.597Zprocess exited: duration 18690 ms, exit code 0
 
4572026-03-10T17:46:32.604Zstarting task 1: "authentication"
4582026-03-10T17:46:32.611Zprocess exited: duration 7 ms, exit code 0
 
4592026-03-10T17:46:32.616Zstarting task 2: "clone repository"
4602026-03-10T17:46:32.619Z+ mkdir -p /work/oxidecomputer/cockroach
4612026-03-10T17:46:32.619Z+ git clone https://github.com/oxidecomputer/cockroach /work/oxidecomputer/cockroach
4622026-03-10T17:46:32.632ZCloning into '/work/oxidecomputer/cockroach'...
4632026-03-10T17:54:00.685Z+ cd /work/oxidecomputer/cockroach
4642026-03-10T17:54:00.685Z+ git fetch origin 21bdff50eb9c7974e74b511a5ae08a75cbc1619c
4652026-03-10T17:54:00.923ZFrom https://github.com/oxidecomputer/cockroach
4662026-03-10T17:54:00.923Z * branch 21bdff50eb9c7974e74b511a5ae08a75cbc1619c -> FETCH_HEAD
4672026-03-10T17:54:00.926Z+ [[ -n iliana/sha256sums ]]
4682026-03-10T17:54:00.929Z++ git branch --show-current
4692026-03-10T17:54:00.929Z+ current=release-22.1-oxide
4702026-03-10T17:54:00.929Z+ [[ release-22.1-oxide != iliana/sha256sums ]]
4712026-03-10T17:54:00.929Z+ git branch -f iliana/sha256sums 21bdff50eb9c7974e74b511a5ae08a75cbc1619c
4722026-03-10T17:54:00.929Z+ git checkout -f iliana/sha256sums
4732026-03-10T17:54:01.504ZSwitched to branch 'iliana/sha256sums'
4742026-03-10T17:54:01.507Z+ git reset --hard 21bdff50eb9c7974e74b511a5ae08a75cbc1619c
4752026-03-10T17:54:01.526ZHEAD is now at 21bdff50eb better handling of checksums in CI
4762026-03-10T17:54:01.529Zprocess exited: duration 448910 ms, exit code 0
 
4772026-03-10T17:54:01.535Zstarting task 3: "build"
4782026-03-10T17:54:01.538Z+ source .github/buildomat/linux-setup.sh
4792026-03-10T17:54:01.538Z++ source .github/buildomat/versions.sh
4802026-03-10T17:54:01.538Z+++ export GOTOOLCHAIN=local
4812026-03-10T17:54:01.538Z+++ GOTOOLCHAIN=local
4822026-03-10T17:54:01.538Z+++ GO_VERSION=1.22.11
4832026-03-10T17:54:01.538Z+++ NODE_VERSION=20.11.1
4842026-03-10T17:54:01.538Z+++ YARN_VERSION=1.22.22
4852026-03-10T17:54:01.538Z++ sudo apt-get install -y --no-install-recommends build-essential autoconf cmake libedit-dev ncurses-dev
4862026-03-10T17:54:01.591ZReading package lists...
4872026-03-10T17:54:01.711ZBuilding dependency tree...
4882026-03-10T17:54:01.714ZReading state information...
4892026-03-10T17:54:01.807Zbuild-essential is already the newest version (12.9ubuntu3).
4902026-03-10T17:54:01.807ZThe following additional packages will be installed:
4912026-03-10T17:54:01.810Z cmake-data dh-elpa-helper emacsen-common libbsd-dev libjsoncpp25 libmd-dev
4922026-03-10T17:54:01.810Z libncurses6 libncursesw6 librhash0 libtinfo6 m4
4932026-03-10T17:54:01.811ZSuggested packages:
4942026-03-10T17:54:01.811Z autoconf-archive gnu-standards autoconf-doc libtool gettext cmake-doc
4952026-03-10T17:54:01.811Z ninja-build cmake-format ncurses-doc m4-doc
4962026-03-10T17:54:01.811ZRecommended packages:
4972026-03-10T17:54:01.811Z automake | automaken
4982026-03-10T17:54:01.833ZThe following NEW packages will be installed:
4992026-03-10T17:54:01.836Z autoconf cmake cmake-data dh-elpa-helper emacsen-common libbsd-dev
5002026-03-10T17:54:01.836Z libedit-dev libjsoncpp25 libmd-dev libncurses-dev librhash0 m4
5012026-03-10T17:54:01.836ZThe following packages will be upgraded:
5022026-03-10T17:54:01.836Z libncurses6 libncursesw6 libtinfo6
5032026-03-10T17:54:01.849Z3 upgraded, 12 newly installed, 0 to remove and 298 not upgraded.
5042026-03-10T17:54:01.870ZNeed to get 8,757 kB of archives.
5052026-03-10T17:54:01.870ZAfter this operation, 38.1 MB of additional disk space will be used.
5062026-03-10T17:54:01.870ZGet:1 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libncursesw6 amd64 6.3-2ubuntu0.1 [147 kB]
5072026-03-10T17:54:01.876ZGet:2 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libncurses6 amd64 6.3-2ubuntu0.1 [111 kB]
5082026-03-10T17:54:01.879ZGet:3 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libtinfo6 amd64 6.3-2ubuntu0.1 [105 kB]
5092026-03-10T17:54:01.882ZGet:4 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 m4 amd64 1.4.18-5ubuntu2 [199 kB]
5102026-03-10T17:54:01.885ZGet:5 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 autoconf all 2.71-2 [338 kB]
5112026-03-10T17:54:01.889ZGet:6 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 libjsoncpp25 amd64 1.9.5-3 [80.0 kB]
5122026-03-10T17:54:01.892ZGet:7 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 librhash0 amd64 1.4.2-1ubuntu1 [125 kB]
5132026-03-10T17:54:01.895ZGet:8 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 dh-elpa-helper all 2.0.9ubuntu1 [7,610 B]
5142026-03-10T17:54:01.898ZGet:9 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 emacsen-common all 3.0.4 [14.9 kB]
5152026-03-10T17:54:01.898ZGet:10 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 cmake-data all 3.22.1-1ubuntu1.22.04.2 [1,913 kB]
5162026-03-10T17:54:01.914ZGet:11 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 cmake amd64 3.22.1-1ubuntu1.22.04.2 [5,010 kB]
5172026-03-10T17:54:01.948ZGet:12 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 libmd-dev amd64 1.0.4-1build1 [40.0 kB]
5182026-03-10T17:54:01.951ZGet:13 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 libbsd-dev amd64 0.11.5-1 [168 kB]
5192026-03-10T17:54:01.954ZGet:14 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libncurses-dev amd64 6.3-2ubuntu0.1 [381 kB]
5202026-03-10T17:54:01.958ZGet:15 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 libedit-dev amd64 3.1-20210910-1build1 [118 kB]
5212026-03-10T17:54:02.046Zdebconf: unable to initialize frontend: Dialog
5222026-03-10T17:54:02.046Zdebconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.)
5232026-03-10T17:54:02.046Zdebconf: falling back to frontend: Readline
5242026-03-10T17:54:02.049Zdebconf: unable to initialize frontend: Readline
5252026-03-10T17:54:02.049Zdebconf: (This frontend requires a controlling tty.)
5262026-03-10T17:54:02.049Zdebconf: falling back to frontend: Teletype
5272026-03-10T17:54:02.052Zdpkg-preconfigure: unable to re-open stdin:
5282026-03-10T17:54:02.064ZFetched 8,757 kB in 0s (86.6 MB/s)
5292026-03-10T17:54:02.101Z(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 69365 files and directories currently installed.)
5302026-03-10T17:54:02.104ZPreparing to unpack .../libncursesw6_6.3-2ubuntu0.1_amd64.deb ...
5312026-03-10T17:54:02.125ZUnpacking libncursesw6:amd64 (6.3-2ubuntu0.1) over (6.3-2) ...
5322026-03-10T17:54:02.177ZPreparing to unpack .../libncurses6_6.3-2ubuntu0.1_amd64.deb ...
5332026-03-10T17:54:02.191ZUnpacking libncurses6:amd64 (6.3-2ubuntu0.1) over (6.3-2) ...
5342026-03-10T17:54:02.240ZPreparing to unpack .../libtinfo6_6.3-2ubuntu0.1_amd64.deb ...
5352026-03-10T17:54:02.256ZUnpacking libtinfo6:amd64 (6.3-2ubuntu0.1) over (6.3-2) ...
5362026-03-10T17:54:02.301ZSetting up libtinfo6:amd64 (6.3-2ubuntu0.1) ...
5372026-03-10T17:54:02.340ZSelecting previously unselected package m4.
5382026-03-10T17:54:02.350Z(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 69365 files and directories currently installed.)
5392026-03-10T17:54:02.354ZPreparing to unpack .../00-m4_1.4.18-5ubuntu2_amd64.deb ...
5402026-03-10T17:54:02.358ZUnpacking m4 (1.4.18-5ubuntu2) ...
5412026-03-10T17:54:02.403ZSelecting previously unselected package autoconf.
5422026-03-10T17:54:02.407ZPreparing to unpack .../01-autoconf_2.71-2_all.deb ...
5432026-03-10T17:54:02.410ZUnpacking autoconf (2.71-2) ...
5442026-03-10T17:54:02.459ZSelecting previously unselected package libjsoncpp25:amd64.
5452026-03-10T17:54:02.462ZPreparing to unpack .../02-libjsoncpp25_1.9.5-3_amd64.deb ...
5462026-03-10T17:54:02.464ZUnpacking libjsoncpp25:amd64 (1.9.5-3) ...
5472026-03-10T17:54:02.513ZSelecting previously unselected package librhash0:amd64.
5482026-03-10T17:54:02.516ZPreparing to unpack .../03-librhash0_1.4.2-1ubuntu1_amd64.deb ...
5492026-03-10T17:54:02.516ZUnpacking librhash0:amd64 (1.4.2-1ubuntu1) ...
5502026-03-10T17:54:02.545ZSelecting previously unselected package dh-elpa-helper.
5512026-03-10T17:54:02.549ZPreparing to unpack .../04-dh-elpa-helper_2.0.9ubuntu1_all.deb ...
5522026-03-10T17:54:02.552ZUnpacking dh-elpa-helper (2.0.9ubuntu1) ...
5532026-03-10T17:54:02.595ZSelecting previously unselected package emacsen-common.
5542026-03-10T17:54:02.598ZPreparing to unpack .../05-emacsen-common_3.0.4_all.deb ...
5552026-03-10T17:54:02.607ZUnpacking emacsen-common (3.0.4) ...
5562026-03-10T17:54:02.653ZSelecting previously unselected package cmake-data.
5572026-03-10T17:54:02.656ZPreparing to unpack .../06-cmake-data_3.22.1-1ubuntu1.22.04.2_all.deb ...
5582026-03-10T17:54:02.661ZUnpacking cmake-data (3.22.1-1ubuntu1.22.04.2) ...
5592026-03-10T17:54:02.867ZSelecting previously unselected package cmake.
5602026-03-10T17:54:02.870ZPreparing to unpack .../07-cmake_3.22.1-1ubuntu1.22.04.2_amd64.deb ...
5612026-03-10T17:54:02.876ZUnpacking cmake (3.22.1-1ubuntu1.22.04.2) ...
5622026-03-10T17:54:02.964ZSelecting previously unselected package libmd-dev:amd64.
5632026-03-10T17:54:02.967ZPreparing to unpack .../08-libmd-dev_1.0.4-1build1_amd64.deb ...
5642026-03-10T17:54:02.971ZUnpacking libmd-dev:amd64 (1.0.4-1build1) ...
5652026-03-10T17:54:03.008ZSelecting previously unselected package libbsd-dev:amd64.
5662026-03-10T17:54:03.013ZPreparing to unpack .../09-libbsd-dev_0.11.5-1_amd64.deb ...
5672026-03-10T17:54:03.016ZUnpacking libbsd-dev:amd64 (0.11.5-1) ...
5682026-03-10T17:54:03.060ZSelecting previously unselected package libncurses-dev:amd64.
5692026-03-10T17:54:03.066ZPreparing to unpack .../10-libncurses-dev_6.3-2ubuntu0.1_amd64.deb ...
5702026-03-10T17:54:03.069ZUnpacking libncurses-dev:amd64 (6.3-2ubuntu0.1) ...
5712026-03-10T17:54:03.109ZSelecting previously unselected package libedit-dev:amd64.
5722026-03-10T17:54:03.113ZPreparing to unpack .../11-libedit-dev_3.1-20210910-1build1_amd64.deb ...
5732026-03-10T17:54:03.117ZUnpacking libedit-dev:amd64 (3.1-20210910-1build1) ...
5742026-03-10T17:54:03.157ZSetting up m4 (1.4.18-5ubuntu2) ...
5752026-03-10T17:54:03.173ZSetting up emacsen-common (3.0.4) ...
5762026-03-10T17:54:03.200ZSetting up dh-elpa-helper (2.0.9ubuntu1) ...
5772026-03-10T17:54:03.215ZSetting up libncurses6:amd64 (6.3-2ubuntu0.1) ...
5782026-03-10T17:54:03.230ZSetting up libjsoncpp25:amd64 (1.9.5-3) ...
5792026-03-10T17:54:03.244ZSetting up libncursesw6:amd64 (6.3-2ubuntu0.1) ...
5802026-03-10T17:54:03.258ZSetting up autoconf (2.71-2) ...
5812026-03-10T17:54:03.278ZSetting up librhash0:amd64 (1.4.2-1ubuntu1) ...
5822026-03-10T17:54:03.295ZSetting up libmd-dev:amd64 (1.0.4-1build1) ...
5832026-03-10T17:54:03.306ZSetting up cmake-data (3.22.1-1ubuntu1.22.04.2) ...
5842026-03-10T17:54:03.334ZSetting up libbsd-dev:amd64 (0.11.5-1) ...
5852026-03-10T17:54:03.348ZSetting up libncurses-dev:amd64 (6.3-2ubuntu0.1) ...
5862026-03-10T17:54:03.362ZSetting up libedit-dev:amd64 (3.1-20210910-1build1) ...
5872026-03-10T17:54:03.376ZSetting up cmake (3.22.1-1ubuntu1.22.04.2) ...
5882026-03-10T17:54:03.391ZProcessing triggers for libc-bin (2.35-0ubuntu3) ...
5892026-03-10T17:54:03.411ZProcessing triggers for man-db (2.10.2-1) ...
5902026-03-10T17:54:03.580ZProcessing triggers for install-info (6.8-4build1) ...
5912026-03-10T17:54:03.832Z
5922026-03-10T17:54:03.832ZRunning kernel seems to be up-to-date.
5932026-03-10T17:54:03.832Z
5942026-03-10T17:54:03.832ZServices to be restarted:
5952026-03-10T17:54:03.837Z systemctl restart acpid.service
5962026-03-10T17:54:03.837Z systemctl restart buildomat-agent.service
5972026-03-10T17:54:03.837Z systemctl restart chrony.service
5982026-03-10T17:54:03.837Z systemctl restart cron.service
5992026-03-10T17:54:03.837Z systemctl restart irqbalance.service
6002026-03-10T17:54:03.837Z systemctl restart multipathd.service
6012026-03-10T17:54:03.837Z systemctl restart packagekit.service
6022026-03-10T17:54:03.838Z systemctl restart polkit.service
6032026-03-10T17:54:03.838Z systemctl restart rsyslog.service
6042026-03-10T17:54:03.843Z systemctl restart serial-getty@ttyS0.service
6052026-03-10T17:54:03.843Z systemctl restart snapd.service
6062026-03-10T17:54:03.843Z systemctl restart ssh.service
6072026-03-10T17:54:03.848Z systemctl restart systemd-journald.service
6082026-03-10T17:54:03.848Z systemctl restart systemd-networkd.service
6092026-03-10T17:54:03.851Z systemctl restart systemd-resolved.service
6102026-03-10T17:54:03.851Z systemctl restart systemd-udevd.service
6112026-03-10T17:54:03.851Z systemctl restart udisks2.service
6122026-03-10T17:54:03.856Z
6132026-03-10T17:54:03.856ZService restarts being deferred:
6142026-03-10T17:54:03.856Z systemctl restart ModemManager.service
6152026-03-10T17:54:03.856Z /etc/needrestart/restart.d/dbus.service
6162026-03-10T17:54:03.856Z systemctl restart getty@tty1.service
6172026-03-10T17:54:03.856Z systemctl restart networkd-dispatcher.service
6182026-03-10T17:54:03.856Z systemctl restart systemd-logind.service
6192026-03-10T17:54:03.856Z systemctl restart unattended-upgrades.service
6202026-03-10T17:54:03.856Z
6212026-03-10T17:54:03.856ZNo containers need to be restarted.
6222026-03-10T17:54:03.856Z
6232026-03-10T17:54:03.856ZNo user sessions are running outdated binaries.
6242026-03-10T17:54:03.856Z
6252026-03-10T17:54:03.856ZNo VM guests are running outdated hypervisor (qemu) binaries on this host.
6262026-03-10T17:54:06.118Z++ pushd /work
6272026-03-10T17:54:06.118Z/work /work/oxidecomputer/cockroach
6282026-03-10T17:54:06.118Z++ curl -sSfL --retry 10 -O https://go.dev/dl/go1.22.11.linux-amd64.tar.gz
6292026-03-10T17:54:06.980Z++ curl -sSfL --retry 10 -O https://nodejs.org/dist/v20.11.1/node-v20.11.1-linux-x64.tar.xz
6302026-03-10T17:54:07.199Z++ curl -sSfL --retry 10 -O https://github.com/yarnpkg/yarn/releases/download/v1.22.22/yarn-1.22.22.js
6312026-03-10T17:54:07.421Z++ sha256sum -c /work/oxidecomputer/cockroach/.github/buildomat/SHA256SUMS.linux
6322026-03-10T17:54:07.542Zgo1.22.11.linux-amd64.tar.gz: OK
6332026-03-10T17:54:07.587Znode-v20.11.1-linux-x64.tar.xz: OK
6342026-03-10T17:54:07.596Zyarn-1.22.22.js: OK
6352026-03-10T17:54:07.599Z++ tar xf go1.22.11.linux-amd64.tar.gz
6362026-03-10T17:54:08.530Z++ tar xf node-v20.11.1-linux-x64.tar.xz
6372026-03-10T17:54:09.687Z++ mv yarn-1.22.22.js node-v20.11.1-linux-x64/bin/yarn
6382026-03-10T17:54:09.691Z++ chmod a+x node-v20.11.1-linux-x64/bin/yarn
6392026-03-10T17:54:09.691Z++ popd
6402026-03-10T17:54:09.691Z++ export PATH=/work/go/bin:/work/node-v20.11.1-linux-x64/bin:/home/build/.cargo/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/ooce/bin:/opt/ooce/sbin
6412026-03-10T17:54:09.691Z++ PATH=/work/go/bin:/work/node-v20.11.1-linux-x64/bin:/home/build/.cargo/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/ooce/bin:/opt/ooce/sbin
6422026-03-10T17:54:09.691Z/work/oxidecomputer/cockroach
6432026-03-10T17:54:09.691Z++ nproc
6442026-03-10T17:54:09.691Z+ gmake -j8 test
6452026-03-10T17:54:09.698ZWarning: 'ccache' not found, consider installing it for faster builds
6462026-03-10T17:54:09.701ZWarning: 'bazel' not found (`brew install bazelisk` for macs)
6472026-03-10T17:54:09.740ZDetected change in build system. Rebooting make.
6482026-03-10T17:54:09.750ZWarning: 'ccache' not found, consider installing it for faster builds
6492026-03-10T17:54:09.753ZWarning: 'bazel' not found (`brew install bazelisk` for macs)
6502026-03-10T17:54:09.790Zgitdir=$(git rev-parse --git-dir 2>/dev/null || true); \
6512026-03-10T17:54:09.791Zif test -n "$gitdir"; then \
6522026-03-10T17:54:09.791Z git submodule update --init --recursive; \
6532026-03-10T17:54:09.791Zfi
6542026-03-10T17:54:09.807ZSubmodule 'c-deps/geos' (https://github.com/oxidecomputer/geos.git) registered for path 'c-deps/geos'
6552026-03-10T17:54:09.811ZSubmodule 'c-deps/jemalloc' (https://github.com/cockroachdb/jemalloc.git) registered for path 'c-deps/jemalloc'
6562026-03-10T17:54:09.811ZSubmodule 'c-deps/proj' (https://github.com/oxidecomputer/PROJ.git) registered for path 'c-deps/proj'
6572026-03-10T17:54:09.811Zmkdir -p pkg/sql/parser/gen
6582026-03-10T17:54:09.815ZCloning into '/work/oxidecomputer/cockroach/c-deps/geos'...
6592026-03-10T17:54:09.815Zset -euo pipefail; \
6602026-03-10T17:54:09.815Zawk '/func.*sqlSymUnion/ {print $(NF - 1)}' pkg/sql/parser/sql.y | \
6612026-03-10T17:54:09.815Zsed -e 's/[]\/$*.^|[]/\\&/g' | \
6622026-03-10T17:54:09.815Zsed -e "s/^/s_(type|token) <(/" | \
6632026-03-10T17:54:09.815Zawk '{print $0")>_\\1 <union> /* <\\2> */_"}' > pkg/sql/parser/gen/types_regex.tmp; \
6642026-03-10T17:54:09.815Zsed -E -f pkg/sql/parser/gen/types_regex.tmp < pkg/sql/parser/sql.y | \
6652026-03-10T17:54:09.815Zawk -f pkg/sql/parser/replace_help_rules.awk | \
6662026-03-10T17:54:09.815Zsed -Ee 's,//.*$,,g;s,/[*]([^*]|[*][^/])*[*]/, ,g;s/ +$//g' > pkg/sql/parser/gen/sql-gen.y.tmp || rm pkg/sql/parser/gen/sql-gen.y.tmp
6672026-03-10T17:54:09.884Zmv -f pkg/sql/parser/gen/sql-gen.y.tmp pkg/sql/parser/gen/sql-gen.y
6682026-03-10T17:54:09.887Zrm pkg/sql/parser/gen/types_regex.tmp
6692026-03-10T17:54:25.815ZCloning into '/work/oxidecomputer/cockroach/c-deps/jemalloc'...
6702026-03-10T17:54:28.134ZCloning into '/work/oxidecomputer/cockroach/c-deps/proj'...
6712026-03-10T17:54:57.318ZSubmodule path 'c-deps/geos': checked out '8bfb91ff2cfee16b29316894fd65d0f2a8a09607'
6722026-03-10T17:54:57.335ZSubmodule path 'c-deps/jemalloc': checked out 'fd00c883c119539844b3ffd5ad8640968e0cb1a3'
6732026-03-10T17:54:57.453ZSubmodule path 'c-deps/proj': checked out '47f32de4d4b683af4891844d6a6aefce5801a082'
6742026-03-10T17:54:57.456Zmkdir -p bin
6752026-03-10T17:54:57.459Ztouch bin/.submodules-initialized
6762026-03-10T17:54:57.459Zcd /work/oxidecomputer/cockroach/c-deps/jemalloc && autoconf
6772026-03-10T17:54:57.459Zrm -rf /work/oxidecomputer/cockroach/lib/build/geos
6782026-03-10T17:54:57.459Zrm -rf /work/oxidecomputer/cockroach/lib/build/proj
6792026-03-10T17:54:57.459Zmkdir -p /work/oxidecomputer/cockroach/lib/build/geos
6802026-03-10T17:54:57.459Zmkdir -p /work/oxidecomputer/cockroach/lib/build/proj
6812026-03-10T17:54:57.462Zcd /work/oxidecomputer/cockroach/lib/build/proj && cmake -DCMAKE_TARGET_MESSAGES=OFF /work/oxidecomputer/cockroach/c-deps/proj -DCMAKE_BUILD_TYPE=Release -DBUILD_LIBPROJ_SHARED=OFF
6822026-03-10T17:54:57.462Zcd /work/oxidecomputer/cockroach/lib/build/geos && \
6832026-03-10T17:54:57.462Z cmake -DCMAKE_TARGET_MESSAGES=OFF /work/oxidecomputer/cockroach/c-deps/geos -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_FLAGS=-fPIC -DCMAKE_CXX_FLAGS=-fPIC
6842026-03-10T17:54:57.465Z-- GEOS: Build type: Release
6852026-03-10T17:54:57.465Z-- GEOS: Run-time output: /work/oxidecomputer/cockroach/lib/build/geos/bin
6862026-03-10T17:54:57.465Z-- GEOS: Archives output: /work/oxidecomputer/cockroach/lib/build/geos/lib
6872026-03-10T17:54:57.509Z-- The C compiler identification is GNU 11.4.0
6882026-03-10T17:54:57.510Z-- The C compiler identification is GNU 11.4.0
6892026-03-10T17:54:57.513Z-- Detecting C compiler ABI info
6902026-03-10T17:54:57.535Z-- The CXX compiler identification is GNU 11.4.0
6912026-03-10T17:54:57.538Z-- Detecting C compiler ABI info
6922026-03-10T17:54:57.551Z-- Detecting C compiler ABI info - done
6932026-03-10T17:54:57.555Z-- Check for working C compiler: /usr/bin/cc - skipped
6942026-03-10T17:54:57.555Z-- Detecting C compile features
6952026-03-10T17:54:57.555Z-- Detecting C compile features - done
6962026-03-10T17:54:57.555Z--
6972026-03-10T17:54:57.555Z-- Configuring PROJ:
6982026-03-10T17:54:57.555Z--
6992026-03-10T17:54:57.555Z-- PROJ_VERSION = 4.9.3
7002026-03-10T17:54:57.555Z-- Looking for include file dlfcn.h
7012026-03-10T17:54:57.562Z-- Detecting C compiler ABI info - done
7022026-03-10T17:54:57.565Z-- Check for working C compiler: /usr/bin/cc - skipped
7032026-03-10T17:54:57.569Z-- Detecting C compile features
7042026-03-10T17:54:57.569Z-- Detecting C compile features - done
7052026-03-10T17:54:57.569Z-- Detecting CXX compiler ABI info
7062026-03-10T17:54:57.584Z-- Looking for include file dlfcn.h - found
7072026-03-10T17:54:57.597Z-- Looking for include file inttypes.h
7082026-03-10T17:54:57.600Z-- Looking for include file inttypes.h - found
7092026-03-10T17:54:57.600Z-- Detecting CXX compiler ABI info - done
7102026-03-10T17:54:57.601Z-- Looking for include file jni.h
7112026-03-10T17:54:57.612Z-- Check for working CXX compiler: /usr/bin/c++ - skipped
7122026-03-10T17:54:57.612Z-- Detecting CXX compile features
7132026-03-10T17:54:57.613Z-- Detecting CXX compile features - done
7142026-03-10T17:54:57.613Z-- GEOS: Version 3.8.1
7152026-03-10T17:54:57.613Z-- GEOS: C API Version 1.13.3
7162026-03-10T17:54:57.613Z-- GEOS: JTS port 1.13.0
7172026-03-10T17:54:57.613Z-- GEOS: Require C++11
7182026-03-10T17:54:57.616Zrm -rf /work/oxidecomputer/cockroach/lib/build/jemalloc
7192026-03-10T17:54:57.616Zmkdir -p /work/oxidecomputer/cockroach/lib/build/jemalloc
7202026-03-10T17:54:57.616Zcd /work/oxidecomputer/cockroach/lib/build/jemalloc && /work/oxidecomputer/cockroach/c-deps/jemalloc/configure --enable-prof
7212026-03-10T17:54:57.616Z-- Looking for include file jni.h - not found
7222026-03-10T17:54:57.616Z-- Looking for include file memory.h
7232026-03-10T17:54:57.645Z-- Looking for include file memory.h - found
7242026-03-10T17:54:57.658Z-- Looking for include file stdint.h
7252026-03-10T17:54:57.658Zchecking for xsltproc... false
7262026-03-10T17:54:57.659Zchecking for gcc... gcc
7272026-03-10T17:54:57.659Z-- GEOS: Configured 'dist' target
7282026-03-10T17:54:57.659Z-- GEOS: Configured 'distcheck' target
7292026-03-10T17:54:57.659Z-- Configuring done
7302026-03-10T17:54:57.662Zchecking whether the C compiler works... yes
7312026-03-10T17:54:57.662Zchecking for C compiler default output file name... a.out
7322026-03-10T17:54:57.676Zchecking for suffix of executables... -- Looking for include file stdint.h - found
7332026-03-10T17:54:57.676Z-- Looking for include file stdlib.h
7342026-03-10T17:54:57.679Z
7352026-03-10T17:54:57.686Zchecking whether we are cross compiling... -- Generating done
7362026-03-10T17:54:57.689Z-- Build files have been written to: /work/oxidecomputer/cockroach/lib/build/geos
7372026-03-10T17:54:57.692Zno
7382026-03-10T17:54:57.695Zchecking for suffix of object files... mkdir /work/oxidecomputer/cockroach/lib/build/geos/capi/geos
7392026-03-10T17:54:57.695Zcp /work/oxidecomputer/cockroach/c-deps/geos/include/geos/export.h /work/oxidecomputer/cockroach/lib/build/geos/capi/geos
7402026-03-10T17:54:57.695Z-- Looking for include file stdlib.h - found
7412026-03-10T17:54:57.695Z-- Looking for include file string.h
7422026-03-10T17:54:57.698Zo
7432026-03-10T17:54:57.704Zchecking whether the compiler supports GNU C... yes
7442026-03-10T17:54:57.710Zchecking whether gcc accepts -g... yes
7452026-03-10T17:54:57.722Zchecking for gcc option to enable C11 features... -- Looking for include file string.h - found
7462026-03-10T17:54:57.725Z-- Looking for include file sys/stat.h
7472026-03-10T17:54:57.725Znone needed
7482026-03-10T17:54:57.731Zchecking whether compiler is cray... no
7492026-03-10T17:54:57.736Zchecking whether compiler supports -std=gnu11... yes
7502026-03-10T17:54:57.742Zchecking whether compiler supports -Wall... yes
7512026-03-10T17:54:57.748Zchecking whether compiler supports -Werror=declaration-after-statement... yes
7522026-03-10T17:54:57.751Zchecking whether compiler supports -Wshorten-64-to-32... -- Looking for include file sys/stat.h - found
7532026-03-10T17:54:57.751Z-- Looking for include file sys/types.h
7542026-03-10T17:54:57.758Zno
7552026-03-10T17:54:57.763Zchecking whether compiler supports -Wsign-compare... yes
7562026-03-10T17:54:57.768Zchecking whether compiler supports -pipe... yes
7572026-03-10T17:54:57.776Zchecking whether compiler supports -g3... yes
7582026-03-10T17:54:57.779Zchecking how to run the C preprocessor... -- Looking for include file sys/types.h - found
7592026-03-10T17:54:57.779Z-- Looking for include file unistd.h
7602026-03-10T17:54:57.782Zgcc -E
7612026-03-10T17:54:57.795Zchecking for stdio.h... yes
7622026-03-10T17:54:57.802Zchecking for stdlib.h... yes
7632026-03-10T17:54:57.805Zchecking for string.h... -- Looking for include file unistd.h - found
7642026-03-10T17:54:57.808Z-- Looking for 4 include files stdlib.h, ..., float.h
7652026-03-10T17:54:57.811Zyes
7662026-03-10T17:54:57.815Zchecking for inttypes.h... yes
7672026-03-10T17:54:57.822Zchecking for stdint.h... yes
7682026-03-10T17:54:57.829Zchecking for strings.h... yes
7692026-03-10T17:54:57.834Zchecking for sys/stat.h... -- Looking for 4 include files stdlib.h, ..., float.h - found
7702026-03-10T17:54:57.838Z-- Looking for localeconv
7712026-03-10T17:54:57.838Zyes
7722026-03-10T17:54:57.843Zchecking for sys/types.h... yes
7732026-03-10T17:54:57.851Zchecking for unistd.h... yes
7742026-03-10T17:54:57.862Zchecking whether byte ordering is bigendian... -- Looking for localeconv - found
7752026-03-10T17:54:57.865Z-- Looking for ceil in m
7762026-03-10T17:54:57.872Zno
7772026-03-10T17:54:57.890Zchecking size of void *... 8
7782026-03-10T17:54:57.893Zchecking size of int... -- Looking for ceil in m - found
7792026-03-10T17:54:57.893Z-- Looking for pthread.h
7802026-03-10T17:54:57.910Z4
7812026-03-10T17:54:57.921Zchecking size of long... -- Looking for pthread.h - found
7822026-03-10T17:54:57.924Z-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
7832026-03-10T17:54:57.928Z8
7842026-03-10T17:54:57.947Zchecking size of long long... 8
7852026-03-10T17:54:57.950Zchecking size of intmax_t... -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
7862026-03-10T17:54:57.955Z-- Found Threads: TRUE
7872026-03-10T17:54:57.955Z-- Performing Test HAVE_PTHREAD_MUTEX_RECURSIVE
7882026-03-10T17:54:57.966Z8
7892026-03-10T17:54:57.980Zchecking build system type... -- Performing Test HAVE_PTHREAD_MUTEX_RECURSIVE - Success
7902026-03-10T17:54:57.983Z-- PROJ_PLATFORM_NAME = x64
7912026-03-10T17:54:57.983Z-- PROJ_COMPILER_NAME = gcc-11
7922026-03-10T17:54:57.983Z-- PROJ4_TESTS = ON
7932026-03-10T17:54:57.986Zx86_64-pc-linux-gnu
7942026-03-10T17:54:57.986Zchecking host system type... x86_64-pc-linux-gnu
7952026-03-10T17:54:57.997Zchecking whether pause instruction is compilable... yes
7962026-03-10T17:54:58.000Zchecking for ar... ar
7972026-03-10T17:54:58.003Zchecking for malloc.h... --
7982026-03-10T17:54:58.003Z-- Configuring proj library:
7992026-03-10T17:54:58.003Z--
8002026-03-10T17:54:58.006Zyes
8012026-03-10T17:54:58.009Zchecking whether malloc_usable_size definition can use const argument... -- JNI_SUPPORT = OFF
8022026-03-10T17:54:58.009Z-- PROJ_CORE_TARGET = proj
8032026-03-10T17:54:58.009Z-- PROJ_CORE_TARGET_OUTPUT_NAME = proj
8042026-03-10T17:54:58.010Z-- PROJ_LIBRARY_TYPE = STATIC
8052026-03-10T17:54:58.010Z-- PROJ_LIBRARIES = proj
8062026-03-10T17:54:58.010Z-- Configuring done
8072026-03-10T17:54:58.015Zno
8082026-03-10T17:54:58.022Zchecking for library containing log... -- Generating done
8092026-03-10T17:54:58.025Z-- Build files have been written to: /work/oxidecomputer/cockroach/lib/build/proj
8102026-03-10T17:54:58.049Z-lm
8112026-03-10T17:54:58.059Zchecking whether __attribute__ syntax is compilable... yes
8122026-03-10T17:54:58.064Zchecking whether compiler supports -fvisibility=hidden... yes
8132026-03-10T17:54:58.069Zchecking whether compiler supports -Werror... yes
8142026-03-10T17:54:58.104Zchecking whether compiler supports -herror_on_warning... yes
8152026-03-10T17:54:58.104Zchecking whether tls_model attribute is compilable... yes
8162026-03-10T17:54:58.104Zchecking whether compiler supports -Werror... yes
8172026-03-10T17:54:58.104Zchecking whether compiler supports -herror_on_warning... yes
8182026-03-10T17:54:58.114Zchecking whether alloc_size attribute is compilable... yes
8192026-03-10T17:54:58.119Zchecking whether compiler supports -Werror... yes
8202026-03-10T17:54:58.126Zchecking whether compiler supports -herror_on_warning... yes
8212026-03-10T17:54:58.140Zchecking whether format(gnu_printf, ...) attribute is compilable... yes
8222026-03-10T17:54:58.145Zchecking whether compiler supports -Werror... yes
8232026-03-10T17:54:58.151Zchecking whether compiler supports -herror_on_warning... yes
8242026-03-10T17:54:58.166Zchecking whether format(printf, ...) attribute is compilable... yes
8252026-03-10T17:54:58.171Zchecking for a BSD-compatible install... /usr/bin/install -c
8262026-03-10T17:54:58.174Zchecking for ranlib... ranlib
8272026-03-10T17:54:58.174Zchecking for ld... /usr/bin/ld
8282026-03-10T17:54:58.174Zchecking for autoconf... /usr/bin/autoconf
8292026-03-10T17:54:58.185Zchecking for memalign... yes
8302026-03-10T17:54:58.199Zchecking for valloc... yes
8312026-03-10T17:54:58.205Zchecking whether compiler supports -O3... yes
8322026-03-10T17:54:58.211Zchecking whether compiler supports -funroll-loops... yes
8332026-03-10T17:54:58.223Zchecking for unwind.h... yes
8342026-03-10T17:54:58.238Zchecking for _Unwind_Backtrace in -lgcc... yes
8352026-03-10T17:54:58.240Zchecking configured backtracing method... libgcc
8362026-03-10T17:54:58.253Zchecking for sbrk... yes
8372026-03-10T17:54:58.264Zchecking whether utrace(2) is compilable... no
8382026-03-10T17:54:58.270Zchecking whether valgrind is compilable... no
8392026-03-10T17:54:58.286Zchecking whether a program using __builtin_unreachable is compilable... yes
8402026-03-10T17:54:58.304Zchecking whether a program using __builtin_ffsl is compilable... yes
8412026-03-10T17:54:58.326Zchecking LG_PAGE... 12
8422026-03-10T17:54:58.333ZMissing VERSION file, and unable to generate it; creating bogus VERSION
8432026-03-10T17:54:58.351Zchecking for pthread.h... yes
8442026-03-10T17:54:58.366Zchecking for pthread_create in -lpthread... yes
8452026-03-10T17:54:58.383Zchecking whether pthread_atfork(3) is compilable... yes
8462026-03-10T17:54:58.399Zchecking for library containing clock_gettime... none required
8472026-03-10T17:54:58.416Zchecking whether clock_gettime(CLOCK_MONOTONIC_COARSE, ...) is compilable... yes
8482026-03-10T17:54:58.434Zchecking whether clock_gettime(CLOCK_MONOTONIC, ...) is compilable... yes
8492026-03-10T17:54:58.440Zchecking whether mach_absolute_time() is compilable... no
8502026-03-10T17:54:58.446Zchecking whether compiler supports -Werror... yes
8512026-03-10T17:54:58.464Zchecking whether syscall(2) is compilable... yes
8522026-03-10T17:54:58.480Zchecking for secure_getenv... yes
8532026-03-10T17:54:58.497Zchecking for issetugid... no
8542026-03-10T17:54:58.515Zchecking for _malloc_thread_cleanup... no
8552026-03-10T17:54:58.522Zchecking whether Darwin OSSpin*() is compilable... no
8562026-03-10T17:54:58.538Zchecking for _pthread_mutex_init_calloc_cb... no
8572026-03-10T17:54:58.544Zchecking for TLS... yes
8582026-03-10T17:54:58.560Zchecking whether C11 atomics is compilable... yes
8592026-03-10T17:54:58.569Zchecking whether atomic(9) is compilable... no
8602026-03-10T17:54:58.574Zchecking whether Darwin OSAtomic*() is compilable... no
8612026-03-10T17:54:58.590Zchecking whether madvise(2) is compilable... yes
8622026-03-10T17:54:58.606Zchecking whether madvise(..., MADV_FREE) is compilable... yes
8632026-03-10T17:54:58.621Zchecking whether madvise(..., MADV_DONTNEED) is compilable... yes
8642026-03-10T17:54:58.638Zchecking whether madvise(..., MADV_[NO]HUGEPAGE) is compilable... yes
8652026-03-10T17:54:58.644Zchecking whether to force 32-bit __sync_{add,sub}_and_fetch()... no
8662026-03-10T17:54:58.650Zchecking whether to force 64-bit __sync_{add,sub}_and_fetch()... no
8672026-03-10T17:54:58.667Zchecking for __builtin_clz... yes
8682026-03-10T17:54:58.673Zchecking whether Darwin os_unfair_lock_*() is compilable... no
8692026-03-10T17:54:58.691Zchecking whether glibc malloc hook is compilable... no
8702026-03-10T17:54:58.708Zchecking whether glibc memalign hook is compilable... no
8712026-03-10T17:54:58.726Zchecking whether pthreads adaptive mutexes is compilable... yes
8722026-03-10T17:54:58.751Zchecking for _Bool... yes
8732026-03-10T17:54:58.761Zchecking for stdbool.h that conforms to C99... no
8742026-03-10T17:54:58.768Zconfigure: creating ./config.status
8752026-03-10T17:54:58.794Zconfig.status: creating Makefile
8762026-03-10T17:54:58.798Zconfig.status: creating jemalloc.pc
8772026-03-10T17:54:58.802Zconfig.status: creating doc/html.xsl
8782026-03-10T17:54:58.807Zconfig.status: creating doc/manpages.xsl
8792026-03-10T17:54:58.812Zconfig.status: creating doc/jemalloc.xml
8802026-03-10T17:54:58.820Zconfig.status: creating include/jemalloc/jemalloc_macros.h
8812026-03-10T17:54:58.823Zconfig.status: creating include/jemalloc/jemalloc_protos.h
8822026-03-10T17:54:58.828Zconfig.status: creating include/jemalloc/jemalloc_typedefs.h
8832026-03-10T17:54:58.833Zconfig.status: creating include/jemalloc/internal/jemalloc_internal.h
8842026-03-10T17:54:58.839Zconfig.status: creating test/test.sh
8852026-03-10T17:54:58.844Zconfig.status: creating test/include/test/jemalloc_test.h
8862026-03-10T17:54:58.850Zconfig.status: creating config.stamp
8872026-03-10T17:54:58.853Zconfig.status: creating bin/jemalloc-config
8882026-03-10T17:54:58.859Zconfig.status: creating bin/jemalloc.sh
8892026-03-10T17:54:58.863Zconfig.status: creating bin/jeprof
8902026-03-10T17:54:58.870Zconfig.status: creating include/jemalloc/jemalloc_defs.h
8912026-03-10T17:54:58.874Zconfig.status: creating include/jemalloc/internal/jemalloc_internal_defs.h
8922026-03-10T17:54:58.878Zconfig.status: creating test/include/test/jemalloc_test_defs.h
8932026-03-10T17:54:58.883Zconfig.status: executing include/jemalloc/internal/private_namespace.h commands
8942026-03-10T17:54:58.886Zconfig.status: executing include/jemalloc/internal/private_unnamespace.h commands
8952026-03-10T17:54:58.889Zconfig.status: executing include/jemalloc/internal/public_symbols.txt commands
8962026-03-10T17:54:58.892Zconfig.status: executing include/jemalloc/internal/public_namespace.h commands
8972026-03-10T17:54:58.911Zconfig.status: executing include/jemalloc/internal/public_unnamespace.h commands
8982026-03-10T17:54:58.923Zconfig.status: executing include/jemalloc/internal/size_classes.h commands
8992026-03-10T17:54:59.009Zconfig.status: executing include/jemalloc/jemalloc_protos_jet.h commands
9002026-03-10T17:54:59.012Zconfig.status: executing include/jemalloc/jemalloc_rename.h commands
9012026-03-10T17:54:59.040Zconfig.status: executing include/jemalloc/jemalloc_mangle.h commands
9022026-03-10T17:54:59.069Zconfig.status: executing include/jemalloc/jemalloc_mangle_jet.h commands
9032026-03-10T17:54:59.099Zconfig.status: executing include/jemalloc/jemalloc.h commands
9042026-03-10T17:54:59.103Z===============================================================================
9052026-03-10T17:54:59.103Zjemalloc version : 0.0.0-0-g0000000000000000000000000000000000000000
9062026-03-10T17:54:59.106Zlibrary revision : 2
9072026-03-10T17:54:59.106Z
9082026-03-10T17:54:59.106ZCONFIG : --enable-prof CFLAGS=-g1 LDFLAGS=
9092026-03-10T17:54:59.106ZCC : gcc
9102026-03-10T17:54:59.106ZCONFIGURE_CFLAGS : -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops
9112026-03-10T17:54:59.106ZSPECIFIED_CFLAGS : -g1
9122026-03-10T17:54:59.106ZEXTRA_CFLAGS :
9132026-03-10T17:54:59.106ZCPPFLAGS : -D_GNU_SOURCE -D_REENTRANT
9142026-03-10T17:54:59.106ZLDFLAGS :
9152026-03-10T17:54:59.106ZEXTRA_LDFLAGS :
9162026-03-10T17:54:59.106ZLIBS : -lm -lgcc -lm -lpthread
9172026-03-10T17:54:59.106ZRPATH_EXTRA :
9182026-03-10T17:54:59.106Z
9192026-03-10T17:54:59.106ZXSLTPROC : false
9202026-03-10T17:54:59.106ZXSLROOT :
9212026-03-10T17:54:59.106Z
9222026-03-10T17:54:59.106ZPREFIX : /usr/local
9232026-03-10T17:54:59.106ZBINDIR : /usr/local/bin
9242026-03-10T17:54:59.106ZDATADIR : /usr/local/share
9252026-03-10T17:54:59.106ZINCLUDEDIR : /usr/local/include
9262026-03-10T17:54:59.106ZLIBDIR : /usr/local/lib
9272026-03-10T17:54:59.107ZMANDIR : /usr/local/share/man
9282026-03-10T17:54:59.107Z
9292026-03-10T17:54:59.107Zsrcroot : /work/oxidecomputer/cockroach/c-deps/jemalloc/
9302026-03-10T17:54:59.107Zabs_srcroot : /work/oxidecomputer/cockroach/c-deps/jemalloc/
9312026-03-10T17:54:59.107Zobjroot :
9322026-03-10T17:54:59.107Zabs_objroot : /work/oxidecomputer/cockroach/lib/build/jemalloc/
9332026-03-10T17:54:59.107Z
9342026-03-10T17:54:59.107ZJEMALLOC_PREFIX :
9352026-03-10T17:54:59.107ZJEMALLOC_PRIVATE_NAMESPACE
9362026-03-10T17:54:59.107Z : je_
9372026-03-10T17:54:59.107Zinstall_suffix :
9382026-03-10T17:54:59.107Zmalloc_conf :
9392026-03-10T17:54:59.107Zautogen : 0
9402026-03-10T17:54:59.107Zcc-silence : 1
9412026-03-10T17:54:59.107Zdebug : 0
9422026-03-10T17:54:59.107Zcode-coverage : 0
9432026-03-10T17:54:59.107Zstats : 1
9442026-03-10T17:54:59.107Zprof : 1
9452026-03-10T17:54:59.107Zprof-libunwind : 0
9462026-03-10T17:54:59.107Zprof-libgcc : 1
9472026-03-10T17:54:59.107Zprof-gcc : 0
9482026-03-10T17:54:59.107Ztcache : 1
9492026-03-10T17:54:59.107Zthp : 1
9502026-03-10T17:54:59.107Zfill : 1
9512026-03-10T17:54:59.107Zutrace : 0
9522026-03-10T17:54:59.107Zvalgrind : 0
9532026-03-10T17:54:59.107Zxmalloc : 0
9542026-03-10T17:54:59.107Zmunmap : 0
9552026-03-10T17:54:59.107Zlazy_lock : 0
9562026-03-10T17:54:59.107Ztls : 1
9572026-03-10T17:54:59.107Zcache-oblivious : 1
9582026-03-10T17:54:59.107Z===============================================================================
9592026-03-10T17:54:59.110Zgo mod download
9602026-03-10T17:55:15.231Zgo mod vendor
9612026-03-10T17:55:16.254ZGOBIN='/work/oxidecomputer/cockroach/bin' GOFLAGS= go install -mod=vendor -v github.com/goware/modvendor
9622026-03-10T17:55:16.265Zinternal/goos
9632026-03-10T17:55:16.267Zinternal/unsafeheader
9642026-03-10T17:55:16.268Zinternal/coverage/rtcov
9652026-03-10T17:55:16.268Zinternal/goarch
9662026-03-10T17:55:16.268Zinternal/godebugs
9672026-03-10T17:55:16.268Zinternal/goexperiment
9682026-03-10T17:55:16.268Zruntime/internal/atomic
9692026-03-10T17:55:16.268Zruntime/internal/syscall
9702026-03-10T17:55:16.268Zinternal/cpu
9712026-03-10T17:55:16.271Zinternal/abi
9722026-03-10T17:55:16.271Zinternal/chacha8rand
9732026-03-10T17:55:16.271Zruntime/internal/math
9742026-03-10T17:55:16.271Zruntime/internal/sys
9752026-03-10T17:55:16.271Zinternal/race
9762026-03-10T17:55:16.278Zsync/atomic
9772026-03-10T17:55:16.285Zunicode
9782026-03-10T17:55:16.285Zunicode/utf8
9792026-03-10T17:55:16.290Zencoding
9802026-03-10T17:55:16.290Zinternal/itoa
9812026-03-10T17:55:16.296Zmath/bits
9822026-03-10T17:55:16.296Zinternal/bytealg
9832026-03-10T17:55:16.296Zcmp
9842026-03-10T17:55:16.299Zslices
9852026-03-10T17:55:16.299Zmath
9862026-03-10T17:55:16.336Zruntime
9872026-03-10T17:55:17.230Zinternal/reflectlite
9882026-03-10T17:55:17.233Zsync
9892026-03-10T17:55:17.259Zinternal/testlog
9902026-03-10T17:55:17.274Zerrors
9912026-03-10T17:55:17.278Zsort
9922026-03-10T17:55:17.283Zio
9932026-03-10T17:55:17.286Zpath
9942026-03-10T17:55:17.286Zinternal/oserror
9952026-03-10T17:55:17.286Zinternal/safefilepath
9962026-03-10T17:55:17.286Zstrconv
9972026-03-10T17:55:17.289Zsyscall
9982026-03-10T17:55:17.308Zbytes
9992026-03-10T17:55:17.311Zstrings
10002026-03-10T17:55:17.350Zreflect
10012026-03-10T17:55:17.355Zbufio
10022026-03-10T17:55:17.358Zregexp/syntax
10032026-03-10T17:55:17.434Zregexp
10042026-03-10T17:55:17.445Zinternal/syscall/execenv
10052026-03-10T17:55:17.448Zinternal/syscall/unix
10062026-03-10T17:55:17.449Ztime
10072026-03-10T17:55:17.561Zio/fs
10082026-03-10T17:55:17.564Zinternal/poll
10092026-03-10T17:55:17.571Zinternal/fmtsort
10102026-03-10T17:55:17.605Zos
10112026-03-10T17:55:17.693Zpath/filepath
10122026-03-10T17:55:17.693Zfmt
10132026-03-10T17:55:17.748Zflag
10142026-03-10T17:55:17.748Zgithub.com/mattn/go-zglob/fastwalk
10152026-03-10T17:55:17.759Zgithub.com/mattn/go-zglob
10162026-03-10T17:55:17.779Zgithub.com/goware/modvendor
10172026-03-10T17:55:17.857Zmodvendor -copy="**/*.c **/*.h **/*.proto" -include 'github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis/google/api,github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis/google/rpc,github.com/prometheus/client_model'
10182026-03-10T17:55:18.077Zfor patch in patches/*.patch; do patch -p1 <$patch; done
10192026-03-10T17:55:18.080Zpatching file vendor/github.com/knz/go-libedit/unix/editline_unix.go
10202026-03-10T17:55:18.083Zpatching file vendor/github.com/elastic/gosigar/sigar_stub.go
10212026-03-10T17:55:18.083Zpatching file vendor/github.com/elastic/gosigar/sigar_illumos.go
10222026-03-10T17:55:18.083Zpatching file vendor/github.com/elastic/gosigar/sigar_unix_common.go
10232026-03-10T17:55:18.083Zpatching file vendor/github.com/cockroachdb/pebble/vfs/errors_unix.go
10242026-03-10T17:55:18.083Zpatching file vendor/github.com/cockroachdb/pebble/vfs/disk_usage_illumos.go
10252026-03-10T17:55:18.083Zgo install -v ./pkg/cmd/prereqs
10262026-03-10T17:55:18.083Zregenerating pkg/cli/zcgo_flags.go
10272026-03-10T17:55:18.083Zregenerating pkg/cli/clisqlshell/zcgo_flags.go
10282026-03-10T17:55:18.083Zregenerating pkg/server/status/zcgo_flags.go
10292026-03-10T17:55:18.083Zregenerating pkg/geo/geoproj/zcgo_flags.go
10302026-03-10T17:55:18.083Zregenerating vendor/github.com/knz/go-libedit/unix/zcgo_flags.go
10312026-03-10T17:55:18.083Zregenerating pkg/cli/zcgo_flags_x86_64_linux_gnu.go
10322026-03-10T17:55:18.090Zregenerating pkg/cli/clisqlshell/zcgo_flags_x86_64_linux_gnu.go
10332026-03-10T17:55:18.093Zregenerating pkg/server/status/zcgo_flags_x86_64_linux_gnu.go
10342026-03-10T17:55:18.093Zregenerating pkg/geo/geoproj/zcgo_flags_x86_64_linux_gnu.go
10352026-03-10T17:55:18.093Zregenerating vendor/github.com/knz/go-libedit/unix/zcgo_flags_x86_64_linux_gnu.go
10362026-03-10T17:55:18.093Zregenerating vendor/github.com/knz/go-libedit/unix/zcgo_flags_extra.go
10372026-03-10T17:55:18.121Zgo: downloading github.com/bufbuild/buf v0.56.0
10382026-03-10T17:55:18.279Zunicode/utf16
10392026-03-10T17:55:18.279Zinternal/goversion
10402026-03-10T17:55:18.279Zinternal/platform
10412026-03-10T17:55:18.279Zhash/maphash
10422026-03-10T17:55:18.282Zinternal/gover
10432026-03-10T17:55:18.282Zlog/internal
10442026-03-10T17:55:18.282Zgolang.org/x/tools/internal/packagesinternal
10452026-03-10T17:55:18.292Zcontainer/heap
10462026-03-10T17:55:18.292Zgolang.org/x/mod/semver
10472026-03-10T17:55:18.295Zgo/build/constraint
10482026-03-10T17:55:18.296Zinternal/bisect
10492026-03-10T17:55:18.296Zencoding/binary
10502026-03-10T17:55:18.296Zgo/version
10512026-03-10T17:55:18.310Zinternal/types/errors
10522026-03-10T17:55:18.313Zcontext
10532026-03-10T17:55:18.313Zinternal/godebug
10542026-03-10T17:55:18.313Zinternal/buildcfg
10552026-03-10T17:55:18.316Zgo/token
10562026-03-10T17:55:18.316Zgo/doc/comment
10572026-03-10T17:55:18.320Zinternal/lazyregexp
10582026-03-10T17:55:18.329Zgolang.org/x/tools/internal/stdlib
10592026-03-10T17:55:18.329Zhash
10602026-03-10T17:55:18.332Zcrypto
10612026-03-10T17:55:18.339Zgolang.org/x/sync/errgroup
10622026-03-10T17:55:18.345Zgolang.org/x/tools/internal/event/label
10632026-03-10T17:55:18.350Zencoding/base64
10642026-03-10T17:55:18.352Zmath/rand
10652026-03-10T17:55:18.355Zos/exec
10662026-03-10T17:55:18.370Zgo/scanner
10672026-03-10T17:55:18.377Zencoding/json
10682026-03-10T17:55:18.380Zcrypto/md5
10692026-03-10T17:55:18.380Zmath/big
10702026-03-10T17:55:18.383Zgolang.org/x/tools/internal/event/keys
10712026-03-10T17:55:18.424Zgo/ast
10722026-03-10T17:55:18.427Zinternal/goroot
10732026-03-10T17:55:18.431Zlog
10742026-03-10T17:55:18.431Zgolang.org/x/tools/internal/event/core
10752026-03-10T17:55:18.461Zgolang.org/x/tools/internal/event
10762026-03-10T17:55:18.516Zgo/internal/typeparams
10772026-03-10T17:55:18.520Zgo/doc
10782026-03-10T17:55:18.523Zgolang.org/x/tools/internal/gocommand
10792026-03-10T17:55:18.523Zgo/parser
10802026-03-10T17:55:18.628Zgo/constant
10812026-03-10T17:55:18.652Zgo/build
10822026-03-10T17:55:18.667Zgolang.org/x/tools/internal/pkgbits
10832026-03-10T17:55:18.670Zgo/types
10842026-03-10T17:55:19.143Zgolang.org/x/tools/internal/aliases
10852026-03-10T17:55:19.143Zgolang.org/x/tools/internal/versions
10862026-03-10T17:55:19.154Zgolang.org/x/tools/internal/typeparams
10872026-03-10T17:55:19.181Zgolang.org/x/tools/go/types/typeutil
10882026-03-10T17:55:19.204Zgolang.org/x/tools/internal/typesinternal
10892026-03-10T17:55:19.233Zgolang.org/x/tools/go/types/objectpath
10902026-03-10T17:55:19.257Zgolang.org/x/tools/internal/gcimporter
10912026-03-10T17:55:19.362Zgolang.org/x/tools/go/gcexportdata
10922026-03-10T17:55:19.376Zgolang.org/x/tools/go/packages
10932026-03-10T17:55:19.443Zgithub.com/cockroachdb/cockroach/pkg/cmd/prereqs
10942026-03-10T17:55:19.561Zgo install -v uptodate
10952026-03-10T17:55:19.565ZGOFLAGS= bin/prereqs ./pkg/cmd/uptodate > bin/uptodate.d.tmp
10962026-03-10T17:55:19.565Zgo install -v protoc-gen-gogoroach
10972026-03-10T17:55:19.565ZGOFLAGS= bin/prereqs ./pkg/cmd/protoc-gen-gogoroach > bin/protoc-gen-gogoroach.d.tmp
10982026-03-10T17:55:19.565Zgo install -v terraformgen
10992026-03-10T17:55:19.565ZGOFLAGS= bin/prereqs ./pkg/roachprod/vm/aws/terraformgen > bin/terraformgen.d.tmp
11002026-03-10T17:55:19.569Zgo install -v optgen
11012026-03-10T17:55:19.569ZGOFLAGS= bin/prereqs ./pkg/sql/opt/optgen/cmd/optgen > bin/optgen.d.tmp
11022026-03-10T17:55:19.583Zmv -f bin/uptodate.d.tmp bin/uptodate.d
11032026-03-10T17:55:19.586Zmv -f bin/optgen.d.tmp bin/optgen.d
11042026-03-10T17:55:19.586ZGOBIN='/work/oxidecomputer/cockroach/bin' GOFLAGS= go install -mod=vendor -tags 'make x86_64_linux_gnu crdb_test' -ldflags '-X github.com/cockroachdb/cockroach/pkg/build.typ=development -extldflags "" -X "github.com/cockroachdb/cockroach/pkg/build.tag=v22.1.22-53-g21bdff50eb" -X "github.com/cockroachdb/cockroach/pkg/build.rev=21bdff50eb9c7974e74b511a5ae08a75cbc1619c" -X "github.com/cockroachdb/cockroach/pkg/build.cgoTargetTriple=x86_64-linux-gnu" ' -v ./pkg/cmd/uptodate
11052026-03-10T17:55:19.586Zmv -f bin/protoc-gen-gogoroach.d.tmp bin/protoc-gen-gogoroach.d
11062026-03-10T17:55:19.586ZGOBIN='/work/oxidecomputer/cockroach/bin' GOFLAGS= go install -mod=vendor -tags 'make x86_64_linux_gnu crdb_test' -ldflags '-X github.com/cockroachdb/cockroach/pkg/build.typ=development -extldflags "" -X "github.com/cockroachdb/cockroach/pkg/build.tag=v22.1.22-53-g21bdff50eb" -X "github.com/cockroachdb/cockroach/pkg/build.rev=21bdff50eb9c7974e74b511a5ae08a75cbc1619c" -X "github.com/cockroachdb/cockroach/pkg/build.cgoTargetTriple=x86_64-linux-gnu" ' -v ./pkg/sql/opt/optgen/cmd/optgen
11072026-03-10T17:55:19.587ZGOBIN='/work/oxidecomputer/cockroach/bin' GOFLAGS= go install -mod=vendor -tags 'make x86_64_linux_gnu crdb_test' -ldflags '-X github.com/cockroachdb/cockroach/pkg/build.typ=development -extldflags "" -X "github.com/cockroachdb/cockroach/pkg/build.tag=v22.1.22-53-g21bdff50eb" -X "github.com/cockroachdb/cockroach/pkg/build.rev=21bdff50eb9c7974e74b511a5ae08a75cbc1619c" -X "github.com/cockroachdb/cockroach/pkg/build.cgoTargetTriple=x86_64-linux-gnu" ' -v ./pkg/cmd/protoc-gen-gogoroach
11082026-03-10T17:55:19.587Zmv -f bin/terraformgen.d.tmp bin/terraformgen.d
11092026-03-10T17:55:19.590ZGOBIN='/work/oxidecomputer/cockroach/bin' GOFLAGS= go install -mod=vendor -tags 'make x86_64_linux_gnu crdb_test' -ldflags '-X github.com/cockroachdb/cockroach/pkg/build.typ=development -extldflags "" -X "github.com/cockroachdb/cockroach/pkg/build.tag=v22.1.22-53-g21bdff50eb" -X "github.com/cockroachdb/cockroach/pkg/build.rev=21bdff50eb9c7974e74b511a5ae08a75cbc1619c" -X "github.com/cockroachdb/cockroach/pkg/build.cgoTargetTriple=x86_64-linux-gnu" ' -v ./pkg/roachprod/vm/aws/terraformgen
11102026-03-10T17:55:19.625Zcrypto/internal/alias
11112026-03-10T17:55:19.625Zcrypto/internal/boring/sig
11122026-03-10T17:55:19.625Zcrypto/subtle
11132026-03-10T17:55:19.630Zhash/crc32
11142026-03-10T17:55:19.634Ztext/tabwriter
11152026-03-10T17:55:19.639Zio/ioutil
11162026-03-10T17:55:19.642Zcompress/flate
11172026-03-10T17:55:19.642Zencoding/hex
11182026-03-10T17:55:19.646Zgithub.com/gogo/protobuf/proto
11192026-03-10T17:55:19.653Zcrypto/cipher
11202026-03-10T17:55:19.653Zgithub.com/gogo/protobuf/protoc-gen-gogo/generator/internal/remap
11212026-03-10T17:55:19.657Zgo/printer
11222026-03-10T17:55:19.690Zcrypto/internal/boring
11232026-03-10T17:55:19.694Zinternal/nettrace
11242026-03-10T17:55:19.694Zgithub.com/cockroachdb/errors/stdstrings
11252026-03-10T17:55:19.697Zcontainer/list
11262026-03-10T17:55:19.697Zgithub.com/cockroachdb/errors/stdstrings
11272026-03-10T17:55:19.697Zvendor/golang.org/x/crypto/cryptobyte/asn1
11282026-03-10T17:55:19.697Zinternal/nettrace
11292026-03-10T17:55:19.697Zcontainer/list
11302026-03-10T17:55:19.697Zvendor/golang.org/x/crypto/cryptobyte/asn1
11312026-03-10T17:55:19.697Zinternal/nettrace
11322026-03-10T17:55:19.698Zvendor/golang.org/x/crypto/internal/alias
11332026-03-10T17:55:19.701Zvendor/golang.org/x/crypto/internal/alias
11342026-03-10T17:55:19.704Zcrypto/sha256
11352026-03-10T17:55:19.707Zcompress/gzip
11362026-03-10T17:55:19.714Zvendor/golang.org/x/net/dns/dnsmessage
11372026-03-10T17:55:19.721Zgithub.com/cockroachdb/redact/internal/rfmt/fmtsort
11382026-03-10T17:55:19.721Zinternal/singleflight
11392026-03-10T17:55:19.721Zinternal/intern
11402026-03-10T17:55:19.721Zgithub.com/cockroachdb/redact/interfaces
11412026-03-10T17:55:19.721Zgithub.com/cockroachdb/redact/internal/fmtforward
11422026-03-10T17:55:19.730Zencoding/csv
11432026-03-10T17:55:19.737Zgithub.com/gogo/protobuf/sortkeys
11442026-03-10T17:55:19.737Zgithub.com/kr/text
11452026-03-10T17:55:19.744Zcrypto/internal/randutil
11462026-03-10T17:55:19.744Zgithub.com/cockroachdb/redact/internal/rfmt/fmtsort
11472026-03-10T17:55:19.744Zgithub.com/kr/text
11482026-03-10T17:55:19.744Zcrypto/internal/nistec/fiat
11492026-03-10T17:55:19.744Zembed
11502026-03-10T17:55:19.744Zcrypto/rc4
11512026-03-10T17:55:19.744Zgithub.com/gogo/protobuf/sortkeys
11522026-03-10T17:55:19.744Zgithub.com/rogpeppe/go-internal/fmtsort
11532026-03-10T17:55:19.744Zvendor/golang.org/x/net/dns/dnsmessage
11542026-03-10T17:55:19.744Zcrypto/internal/randutil
11552026-03-10T17:55:19.757Zgithub.com/gogo/protobuf/proto
11562026-03-10T17:55:19.761Zcrypto/aes
11572026-03-10T17:55:19.761Zruntime/cgo
11582026-03-10T17:55:19.761Znet/netip
11592026-03-10T17:55:19.761Zcrypto/des
11602026-03-10T17:55:19.761Znet/url
11612026-03-10T17:55:19.765Zcrypto/internal/edwards25519/field
11622026-03-10T17:55:19.772Ztext/template/parse
11632026-03-10T17:55:19.772Zgithub.com/cockroachdb/redact/interfaces
11642026-03-10T17:55:19.772Zcrypto/internal/nistec/fiat
11652026-03-10T17:55:19.772Zgithub.com/rogpeppe/go-internal/fmtsort
11662026-03-10T17:55:19.776Zgithub.com/pkg/errors
11672026-03-10T17:55:19.781Zgithub.com/MichaelTJones/walk
11682026-03-10T17:55:19.784Zgithub.com/gogo/protobuf/proto
11692026-03-10T17:55:19.788Zgithub.com/cockroachdb/redact/interfaces
11702026-03-10T17:55:19.792Zgithub.com/kr/pretty
11712026-03-10T17:55:19.795Zgithub.com/cockroachdb/logtags
11722026-03-10T17:55:19.803Zcrypto/rand
11732026-03-10T17:55:19.818Zgithub.com/cockroachdb/redact/internal/markers
11742026-03-10T17:55:19.821Zgithub.com/cockroachdb/redact/internal/markers
11752026-03-10T17:55:19.821Zgithub.com/cockroachdb/redact/internal/redact
11762026-03-10T17:55:19.840Zgithub.com/cockroachdb/redact/internal/redact
11772026-03-10T17:55:19.853Zgithub.com/cockroachdb/redact/internal/markers
11782026-03-10T17:55:19.857Zgithub.com/cockroachdb/redact/internal/redact
11792026-03-10T17:55:19.878Zgithub.com/cockroachdb/logtags
11802026-03-10T17:55:19.891Zgithub.com/cockroachdb/redact/internal/escape
11812026-03-10T17:55:19.894Zgithub.com/cockroachdb/redact/internal/escape
11822026-03-10T17:55:19.906Zgithub.com/kr/pretty
11832026-03-10T17:55:19.910Zgo/format
11842026-03-10T17:55:19.916Zgithub.com/cockroachdb/redact/internal/buffer
11852026-03-10T17:55:19.920Zcrypto/internal/edwards25519/field
11862026-03-10T17:55:19.920Zcrypto/internal/bigmod
11872026-03-10T17:55:19.923Zcrypto/aes
11882026-03-10T17:55:19.939Zcrypto/internal/boring/bbig
11892026-03-10T17:55:19.943Zcrypto/internal/bigmod
11902026-03-10T17:55:19.949Zcrypto/sha512
11912026-03-10T17:55:19.969Zencoding/asn1
11922026-03-10T17:55:19.978Zcrypto/hmac
11932026-03-10T17:55:19.983Zgithub.com/cockroachdb/redact/internal/rfmt
11942026-03-10T17:55:19.987Zcrypto/sha512
11952026-03-10T17:55:20.012Zgithub.com/cockroachdb/redact/internal/rfmt
11962026-03-10T17:55:20.016Zcrypto/sha1
11972026-03-10T17:55:20.032Zencoding/asn1
11982026-03-10T17:55:20.036Zcrypto/internal/edwards25519
11992026-03-10T17:55:20.043Zcrypto/rsa
12002026-03-10T17:55:20.078Zcrypto/internal/nistec
12012026-03-10T17:55:20.092Zcrypto/dsa
12022026-03-10T17:55:20.111Zcrypto/internal/nistec
12032026-03-10T17:55:20.146Zcrypto/ed25519
12042026-03-10T17:55:20.151Zencoding/pem
12052026-03-10T17:55:20.159Zcrypto/sha1
12062026-03-10T17:55:20.197Zvendor/golang.org/x/crypto/cryptobyte
12072026-03-10T17:55:20.197Zcrypto/dsa
12082026-03-10T17:55:20.197Zgithub.com/cockroachdb/redact/builder
12092026-03-10T17:55:20.197Ztext/template
12102026-03-10T17:55:20.198Zcrypto/x509/pkix
12112026-03-10T17:55:20.198Zgithub.com/cockroachdb/redact/builder
12122026-03-10T17:55:20.229Zvendor/golang.org/x/crypto/chacha20
12132026-03-10T17:55:20.232Zcrypto/ed25519
12142026-03-10T17:55:20.232Zgithub.com/cockroachdb/redact
12152026-03-10T17:55:20.262Zvendor/golang.org/x/crypto/chacha20
12162026-03-10T17:55:20.265Zgithub.com/cockroachdb/redact
12172026-03-10T17:55:20.283Zgithub.com/cockroachdb/redact
12182026-03-10T17:55:20.286Zvendor/golang.org/x/crypto/internal/poly1305
12192026-03-10T17:55:20.290Zvendor/golang.org/x/crypto/cryptobyte
12202026-03-10T17:55:20.293Zvendor/golang.org/x/sys/cpu
12212026-03-10T17:55:20.293Zvendor/golang.org/x/crypto/internal/poly1305
12222026-03-10T17:55:20.303Zvendor/golang.org/x/sys/cpu
12232026-03-10T17:55:20.306Zgo: downloading google.golang.org/protobuf v1.27.1
12242026-03-10T17:55:20.306Zgo: downloading go.uber.org/multierr v1.7.0
12252026-03-10T17:55:20.306Zgo: downloading golang.org/x/term v0.0.0-20210615171337-6886f2dfbf5b
12262026-03-10T17:55:20.306Zgo: downloading go.uber.org/zap v1.19.0
12272026-03-10T17:55:20.306Zgo: downloading github.com/pkg/profile v1.6.0
12282026-03-10T17:55:20.310Zcrypto/ecdh
12292026-03-10T17:55:20.310Zcrypto/elliptic
12302026-03-10T17:55:20.313Zvendor/golang.org/x/crypto/hkdf
12312026-03-10T17:55:20.316Zgithub.com/getsentry/sentry-go/internal/crypto/randutil
12322026-03-10T17:55:20.319Zgo: downloading github.com/twitchtv/twirp v8.1.0+incompatible
12332026-03-10T17:55:20.323Zgo: downloading google.golang.org/grpc v1.41.0-dev.0.20210907181116-2f3355d2244e
12342026-03-10T17:55:20.326Zvendor/golang.org/x/crypto/hkdf
12352026-03-10T17:55:20.348Zruntime/cgo
12362026-03-10T17:55:20.353Zgithub.com/cockroachdb/cockroach/pkg/cli/exit
12372026-03-10T17:55:20.364Zruntime/cgo
12382026-03-10T17:55:20.375Zgithub.com/getsentry/sentry-go/internal/crypto/randutil
12392026-03-10T17:55:20.387Zcrypto/ecdh
12402026-03-10T17:55:20.387Zvendor/golang.org/x/text/transform
12412026-03-10T17:55:20.412Zcrypto/elliptic
12422026-03-10T17:55:20.429Zvendor/golang.org/x/text/unicode/bidi
12432026-03-10T17:55:20.429Zvendor/golang.org/x/net/http2/hpack
12442026-03-10T17:55:20.433Zcrypto/ecdsa
12452026-03-10T17:55:20.440Zmime
12462026-03-10T17:55:20.506Zvendor/golang.org/x/text/transform
12472026-03-10T17:55:20.509Zvendor/golang.org/x/text/unicode/norm
12482026-03-10T17:55:20.513Zvendor/golang.org/x/text/unicode/bidi
12492026-03-10T17:55:20.532Zmime/quotedprintable
12502026-03-10T17:55:20.536Zmime
12512026-03-10T17:55:20.536Zvendor/golang.org/x/crypto/chacha20poly1305
12522026-03-10T17:55:20.543Zmime/quotedprintable
12532026-03-10T17:55:20.582Zcrypto/ecdsa
12542026-03-10T17:55:20.589Zvendor/golang.org/x/crypto/chacha20poly1305
12552026-03-10T17:55:20.594Znet/http/internal
12562026-03-10T17:55:20.617Zvendor/golang.org/x/text/unicode/norm
12572026-03-10T17:55:20.633Znet/http/internal
12582026-03-10T17:55:20.645Znet/http/internal/ascii
12592026-03-10T17:55:20.645Zvendor/golang.org/x/text/secure/bidirule
12602026-03-10T17:55:20.656Zgolang.org/x/sys/execabs
12612026-03-10T17:55:20.656Zgo: downloading github.com/jhump/protoreflect v1.9.1-0.20210817181203-db1a327a393e
12622026-03-10T17:55:20.667Zvendor/golang.org/x/text/secure/bidirule
12632026-03-10T17:55:20.667Znet/http/internal/ascii
12642026-03-10T17:55:20.684Zruntime/debug
12652026-03-10T17:55:20.694Zgolang.org/x/sys/execabs
12662026-03-10T17:55:20.702Zruntime/debug
12672026-03-10T17:55:20.705Zgo: downloading golang.org/x/sync v0.0.0-20210220032951-036812b2e83c
12682026-03-10T17:55:20.708Zgo: downloading github.com/bgentry/go-netrc v0.0.0-20140422174119-9fd32a8b3d3d
12692026-03-10T17:55:20.708Zgo: downloading github.com/gofrs/uuid v4.0.0+incompatible
12702026-03-10T17:55:20.708Zgo: downloading github.com/gofrs/flock v0.8.1
12712026-03-10T17:55:20.708Zgo: downloading github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e
12722026-03-10T17:55:20.746Zgo: downloading gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b
12732026-03-10T17:55:20.753Zgo: downloading go.uber.org/atomic v1.7.0
12742026-03-10T17:55:20.756Zgo: downloading golang.org/x/sys v0.0.0-20210908143011-c212e7322662
12752026-03-10T17:55:20.756Zgo: downloading golang.org/x/net v0.0.0-20210907225631-ff17edfbf26d
12762026-03-10T17:55:20.759Zgo: downloading github.com/klauspost/compress v1.13.5
12772026-03-10T17:55:20.762Zgo: downloading github.com/klauspost/pgzip v1.2.5
12782026-03-10T17:55:20.765Zgo: downloading google.golang.org/genproto v0.0.0-20210903162649-d08c68adba83
12792026-03-10T17:55:20.828Zgo: downloading github.com/golang/protobuf v1.5.2
12802026-03-10T17:55:20.930Zvendor/golang.org/x/net/idna
12812026-03-10T17:55:20.933Zvendor/golang.org/x/net/idna
12822026-03-10T17:55:20.960Zgithub.com/gogo/protobuf/protoc-gen-gogo/descriptor
12832026-03-10T17:55:21.059Zgithub.com/gogo/protobuf/types
12842026-03-10T17:55:21.062Zgithub.com/gogo/protobuf/protoc-gen-gogo/descriptor
12852026-03-10T17:55:21.115Zgithub.com/gogo/protobuf/types
12862026-03-10T17:55:21.118Zgithub.com/gogo/protobuf/protoc-gen-gogo/descriptor
12872026-03-10T17:55:21.153Zgo: downloading golang.org/x/text v0.3.7
12882026-03-10T17:55:21.158Zgithub.com/gogo/protobuf/gogoproto
12892026-03-10T17:55:21.158Zgithub.com/gogo/protobuf/protoc-gen-gogo/plugin
12902026-03-10T17:55:21.187Zgithub.com/gogo/protobuf/vanity
12912026-03-10T17:55:21.187Zgithub.com/gogo/protobuf/protoc-gen-gogo/generator
12922026-03-10T17:55:21.244Znet
12932026-03-10T17:55:21.274Znet
12942026-03-10T17:55:21.362Znet
12952026-03-10T17:55:21.469Zgithub.com/gogo/protobuf/plugin/testgen
12962026-03-10T17:55:21.473Zgithub.com/gogo/protobuf/plugin/defaultcheck
12972026-03-10T17:55:21.473Zgithub.com/gogo/protobuf/plugin/embedcheck
12982026-03-10T17:55:21.473Zgithub.com/gogo/protobuf/plugin/enumstringer
12992026-03-10T17:55:21.473Zgithub.com/gogo/protobuf/plugin/marshalto
13002026-03-10T17:55:21.473Zgithub.com/gogo/protobuf/plugin/oneofcheck
13012026-03-10T17:55:21.494Zgithub.com/gogo/protobuf/plugin/populate
13022026-03-10T17:55:21.497Zgithub.com/gogo/protobuf/plugin/unmarshal
13032026-03-10T17:55:21.500Zgithub.com/cockroachdb/errors/errorspb
13042026-03-10T17:55:21.532Zgithub.com/gogo/protobuf/protoc-gen-gogo/grpc
13052026-03-10T17:55:21.665Zgithub.com/cockroachdb/errors/errbase
13062026-03-10T17:55:21.727Zgithub.com/cockroachdb/errors/markers
13072026-03-10T17:55:21.727Zgithub.com/cockroachdb/errors/barriers
13082026-03-10T17:55:21.727Zgithub.com/cockroachdb/errors/contexttags
13092026-03-10T17:55:21.747Zgithub.com/cockroachdb/errors/secondary
13102026-03-10T17:55:21.747Zgithub.com/cockroachdb/errors/hintdetail
13112026-03-10T17:55:21.747Zgithub.com/cockroachdb/errors/safedetails
13122026-03-10T17:55:21.747Zgithub.com/cockroachdb/errors/telemetrykeys
13132026-03-10T17:55:21.755Zgithub.com/cockroachdb/errors/domains
13142026-03-10T17:55:21.758Zgithub.com/cockroachdb/errors/markers
13152026-03-10T17:55:21.761Zgithub.com/cockroachdb/errors/secondary
13162026-03-10T17:55:21.761Zgithub.com/cockroachdb/errors/domains
13172026-03-10T17:55:21.761Zgithub.com/cockroachdb/errors/assert
13182026-03-10T17:55:21.761Zgithub.com/cockroachdb/errors/issuelink
13192026-03-10T17:55:21.773Zgithub.com/gogo/protobuf/plugin/compare
13202026-03-10T17:55:21.774Zgithub.com/gogo/protobuf/plugin/description
13212026-03-10T17:55:21.774Zgithub.com/gogo/protobuf/plugin/equal
13222026-03-10T17:55:21.777Zgithub.com/gogo/protobuf/plugin/face
13232026-03-10T17:55:21.777Zgithub.com/gogo/protobuf/plugin/gostring
13242026-03-10T17:55:21.777Zgithub.com/gogo/protobuf/plugin/size
13252026-03-10T17:55:21.777Zgithub.com/gogo/protobuf/plugin/stringer
13262026-03-10T17:55:21.784Zgithub.com/gogo/protobuf/plugin/union
13272026-03-10T17:55:21.787Zgithub.com/cockroachdb/errors/issuelink
13282026-03-10T17:55:21.974Zgithub.com/gogo/protobuf/vanity/command
13292026-03-10T17:55:21.988Zgithub.com/cockroachdb/cockroach/pkg/cmd/protoc-gen-gogoroach
13302026-03-10T17:55:22.262Zgoogle.golang.org/protobuf/internal/set
13312026-03-10T17:55:22.262Zgithub.com/golang/groupcache/lru
13322026-03-10T17:55:22.262Zgoogle.golang.org/protobuf/internal/flags
13332026-03-10T17:55:22.265Zgo.opencensus.io
13342026-03-10T17:55:22.265Zgo.opencensus.io/trace/internal
13352026-03-10T17:55:22.283Zgithub.com/twitchtv/twirp/internal/contextkeys
13362026-03-10T17:55:22.283Zgoogle.golang.org/grpc/serviceconfig
13372026-03-10T17:55:22.286Zgithub.com/klauspost/compress
13382026-03-10T17:55:22.313Zhash/fnv
13392026-03-10T17:55:22.313Zgoogle.golang.org/protobuf/internal/pragma
13402026-03-10T17:55:22.313Zgo.uber.org/zap/buffer
13412026-03-10T17:55:22.313Zgithub.com/bufbuild/buf/private/usage
13422026-03-10T17:55:22.316Zos/signal
13432026-03-10T17:55:22.316Zgo.uber.org/atomic
13442026-03-10T17:55:22.341Zgopkg.in/yaml.v3
13452026-03-10T17:55:22.345Zgoogle.golang.org/protobuf/internal/version
13462026-03-10T17:55:22.348Zgo.opencensus.io/internal
13472026-03-10T17:55:22.351Zgo.uber.org/multierr
13482026-03-10T17:55:22.356Zgithub.com/bufbuild/buf/private/bufpkg/bufanalysis
13492026-03-10T17:55:22.376Zgithub.com/bufbuild/buf/private/pkg/stringutil
13502026-03-10T17:55:22.379Zgithub.com/bufbuild/buf/private/pkg/interrupt
13512026-03-10T17:55:22.393Zgithub.com/bufbuild/buf/private/pkg/protoversion
13522026-03-10T17:55:22.406Zgoogle.golang.org/protobuf/internal/detrand
13532026-03-10T17:55:22.406Znet
13542026-03-10T17:55:22.409Zgo.opencensus.io/trace/tracestate
13552026-03-10T17:55:22.409Zgithub.com/bufbuild/buf/private/pkg/ioextended
13562026-03-10T17:55:22.417Zgithub.com/bufbuild/buf/private/buf/bufcheck
13572026-03-10T17:55:22.420Zgoogle.golang.org/protobuf/internal/errors
13582026-03-10T17:55:22.425Zgithub.com/bufbuild/buf/private/pkg/normalpath
13592026-03-10T17:55:22.425Zgithub.com/bufbuild/buf/private/pkg/thread
13602026-03-10T17:55:22.429Zruntime/trace
13612026-03-10T17:55:22.435Zgo.uber.org/zap/internal/bufferpool
13622026-03-10T17:55:22.435Zgoogle.golang.org/protobuf/encoding/protowire
13632026-03-10T17:55:22.435Zgithub.com/bufbuild/buf/private/pkg/app
13642026-03-10T17:55:22.442Zgo.uber.org/zap/internal/color
13652026-03-10T17:55:22.463Zgoogle.golang.org/protobuf/reflect/protoreflect
13662026-03-10T17:55:22.463Zgo.opencensus.io/trace
13672026-03-10T17:55:22.491Zgo.uber.org/zap/internal/exit
13682026-03-10T17:55:22.501Zgithub.com/bufbuild/buf/private/pkg/diff
13692026-03-10T17:55:22.501Zgithub.com/bufbuild/buf/private/pkg/storage/storageutil
13702026-03-10T17:55:22.501Zdatabase/sql/driver
13712026-03-10T17:55:22.509Zgithub.com/spf13/pflag
13722026-03-10T17:55:22.513Zgo.uber.org/zap/zapcore
13732026-03-10T17:55:22.534Zgithub.com/bufbuild/buf/private/bufpkg/bufmodule/internal
13742026-03-10T17:55:22.537Zgithub.com/bufbuild/buf/private/pkg/storage
13752026-03-10T17:55:22.547Znet/textproto
13762026-03-10T17:55:22.547Zvendor/golang.org/x/net/http/httpproxy
13772026-03-10T17:55:22.550Zcrypto/x509
13782026-03-10T17:55:22.561Zgithub.com/bufbuild/buf/private/pkg/osextended
13792026-03-10T17:55:22.565Zgoogle.golang.org/protobuf/internal/encoding/messageset
13802026-03-10T17:55:22.577Zgoogle.golang.org/protobuf/internal/genid
13812026-03-10T17:55:22.580Zgoogle.golang.org/protobuf/internal/order
13822026-03-10T17:55:22.590Znet/textproto
13832026-03-10T17:55:22.594Zvendor/golang.org/x/net/http/httpproxy
13842026-03-10T17:55:22.597Zgoogle.golang.org/protobuf/internal/strs
13852026-03-10T17:55:22.600Zgoogle.golang.org/protobuf/reflect/protoregistry
13862026-03-10T17:55:22.618Zgoogle.golang.org/protobuf/runtime/protoiface
13872026-03-10T17:55:22.622Zgoogle.golang.org/protobuf/internal/descfmt
13882026-03-10T17:55:22.630Zgithub.com/spf13/pflag
13892026-03-10T17:55:22.633Zcrypto/x509
13902026-03-10T17:55:22.672Zgoogle.golang.org/protobuf/internal/descopts
13912026-03-10T17:55:22.685Zgithub.com/bufbuild/buf/private/pkg/storage/storagemem
13922026-03-10T17:55:22.691Zgithub.com/bufbuild/buf/private/pkg/filepathextended
13932026-03-10T17:55:22.698Zvendor/golang.org/x/net/http/httpguts
13942026-03-10T17:55:22.710Zgithub.com/bufbuild/buf/private/pkg/storage/storageos
13952026-03-10T17:55:22.713Zgoogle.golang.org/protobuf/internal/encoding/text
13962026-03-10T17:55:22.727Zvendor/golang.org/x/net/http/httpguts
13972026-03-10T17:55:22.730Zmime/multipart
13982026-03-10T17:55:22.733Zgithub.com/bgentry/go-netrc/netrc
13992026-03-10T17:55:22.736Zgithub.com/bufbuild/buf/private/gen/data/datawkt
14002026-03-10T17:55:22.742Zmime/multipart
14012026-03-10T17:55:22.749Zos/user
14022026-03-10T17:55:22.752Zgoogle.golang.org/protobuf/proto
14032026-03-10T17:55:22.781Zgithub.com/klauspost/compress/flate
14042026-03-10T17:55:22.789Zgithub.com/bufbuild/buf/private/pkg/encoding
14052026-03-10T17:55:22.825Zgithub.com/bufbuild/buf/private/pkg/netrc
14062026-03-10T17:55:22.825Zgithub.com/bufbuild/buf/private/bufpkg/buflock
14072026-03-10T17:55:22.829Zgoogle.golang.org/protobuf/internal/encoding/defval
14082026-03-10T17:55:22.863Zgithub.com/klauspost/compress/fse
14092026-03-10T17:55:22.881Zgoogle.golang.org/protobuf/encoding/prototext
14102026-03-10T17:55:22.894Zgoogle.golang.org/protobuf/internal/filedesc
14112026-03-10T17:55:22.910Zgithub.com/klauspost/compress/internal/snapref
14122026-03-10T17:55:22.920Zcrypto/tls
14132026-03-10T17:55:22.923Zgithub.com/spf13/cobra
14142026-03-10T17:55:22.933Zgithub.com/klauspost/compress/huff0
14152026-03-10T17:55:23.006Zgithub.com/klauspost/compress/zstd/internal/xxhash
14162026-03-10T17:55:23.021Zgoogle.golang.org/protobuf/internal/encoding/json
14172026-03-10T17:55:23.082Zgithub.com/jhump/protoreflect/desc/protoparse/ast
14182026-03-10T17:55:23.089Ztext/scanner
14192026-03-10T17:55:23.118Zgithub.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/terraformgen
14202026-03-10T17:55:23.121Zgolang.org/x/net/internal/timeseries
14212026-03-10T17:55:23.134Zgithub.com/klauspost/compress/zstd
14222026-03-10T17:55:23.152Zcrypto/tls
14232026-03-10T17:55:23.159Zhtml
14242026-03-10T17:55:23.163Zgithub.com/klauspost/compress/zip
14252026-03-10T17:55:23.179Zgoogle.golang.org/protobuf/internal/encoding/tag
14262026-03-10T17:55:23.198Zgithub.com/klauspost/pgzip
14272026-03-10T17:55:23.216Zgoogle.golang.org/protobuf/encoding/protojson
14282026-03-10T17:55:23.226Zgoogle.golang.org/protobuf/internal/impl
14292026-03-10T17:55:23.268Zhtml/template
14302026-03-10T17:55:23.300Zgoogle.golang.org/grpc/backoff
14312026-03-10T17:55:23.329Zgoogle.golang.org/grpc/internal/grpclog
14322026-03-10T17:55:23.375Zgoogle.golang.org/grpc/attributes
14332026-03-10T17:55:23.378Zgoogle.golang.org/grpc/grpclog
14342026-03-10T17:55:23.381Zgoogle.golang.org/grpc/metadata
14352026-03-10T17:55:23.394Zgoogle.golang.org/grpc/internal/grpcrand
14362026-03-10T17:55:23.434Zgoogle.golang.org/grpc/codes
14372026-03-10T17:55:23.454Zgoogle.golang.org/grpc/encoding
14382026-03-10T17:55:23.486Zgoogle.golang.org/grpc/internal/backoff
14392026-03-10T17:55:23.495Zgoogle.golang.org/grpc/internal/balancerload
14402026-03-10T17:55:23.501Zgoogle.golang.org/grpc/internal/buffer
14412026-03-10T17:55:23.501Zgoogle.golang.org/grpc/connectivity
14422026-03-10T17:55:23.504Zgolang.org/x/sys/internal/unsafeheader
14432026-03-10T17:55:23.504Zgoogle.golang.org/grpc/internal/envconfig
14442026-03-10T17:55:23.504Zgoogle.golang.org/grpc/internal/grpcsync
14452026-03-10T17:55:23.523Zgolang.org/x/sys/unix
14462026-03-10T17:55:23.526Zgolang.org/x/text/transform
14472026-03-10T17:55:23.533Zgolang.org/x/text/unicode/bidi
14482026-03-10T17:55:23.533Zgolang.org/x/net/http2/hpack
14492026-03-10T17:55:23.577Zgoogle.golang.org/grpc/keepalive
14502026-03-10T17:55:23.581Zgolang.org/x/text/unicode/norm
14512026-03-10T17:55:23.586Zgoogle.golang.org/grpc/tap
14522026-03-10T17:55:23.604Zgithub.com/bufbuild/buf/private/pkg/rpc
14532026-03-10T17:55:23.640Zgolang.org/x/text/secure/bidirule
14542026-03-10T17:55:23.643Zgithub.com/bufbuild/buf/private/pkg/rpc/rpcheader
14552026-03-10T17:55:23.643Zarchive/tar
14562026-03-10T17:55:23.674Zgo.opencensus.io/resource
14572026-03-10T17:55:23.674Znet/http/httptrace
14582026-03-10T17:55:23.678Zruntime/pprof
14592026-03-10T17:55:23.692Zgo.opencensus.io/internal/tagencoding
14602026-03-10T17:55:23.697Zgo.opencensus.io/metric/metricdata
14612026-03-10T17:55:23.714Znet/http
14622026-03-10T17:55:23.772Zgo.opencensus.io/metric/metricproducer
14632026-03-10T17:55:23.775Zgoogle.golang.org/grpc/encoding/gzip
14642026-03-10T17:55:23.805Zgolang.org/x/net/idna
14652026-03-10T17:55:23.805Zgithub.com/gofrs/flock
14662026-03-10T17:55:23.809Zgithub.com/bufbuild/buf/private/pkg/verbose
14672026-03-10T17:55:23.831Zgithub.com/bufbuild/buf/private/pkg/storage/storagearchive
14682026-03-10T17:55:23.852Znet/http
14692026-03-10T17:55:23.856Zgithub.com/bufbuild/buf/private/pkg/filelock
14702026-03-10T17:55:23.856Zgo.opencensus.io/tag
14712026-03-10T17:55:23.865Zgithub.com/bufbuild/buf/private/bufpkg/bufrpc
14722026-03-10T17:55:23.916Zgithub.com/bufbuild/buf/private/bufpkg/buftransport
14732026-03-10T17:55:23.921Zgithub.com/bufbuild/buf/private/pkg/app/appverbose
14742026-03-10T17:55:23.927Zgithub.com/pkg/profile
14752026-03-10T17:55:23.938Zgo.opencensus.io/stats/internal
14762026-03-10T17:55:23.941Zgolang.org/x/sync/errgroup
14772026-03-10T17:55:23.952Zgolang.org/x/term
14782026-03-10T17:55:23.961Zgo.opencensus.io/stats
14792026-03-10T17:55:23.984Zgo.opencensus.io/stats/view
14802026-03-10T17:55:24.112Zgoogle.golang.org/protobuf/internal/filetype
14812026-03-10T17:55:24.145Zgoogle.golang.org/protobuf/runtime/protoimpl
14822026-03-10T17:55:24.157Zgithub.com/bufbuild/buf/private/pkg/app/appname
14832026-03-10T17:55:24.157Zgoogle.golang.org/grpc/internal/credentials
14842026-03-10T17:55:24.157Zgolang.org/x/net/http/httpguts
14852026-03-10T17:55:24.157Zgoogle.golang.org/grpc/internal/syscall
14862026-03-10T17:55:24.157Znet/http
14872026-03-10T17:55:24.157Zgithub.com/gofrs/uuid
14882026-03-10T17:55:24.157Zgithub.com/bufbuild/buf/private/pkg/netextended
14892026-03-10T17:55:24.171Zgoogle.golang.org/grpc/stats
14902026-03-10T17:55:24.171Zgithub.com/spf13/pflag
14912026-03-10T17:55:24.176Zgithub.com/bufbuild/buf/private/pkg/cert/certclient
14922026-03-10T17:55:24.179Zgoogle.golang.org/protobuf/types/descriptorpb
14932026-03-10T17:55:24.179Zgoogle.golang.org/protobuf/types/known/timestamppb
14942026-03-10T17:55:24.185Zgoogle.golang.org/protobuf/types/known/anypb
14952026-03-10T17:55:24.194Zgoogle.golang.org/protobuf/types/known/durationpb
14962026-03-10T17:55:24.198Zgithub.com/bufbuild/buf/private/pkg/uuidutil
14972026-03-10T17:55:24.226Zgithub.com/bufbuild/buf/private/gen/proto/go/buf/alpha/module/v1alpha1
14982026-03-10T17:55:24.239Zgithub.com/bufbuild/buf/private/pkg/prototime
14992026-03-10T17:55:24.250Zgithub.com/golang/protobuf/ptypes/duration
15002026-03-10T17:55:24.255Zgithub.com/bufbuild/buf/private/buf/bufapp
15012026-03-10T17:55:24.260Zgithub.com/golang/protobuf/ptypes/any
15022026-03-10T17:55:24.266Zgoogle.golang.org/protobuf/types/known/emptypb
15032026-03-10T17:55:24.269Zgithub.com/bufbuild/buf/private/pkg/tmp
15042026-03-10T17:55:24.278Zgoogle.golang.org/protobuf/types/known/structpb
15052026-03-10T17:55:24.285Zgithub.com/golang/protobuf/ptypes/timestamp
15062026-03-10T17:55:24.288Zgoogle.golang.org/protobuf/types/known/sourcecontextpb
15072026-03-10T17:55:24.288Zgoogle.golang.org/protobuf/types/known/wrapperspb
15082026-03-10T17:55:24.291Zgoogle.golang.org/protobuf/types/pluginpb
15092026-03-10T17:55:24.310Zgithub.com/bufbuild/buf/private/gen/proto/go/buf/alpha/image/v1
15102026-03-10T17:55:24.322Zgoogle.golang.org/protobuf/reflect/protodesc
15112026-03-10T17:55:24.341Zgithub.com/golang/protobuf/protoc-gen-go/descriptor
15122026-03-10T17:55:24.344Zgithub.com/golang/protobuf/ptypes/empty
15132026-03-10T17:55:24.370Zgithub.com/bufbuild/buf/private/pkg/protodescriptor
15142026-03-10T17:55:24.374Zgithub.com/golang/protobuf/protoc-gen-go/plugin
15152026-03-10T17:55:24.374Zgithub.com/golang/protobuf/ptypes/struct
15162026-03-10T17:55:24.393Zgithub.com/golang/protobuf/ptypes/wrappers
15172026-03-10T17:55:24.406Zgoogle.golang.org/protobuf/types/known/typepb
15182026-03-10T17:55:24.414Zgithub.com/bufbuild/buf/private/pkg/protosource
15192026-03-10T17:55:24.417Zgithub.com/bufbuild/buf/private/bufpkg/bufmodule
15202026-03-10T17:55:24.421Zgoogle.golang.org/protobuf/types/known/fieldmaskpb
15212026-03-10T17:55:24.447Zgoogle.golang.org/protobuf/types/known/apipb
15222026-03-10T17:55:24.470Zgoogle.golang.org/genproto/protobuf/api
15232026-03-10T17:55:24.480Zgoogle.golang.org/genproto/protobuf/ptype
15242026-03-10T17:55:24.484Zgoogle.golang.org/genproto/protobuf/source_context
15252026-03-10T17:55:24.486Zgoogle.golang.org/genproto/protobuf/field_mask
15262026-03-10T17:55:24.489Zgithub.com/golang/protobuf/proto
15272026-03-10T17:55:24.507Zgoogle.golang.org/protobuf/types/dynamicpb
15282026-03-10T17:55:24.517Zgoogle.golang.org/genproto/googleapis/rpc/status
15292026-03-10T17:55:24.549Zgithub.com/bufbuild/buf/private/bufpkg/bufimage
15302026-03-10T17:55:24.563Zgithub.com/bufbuild/buf/private/bufpkg/bufmodule/bufmoduleconfig
15312026-03-10T17:55:24.571Zgithub.com/spf13/cobra
15322026-03-10T17:55:24.660Zgithub.com/bufbuild/buf/private/pkg/protoencoding
15332026-03-10T17:55:24.671Zgithub.com/jhump/protoreflect/internal
15342026-03-10T17:55:24.674Zgithub.com/jhump/protoreflect/internal/codec
15352026-03-10T17:55:24.674Zgithub.com/golang/protobuf/jsonpb
15362026-03-10T17:55:24.674Zgoogle.golang.org/grpc/credentials
15372026-03-10T17:55:24.698Zgoogle.golang.org/grpc/encoding/proto
15382026-03-10T17:55:24.722Zgithub.com/golang/protobuf/ptypes
15392026-03-10T17:55:24.741Zgithub.com/bufbuild/buf/private/bufpkg/bufimage/bufimageutil
15402026-03-10T17:55:24.758Zgithub.com/jhump/protoreflect/desc/internal
15412026-03-10T17:55:24.786Zgoogle.golang.org/grpc/binarylog/grpc_binarylog_v1
15422026-03-10T17:55:24.794Zgoogle.golang.org/grpc/resolver
15432026-03-10T17:55:24.798Zgoogle.golang.org/grpc/internal/channelz
15442026-03-10T17:55:24.815Zgoogle.golang.org/grpc/peer
15452026-03-10T17:55:24.818Zgoogle.golang.org/grpc/internal/status
15462026-03-10T17:55:24.837Zgithub.com/bufbuild/buf/private/pkg/app/appcmd
15472026-03-10T17:55:24.837Zgoogle.golang.org/grpc/internal
15482026-03-10T17:55:24.853Zgoogle.golang.org/grpc/internal/grpcutil
15492026-03-10T17:55:24.857Zgithub.com/jhump/protoreflect/desc
15502026-03-10T17:55:24.860Zgoogle.golang.org/grpc/balancer/grpclb/state
15512026-03-10T17:55:24.872Zgoogle.golang.org/grpc/balancer
15522026-03-10T17:55:24.895Zgoogle.golang.org/grpc/status
15532026-03-10T17:55:24.895Zgoogle.golang.org/grpc/internal/resolver/passthrough
15542026-03-10T17:55:24.901Zgoogle.golang.org/grpc/internal/transport/networktype
15552026-03-10T17:55:24.913Zgoogle.golang.org/grpc/internal/resolver/dns
15562026-03-10T17:55:24.921Zgoogle.golang.org/grpc/balancer/base
15572026-03-10T17:55:24.938Zgoogle.golang.org/grpc/internal/serviceconfig
15582026-03-10T17:55:24.942Zgoogle.golang.org/grpc/internal/binarylog
15592026-03-10T17:55:24.958Zgoogle.golang.org/grpc/internal/resolver
15602026-03-10T17:55:24.964Zgoogle.golang.org/grpc/internal/resolver/unix
15612026-03-10T17:55:24.969Zgoogle.golang.org/grpc/balancer/roundrobin
15622026-03-10T17:55:24.972Zgoogle.golang.org/grpc/internal/metadata
15632026-03-10T17:55:25.009Zgithub.com/jhump/protoreflect/codec
15642026-03-10T17:55:25.020Znet/http/httputil
15652026-03-10T17:55:25.023Zgithub.com/getsentry/sentry-go/internal/ratelimit
15662026-03-10T17:55:25.047Zgithub.com/jhump/protoreflect/dynamic
15672026-03-10T17:55:25.059Znet/http/httputil
15682026-03-10T17:55:25.124Zgithub.com/getsentry/sentry-go/internal/debug
15692026-03-10T17:55:25.131Zgithub.com/getsentry/sentry-go/internal/debug
15702026-03-10T17:55:25.136Zgithub.com/getsentry/sentry-go
15712026-03-10T17:55:25.148Zgithub.com/getsentry/sentry-go
15722026-03-10T17:55:25.247Zgithub.com/twitchtv/twirp
15732026-03-10T17:55:25.256Zgithub.com/twitchtv/twirp/ctxsetters
15742026-03-10T17:55:25.256Zgolang.org/x/net/trace
15752026-03-10T17:55:25.256Zgo.opencensus.io/trace/propagation
15762026-03-10T17:55:25.256Zgolang.org/x/net/http2
15772026-03-10T17:55:25.256Zgithub.com/bufbuild/buf/private/pkg/httpauth
15782026-03-10T17:55:25.256Zgo.uber.org/zap
15792026-03-10T17:55:25.269Zgithub.com/bufbuild/buf/private/pkg/rpc/rpchttp
15802026-03-10T17:55:25.279Zgithub.com/bufbuild/buf/private/pkg/rpc/rpcauth
15812026-03-10T17:55:25.288Zgo.opencensus.io/plugin/ocgrpc
15822026-03-10T17:55:25.288Zgo.opencensus.io/plugin/ochttp/propagation/b3
15832026-03-10T17:55:25.304Zgithub.com/jhump/protoreflect/desc/protoparse
15842026-03-10T17:55:25.304Zgithub.com/cockroachdb/errors/withstack
15852026-03-10T17:55:25.307Zgithub.com/bufbuild/buf/private/pkg/rpc/rpctwirp
15862026-03-10T17:55:25.317Zgo.opencensus.io/plugin/ochttp
15872026-03-10T17:55:25.320Zgithub.com/cockroachdb/errors/withstack
15882026-03-10T17:55:25.342Zgithub.com/cockroachdb/errors/report
15892026-03-10T17:55:25.342Zgithub.com/cockroachdb/errors/errutil
15902026-03-10T17:55:25.370Zgithub.com/bufbuild/buf/private/pkg/transport/twirp/twirpclient
15912026-03-10T17:55:25.379Zgithub.com/cockroachdb/errors/report
15922026-03-10T17:55:25.390Zgithub.com/cockroachdb/errors
15932026-03-10T17:55:25.418Zgithub.com/cockroachdb/errors
15942026-03-10T17:55:25.422Zgithub.com/bufbuild/buf/private/buf/bufcheck/internal
15952026-03-10T17:55:25.422Zgithub.com/bufbuild/buf/private/bufpkg/bufmodule/bufmodulebuild
15962026-03-10T17:55:25.422Zgithub.com/bufbuild/buf/private/pkg/zaputil
15972026-03-10T17:55:25.422Zgithub.com/bufbuild/buf/private/bufpkg/bufmodule/bufmodulecache
15982026-03-10T17:55:25.427Zgithub.com/bufbuild/buf/private/pkg/git
15992026-03-10T17:55:25.427Zgithub.com/cockroachdb/errors/oserror
16002026-03-10T17:55:25.447Zgithub.com/bufbuild/buf/private/pkg/app/applog
16012026-03-10T17:55:25.450Zgithub.com/cockroachdb/cockroach/pkg/sql/opt/optgen/lang
16022026-03-10T17:55:25.450Zgithub.com/bufbuild/buf/private/pkg/app/appproto
16032026-03-10T17:55:25.461Zgithub.com/bufbuild/buf/private/buf/buffetch/internal
16042026-03-10T17:55:25.461Zgithub.com/bufbuild/buf/private/bufpkg/bufimage/bufimagemodify
16052026-03-10T17:55:25.540Zgithub.com/bufbuild/buf/private/pkg/observability
16062026-03-10T17:55:25.545Zgithub.com/cockroachdb/cockroach/pkg/cmd/uptodate
16072026-03-10T17:55:25.550Zgithub.com/bufbuild/buf/private/buf/bufcheck/buflint/internal/buflintcheck
16082026-03-10T17:55:25.554Zgithub.com/bufbuild/buf/private/buf/bufcheck/bufbreaking/internal/bufbreakingcheck
16092026-03-10T17:55:25.571Zgithub.com/cockroachdb/cockroach/pkg/sql/opt/optgen/cmd/optgen
16102026-03-10T17:55:25.588Zgithub.com/bufbuild/buf/private/pkg/app/appproto/appprotoexec
16112026-03-10T17:55:25.609Zgithub.com/bufbuild/buf/private/pkg/transport/http/httpclient
16122026-03-10T17:55:25.614Zgithub.com/bufbuild/buf/private/pkg/observability/observabilityzap
16132026-03-10T17:55:25.667Zgithub.com/bufbuild/buf/private/pkg/app/appflag
16142026-03-10T17:55:25.683Zgithub.com/bufbuild/buf/private/pkg/app/appproto/appprotoos
16152026-03-10T17:55:25.716Zgithub.com/bufbuild/buf/private/buf/bufcheck/buflint/internal/buflintbuild
16162026-03-10T17:55:25.748Zgithub.com/bufbuild/buf/private/buf/bufcheck/buflint/internal/buflintv1
16172026-03-10T17:55:25.751Zgithub.com/bufbuild/buf/private/buf/bufcheck/buflint/internal/buflintv1beta1
16182026-03-10T17:55:25.751Zgoogle.golang.org/grpc/internal/transport
16192026-03-10T17:55:25.754Zgithub.com/bufbuild/buf/private/buf/bufcheck/bufbreaking/internal/bufbreakingbuild
16202026-03-10T17:55:25.778Zgithub.com/bufbuild/buf/private/buf/bufcheck/buflint/buflintconfig
16212026-03-10T17:55:25.794Zgithub.com/bufbuild/buf/private/buf/bufcheck/buflint
16222026-03-10T17:55:25.798Zgithub.com/bufbuild/buf/private/buf/bufcheck/bufbreaking/internal/bufbreakingv1
16232026-03-10T17:55:25.798Zgithub.com/bufbuild/buf/private/buf/bufcheck/bufbreaking/internal/bufbreakingv1beta1
16242026-03-10T17:55:25.807Zgithub.com/bufbuild/buf/private/buf/bufcheck/bufbreaking/bufbreakingconfig
16252026-03-10T17:55:25.821Zgithub.com/bufbuild/buf/private/buf/bufconfig
16262026-03-10T17:55:25.821Zgithub.com/bufbuild/buf/private/buf/bufcheck/bufbreaking
16272026-03-10T17:55:25.836Zgithub.com/bufbuild/buf/private/bufpkg/bufmodule/bufmoduleprotoparse
16282026-03-10T17:55:25.847Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/config/internal
16292026-03-10T17:55:25.847Zgithub.com/bufbuild/buf/private/buf/bufwork
16302026-03-10T17:55:25.850Zgithub.com/bufbuild/buf/private/bufpkg/bufimage/bufimagebuild
16312026-03-10T17:55:25.857Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/config/configlsbreakingrules
16322026-03-10T17:55:25.857Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/config/configlslintrules
16332026-03-10T17:55:25.891Zgithub.com/bufbuild/buf/private/buf/buffetch
16342026-03-10T17:55:25.891Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/jemalloc.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/jemalloc.c
16352026-03-10T17:55:25.901Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/arena.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/arena.c
16362026-03-10T17:55:25.911Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/atomic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/atomic.c
16372026-03-10T17:55:25.915Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/base.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/base.c
16382026-03-10T17:55:25.979Zgithub.com/bufbuild/buf/private/buf/bufwire
16392026-03-10T17:55:25.984Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/bitmap.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/bitmap.c
16402026-03-10T17:55:26.019Z[ 1%] Building CXX object CMakeFiles/geos.dir/src/algorithm/Angle.cpp.o
16412026-03-10T17:55:26.039Z[ 1%] Building C object src/CMakeFiles/proj.dir/nad_init.c.o
16422026-03-10T17:55:26.044Zgoogle.golang.org/grpc
16432026-03-10T17:55:26.053Z[ 1%] Building C object src/CMakeFiles/proj.dir/PJ_aea.c.o
16442026-03-10T17:55:26.071Z[ 2%] Building C object src/CMakeFiles/proj.dir/PJ_aeqd.c.o
16452026-03-10T17:55:26.082Z[ 2%] Building C object src/CMakeFiles/proj.dir/PJ_airy.c.o
16462026-03-10T17:55:26.087Zoptgen -out pkg/sql/opt/memo/expr.og.go exprs pkg/sql/opt/ops/*.opt
16472026-03-10T17:55:26.114Z[ 3%] Building C object src/CMakeFiles/proj.dir/PJ_aitoff.c.o
16482026-03-10T17:55:26.118Z[ 4%] Building C object src/CMakeFiles/proj.dir/PJ_august.c.o
16492026-03-10T17:55:26.136Z[ 4%] Building C object src/CMakeFiles/proj.dir/PJ_bacon.c.o
16502026-03-10T17:55:26.150Zoptgen -out pkg/sql/opt/operator.og.go ops pkg/sql/opt/ops/*.opt
16512026-03-10T17:55:26.159Z[ 5%] Building C object src/CMakeFiles/proj.dir/PJ_bipc.c.o
16522026-03-10T17:55:26.177Z[ 5%] Building C object src/CMakeFiles/proj.dir/PJ_boggs.c.o
16532026-03-10T17:55:26.180Zoptgen -out pkg/sql/opt/xform/explorer.og.go explorer pkg/sql/opt/ops/*.opt pkg/sql/opt/xform/rules/*.opt
16542026-03-10T17:55:26.184Z[ 6%] Building C object src/CMakeFiles/proj.dir/PJ_bonne.c.o
16552026-03-10T17:55:26.201Z[ 1%] Building CXX object CMakeFiles/geos.dir/src/algorithm/Area.cpp.o
16562026-03-10T17:55:26.208Z[ 6%] Building C object src/CMakeFiles/proj.dir/PJ_calcofi.c.o
16572026-03-10T17:55:26.221Zoptgen -out pkg/sql/opt/norm/factory.og.go factory pkg/sql/opt/ops/*.opt pkg/sql/opt/norm/rules/*.opt
16582026-03-10T17:55:26.224Z[ 1%] Building CXX object CMakeFiles/geos.dir/src/algorithm/BoundaryNodeRule.cpp.o
16592026-03-10T17:55:26.260Zoptgen -out pkg/sql/opt/rule_name.og.go rulenames pkg/sql/opt/ops/*.opt pkg/sql/opt/norm/rules/*.opt pkg/sql/opt/xform/rules/*.opt
16602026-03-10T17:55:26.274Z[ 7%] Building C object src/CMakeFiles/proj.dir/PJ_cass.c.o
16612026-03-10T17:55:26.288Z[ 1%] Building CXX object CMakeFiles/geos.dir/src/algorithm/CGAlgorithmsDD.cpp.o
16622026-03-10T17:55:26.309Zoptgen -out pkg/sql/opt/exec/factory.og.go execfactory pkg/sql/opt/exec/factory.opt
16632026-03-10T17:55:26.319Zoptgen -out pkg/sql/opt/exec/explain/explain_factory.og.go execexplain pkg/sql/opt/exec/factory.opt
16642026-03-10T17:55:26.322Z[ 7%] Building C object src/CMakeFiles/proj.dir/PJ_cc.c.o
16652026-03-10T17:55:26.330Zoptgen -out pkg/sql/opt/exec/explain/plan_gist_factory.og.go execplangist pkg/sql/opt/exec/factory.opt
16662026-03-10T17:55:26.333Zgithub.com/bufbuild/buf/private/pkg/rpc/rpcgrpc
16672026-03-10T17:55:26.333Zgithub.com/bufbuild/buf/private/gen/proto/go/buf/alpha/registry/v1alpha1
16682026-03-10T17:55:26.353Z[ 1%] Building CXX object CMakeFiles/geos.dir/src/algorithm/Centroid.cpp.o
16692026-03-10T17:55:26.356Zgithub.com/bufbuild/buf/private/pkg/transport/grpc/grpcclient
16702026-03-10T17:55:26.360Z[ 3%] Building CXX object CMakeFiles/geos.dir/src/algorithm/ConvexHull.cpp.o
16712026-03-10T17:55:26.373Z[ 8%] Building C object src/CMakeFiles/proj.dir/PJ_cea.c.o
16722026-03-10T17:55:26.375Z[ 9%] Building C object src/CMakeFiles/proj.dir/PJ_chamb.c.o
16732026-03-10T17:55:26.424Z[ 9%] Building C object src/CMakeFiles/proj.dir/PJ_collg.c.o
16742026-03-10T17:55:26.449Z[ 10%] Building C object src/CMakeFiles/proj.dir/PJ_comill.c.o
16752026-03-10T17:55:26.476Z[ 11%] Building C object src/CMakeFiles/proj.dir/PJ_crast.c.o
16762026-03-10T17:55:26.476Z[ 11%] Building C object src/CMakeFiles/proj.dir/PJ_denoy.c.o
16772026-03-10T17:55:26.523Z[ 3%] Building CXX object CMakeFiles/geos.dir/src/algorithm/Distance.cpp.o
16782026-03-10T17:55:26.528Z[ 11%] Building C object src/CMakeFiles/proj.dir/PJ_eck1.c.o
16792026-03-10T17:55:26.531Z[ 13%] Building C object src/CMakeFiles/proj.dir/PJ_eck2.c.o
16802026-03-10T17:55:26.560Z[ 13%] Building C object src/CMakeFiles/proj.dir/PJ_eck3.c.o
16812026-03-10T17:55:26.601Z[ 3%] Building CXX object CMakeFiles/geos.dir/src/algorithm/HCoordinate.cpp.o
16822026-03-10T17:55:26.604Z[ 14%] Building C object src/CMakeFiles/proj.dir/PJ_eck4.c.o
16832026-03-10T17:55:26.635Z[ 14%] Building C object src/CMakeFiles/proj.dir/PJ_eck5.c.o
16842026-03-10T17:55:26.661Z[ 15%] Building C object src/CMakeFiles/proj.dir/PJ_eqc.c.o
16852026-03-10T17:55:26.670Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/chunk.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/chunk.c
16862026-03-10T17:55:26.691Z[ 16%] Building C object src/CMakeFiles/proj.dir/PJ_eqdc.c.o
16872026-03-10T17:55:26.732Z[ 3%] Building CXX object CMakeFiles/geos.dir/src/algorithm/InteriorPointArea.cpp.o
16882026-03-10T17:55:26.764Z[ 16%] Building C object src/CMakeFiles/proj.dir/PJ_fahey.c.o
16892026-03-10T17:55:26.796Z[ 17%] Building C object src/CMakeFiles/proj.dir/PJ_fouc_s.c.o
16902026-03-10T17:55:26.830Z[ 3%] Building CXX object CMakeFiles/geos.dir/src/algorithm/InteriorPointLine.cpp.o
16912026-03-10T17:55:26.833Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/chunk_dss.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/chunk_dss.c
16922026-03-10T17:55:26.834Z[ 17%] Building C object src/CMakeFiles/proj.dir/PJ_gall.c.o
16932026-03-10T17:55:26.857Z[ 18%] Building C object src/CMakeFiles/proj.dir/PJ_geos.c.o
16942026-03-10T17:55:26.903Z[ 4%] Building CXX object CMakeFiles/geos.dir/src/algorithm/InteriorPointPoint.cpp.o
16952026-03-10T17:55:26.906Z[ 18%] Building C object src/CMakeFiles/proj.dir/PJ_gins8.c.o
16962026-03-10T17:55:26.911Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/chunk_mmap.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/chunk_mmap.c
16972026-03-10T17:55:26.928Z[ 19%] Building C object src/CMakeFiles/proj.dir/PJ_gnom.c.o
16982026-03-10T17:55:26.979Z[ 19%] Building C object src/CMakeFiles/proj.dir/PJ_gn_sinu.c.o
16992026-03-10T17:55:26.982Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/ckh.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/ckh.c
17002026-03-10T17:55:27.035Z[ 4%] Building CXX object CMakeFiles/geos.dir/src/algorithm/Intersection.cpp.o
17012026-03-10T17:55:27.039Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/ctl.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/ctl.c
17022026-03-10T17:55:27.045Z[ 20%] Building C object src/CMakeFiles/proj.dir/PJ_goode.c.o
17032026-03-10T17:55:27.080Z[ 4%] Building CXX object CMakeFiles/geos.dir/src/algorithm/Length.cpp.o
17042026-03-10T17:55:27.099Z[ 21%] Building C object src/CMakeFiles/proj.dir/PJ_gstmerc.c.o
17052026-03-10T17:55:27.151Z[ 21%] Building C object src/CMakeFiles/proj.dir/PJ_hammer.c.o
17062026-03-10T17:55:27.155Z[ 4%] Building CXX object CMakeFiles/geos.dir/src/algorithm/LineIntersector.cpp.o
17072026-03-10T17:55:27.186Z[ 22%] Building C object src/CMakeFiles/proj.dir/PJ_hatano.c.o
17082026-03-10T17:55:27.190Z[ 4%] Building CXX object CMakeFiles/geos.dir/src/algorithm/MinimumBoundingCircle.cpp.o
17092026-03-10T17:55:27.212Z[ 6%] Building CXX object CMakeFiles/geos.dir/src/algorithm/MinimumDiameter.cpp.o
17102026-03-10T17:55:27.219Z[ 6%] Building CXX object CMakeFiles/geos.dir/src/algorithm/NotRepresentableException.cpp.o
17112026-03-10T17:55:27.224Z[ 6%] Building CXX object CMakeFiles/geos.dir/src/algorithm/Orientation.cpp.o
17122026-03-10T17:55:27.266Z[ 22%] Building C object src/CMakeFiles/proj.dir/PJ_igh.c.o
17132026-03-10T17:55:27.323Z[ 6%] Building CXX object CMakeFiles/geos.dir/src/algorithm/PointLocation.cpp.o
17142026-03-10T17:55:27.368Z[ 23%] Building C object src/CMakeFiles/proj.dir/PJ_isea.c.o
17152026-03-10T17:55:27.438Z[ 6%] Building CXX object CMakeFiles/geos.dir/src/algorithm/PointLocator.cpp.o
17162026-03-10T17:55:27.493Z[ 8%] Building CXX object CMakeFiles/geos.dir/src/algorithm/RayCrossingCounter.cpp.o
17172026-03-10T17:55:27.498Z[ 8%] Building CXX object CMakeFiles/geos.dir/src/algorithm/RayCrossingCounterDD.cpp.o
17182026-03-10T17:55:27.499Z[ 23%] Building C object src/CMakeFiles/proj.dir/PJ_imw_p.c.o
17192026-03-10T17:55:27.591Z[ 25%] Building C object src/CMakeFiles/proj.dir/PJ_krovak.c.o
17202026-03-10T17:55:27.630Z[ 8%] Building CXX object CMakeFiles/geos.dir/src/algorithm/RobustDeterminant.cpp.o
17212026-03-10T17:55:27.636Z[ 8%] Building CXX object CMakeFiles/geos.dir/src/algorithm/SimplePointInRing.cpp.o
17222026-03-10T17:55:27.747Z[ 8%] Building CXX object CMakeFiles/geos.dir/src/algorithm/distance/DiscreteFrechetDistance.cpp.o
17232026-03-10T17:55:27.781Z[ 25%] Building C object src/CMakeFiles/proj.dir/PJ_labrd.c.o
17242026-03-10T17:55:27.801Z[ 9%] Building CXX object CMakeFiles/geos.dir/src/algorithm/distance/DiscreteHausdorffDistance.cpp.o
17252026-03-10T17:55:27.805Z[ 9%] Building CXX object CMakeFiles/geos.dir/src/algorithm/distance/DistanceToPoint.cpp.o
17262026-03-10T17:55:27.810Z[ 9%] Building CXX object CMakeFiles/geos.dir/src/algorithm/locate/IndexedPointInAreaLocator.cpp.o
17272026-03-10T17:55:27.894Z[ 26%] Building C object src/CMakeFiles/proj.dir/PJ_laea.c.o
17282026-03-10T17:55:27.924Z[ 9%] Building CXX object CMakeFiles/geos.dir/src/algorithm/locate/PointOnGeometryLocator.cpp.o
17292026-03-10T17:55:27.957Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/extent.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/extent.c
17302026-03-10T17:55:28.061Z[ 26%] Building C object src/CMakeFiles/proj.dir/PJ_lagrng.c.o
17312026-03-10T17:55:28.079Z[ 11%] Building CXX object CMakeFiles/geos.dir/src/geom/Coordinate.cpp.o
17322026-03-10T17:55:28.083Z[ 11%] Building CXX object CMakeFiles/geos.dir/src/algorithm/locate/SimplePointInAreaLocator.cpp.o
17332026-03-10T17:55:28.116Z[ 11%] Building CXX object CMakeFiles/geos.dir/src/geom/CoordinateArraySequence.cpp.o
17342026-03-10T17:55:28.127Z[ 27%] Building C object src/CMakeFiles/proj.dir/PJ_larr.c.o
17352026-03-10T17:55:28.135Z[ 11%] Building CXX object CMakeFiles/geos.dir/src/geom/CoordinateArraySequenceFactory.cpp.o
17362026-03-10T17:55:28.154Z[ 28%] Building C object src/CMakeFiles/proj.dir/PJ_lask.c.o
17372026-03-10T17:55:28.228Z[ 28%] Building C object src/CMakeFiles/proj.dir/PJ_lcca.c.o
17382026-03-10T17:55:28.232Z[ 11%] Building CXX object CMakeFiles/geos.dir/src/geom/CoordinateSequence.cpp.o
17392026-03-10T17:55:28.270Z[ 11%] Building CXX object CMakeFiles/geos.dir/src/geom/DefaultCoordinateSequenceFactory.cpp.o
17402026-03-10T17:55:28.309Z[ 13%] Building CXX object CMakeFiles/geos.dir/src/geom/Dimension.cpp.o
17412026-03-10T17:55:28.360Z[ 29%] Building C object src/CMakeFiles/proj.dir/PJ_lcc.c.o
17422026-03-10T17:55:28.400Z[ 13%] Building CXX object CMakeFiles/geos.dir/src/geom/Envelope.cpp.o
17432026-03-10T17:55:28.419Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/hash.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/hash.c
17442026-03-10T17:55:28.480Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/huge.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/huge.c
17452026-03-10T17:55:28.533Z[ 29%] Building C object src/CMakeFiles/proj.dir/PJ_loxim.c.o
17462026-03-10T17:55:28.554Z[ 13%] Building CXX object CMakeFiles/geos.dir/src/geom/Geometry.cpp.o
17472026-03-10T17:55:28.564Z[ 13%] Building CXX object CMakeFiles/geos.dir/src/geom/GeometryCollection.cpp.o
17482026-03-10T17:55:28.572Z[ 30%] Building C object src/CMakeFiles/proj.dir/PJ_lsat.c.o
17492026-03-10T17:55:28.614Z[ 13%] Building CXX object CMakeFiles/geos.dir/src/geom/GeometryComponentFilter.cpp.o
17502026-03-10T17:55:28.702Z[ 30%] Building C object src/CMakeFiles/proj.dir/PJ_misrsom.c.o
17512026-03-10T17:55:28.730Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/mb.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/mb.c
17522026-03-10T17:55:28.740Z[ 14%] Building CXX object CMakeFiles/geos.dir/src/geom/GeometryFactory.cpp.o
17532026-03-10T17:55:28.849Z[ 31%] Building C object src/CMakeFiles/proj.dir/PJ_mbt_fps.c.o
17542026-03-10T17:55:28.856Z[ 14%] Building CXX object CMakeFiles/geos.dir/src/geom/IntersectionMatrix.cpp.o
17552026-03-10T17:55:28.859Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/mutex.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/mutex.c
17562026-03-10T17:55:28.894Z[ 31%] Building C object src/CMakeFiles/proj.dir/PJ_mbtfpp.c.o
17572026-03-10T17:55:28.927Z[ 32%] Building C object src/CMakeFiles/proj.dir/PJ_mbtfpq.c.o
17582026-03-10T17:55:28.946Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/nstime.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/nstime.c
17592026-03-10T17:55:28.959Z[ 14%] Building CXX object CMakeFiles/geos.dir/src/geom/LineSegment.cpp.o
17602026-03-10T17:55:28.964Z[ 33%] Building C object src/CMakeFiles/proj.dir/PJ_merc.c.o
17612026-03-10T17:55:29.017Z[ 33%] Building C object src/CMakeFiles/proj.dir/PJ_mill.c.o
17622026-03-10T17:55:29.022Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/pages.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/pages.c
17632026-03-10T17:55:29.061Z[ 34%] Building C object src/CMakeFiles/proj.dir/PJ_mod_ster.c.o
17642026-03-10T17:55:29.103Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/prng.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/prng.c
17652026-03-10T17:55:29.117Z[ 14%] Building CXX object CMakeFiles/geos.dir/src/geom/LinearRing.cpp.o
17662026-03-10T17:55:29.118Z[ 14%] Building CXX object CMakeFiles/geos.dir/src/geom/LineString.cpp.o
17672026-03-10T17:55:29.131Zgithub.com/bufbuild/buf/private/bufpkg/bufplugin
17682026-03-10T17:55:29.131Zgithub.com/bufbuild/buf/private/gen/proto/api/buf/alpha/registry/v1alpha1/registryv1alpha1api
17692026-03-10T17:55:29.163Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/prof.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/prof.c
17702026-03-10T17:55:29.167Z[ 34%] Building C object src/CMakeFiles/proj.dir/PJ_moll.c.o
17712026-03-10T17:55:29.171Zgithub.com/bufbuild/buf/private/gen/proto/apiclient/buf/alpha/registry/v1alpha1/registryv1alpha1apiclient
17722026-03-10T17:55:29.198Zgithub.com/bufbuild/buf/private/gen/proto/apiclienttwirp/buf/alpha/registry/v1alpha1/registryv1alpha1apiclienttwirp
17732026-03-10T17:55:29.198Zgithub.com/bufbuild/buf/private/bufpkg/bufapimodule
17742026-03-10T17:55:29.198Zgithub.com/bufbuild/buf/private/buf/bufgen
17752026-03-10T17:55:29.198Zgithub.com/bufbuild/buf/private/buf/bufprint
17762026-03-10T17:55:29.201Zgithub.com/bufbuild/buf/private/gen/proto/apiclientgrpc/buf/alpha/registry/v1alpha1/registryv1alpha1apiclientgrpc
17772026-03-10T17:55:29.269Z[ 35%] Building C object src/CMakeFiles/proj.dir/PJ_natearth.c.o
17782026-03-10T17:55:29.301Zgithub.com/bufbuild/buf/private/buf/bufmigrate
17792026-03-10T17:55:29.344Z[ 35%] Building C object src/CMakeFiles/proj.dir/PJ_natearth2.c.o
17802026-03-10T17:55:29.361Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/config/configmigratev1beta1
17812026-03-10T17:55:29.366Z[ 36%] Building C object src/CMakeFiles/proj.dir/PJ_nell.c.o
17822026-03-10T17:55:29.370Zgithub.com/bufbuild/buf/private/bufpkg/bufapiclient
17832026-03-10T17:55:29.382Z[ 36%] Building C object src/CMakeFiles/proj.dir/PJ_nell_h.c.o
17842026-03-10T17:55:29.406Z[ 38%] Building C object src/CMakeFiles/proj.dir/PJ_nocol.c.o
17852026-03-10T17:55:29.409Zgithub.com/bufbuild/buf/private/buf/bufcli
17862026-03-10T17:55:29.421Z[ 38%] Building C object src/CMakeFiles/proj.dir/PJ_nsper.c.o
17872026-03-10T17:55:29.459Z[ 39%] Building C object src/CMakeFiles/proj.dir/PJ_nzmg.c.o
17882026-03-10T17:55:29.470Z[ 16%] Building CXX object CMakeFiles/geos.dir/src/geom/Location.cpp.o
17892026-03-10T17:55:29.473Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/alpha/registry/token/tokencreate
17902026-03-10T17:55:29.473Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/alpha/registry/token/tokendelete
17912026-03-10T17:55:29.473Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/alpha/registry/token/tokenget
17922026-03-10T17:55:29.473Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/alpha/registry/token/tokenlist
17932026-03-10T17:55:29.473Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/beta/registry/commit/commitget
17942026-03-10T17:55:29.473Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/beta/registry/commit/commitlist
17952026-03-10T17:55:29.473Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/beta/registry/organization/organizationcreate
17962026-03-10T17:55:29.473Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/beta/registry/organization/organizationdelete
17972026-03-10T17:55:29.473Z[ 40%] Building C object src/CMakeFiles/proj.dir/PJ_ob_tran.c.o
17982026-03-10T17:55:29.499Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/beta/registry/organization/organizationget
17992026-03-10T17:55:29.503Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/beta/registry/plugin/plugincreate
18002026-03-10T17:55:29.506Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/beta/registry/plugin/plugindelete
18012026-03-10T17:55:29.509Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/beta/registry/plugin/pluginlist
18022026-03-10T17:55:29.514Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/beta/registry/plugin/pluginversion/pluginversionlist
18032026-03-10T17:55:29.517Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/beta/registry/repository/repositorycreate
18042026-03-10T17:55:29.521Z[ 40%] Building C object src/CMakeFiles/proj.dir/PJ_ocea.c.o
18052026-03-10T17:55:29.532Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/beta/registry/repository/repositoryget
18062026-03-10T17:55:29.535Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/beta/registry/repository/repositorylist
18072026-03-10T17:55:29.535Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/beta/registry/repository/repositorydelete
18082026-03-10T17:55:29.535Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/beta/registry/tag/tagcreate
18092026-03-10T17:55:29.539Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/beta/registry/tag/taglist
18102026-03-10T17:55:29.555Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/beta/registry/template/templatecreate
18112026-03-10T17:55:29.559Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/beta/registry/template/templatedelete
18122026-03-10T17:55:29.575Z[ 16%] Building CXX object CMakeFiles/geos.dir/src/geom/MultiLineString.cpp.o
18132026-03-10T17:55:29.578Z[ 41%] Building C object src/CMakeFiles/proj.dir/PJ_oea.c.o
18142026-03-10T17:55:29.578Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/beta/registry/template/templatelist
18152026-03-10T17:55:29.583Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/beta/registry/template/templateversion/templateversioncreate
18162026-03-10T17:55:29.586Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/beta/registry/template/templateversion/templateversionlist
18172026-03-10T17:55:29.592Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/breaking
18182026-03-10T17:55:29.597Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/build
18192026-03-10T17:55:29.600Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/convert
18202026-03-10T17:55:29.606Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/export
18212026-03-10T17:55:29.609Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/generate
18222026-03-10T17:55:29.616Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/lint
18232026-03-10T17:55:29.624Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/login
18242026-03-10T17:55:29.627Z[ 41%] Building C object src/CMakeFiles/proj.dir/PJ_omerc.c.o
18252026-03-10T17:55:29.627Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/logout
18262026-03-10T17:55:29.630Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/lsfiles
18272026-03-10T17:55:29.637Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/mod/modclearcache
18282026-03-10T17:55:29.658Z[ 16%] Building CXX object CMakeFiles/geos.dir/src/geom/MultiPoint.cpp.o
18292026-03-10T17:55:29.658Z[ 16%] Building CXX object CMakeFiles/geos.dir/src/geom/MultiPolygon.cpp.o
18302026-03-10T17:55:29.658Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/mod/modinit
18312026-03-10T17:55:29.663Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/mod/modprune
18322026-03-10T17:55:29.666Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/mod/modupdate
18332026-03-10T17:55:29.666Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/protoc
18342026-03-10T17:55:29.676Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/push
18352026-03-10T17:55:29.699Z[ 16%] Building CXX object CMakeFiles/geos.dir/src/geom/Point.cpp.o
18362026-03-10T17:55:29.708Z[ 42%] Building C object src/CMakeFiles/proj.dir/PJ_ortho.c.o
18372026-03-10T17:55:29.713Zgithub.com/bufbuild/buf/private/buf/cmd/buf
18382026-03-10T17:55:29.742Z[ 42%] Building C object src/CMakeFiles/proj.dir/PJ_patterson.c.o
18392026-03-10T17:55:29.745Zgithub.com/bufbuild/buf/cmd/buf
18402026-03-10T17:55:29.751Z[ 43%] Building C object src/CMakeFiles/proj.dir/PJ_poly.c.o
18412026-03-10T17:55:29.769Z[ 43%] Building C object src/CMakeFiles/proj.dir/PJ_putp2.c.o
18422026-03-10T17:55:29.803Z[ 45%] Building C object src/CMakeFiles/proj.dir/PJ_putp3.c.o
18432026-03-10T17:55:29.803Z[ 45%] Building C object src/CMakeFiles/proj.dir/PJ_putp4p.c.o
18442026-03-10T17:55:29.837Z[ 45%] Building C object src/CMakeFiles/proj.dir/PJ_putp5.c.o
18452026-03-10T17:55:29.840Z[ 46%] Building C object src/CMakeFiles/proj.dir/PJ_putp6.c.o
18462026-03-10T17:55:29.849Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/quarantine.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/quarantine.c
18472026-03-10T17:55:29.875Z[ 47%] Building C object src/CMakeFiles/proj.dir/PJ_qsc.c.o
18482026-03-10T17:55:29.875Z[ 47%] Building C object src/CMakeFiles/proj.dir/PJ_robin.c.o
18492026-03-10T17:55:29.911Z[ 47%] Building C object src/CMakeFiles/proj.dir/PJ_rpoly.c.o
18502026-03-10T17:55:29.945Z[ 18%] Building CXX object CMakeFiles/geos.dir/src/geom/Polygon.cpp.o
18512026-03-10T17:55:29.948Z[ 18%] Building CXX object CMakeFiles/geos.dir/src/geom/PrecisionModel.cpp.o
18522026-03-10T17:55:29.948Z[ 48%] Building C object src/CMakeFiles/proj.dir/PJ_sch.c.o
18532026-03-10T17:55:29.976Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/rtree.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/rtree.c
18542026-03-10T17:55:29.992Z[ 18%] Building CXX object CMakeFiles/geos.dir/src/geom/Triangle.cpp.o
18552026-03-10T17:55:29.992Z[ 48%] Building C object src/CMakeFiles/proj.dir/PJ_sconics.c.o
18562026-03-10T17:55:29.996Z[ 18%] Building CXX object CMakeFiles/geos.dir/src/geom/prep/AbstractPreparedPolygonContains.cpp.o
18572026-03-10T17:55:30.076Z[ 50%] Building C object src/CMakeFiles/proj.dir/PJ_somerc.c.o
18582026-03-10T17:55:30.079Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/stats.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/stats.c
18592026-03-10T17:55:30.150Z[ 50%] Building C object src/CMakeFiles/proj.dir/PJ_sterea.c.o
18602026-03-10T17:55:30.163Z[ 18%] Building CXX object CMakeFiles/geos.dir/src/geom/prep/BasicPreparedGeometry.cpp.o
18612026-03-10T17:55:30.188Z[ 51%] Building C object src/CMakeFiles/proj.dir/PJ_stere.c.o
18622026-03-10T17:55:30.191Z[ 19%] Building CXX object CMakeFiles/geos.dir/src/geom/prep/PreparedGeometry.cpp.o
18632026-03-10T17:55:30.199Z[ 19%] Building CXX object CMakeFiles/geos.dir/src/geom/prep/PreparedGeometryFactory.cpp.o
18642026-03-10T17:55:30.274Z[ 19%] Building CXX object CMakeFiles/geos.dir/src/geom/prep/PreparedLineString.cpp.o
18652026-03-10T17:55:30.292Z[ 52%] Building C object src/CMakeFiles/proj.dir/PJ_sts.c.o
18662026-03-10T17:55:30.295Z[ 19%] Building CXX object CMakeFiles/geos.dir/src/geom/prep/PreparedLineStringIntersects.cpp.o
18672026-03-10T17:55:30.326Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/spin.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/spin.c
18682026-03-10T17:55:30.373Z[ 52%] Building C object src/CMakeFiles/proj.dir/PJ_tcc.c.o
18692026-03-10T17:55:30.390Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/tcache.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/tcache.c
18702026-03-10T17:55:30.413Z[ 53%] Building C object src/CMakeFiles/proj.dir/PJ_tcea.c.o
18712026-03-10T17:55:30.417Z[ 53%] Building C object src/CMakeFiles/proj.dir/PJ_times.c.o
18722026-03-10T17:55:30.452Z[ 54%] Building C object src/CMakeFiles/proj.dir/PJ_tmerc.c.o
18732026-03-10T17:55:30.455Z[ 54%] Building C object src/CMakeFiles/proj.dir/PJ_tpeqd.c.o
18742026-03-10T17:55:30.489Z[ 19%] Building CXX object CMakeFiles/geos.dir/src/geom/prep/PreparedPoint.cpp.o
18752026-03-10T17:55:30.496Z[ 55%] Building C object src/CMakeFiles/proj.dir/PJ_urm5.c.o
18762026-03-10T17:55:30.521Z[ 55%] Building C object src/CMakeFiles/proj.dir/PJ_urmfps.c.o
18772026-03-10T17:55:30.540Z[ 56%] Building C object src/CMakeFiles/proj.dir/PJ_vandg.c.o
18782026-03-10T17:55:30.558Z[ 21%] Building CXX object CMakeFiles/geos.dir/src/geom/prep/PreparedPolygon.cpp.o
18792026-03-10T17:55:30.569Z[ 56%] Building C object src/CMakeFiles/proj.dir/PJ_vandg2.c.o
18802026-03-10T17:55:30.582Z[ 21%] Building CXX object CMakeFiles/geos.dir/src/geom/prep/PreparedPolygonContains.cpp.o
18812026-03-10T17:55:30.582Z[ 57%] Building C object src/CMakeFiles/proj.dir/PJ_vandg4.c.o
18822026-03-10T17:55:30.588Z[ 21%] Building CXX object CMakeFiles/geos.dir/src/geom/prep/PreparedPolygonContainsProperly.cpp.o
18832026-03-10T17:55:30.602Zgithub.com/cockroachdb/gostdlib/x/tools/internal/semver
18842026-03-10T17:55:30.606Zgo.mongodb.org/mongo-driver/bson/bsonoptions
18852026-03-10T17:55:30.609Zgo.mongodb.org/mongo-driver/bson/bsontype
18862026-03-10T17:55:30.612Z[ 58%] Building C object src/CMakeFiles/proj.dir/PJ_wag2.c.o
18872026-03-10T17:55:30.617Zgoogle.golang.org/protobuf/internal/flags
18882026-03-10T17:55:30.620Z[ 59%] Building C object src/CMakeFiles/proj.dir/PJ_wag7.c.o
18892026-03-10T17:55:30.620Zgoogle.golang.org/protobuf/internal/set
18902026-03-10T17:55:30.620Zgithub.com/grpc-ecosystem/grpc-gateway/internal/casing
18912026-03-10T17:55:30.623Zgithub.com/pseudomuto/protoc-gen-doc/extensions
18922026-03-10T17:55:30.623Zmaps
18932026-03-10T17:55:30.623Zhonnef.co/go/tools/internal/sync
18942026-03-10T17:55:30.635Zgithub.com/cockroachdb/gostdlib/go/printer
18952026-03-10T17:55:30.635Zgithub.com/cockroachdb/crlfmt/internal/parser
18962026-03-10T17:55:30.638Zgithub.com/cockroachdb/gostdlib/x/tools/go/ast/astutil
18972026-03-10T17:55:30.638Z[ 59%] Building C object src/CMakeFiles/proj.dir/PJ_wag3.c.o
18982026-03-10T17:55:30.642Zgithub.com/cockroachdb/gostdlib/x/tools/go/internal/packagesdriver
18992026-03-10T17:55:30.647Z[ 59%] Building C object src/CMakeFiles/proj.dir/PJ_wink1.c.o
19002026-03-10T17:55:30.653Zgithub.com/cockroachdb/gostdlib/x/tools/internal/fastwalk
19012026-03-10T17:55:30.656Zgithub.com/cockroachdb/gostdlib/x/tools/internal/span
19022026-03-10T17:55:30.660Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/ticker.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/ticker.c
19032026-03-10T17:55:30.663Zgithub.com/client9/misspell
19042026-03-10T17:55:30.670Zgithub.com/cockroachdb/crlfmt/internal/render
19052026-03-10T17:55:30.675Zgithub.com/cockroachdb/gostdlib/x/tools/internal/module
19062026-03-10T17:55:30.695Zgithub.com/cockroachdb/ttycolor
19072026-03-10T17:55:30.710Z[ 60%] Building C object src/CMakeFiles/proj.dir/PJ_wink2.c.o
19082026-03-10T17:55:30.713Zgithub.com/cockroachdb/stress
19092026-03-10T17:55:30.713Zgithub.com/cockroachdb/gostdlib/x/tools/internal/gopathwalk
19102026-03-10T17:55:30.729Zgithub.com/cockroachdb/tools/cmd/stringer
19112026-03-10T17:55:30.732Z[ 60%] Building C object src/CMakeFiles/proj.dir/proj_etmerc.c.o
19122026-03-10T17:55:30.732Zgithub.com/go-openapi/analysis/internal
19132026-03-10T17:55:30.741Zgithub.com/josharian/intern
19142026-03-10T17:55:30.747Zgithub.com/cockroachdb/gostdlib/go/format
19152026-03-10T17:55:30.752Zgithub.com/cockroachdb/gostdlib/cmd/gofmt
19162026-03-10T17:55:30.756Zgithub.com/cockroachdb/gostdlib/x/tools/go/packages
19172026-03-10T17:55:30.760Zgithub.com/mailru/easyjson/buffer
19182026-03-10T17:55:30.775Zgithub.com/mailru/easyjson/jlexer
19192026-03-10T17:55:30.778Z[ 61%] Building C object src/CMakeFiles/proj.dir/aasincos.c.o
19202026-03-10T17:55:30.781Zgithub.com/mailru/easyjson/jwriter
19212026-03-10T17:55:30.801Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/tsd.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/tsd.c
19222026-03-10T17:55:30.804Zgopkg.in/yaml.v2
19232026-03-10T17:55:30.822Zinternal/saferio
19242026-03-10T17:55:30.825Zgithub.com/PuerkitoBio/urlesc
19252026-03-10T17:55:30.825Z[ 61%] Building C object src/CMakeFiles/proj.dir/adjlon.c.o
19262026-03-10T17:55:30.845Zgolang.org/x/text/transform
19272026-03-10T17:55:30.857Zencoding/gob
19282026-03-10T17:55:30.861Z[ 63%] Building C object src/CMakeFiles/proj.dir/bch2bps.c.o
19292026-03-10T17:55:30.866Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/util.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c
19302026-03-10T17:55:30.884Zgolang.org/x/text/unicode/bidi
19312026-03-10T17:55:30.892Zgithub.com/cockroachdb/gostdlib/x/tools/internal/imports
19322026-03-10T17:55:30.910Z[ 64%] Building C object src/CMakeFiles/proj.dir/bchgen.c.o
19332026-03-10T17:55:30.968Zgolang.org/x/text/secure/bidirule
19342026-03-10T17:55:30.973Z[ 64%] Building C object src/CMakeFiles/proj.dir/biveval.c.o
19352026-03-10T17:55:30.981Z/work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c: In function 'je_malloc_vsnprintf':
19362026-03-10T17:55:30.981Z/work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c:359:9: warning: case label value exceeds maximum value for type [-Wswitch-outside-range]
19372026-03-10T17:55:30.981Z 359 | case '?' | 0x80: \
19382026-03-10T17:55:30.981Z | ^~~~
19392026-03-10T17:55:30.981Z/work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c:569:33: note: in expansion of macro 'GET_ARG_NUMERIC'
19402026-03-10T17:55:30.981Z 569 | GET_ARG_NUMERIC(val, 'p');
19412026-03-10T17:55:30.981Z | ^~~~~~~~~~~~~~~
19422026-03-10T17:55:30.981Z/work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c:377:9: warning: case label value exceeds maximum value for type [-Wswitch-outside-range]
19432026-03-10T17:55:30.981Z 377 | case 'j' | 0x80: \
19442026-03-10T17:55:30.981Z | ^~~~
19452026-03-10T17:55:30.981Z/work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c:569:33: note: in expansion of macro 'GET_ARG_NUMERIC'
19462026-03-10T17:55:30.981Z 569 | GET_ARG_NUMERIC(val, 'p');
19472026-03-10T17:55:30.981Z | ^~~~~~~~~~~~~~~
19482026-03-10T17:55:30.981Z/work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c:365:9: warning: case label value exceeds maximum value for type [-Wswitch-outside-range]
19492026-03-10T17:55:30.981Z 365 | case 'l' | 0x80: \
19502026-03-10T17:55:30.981Z | ^~~~
19512026-03-10T17:55:30.981Z/work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c:569:33: note: in expansion of macro 'GET_ARG_NUMERIC'
19522026-03-10T17:55:30.981Z 569 | GET_ARG_NUMERIC(val, 'p');
19532026-03-10T17:55:30.981Z | ^~~~~~~~~~~~~~~
19542026-03-10T17:55:30.981Z/work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c:371:9: warning: case label value exceeds maximum value for type [-Wswitch-outside-range]
19552026-03-10T17:55:30.981Z 371 | case 'q' | 0x80: \
19562026-03-10T17:55:30.981Z | ^~~~
19572026-03-10T17:55:30.981Z/work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c:569:33: note: in expansion of macro 'GET_ARG_NUMERIC'
19582026-03-10T17:55:30.981Z 569 | GET_ARG_NUMERIC(val, 'p');
19592026-03-10T17:55:30.981Z | ^~~~~~~~~~~~~~~
19602026-03-10T17:55:30.981Z/work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c:386:9: warning: case label value exceeds maximum value for type [-Wswitch-outside-range]
19612026-03-10T17:55:30.981Z 386 | case 'z' | 0x80: \
19622026-03-10T17:55:30.981Z | ^~~~
19632026-03-10T17:55:30.981Z/work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c:569:33: note: in expansion of macro 'GET_ARG_NUMERIC'
19642026-03-10T17:55:30.981Z 569 | GET_ARG_NUMERIC(val, 'p');
19652026-03-10T17:55:30.981Z | ^~~~~~~~~~~~~~~
19662026-03-10T17:55:30.981Zgithub.com/client9/misspell/cmd/misspell
19672026-03-10T17:55:30.991Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/witness.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/witness.c
19682026-03-10T17:55:30.998Zgolang.org/x/text/unicode/norm
19692026-03-10T17:55:31.003Z[ 21%] Building CXX object CMakeFiles/geos.dir/src/geom/prep/PreparedPolygonCovers.cpp.o
19702026-03-10T17:55:31.007Zgolang.org/x/text/width
19712026-03-10T17:55:31.014Zgithub.com/asaskevich/govalidator
19722026-03-10T17:55:31.064Zgithub.com/go-openapi/errors
19732026-03-10T17:55:31.069Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/jemalloc.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/jemalloc.c
19742026-03-10T17:55:31.109Zgithub.com/mitchellh/mapstructure
19752026-03-10T17:55:31.137Zgithub.com/oklog/ulid
19762026-03-10T17:55:31.182Z[ 65%] Building C object src/CMakeFiles/proj.dir/dmstor.c.o
19772026-03-10T17:55:31.210Zgo.mongodb.org/mongo-driver/bson/primitive
19782026-03-10T17:55:31.225Z[ 21%] Building CXX object CMakeFiles/geos.dir/src/geom/prep/PreparedPolygonIntersects.cpp.o
19792026-03-10T17:55:31.293Zgithub.com/go-stack/stack
19802026-03-10T17:55:31.298Znet/mail
19812026-03-10T17:55:31.303Z[ 65%] Building C object src/CMakeFiles/proj.dir/emess.c.o
19822026-03-10T17:55:31.314Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/arena.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/arena.c
19832026-03-10T17:55:31.317Zencoding/xml
19842026-03-10T17:55:31.320Zgithub.com/cockroachdb/gostdlib/x/tools/imports
19852026-03-10T17:55:31.323Zgithub.com/cockroachdb/gostdlib/x/tools/cmd/goimports
19862026-03-10T17:55:31.350Zgithub.com/cockroachdb/crlfmt
19872026-03-10T17:55:31.353Zgolang.org/x/net/idna
19882026-03-10T17:55:31.356Zgo.mongodb.org/mongo-driver/x/bsonx/bsoncore
19892026-03-10T17:55:31.377Zgithub.com/go-openapi/runtime/logger
19902026-03-10T17:55:31.381Zgithub.com/go-openapi/swag
19912026-03-10T17:55:31.390Z[ 22%] Building CXX object CMakeFiles/geos.dir/src/geom/prep/PreparedPolygonPredicate.cpp.o
19922026-03-10T17:55:31.397Zgithub.com/go-openapi/runtime/middleware/denco
19932026-03-10T17:55:31.410Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/atomic.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/atomic.c
19942026-03-10T17:55:31.410Z[ 66%] Building C object src/CMakeFiles/proj.dir/geocent.c.o
19952026-03-10T17:55:31.429Zgithub.com/go-openapi/runtime/middleware/header
19962026-03-10T17:55:31.491Zgolang.org/x/tools/go/ast/astutil
19972026-03-10T17:55:31.511Zgithub.com/PuerkitoBio/purell
19982026-03-10T17:55:31.534Zgithub.com/go-openapi/inflect
19992026-03-10T17:55:31.551Z[ 66%] Building C object src/CMakeFiles/proj.dir/geodesic.c.o
20002026-03-10T17:55:31.554Zgo.mongodb.org/mongo-driver/bson/bsonrw
20012026-03-10T17:55:31.574Zgithub.com/go-openapi/jsonpointer
20022026-03-10T17:55:31.578Zgithub.com/go-openapi/loads/fmts
20032026-03-10T17:55:31.594Zgolang.org/x/sys/unix
20042026-03-10T17:55:31.612Zgithub.com/go-openapi/jsonreference
20052026-03-10T17:55:31.621Zgithub.com/hashicorp/hcl/hcl/strconv
20062026-03-10T17:55:31.621Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/base.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/base.c
20072026-03-10T17:55:31.621Zgithub.com/magiconair/properties
20082026-03-10T17:55:31.641Zgithub.com/hashicorp/hcl/hcl/token
20092026-03-10T17:55:31.657Zgithub.com/go-openapi/spec
20102026-03-10T17:55:31.698Zgithub.com/hashicorp/hcl/hcl/ast
20112026-03-10T17:55:31.737Zgithub.com/hashicorp/hcl/hcl/scanner
20122026-03-10T17:55:31.746Zgithub.com/hashicorp/hcl/json/token
20132026-03-10T17:55:31.750Zgithub.com/pelletier/go-toml
20142026-03-10T17:55:31.789Zgithub.com/hashicorp/hcl/json/scanner
20152026-03-10T17:55:31.793Zgithub.com/spf13/afero/mem
20162026-03-10T17:55:31.793Z[ 22%] Building CXX object CMakeFiles/geos.dir/src/geom/util/ComponentCoordinateExtracter.cpp.o
20172026-03-10T17:55:31.797Zgithub.com/hashicorp/hcl/hcl/parser
20182026-03-10T17:55:31.822Zgithub.com/spf13/afero
20192026-03-10T17:55:31.842Zgithub.com/hashicorp/hcl/json/parser
20202026-03-10T17:55:31.858Zgithub.com/hashicorp/hcl/hcl/printer
20212026-03-10T17:55:31.869Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/bitmap.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/bitmap.c
20222026-03-10T17:55:31.892Z[ 22%] Building CXX object CMakeFiles/geos.dir/src/geom/util/CoordinateOperation.cpp.o
20232026-03-10T17:55:31.906Zgo.mongodb.org/mongo-driver/bson/bsoncodec
20242026-03-10T17:55:31.910Zgithub.com/hashicorp/hcl
20252026-03-10T17:55:31.928Z[ 22%] Building CXX object CMakeFiles/geos.dir/src/geom/util/Densifier.cpp.o
20262026-03-10T17:55:31.977Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/chunk.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/chunk.c
20272026-03-10T17:55:32.006Zgithub.com/spf13/cast
20282026-03-10T17:55:32.048Zgithub.com/spf13/jwalterweatherman
20292026-03-10T17:55:32.103Zgithub.com/subosito/gotenv
20302026-03-10T17:55:32.133Zgopkg.in/ini.v1
20312026-03-10T17:55:32.167Zgithub.com/go-swagger/go-swagger/cmd/swagger/commands/diff
20322026-03-10T17:55:32.170Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/chunk_dss.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/chunk_dss.c
20332026-03-10T17:55:32.170Zgithub.com/go-swagger/go-swagger/codescan
20342026-03-10T17:55:32.170Zgolang.org/x/mod/internal/lazyregexp
20352026-03-10T17:55:32.246Zgolang.org/x/mod/module
20362026-03-10T17:55:32.257Zgolang.org/x/tools/internal/gopathwalk
20372026-03-10T17:55:32.263Zhash/crc64
20382026-03-10T17:55:32.311Zinternal/sysinfo
20392026-03-10T17:55:32.337Zgithub.com/go-swagger/go-swagger/cmd/swagger/commands/initcmd
20402026-03-10T17:55:32.347Zgithub.com/felixge/httpsnoop
20412026-03-10T17:55:32.351Zgithub.com/toqueteos/webbrowser
20422026-03-10T17:55:32.351Ztesting
20432026-03-10T17:55:32.359Zgithub.com/fsnotify/fsnotify
20442026-03-10T17:55:32.381Zgithub.com/jessevdk/go-flags
20452026-03-10T17:55:32.404Zgithub.com/golang/mock/mockgen/model
20462026-03-10T17:55:32.410Zhash/adler32
20472026-03-10T17:55:32.417Zgithub.com/spf13/viper
20482026-03-10T17:55:32.459Zgo.mongodb.org/mongo-driver/bson
20492026-03-10T17:55:32.477Zcompress/zlib
20502026-03-10T17:55:32.480Z[ 22%] Building CXX object CMakeFiles/geos.dir/src/geom/util/GeometryCombiner.cpp.o
20512026-03-10T17:55:32.490Zdebug/dwarf
20522026-03-10T17:55:32.498Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/chunk_mmap.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/chunk_mmap.c
20532026-03-10T17:55:32.501Zgithub.com/gorilla/handlers
20542026-03-10T17:55:32.582Zinternal/zstd
20552026-03-10T17:55:32.600Zgithub.com/go-openapi/strfmt
20562026-03-10T17:55:32.655Zgolang.org/x/tools/internal/modindex
20572026-03-10T17:55:32.666Z[ 24%] Building CXX object CMakeFiles/geos.dir/src/geom/util/GeometryEditor.cpp.o
20582026-03-10T17:55:32.669Zinternal/pkgbits
20592026-03-10T17:55:32.686Zgo/internal/srcimporter
20602026-03-10T17:55:32.699Zgolang.org/x/mod/modfile
20612026-03-10T17:55:32.704Zgithub.com/golang/glog/internal/stackdump
20622026-03-10T17:55:32.714Zgoogle.golang.org/protobuf/internal/detrand
20632026-03-10T17:55:32.736Zgithub.com/golang/glog/internal/logsink
20642026-03-10T17:55:32.771Zgoogle.golang.org/protobuf/internal/pragma
20652026-03-10T17:55:32.774Zgo/internal/gcimporter
20662026-03-10T17:55:32.778Zgoogle.golang.org/protobuf/internal/errors
20672026-03-10T17:55:32.782Zgithub.com/golang/glog
20682026-03-10T17:55:32.791Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/ckh.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/ckh.c
20692026-03-10T17:55:32.809Zgoogle.golang.org/protobuf/encoding/protowire
20702026-03-10T17:55:32.815Zgolang.org/x/tools/internal/imports
20712026-03-10T17:55:32.825Zgoogle.golang.org/protobuf/internal/editiondefaults
20722026-03-10T17:55:32.852Zdebug/elf
20732026-03-10T17:55:32.862Zinternal/xcoff
20742026-03-10T17:55:32.865Zgithub.com/go-openapi/analysis
20752026-03-10T17:55:32.894Zgithub.com/go-openapi/runtime
20762026-03-10T17:55:32.894Zgoogle.golang.org/protobuf/reflect/protoreflect
20772026-03-10T17:55:32.899Z[ 67%] Building C object src/CMakeFiles/proj.dir/mk_cheby.c.o
20782026-03-10T17:55:32.903Zgoogle.golang.org/protobuf/internal/version
20792026-03-10T17:55:32.914Z[ 24%] Building CXX object CMakeFiles/geos.dir/src/geom/util/GeometryTransformer.cpp.o
20802026-03-10T17:55:32.924Zgithub.com/ghodss/yaml
20812026-03-10T17:55:32.929Zgithub.com/grpc-ecosystem/grpc-gateway/utilities
20822026-03-10T17:55:32.933Zencoding/base32
20832026-03-10T17:55:32.982Zgithub.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway/httprule
20842026-03-10T17:55:33.008Zgithub.com/go-openapi/runtime/security
20852026-03-10T17:55:33.020Zgolang.org/x/tools/imports
20862026-03-10T17:55:33.040Zgo/internal/gccgoimporter
20872026-03-10T17:55:33.052Zgolang.org/x/tools/go/analysis
20882026-03-10T17:55:33.057Zgithub.com/kevinburke/go-bindata
20892026-03-10T17:55:33.057Z[ 67%] Building C object src/CMakeFiles/proj.dir/nad_cvt.c.o
20902026-03-10T17:55:33.065Zgolang.org/x/tools/cover
20912026-03-10T17:55:33.068Zgithub.com/mibk/dupl/suffixtree
20922026-03-10T17:55:33.068Zgoogle.golang.org/protobuf/internal/encoding/messageset
20932026-03-10T17:55:33.071Zgoogle.golang.org/protobuf/internal/strs
20942026-03-10T17:55:33.097Zgoogle.golang.org/protobuf/internal/genid
20952026-03-10T17:55:33.103Zgoogle.golang.org/protobuf/internal/order
20962026-03-10T17:55:33.109Zgoogle.golang.org/protobuf/internal/encoding/text
20972026-03-10T17:55:33.118Zgoogle.golang.org/protobuf/reflect/protoregistry
20982026-03-10T17:55:33.133Zgoogle.golang.org/protobuf/runtime/protoiface
20992026-03-10T17:55:33.141Z[ 68%] Building C object src/CMakeFiles/proj.dir/nad_intr.c.o
21002026-03-10T17:55:33.160Zgoogle.golang.org/protobuf/internal/descfmt
21012026-03-10T17:55:33.163Zgoogle.golang.org/protobuf/internal/descopts
21022026-03-10T17:55:33.167Zgithub.com/go-openapi/loads
21032026-03-10T17:55:33.170Zgoogle.golang.org/protobuf/internal/encoding/json
21042026-03-10T17:55:33.178Zgithub.com/kevinburke/go-bindata/go-bindata
21052026-03-10T17:55:33.181Z[ 68%] Building C object src/CMakeFiles/proj.dir/pj_apply_gridshift.c.o
21062026-03-10T17:55:33.185Zgoogle.golang.org/protobuf/proto
21072026-03-10T17:55:33.209Zgithub.com/kisielk/errcheck/errcheck
21082026-03-10T17:55:33.213Zgo/importer
21092026-03-10T17:55:33.216Zgoogle.golang.org/protobuf/internal/encoding/defval
21102026-03-10T17:55:33.228Zgithub.com/go-openapi/runtime/middleware/untyped
21112026-03-10T17:55:33.243Zgithub.com/go-openapi/validate
21122026-03-10T17:55:33.247Zgithub.com/mattn/goveralls
21132026-03-10T17:55:33.262Zgithub.com/golang/mock/mockgen
21142026-03-10T17:55:33.270Zgithub.com/mibk/dupl/syntax
21152026-03-10T17:55:33.288Zgithub.com/mmatczuk/go_generics/globals
21162026-03-10T17:55:33.298Z[ 69%] Building C object src/CMakeFiles/proj.dir/pj_apply_vgridshift.c.o
21172026-03-10T17:55:33.301Zgithub.com/kisielk/errcheck
21182026-03-10T17:55:33.304Zgoogle.golang.org/protobuf/encoding/prototext
21192026-03-10T17:55:33.313Zgoogle.golang.org/protobuf/internal/filedesc
21202026-03-10T17:55:33.355Zgithub.com/mibk/dupl/syntax/golang
21212026-03-10T17:55:33.375Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/ctl.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/ctl.c
21222026-03-10T17:55:33.394Z[ 24%] Building CXX object CMakeFiles/geos.dir/src/geom/util/LinearComponentExtracter.cpp.o
21232026-03-10T17:55:33.403Z[ 70%] Building C object src/CMakeFiles/proj.dir/pj_auth.c.o
21242026-03-10T17:55:33.478Zgithub.com/mibk/dupl/job
21252026-03-10T17:55:33.490Zgithub.com/mibk/dupl/output
21262026-03-10T17:55:33.511Zgithub.com/mmatczuk/go_generics/cmd/go_generics
21272026-03-10T17:55:33.521Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/extent.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/extent.c
21282026-03-10T17:55:33.525Zgithub.com/Masterminds/goutils
21292026-03-10T17:55:33.531Z[ 24%] Building CXX object CMakeFiles/geos.dir/src/geom/util/PointExtracter.cpp.o
21302026-03-10T17:55:33.540Z[ 70%] Building C object src/CMakeFiles/proj.dir/pj_ctx.c.o
21312026-03-10T17:55:33.558Zgithub.com/mibk/dupl
21322026-03-10T17:55:33.585Zgithub.com/Masterminds/semver
21332026-03-10T17:55:33.631Zgithub.com/go-openapi/runtime/middleware
21342026-03-10T17:55:33.636Z[ 71%] Building C object src/CMakeFiles/proj.dir/pj_fileapi.c.o
21352026-03-10T17:55:33.636Zgithub.com/go-swagger/go-swagger/generator
21362026-03-10T17:55:33.643Zgoogle.golang.org/protobuf/internal/encoding/tag
21372026-03-10T17:55:33.679Zgoogle.golang.org/protobuf/encoding/protojson
21382026-03-10T17:55:33.721Zgoogle.golang.org/protobuf/internal/impl
21392026-03-10T17:55:33.814Zgithub.com/google/uuid
21402026-03-10T17:55:33.819Zgithub.com/huandu/xstrings
21412026-03-10T17:55:33.822Z[ 71%] Building C object src/CMakeFiles/proj.dir/pj_datum_set.c.o
21422026-03-10T17:55:33.831Zgithub.com/imdario/mergo
21432026-03-10T17:55:33.859Zgithub.com/mitchellh/reflectwalk
21442026-03-10T17:55:33.865Zgolang.org/x/crypto/pbkdf2
21452026-03-10T17:55:33.870Zgithub.com/mwitkow/go-proto-validators
21462026-03-10T17:55:33.897Zgithub.com/wadey/gocovmerge
21472026-03-10T17:55:33.901Zgolang.org/x/perf/internal/stats
21482026-03-10T17:55:33.908Zgolang.org/x/perf/storage/benchfmt
21492026-03-10T17:55:33.915Zgolang.org/x/crypto/scrypt
21502026-03-10T17:55:33.918Z[ 24%] Building CXX object CMakeFiles/geos.dir/src/geom/util/PolygonExtracter.cpp.o
21512026-03-10T17:55:33.923Zgithub.com/mitchellh/copystructure
21522026-03-10T17:55:33.929Zgolang.org/x/tools/cmd/goyacc
21532026-03-10T17:55:33.976Z[ 72%] Building C object src/CMakeFiles/proj.dir/pj_datums.c.o
21542026-03-10T17:55:33.981Zgolang.org/x/tools/internal/astutil/edge
21552026-03-10T17:55:33.981Zgolang.org/x/tools/internal/analysisinternal
21562026-03-10T17:55:33.987Zgolang.org/x/perf/benchstat
21572026-03-10T17:55:33.998Zgithub.com/Masterminds/sprig
21582026-03-10T17:55:34.052Zgolang.org/x/tools/go/analysis/passes/internal/analysisutil
21592026-03-10T17:55:34.052Zgolang.org/x/tools/go/ast/inspector
21602026-03-10T17:55:34.081Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/hash.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/hash.c
21612026-03-10T17:55:34.085Zgolang.org/x/perf/cmd/benchstat
21622026-03-10T17:55:34.099Zgolang.org/x/tools/go/analysis/internal/analysisflags
21632026-03-10T17:55:34.108Z[ 72%] Building C object src/CMakeFiles/proj.dir/pj_deriv.c.o
21642026-03-10T17:55:34.111Zgolang.org/x/tools/go/analysis/internal
21652026-03-10T17:55:34.169Zgolang.org/x/tools/go/analysis/passes/inspect
21662026-03-10T17:55:34.185Zgolang.org/x/tools/internal/diff/lcs
21672026-03-10T17:55:34.199Zgolang.org/x/tools/internal/facts
21682026-03-10T17:55:34.215Z[ 26%] Building CXX object CMakeFiles/geos.dir/src/geom/util/ShortCircuitedGeometryVisitor.cpp.o
21692026-03-10T17:55:34.219Zgolang.org/x/tools/go/analysis/passes/shadow
21702026-03-10T17:55:34.224Z[ 26%] Building CXX object CMakeFiles/geos.dir/src/geom/util/SineStarFactory.cpp.o
21712026-03-10T17:55:34.228Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/huge.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/huge.c
21722026-03-10T17:55:34.237Zgolang.org/x/tools/internal/diff
21732026-03-10T17:55:34.240Zgolang.org/x/tools/go/analysis/checker
21742026-03-10T17:55:34.243Z[ 73%] Building C object src/CMakeFiles/proj.dir/pj_ell_set.c.o
21752026-03-10T17:55:34.294Zgolang.org/x/tools/go/buildutil
21762026-03-10T17:55:34.302Zhonnef.co/go/tools/analysis/facts/tokenfile
21772026-03-10T17:55:34.319Zgithub.com/BurntSushi/toml/internal
21782026-03-10T17:55:34.319Zgithub.com/go-swagger/go-swagger/cmd/swagger/commands/generate
21792026-03-10T17:55:34.326Zgolang.org/x/tools/go/analysis/unitchecker
21802026-03-10T17:55:34.349Z[ 73%] Building C object src/CMakeFiles/proj.dir/pj_ellps.c.o
21812026-03-10T17:55:34.354Zgithub.com/BurntSushi/toml
21822026-03-10T17:55:34.359Zgolang.org/x/tools/go/analysis/internal/checker
21832026-03-10T17:55:34.367Zhonnef.co/go/tools/analysis/lint
21842026-03-10T17:55:34.370Zdebug/macho
21852026-03-10T17:55:34.374Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/mb.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/mb.c
21862026-03-10T17:55:34.378Zhonnef.co/go/tools/internal/robustio
21872026-03-10T17:55:34.390Zhonnef.co/go/tools/analysis/facts/generated
21882026-03-10T17:55:34.412Zgolang.org/x/exp/typeparams
21892026-03-10T17:55:34.416Zgithub.com/go-swagger/go-swagger/cmd/swagger/commands
21902026-03-10T17:55:34.430Zhonnef.co/go/tools/internal/renameio
21912026-03-10T17:55:34.433Z[ 26%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/Depth.cpp.o
21922026-03-10T17:55:34.463Zgolang.org/x/tools/go/analysis/singlechecker
21932026-03-10T17:55:34.467Z[ 75%] Building C object src/CMakeFiles/proj.dir/pj_errno.c.o
21942026-03-10T17:55:34.470Zhonnef.co/go/tools/go/ast/astutil
21952026-03-10T17:55:34.477Zhonnef.co/go/tools/go/buildid
21962026-03-10T17:55:34.485Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/mutex.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/mutex.c
21972026-03-10T17:55:34.503Zhonnef.co/go/tools/go/types/typeutil
21982026-03-10T17:55:34.510Zgolang.org/x/tools/go/analysis/passes/shadow/cmd/shadow
21992026-03-10T17:55:34.513Zhonnef.co/go/tools/lintcmd/cache
22002026-03-10T17:55:34.517Z[ 76%] Building C object src/CMakeFiles/proj.dir/pj_factors.c.o
22012026-03-10T17:55:34.529Zgolang.org/x/tools/go/internal/cgo
22022026-03-10T17:55:34.537Zgithub.com/go-swagger/go-swagger/cmd/swagger
22032026-03-10T17:55:34.582Zhonnef.co/go/tools/knowledge
22042026-03-10T17:55:34.598Zhonnef.co/go/tools/pattern
22052026-03-10T17:55:34.612Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/nstime.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/nstime.c
22062026-03-10T17:55:34.615Zgolang.org/x/tools/go/loader
22072026-03-10T17:55:34.618Zhonnef.co/go/tools/go/ir
22082026-03-10T17:55:34.639Z[ 76%] Building C object src/CMakeFiles/proj.dir/pj_fwd.c.o
22092026-03-10T17:55:34.666Z[ 77%] Building C object src/CMakeFiles/proj.dir/pj_fwd3d.c.o
22102026-03-10T17:55:34.684Zhonnef.co/go/tools/analysis/facts/directives
22112026-03-10T17:55:34.703Z[ 77%] Building C object src/CMakeFiles/proj.dir/pj_gauss.c.o
22122026-03-10T17:55:34.719Zgoogle.golang.org/protobuf/internal/filetype
22132026-03-10T17:55:34.722Zhonnef.co/go/tools/lintcmd/version
22142026-03-10T17:55:34.732Zhonnef.co/go/tools/config
22152026-03-10T17:55:34.745Zhonnef.co/go/tools/sarif
22162026-03-10T17:55:34.753Z[ 78%] Building C object src/CMakeFiles/proj.dir/pj_gc_reader.c.o
22172026-03-10T17:55:34.757Zhonnef.co/go/tools/analysis/facts/deprecated
22182026-03-10T17:55:34.776Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/pages.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/pages.c
22192026-03-10T17:55:34.789Zgoogle.golang.org/protobuf/runtime/protoimpl
22202026-03-10T17:55:34.804Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/prng.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/prng.c
22212026-03-10T17:55:34.808Z[ 78%] Building C object src/CMakeFiles/proj.dir/pj_generic_selftest.c.o
22222026-03-10T17:55:34.819Zhonnef.co/go/tools/staticcheck/fakereflect
22232026-03-10T17:55:34.846Zhonnef.co/go/tools/printf
22242026-03-10T17:55:34.858Zhonnef.co/go/tools/go/loader
22252026-03-10T17:55:34.864Z[ 26%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/DirectedEdge.cpp.o
22262026-03-10T17:55:34.874Zhonnef.co/go/tools/staticcheck/fakejson
22272026-03-10T17:55:34.875Z[ 26%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/DirectedEdgeStar.cpp.o
22282026-03-10T17:55:34.878Z[ 79%] Building C object src/CMakeFiles/proj.dir/pj_geocent.c.o
22292026-03-10T17:55:34.878Zhonnef.co/go/tools/staticcheck/fakexml
22302026-03-10T17:55:34.881Zgoogle.golang.org/protobuf/types/descriptorpb
22312026-03-10T17:55:34.890Zgoogle.golang.org/protobuf/types/known/durationpb
22322026-03-10T17:55:34.894Z[ 27%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/Edge.cpp.o
22332026-03-10T17:55:34.894Z[ 27%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/EdgeEnd.cpp.o
22342026-03-10T17:55:34.921Zgoogle.golang.org/protobuf/types/known/timestamppb
22352026-03-10T17:55:34.997Z[ 79%] Building C object src/CMakeFiles/proj.dir/pj_gridcatalog.c.o
22362026-03-10T17:55:35.000Zhonnef.co/go/tools/analysis/edit
22372026-03-10T17:55:35.073Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/prof.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/prof.c
22382026-03-10T17:55:35.146Zgoogle.golang.org/protobuf/types/pluginpb
22392026-03-10T17:55:35.146Zgithub.com/golang/protobuf/protoc-gen-go/descriptor
22402026-03-10T17:55:35.146Zgoogle.golang.org/genproto/googleapis/api/annotations
22412026-03-10T17:55:35.146Zgithub.com/envoyproxy/protoc-gen-validate/validate
22422026-03-10T17:55:35.146Zgoogle.golang.org/protobuf/types/gofeaturespb
22432026-03-10T17:55:35.198Zgithub.com/golang/protobuf/protoc-gen-go/plugin
22442026-03-10T17:55:35.209Z[ 80%] Building C object src/CMakeFiles/proj.dir/pj_gridinfo.c.o
22452026-03-10T17:55:35.231Zgoogle.golang.org/protobuf/reflect/protodesc
22462026-03-10T17:55:35.329Zgithub.com/pseudomuto/protoc-gen-doc/extensions/google_api_http
22472026-03-10T17:55:35.337Zgithub.com/pseudomuto/protoc-gen-doc/extensions/envoyproxy_validate
22482026-03-10T17:55:35.354Z[ 27%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/EdgeEndStar.cpp.o
22492026-03-10T17:55:35.398Z[ 27%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/EdgeIntersectionList.cpp.o
22502026-03-10T17:55:35.413Zgithub.com/golang/protobuf/proto
22512026-03-10T17:55:35.422Zgithub.com/pseudomuto/protoc-gen-doc/extensions/lyft_validate
22522026-03-10T17:55:35.514Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/quarantine.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/quarantine.c
22532026-03-10T17:55:35.528Z[ 80%] Building C object src/CMakeFiles/proj.dir/pj_gridlist.c.o
22542026-03-10T17:55:35.552Z[ 27%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/EdgeList.cpp.o
22552026-03-10T17:55:35.556Zgithub.com/grpc-ecosystem/grpc-gateway/codegenerator
22562026-03-10T17:55:35.556Zgithub.com/golang/protobuf/jsonpb
22572026-03-10T17:55:35.556Zgithub.com/pseudomuto/protoc-gen-doc/extensions/validator_field
22582026-03-10T17:55:35.556Zgithub.com/pseudomuto/protokit
22592026-03-10T17:55:35.564Z[ 29%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/EdgeNodingValidator.cpp.o
22602026-03-10T17:55:35.573Zhonnef.co/go/tools/internal/passes/buildir
22612026-03-10T17:55:35.573Zhonnef.co/go/tools/go/ir/irutil
22622026-03-10T17:55:35.624Zhonnef.co/go/tools/analysis/facts/nilness
22632026-03-10T17:55:35.629Zgithub.com/pseudomuto/protoc-gen-doc
22642026-03-10T17:55:35.649Z[ 81%] Building C object src/CMakeFiles/proj.dir/PJ_healpix.c.o
22652026-03-10T17:55:35.661Zhonnef.co/go/tools/analysis/facts/purity
22662026-03-10T17:55:35.661Zhonnef.co/go/tools/analysis/facts/typedness
22672026-03-10T17:55:35.689Zgithub.com/pseudomuto/protoc-gen-doc/cmd/protoc-gen-doc
22682026-03-10T17:55:35.692Zgithub.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway/descriptor
22692026-03-10T17:55:35.728Zhonnef.co/go/tools/analysis/code
22702026-03-10T17:55:35.762Zhonnef.co/go/tools/analysis/report
22712026-03-10T17:55:35.780Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/rtree.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/rtree.c
22722026-03-10T17:55:35.787Zgithub.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway/generator
22732026-03-10T17:55:35.804Zhonnef.co/go/tools/unused
22742026-03-10T17:55:35.804Zhonnef.co/go/tools/quickfix/qf1001
22752026-03-10T17:55:35.804Zhonnef.co/go/tools/quickfix/qf1002
22762026-03-10T17:55:35.804Zhonnef.co/go/tools/quickfix/qf1003
22772026-03-10T17:55:35.804Zhonnef.co/go/tools/quickfix/qf1004
22782026-03-10T17:55:35.808Zhonnef.co/go/tools/quickfix/qf1005
22792026-03-10T17:55:35.808Zgithub.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway/internal/gengateway
22802026-03-10T17:55:35.830Zhonnef.co/go/tools/quickfix/qf1006
22812026-03-10T17:55:35.838Zhonnef.co/go/tools/quickfix/qf1007
22822026-03-10T17:55:35.838Zhonnef.co/go/tools/quickfix/qf1008
22832026-03-10T17:55:35.841Z[ 82%] Building C object src/CMakeFiles/proj.dir/pj_init.c.o
22842026-03-10T17:55:35.845Zhonnef.co/go/tools/quickfix/qf1009
22852026-03-10T17:55:35.848Zhonnef.co/go/tools/quickfix/qf1010
22862026-03-10T17:55:35.870Zhonnef.co/go/tools/internal/sharedcheck
22872026-03-10T17:55:35.873Zhonnef.co/go/tools/quickfix/qf1012
22882026-03-10T17:55:35.881Zhonnef.co/go/tools/simple/s1000
22892026-03-10T17:55:35.886Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/stats.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/stats.c
22902026-03-10T17:55:35.890Zhonnef.co/go/tools/simple/s1001
22912026-03-10T17:55:35.890Zhonnef.co/go/tools/simple/s1002
22922026-03-10T17:55:35.899Z[ 29%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/EdgeRing.cpp.o
22932026-03-10T17:55:35.903Zgithub.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway
22942026-03-10T17:55:35.906Zhonnef.co/go/tools/simple/s1003
22952026-03-10T17:55:35.918Zhonnef.co/go/tools/simple/s1004
22962026-03-10T17:55:35.925Zhonnef.co/go/tools/simple/s1005
22972026-03-10T17:55:35.928Zhonnef.co/go/tools/simple/s1006
22982026-03-10T17:55:35.954Z[ 29%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/GeometryGraph.cpp.o
22992026-03-10T17:55:35.957Zhonnef.co/go/tools/simple/s1007
23002026-03-10T17:55:35.960Zhonnef.co/go/tools/quickfix/qf1011
23012026-03-10T17:55:35.960Zhonnef.co/go/tools/simple/s1008
23022026-03-10T17:55:35.960Zhonnef.co/go/tools/simple/s1009
23032026-03-10T17:55:35.980Zhonnef.co/go/tools/simple/s1010
23042026-03-10T17:55:35.984Z[ 29%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/GraphComponent.cpp.o
23052026-03-10T17:55:35.987Zhonnef.co/go/tools/quickfix
23062026-03-10T17:55:35.990Zhonnef.co/go/tools/simple/s1011
23072026-03-10T17:55:35.993Zhonnef.co/go/tools/simple/s1012
23082026-03-10T17:55:35.993Zhonnef.co/go/tools/lintcmd/runner
23092026-03-10T17:55:36.000Zhonnef.co/go/tools/simple/s1016
23102026-03-10T17:55:36.006Zhonnef.co/go/tools/simple/s1017
23112026-03-10T17:55:36.023Zhonnef.co/go/tools/simple/s1018
23122026-03-10T17:55:36.026Zhonnef.co/go/tools/simple/s1019
23132026-03-10T17:55:36.030Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/spin.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/spin.c
23142026-03-10T17:55:36.037Zhonnef.co/go/tools/simple/s1020
23152026-03-10T17:55:36.048Zhonnef.co/go/tools/simple/s1021
23162026-03-10T17:55:36.057Zhonnef.co/go/tools/simple/s1023
23172026-03-10T17:55:36.065Zhonnef.co/go/tools/simple/s1024
23182026-03-10T17:55:36.076Z[ 82%] Building C object src/CMakeFiles/proj.dir/pj_initcache.c.o
23192026-03-10T17:55:36.079Zhonnef.co/go/tools/simple/s1025
23202026-03-10T17:55:36.080Zhonnef.co/go/tools/simple/s1028
23212026-03-10T17:55:36.083Z[ 29%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/Label.cpp.o
23222026-03-10T17:55:36.086Zhonnef.co/go/tools/simple/s1029
23232026-03-10T17:55:36.093Zhonnef.co/go/tools/simple/s1030
23242026-03-10T17:55:36.093Zhonnef.co/go/tools/lintcmd
23252026-03-10T17:55:36.098Zhonnef.co/go/tools/simple/s1031
23262026-03-10T17:55:36.122Zhonnef.co/go/tools/simple/s1032
23272026-03-10T17:55:36.125Zhonnef.co/go/tools/simple/s1033
23282026-03-10T17:55:36.128Zhonnef.co/go/tools/simple/s1034
23292026-03-10T17:55:36.148Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/tcache.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/tcache.c
23302026-03-10T17:55:36.156Z[ 83%] Building C object src/CMakeFiles/proj.dir/pj_inv.c.o
23312026-03-10T17:55:36.159Zhonnef.co/go/tools/simple/s1035
23322026-03-10T17:55:36.162Zhonnef.co/go/tools/simple/s1036
23332026-03-10T17:55:36.173Zhonnef.co/go/tools/simple/s1037
23342026-03-10T17:55:36.189Zhonnef.co/go/tools/simple/s1038
23352026-03-10T17:55:36.193Zhonnef.co/go/tools/simple/s1039
23362026-03-10T17:55:36.201Zhonnef.co/go/tools/simple/s1040
23372026-03-10T17:55:36.201Zhonnef.co/go/tools/analysis/callcheck
23382026-03-10T17:55:36.213Zhonnef.co/go/tools/staticcheck/sa1001
23392026-03-10T17:55:36.238Zhonnef.co/go/tools/staticcheck/sa1004
23402026-03-10T17:55:36.239Zhonnef.co/go/tools/staticcheck/sa1005
23412026-03-10T17:55:36.239Zhonnef.co/go/tools/staticcheck/sa1006
23422026-03-10T17:55:36.258Z[ 83%] Building C object src/CMakeFiles/proj.dir/pj_inv3d.c.o
23432026-03-10T17:55:36.261Zhonnef.co/go/tools/staticcheck/sa1008
23442026-03-10T17:55:36.265Zhonnef.co/go/tools/staticcheck/sa1012
23452026-03-10T17:55:36.271Zhonnef.co/go/tools/staticcheck/sa1013
23462026-03-10T17:55:36.274Zhonnef.co/go/tools/staticcheck/sa1015
23472026-03-10T17:55:36.279Zhonnef.co/go/tools/staticcheck/sa1000
23482026-03-10T17:55:36.282Zhonnef.co/go/tools/simple
23492026-03-10T17:55:36.282Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/ticker.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/ticker.c
23502026-03-10T17:55:36.289Zhonnef.co/go/tools/staticcheck/sa1002
23512026-03-10T17:55:36.295Zhonnef.co/go/tools/staticcheck/sa1003
23522026-03-10T17:55:36.311Zhonnef.co/go/tools/staticcheck/sa1007
23532026-03-10T17:55:36.321Zhonnef.co/go/tools/staticcheck/sa1010
23542026-03-10T17:55:36.321Z[ 84%] Building C object src/CMakeFiles/proj.dir/pj_latlong.c.o
23552026-03-10T17:55:36.327Zhonnef.co/go/tools/staticcheck/sa1011
23562026-03-10T17:55:36.330Z[ 31%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/Node.cpp.o
23572026-03-10T17:55:36.330Zhonnef.co/go/tools/staticcheck/sa1014
23582026-03-10T17:55:36.330Zhonnef.co/go/tools/staticcheck/sa1016
23592026-03-10T17:55:36.341Zhonnef.co/go/tools/staticcheck/sa1017
23602026-03-10T17:55:36.345Zhonnef.co/go/tools/staticcheck/sa1018
23612026-03-10T17:55:36.350Zhonnef.co/go/tools/staticcheck/sa1019
23622026-03-10T17:55:36.362Zhonnef.co/go/tools/staticcheck/sa1020
23632026-03-10T17:55:36.381Zhonnef.co/go/tools/staticcheck/sa1021
23642026-03-10T17:55:36.386Zhonnef.co/go/tools/staticcheck/sa1023
23652026-03-10T17:55:36.386Zhonnef.co/go/tools/staticcheck/sa1024
23662026-03-10T17:55:36.390Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/tsd.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/tsd.c
23672026-03-10T17:55:36.397Zhonnef.co/go/tools/staticcheck/sa1025
23682026-03-10T17:55:36.407Zhonnef.co/go/tools/staticcheck/sa1026
23692026-03-10T17:55:36.411Zhonnef.co/go/tools/staticcheck/sa1027
23702026-03-10T17:55:36.414Zhonnef.co/go/tools/staticcheck/sa1028
23712026-03-10T17:55:36.418Zhonnef.co/go/tools/staticcheck/sa1029
23722026-03-10T17:55:36.427Z[ 31%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/NodeFactory.cpp.o
23732026-03-10T17:55:36.438Zhonnef.co/go/tools/staticcheck/sa1030
23742026-03-10T17:55:36.447Zhonnef.co/go/tools/staticcheck/sa1031
23752026-03-10T17:55:36.459Zhonnef.co/go/tools/staticcheck/sa1032
23762026-03-10T17:55:36.470Zhonnef.co/go/tools/staticcheck/sa2000
23772026-03-10T17:55:36.476Zhonnef.co/go/tools/staticcheck/sa2001
23782026-03-10T17:55:36.476Zhonnef.co/go/tools/staticcheck/sa2002
23792026-03-10T17:55:36.479Z[ 84%] Building C object src/CMakeFiles/proj.dir/pj_list.c.o
23802026-03-10T17:55:36.479Zhonnef.co/go/tools/staticcheck/sa2003
23812026-03-10T17:55:36.487Zhonnef.co/go/tools/staticcheck/sa3000
23822026-03-10T17:55:36.494Zhonnef.co/go/tools/staticcheck/sa3001
23832026-03-10T17:55:36.509Zhonnef.co/go/tools/staticcheck/sa4000
23842026-03-10T17:55:36.513Zhonnef.co/go/tools/staticcheck/sa4001
23852026-03-10T17:55:36.513Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/util.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c
23862026-03-10T17:55:36.513Zhonnef.co/go/tools/staticcheck/sa4003
23872026-03-10T17:55:36.516Zhonnef.co/go/tools/staticcheck/sa4004
23882026-03-10T17:55:36.526Zhonnef.co/go/tools/staticcheck/sa4005
23892026-03-10T17:55:36.531Zhonnef.co/go/tools/staticcheck/sa4006
23902026-03-10T17:55:36.535Zhonnef.co/go/tools/staticcheck/sa4008
23912026-03-10T17:55:36.539Zhonnef.co/go/tools/staticcheck/sa4009
23922026-03-10T17:55:36.542Zhonnef.co/go/tools/staticcheck/sa4010
23932026-03-10T17:55:36.548Zhonnef.co/go/tools/staticcheck/sa4011
23942026-03-10T17:55:36.555Zhonnef.co/go/tools/staticcheck/sa4012
23952026-03-10T17:55:36.568Zhonnef.co/go/tools/staticcheck/sa4013
23962026-03-10T17:55:36.573Z[ 85%] Building C object src/CMakeFiles/proj.dir/pj_log.c.o
23972026-03-10T17:55:36.576Zhonnef.co/go/tools/staticcheck/sa4014
23982026-03-10T17:55:36.580Zhonnef.co/go/tools/staticcheck/sa4015
23992026-03-10T17:55:36.601Zhonnef.co/go/tools/staticcheck/sa4016
24002026-03-10T17:55:36.601Zhonnef.co/go/tools/staticcheck/sa4017
24012026-03-10T17:55:36.606Zhonnef.co/go/tools/staticcheck/sa4018
24022026-03-10T17:55:36.606Zhonnef.co/go/tools/staticcheck/sa4019
24032026-03-10T17:55:36.609Zhonnef.co/go/tools/staticcheck/sa4020
24042026-03-10T17:55:36.613Z/work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c: In function 'je_malloc_vsnprintf':
24052026-03-10T17:55:36.616Z/work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c:359:9: warning: case label value exceeds maximum value for type [-Wswitch-outside-range]
24062026-03-10T17:55:36.616Z 359 | case '?' | 0x80: \
24072026-03-10T17:55:36.616Z | ^~~~
24082026-03-10T17:55:36.616Z/work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c:569:33: note: in expansion of macro 'GET_ARG_NUMERIC'
24092026-03-10T17:55:36.616Z 569 | GET_ARG_NUMERIC(val, 'p');
24102026-03-10T17:55:36.616Z | ^~~~~~~~~~~~~~~
24112026-03-10T17:55:36.616Z/work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c:377:9: warning: case label value exceeds maximum value for type [-Wswitch-outside-range]
24122026-03-10T17:55:36.616Z 377 | case 'j' | 0x80: \
24132026-03-10T17:55:36.616Z | ^~~~
24142026-03-10T17:55:36.616Z/work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c:569:33: note: in expansion of macro 'GET_ARG_NUMERIC'
24152026-03-10T17:55:36.616Z 569 | GET_ARG_NUMERIC(val, 'p');
24162026-03-10T17:55:36.616Z | ^~~~~~~~~~~~~~~
24172026-03-10T17:55:36.616Z/work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c:365:9: warning: case label value exceeds maximum value for type [-Wswitch-outside-range]
24182026-03-10T17:55:36.616Z 365 | case 'l' | 0x80: \
24192026-03-10T17:55:36.616Z | ^~~~
24202026-03-10T17:55:36.616Z/work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c:569:33: note: in expansion of macro 'GET_ARG_NUMERIC'
24212026-03-10T17:55:36.616Z 569 | GET_ARG_NUMERIC(val, 'p');
24222026-03-10T17:55:36.616Z | ^~~~~~~~~~~~~~~
24232026-03-10T17:55:36.617Z/work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c:371:9: warning: case label value exceeds maximum value for type [-Wswitch-outside-range]
24242026-03-10T17:55:36.617Z 371 | case 'q' | 0x80: \
24252026-03-10T17:55:36.617Z | ^~~~
24262026-03-10T17:55:36.617Z/work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c:569:33: note: in expansion of macro 'GET_ARG_NUMERIC'
24272026-03-10T17:55:36.617Z 569 | GET_ARG_NUMERIC(val, 'p');
24282026-03-10T17:55:36.617Z | ^~~~~~~~~~~~~~~
24292026-03-10T17:55:36.617Z/work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c:386:9: warning: case label value exceeds maximum value for type [-Wswitch-outside-range]
24302026-03-10T17:55:36.617Z 386 | case 'z' | 0x80: \
24312026-03-10T17:55:36.617Z | ^~~~
24322026-03-10T17:55:36.617Z/work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c:569:33: note: in expansion of macro 'GET_ARG_NUMERIC'
24332026-03-10T17:55:36.617Z 569 | GET_ARG_NUMERIC(val, 'p');
24342026-03-10T17:55:36.617Z | ^~~~~~~~~~~~~~~
24352026-03-10T17:55:36.628Zhonnef.co/go/tools/staticcheck/sa4021
24362026-03-10T17:55:36.631Zhonnef.co/go/tools/staticcheck/sa4022
24372026-03-10T17:55:36.638Zhonnef.co/go/tools/staticcheck/sa4023
24382026-03-10T17:55:36.638Zhonnef.co/go/tools/staticcheck/sa4024
24392026-03-10T17:55:36.641Zhonnef.co/go/tools/staticcheck/sa4025
24402026-03-10T17:55:36.644Zhonnef.co/go/tools/staticcheck/sa4026
24412026-03-10T17:55:36.647Zhonnef.co/go/tools/staticcheck/sa4027
24422026-03-10T17:55:36.667Zhonnef.co/go/tools/staticcheck/sa4028
24432026-03-10T17:55:36.670Zhonnef.co/go/tools/staticcheck/sa4029
24442026-03-10T17:55:36.677Zhonnef.co/go/tools/staticcheck/sa4030
24452026-03-10T17:55:36.677Zhonnef.co/go/tools/staticcheck/sa4032
24462026-03-10T17:55:36.677Zhonnef.co/go/tools/staticcheck/sa4031
24472026-03-10T17:55:36.682Zhonnef.co/go/tools/staticcheck/sa5000
24482026-03-10T17:55:36.698Zhonnef.co/go/tools/staticcheck/sa5001
24492026-03-10T17:55:36.706Zhonnef.co/go/tools/staticcheck/sa5002
24502026-03-10T17:55:36.709Zhonnef.co/go/tools/staticcheck/sa5003
24512026-03-10T17:55:36.713Zhonnef.co/go/tools/staticcheck/sa5004
24522026-03-10T17:55:36.720Z[ 85%] Building C object src/CMakeFiles/proj.dir/pj_malloc.c.o
24532026-03-10T17:55:36.725Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/witness.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/witness.c
24542026-03-10T17:55:36.729Zhonnef.co/go/tools/staticcheck/sa5005
24552026-03-10T17:55:36.732Zhonnef.co/go/tools/staticcheck/sa5007
24562026-03-10T17:55:36.736Zhonnef.co/go/tools/staticcheck/sa5008
24572026-03-10T17:55:36.736Zhonnef.co/go/tools/staticcheck/sa5009
24582026-03-10T17:55:36.740Zhonnef.co/go/tools/staticcheck/sa5010
24592026-03-10T17:55:36.740Zhonnef.co/go/tools/staticcheck/sa5011
24602026-03-10T17:55:36.761Zhonnef.co/go/tools/staticcheck/sa5012
24612026-03-10T17:55:36.764Zhonnef.co/go/tools/staticcheck/sa6000
24622026-03-10T17:55:36.764Zhonnef.co/go/tools/staticcheck/sa6001
24632026-03-10T17:55:36.775Zhonnef.co/go/tools/staticcheck/sa6002
24642026-03-10T17:55:36.780Zhonnef.co/go/tools/staticcheck/sa6003
24652026-03-10T17:55:36.803Z[ 86%] Building C object src/CMakeFiles/proj.dir/pj_mlfn.c.o
24662026-03-10T17:55:36.806Zhonnef.co/go/tools/staticcheck/sa6005
24672026-03-10T17:55:36.812Zhonnef.co/go/tools/staticcheck/sa6006
24682026-03-10T17:55:36.815Zhonnef.co/go/tools/staticcheck/sa9001
24692026-03-10T17:55:36.815Zhonnef.co/go/tools/staticcheck/sa9002
24702026-03-10T17:55:36.815Z[ 31%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/NodeMap.cpp.o
24712026-03-10T17:55:36.826Zhonnef.co/go/tools/staticcheck/sa9003
24722026-03-10T17:55:36.829Zhonnef.co/go/tools/staticcheck/sa9004
24732026-03-10T17:55:36.829Zhonnef.co/go/tools/staticcheck/sa9005
24742026-03-10T17:55:36.829Zhonnef.co/go/tools/staticcheck/sa9006
24752026-03-10T17:55:36.837Zhonnef.co/go/tools/staticcheck/sa9007
24762026-03-10T17:55:36.845Zhonnef.co/go/tools/staticcheck/sa9008
24772026-03-10T17:55:36.845Z[ 31%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/PlanarGraph.cpp.o
24782026-03-10T17:55:36.845Zhonnef.co/go/tools/staticcheck/sa9009
24792026-03-10T17:55:36.848Zhonnef.co/go/tools/stylecheck/st1000
24802026-03-10T17:55:36.852Z[ 88%] Building C object src/CMakeFiles/proj.dir/pj_msfn.c.o
24812026-03-10T17:55:36.855Zhonnef.co/go/tools/stylecheck/st1001
24822026-03-10T17:55:36.866Zhonnef.co/go/tools/stylecheck/st1003
24832026-03-10T17:55:36.869Zhonnef.co/go/tools/stylecheck/st1005
24842026-03-10T17:55:36.869Zhonnef.co/go/tools/stylecheck/st1006
24852026-03-10T17:55:36.875Zhonnef.co/go/tools/stylecheck/st1008
24862026-03-10T17:55:36.879Zhonnef.co/go/tools/stylecheck/st1011
24872026-03-10T17:55:36.882Z[ 88%] Building C object src/CMakeFiles/proj.dir/pj_mutex.c.o
24882026-03-10T17:55:36.886Zhonnef.co/go/tools/stylecheck/st1012
24892026-03-10T17:55:36.893Zhonnef.co/go/tools/stylecheck/st1013
24902026-03-10T17:55:36.899Zhonnef.co/go/tools/stylecheck/st1015
24912026-03-10T17:55:36.903Zhonnef.co/go/tools/staticcheck
24922026-03-10T17:55:36.912Z[ 31%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/Position.cpp.o
24932026-03-10T17:55:36.915Zhonnef.co/go/tools/stylecheck/st1016
24942026-03-10T17:55:36.918Zar crus lib/libjemalloc.a src/jemalloc.o src/arena.o src/atomic.o src/base.o src/bitmap.o src/chunk.o src/chunk_dss.o src/chunk_mmap.o src/ckh.o src/ctl.o src/extent.o src/hash.o src/huge.o src/mb.o src/mutex.o src/nstime.o src/pages.o src/prng.o src/prof.o src/quarantine.o src/rtree.o src/stats.o src/spin.o src/tcache.o src/ticker.o src/tsd.o src/util.o src/witness.o
24952026-03-10T17:55:36.921Zar: `u' modifier ignored since `D' is the default (see `U')
24962026-03-10T17:55:36.921Zhonnef.co/go/tools/stylecheck/st1017
24972026-03-10T17:55:36.924Zhonnef.co/go/tools/stylecheck/st1018
24982026-03-10T17:55:36.927Z[ 89%] Building C object src/CMakeFiles/proj.dir/pj_open_lib.c.o
24992026-03-10T17:55:36.927Zhonnef.co/go/tools/stylecheck/st1019
25002026-03-10T17:55:36.927Zhonnef.co/go/tools/stylecheck/st1020
25012026-03-10T17:55:36.944Zar crus lib/libjemalloc_pic.a src/jemalloc.pic.o src/arena.pic.o src/atomic.pic.o src/base.pic.o src/bitmap.pic.o src/chunk.pic.o src/chunk_dss.pic.o src/chunk_mmap.pic.o src/ckh.pic.o src/ctl.pic.o src/extent.pic.o src/hash.pic.o src/huge.pic.o src/mb.pic.o src/mutex.pic.o src/nstime.pic.o src/pages.pic.o src/prng.pic.o src/prof.pic.o src/quarantine.pic.o src/rtree.pic.o src/stats.pic.o src/spin.pic.o src/tcache.pic.o src/ticker.pic.o src/tsd.pic.o src/util.pic.o src/witness.pic.o
25022026-03-10T17:55:36.944Zhonnef.co/go/tools/stylecheck/st1021
25032026-03-10T17:55:36.948Z[ 89%] Building C object src/CMakeFiles/proj.dir/pj_param.c.o
25042026-03-10T17:55:36.951Zhonnef.co/go/tools/stylecheck/st1022
25052026-03-10T17:55:36.951Zhonnef.co/go/tools/stylecheck/st1023
25062026-03-10T17:55:36.964Zar: `u' modifier ignored since `D' is the default (see `U')
25072026-03-10T17:55:36.971Zhonnef.co/go/tools/stylecheck
25082026-03-10T17:55:36.975Z[ 89%] Building C object src/CMakeFiles/proj.dir/pj_pr_list.c.o
25092026-03-10T17:55:36.975Z[ 32%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/Quadrant.cpp.o
25102026-03-10T17:55:36.978Z[ 90%] Building C object src/CMakeFiles/proj.dir/pj_phi2.c.o
25112026-03-10T17:55:36.985Zhonnef.co/go/tools/cmd/staticcheck
25122026-03-10T17:55:36.990Z[ 91%] Building C object src/CMakeFiles/proj.dir/pj_qsfn.c.o
25132026-03-10T17:55:37.012Z[ 91%] Building C object src/CMakeFiles/proj.dir/pj_release.c.o
25142026-03-10T17:55:37.012Z[ 32%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/TopologyLocation.cpp.o
25152026-03-10T17:55:37.015Z[ 92%] Building C object src/CMakeFiles/proj.dir/pj_run_selftests.c.o
25162026-03-10T17:55:37.027Z[ 92%] Building C object src/CMakeFiles/proj.dir/pj_strerrno.c.o
25172026-03-10T17:55:37.036Z[ 93%] Building C object src/CMakeFiles/proj.dir/pj_transform.c.o
25182026-03-10T17:55:37.058Z[ 32%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/index/MonotoneChainEdge.cpp.o
25192026-03-10T17:55:37.091Z[ 94%] Building C object src/CMakeFiles/proj.dir/pj_tsfn.c.o
25202026-03-10T17:55:37.113Z[ 94%] Building C object src/CMakeFiles/proj.dir/pj_units.c.o
25212026-03-10T17:55:37.130Z[ 32%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/index/MonotoneChainIndexer.cpp.o
25222026-03-10T17:55:37.135Z[ 95%] Building C object src/CMakeFiles/proj.dir/pj_utils.c.o
25232026-03-10T17:55:37.143Z[ 32%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/index/SegmentIntersector.cpp.o
25242026-03-10T17:55:37.169Z[ 34%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/index/SimpleEdgeSetIntersector.cpp.o
25252026-03-10T17:55:37.180Z[ 95%] Building C object src/CMakeFiles/proj.dir/pj_zpoly1.c.o
25262026-03-10T17:55:37.180Z[ 96%] Building C object src/CMakeFiles/proj.dir/proj_mdist.c.o
25272026-03-10T17:55:37.205Z[ 96%] Building C object src/CMakeFiles/proj.dir/proj_rouss.c.o
25282026-03-10T17:55:37.229Z[ 97%] Building C object src/CMakeFiles/proj.dir/rtodms.c.o
25292026-03-10T17:55:37.248Z[ 97%] Building C object src/CMakeFiles/proj.dir/vector1.c.o
25302026-03-10T17:55:37.288Z[ 98%] Building C object src/CMakeFiles/proj.dir/pj_strtod.c.o
25312026-03-10T17:55:37.322Z[ 34%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/index/SimpleMCSweepLineIntersector.cpp.o
25322026-03-10T17:55:37.322Z[ 34%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/index/SimpleSweepLineIntersector.cpp.o
25332026-03-10T17:55:37.358Z[100%] Linking C static library ../lib/libproj.a
25342026-03-10T17:55:37.358Z[ 34%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/index/SweepLineEvent.cpp.o
25352026-03-10T17:55:37.369Z[ 34%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/index/SweepLineSegment.cpp.o
25362026-03-10T17:55:37.375Z[ 36%] Building CXX object CMakeFiles/geos.dir/src/index/bintree/Bintree.cpp.o
25372026-03-10T17:55:37.376Z[ 36%] Building CXX object CMakeFiles/geos.dir/src/index/bintree/Interval.cpp.o
25382026-03-10T17:55:37.398Z[ 36%] Building CXX object CMakeFiles/geos.dir/src/index/bintree/Key.cpp.o
25392026-03-10T17:55:37.423Z[ 36%] Building CXX object CMakeFiles/geos.dir/src/index/bintree/Node.cpp.o
25402026-03-10T17:55:37.434Ztouch bin/.bootstrap
25412026-03-10T17:55:37.434Zfind ./pkg -name node_modules -prune -o -type f -name '*.pb.go' -exec rm {} +
25422026-03-10T17:55:37.445Zset -e; for dir in ./pkg/acceptance/cluster/ ./pkg/blobs/blobspb/ ./pkg/build/ ./pkg/clusterversion/ ./pkg/config/ ./pkg/config/zonepb/ ./pkg/geo/geoindex/ ./pkg/geo/geopb/ ./pkg/gossip/ ./pkg/jobs/jobspb/ ./pkg/kv/kvnemesis/ ./pkg/kv/kvserver/ ./pkg/kv/kvserver/closedts/ctpb/ ./pkg/kv/kvserver/concurrency/lock/ ./pkg/kv/kvserver/concurrency/poison/ ./pkg/kv/kvserver/kvserverpb/ ./pkg/kv/kvserver/liveness/livenesspb/ ./pkg/kv/kvserver/loqrecovery/loqrecoverypb/ ./pkg/kv/kvserver/protectedts/ptpb/ ./pkg/kv/kvserver/protectedts/ptstorage/ ./pkg/kv/kvserver/readsummary/rspb/ ./pkg/roachpb/ ./pkg/rpc/ ./pkg/server/diagnostics/diagnosticspb/ ./pkg/server/serverpb/ ./pkg/server/status/statuspb/ ./pkg/settings/ ./pkg/sql/catalog/catpb/ ./pkg/sql/catalog/descpb/ ./pkg/sql/contentionpb/ ./pkg/sql/execinfrapb/ ./pkg/sql/inverted/ ./pkg/sql/lex/ ./pkg/sql/pgwire/pgerror/ ./pkg/sql/protoreflect/test/ ./pkg/sql/rowenc/rowencpb/ ./pkg/sql/schemachanger/scpb/ ./pkg/sql/sessiondatapb/ ./pkg/sql/sqlstats/persistedsqlstats/ ./pkg/sql/stats/ ./pkg/sql/types/ ./pkg/startupmigrations/leasemanager/ ./pkg/storage/enginepb/ ./pkg/testutils/grpcutils/ ./pkg/ts/catalog/ ./pkg/ts/tspb/ ./pkg/util/ ./pkg/util/duration/ ./pkg/util/hlc/ ./pkg/util/log/eventpb/ ./pkg/util/log/logpb/ ./pkg/util/metric/ ./pkg/util/optional/ ./pkg/util/protoutil/ ./pkg/util/timeutil/pgdate/ ./pkg/util/tracing/tracingpb/ ./pkg/util/tracing/tracingservicepb/; do \
25432026-03-10T17:55:37.445Z buf protoc -Ipkg -I./vendor/github.com/gogo/protobuf -I./vendor/go.etcd.io -I./vendor/github.com/prometheus/client_model -I./vendor/github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis -I./vendor/github.com/cockroachdb/errors --gogoroach_out=Mgoogle/api/annotations.proto=google.golang.org/genproto/googleapis/api/annotations,Mgoogle/protobuf/timestamp.proto=github.com/gogo/protobuf/types,Mgoogle/protobuf/any.proto=github.com/gogo/protobuf/types,Mgoogle/protobuf/duration.proto=github.com/gogo/protobuf/types,plugins=grpc,import_prefix=github.com/cockroachdb/cockroach/pkg/:./pkg $dir/*.proto; \
25442026-03-10T17:55:37.445Zdone
25452026-03-10T17:55:37.460Z[ 36%] Building CXX object CMakeFiles/geos.dir/src/index/bintree/NodeBase.cpp.o
25462026-03-10T17:55:37.460Z[ 37%] Building CXX object CMakeFiles/geos.dir/src/index/bintree/Root.cpp.o
25472026-03-10T17:55:37.470Z[ 37%] Building CXX object CMakeFiles/geos.dir/src/index/chain/MonotoneChain.cpp.o
25482026-03-10T17:55:37.505Z[ 37%] Building CXX object CMakeFiles/geos.dir/src/index/chain/MonotoneChainBuilder.cpp.o
25492026-03-10T17:55:37.522Z[ 37%] Building CXX object CMakeFiles/geos.dir/src/index/chain/MonotoneChainOverlapAction.cpp.o
25502026-03-10T17:55:37.571Z[ 37%] Building CXX object CMakeFiles/geos.dir/src/index/chain/MonotoneChainSelectAction.cpp.o
25512026-03-10T17:55:37.584Z[ 39%] Building CXX object CMakeFiles/geos.dir/src/index/intervalrtree/IntervalRTreeBranchNode.cpp.o
25522026-03-10T17:55:37.598Z[ 39%] Building CXX object CMakeFiles/geos.dir/src/index/intervalrtree/IntervalRTreeLeafNode.cpp.o
25532026-03-10T17:55:37.644Z[ 39%] Building CXX object CMakeFiles/geos.dir/src/index/intervalrtree/SortedPackedIntervalRTree.cpp.o
25542026-03-10T17:55:37.666Z[ 39%] Building CXX object CMakeFiles/geos.dir/src/index/quadtree/DoubleBits.cpp.o
25552026-03-10T17:55:37.679Zfind ./pkg -name node_modules -prune -o -type f -name '*.pb.gw.go' -exec rm {} +
25562026-03-10T17:55:37.679Z[ 39%] Building CXX object CMakeFiles/geos.dir/src/index/quadtree/IntervalSize.cpp.o
25572026-03-10T17:55:37.679Zbuf protoc -Ipkg -I./vendor/github.com/gogo/protobuf -I./vendor/github.com/cockroachdb/errors -I./vendor/go.etcd.io -I./vendor/github.com/prometheus/client_model -I./vendor/github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis --grpc-gateway_out=logtostderr=true,request_context=true:./pkg ./pkg/server/serverpb/admin.proto ./pkg/server/serverpb/status.proto ./pkg/server/serverpb/authentication.proto
25582026-03-10T17:55:37.698Z(cd pkg/roachprod/vm/aws && go generate)
25592026-03-10T17:55:37.738Z[ 40%] Building CXX object CMakeFiles/geos.dir/src/index/quadtree/Key.cpp.o
25602026-03-10T17:55:37.745Z[ 40%] Building CXX object CMakeFiles/geos.dir/src/index/quadtree/Node.cpp.o
25612026-03-10T17:55:37.788Z[ 40%] Building CXX object CMakeFiles/geos.dir/src/index/quadtree/Quadtree.cpp.o
25622026-03-10T17:55:37.801Z[ 40%] Building CXX object CMakeFiles/geos.dir/src/index/quadtree/NodeBase.cpp.o
25632026-03-10T17:55:37.834Z(cd pkg/security/securitytest && go generate)
25642026-03-10T17:55:37.906Z[ 40%] Building CXX object CMakeFiles/geos.dir/src/index/quadtree/Root.cpp.o
25652026-03-10T17:55:37.916Z[ 42%] Building CXX object CMakeFiles/geos.dir/src/index/strtree/AbstractSTRtree.cpp.o
25662026-03-10T17:55:37.947Zbuf protoc -Ipkg -I./vendor/github.com/gogo/protobuf -I./vendor/github.com/cockroachdb/errors -I./vendor/go.etcd.io -I./vendor/github.com/prometheus/client_model -I./vendor/github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis --grpc-gateway_out=logtostderr=true,request_context=true:./pkg ./pkg/ts/tspb/timeseries.proto
25672026-03-10T17:55:38.007Z[ 42%] Building CXX object CMakeFiles/geos.dir/src/index/strtree/BoundablePair.cpp.o
25682026-03-10T17:55:38.014Z[ 42%] Building CXX object CMakeFiles/geos.dir/src/index/strtree/EnvelopeUtil.cpp.o
25692026-03-10T17:55:38.019Z[ 42%] Building CXX object CMakeFiles/geos.dir/src/index/strtree/GeometryItemDistance.cpp.o
25702026-03-10T17:55:38.024Zgofmt -s -w ./pkg/server/serverpb/admin.pb.gw.go ./pkg/server/serverpb/status.pb.gw.go ./pkg/server/serverpb/authentication.pb.gw.go ./pkg/ts/tspb/timeseries.pb.gw.go
25712026-03-10T17:55:38.075Zgoimports -w ./pkg/server/serverpb/admin.pb.gw.go ./pkg/server/serverpb/status.pb.gw.go ./pkg/server/serverpb/authentication.pb.gw.go ./pkg/ts/tspb/timeseries.pb.gw.go
25722026-03-10T17:55:38.100Z[ 42%] Building CXX object CMakeFiles/geos.dir/src/index/strtree/Interval.cpp.o
25732026-03-10T17:55:38.128Ztouch bin/.gw_protobuf_sources
25742026-03-10T17:55:38.132Zset -euo pipefail; \
25752026-03-10T17:55:38.132Z ret=$(cd pkg/sql/parser/gen && goyacc -p sql -o sql.go.tmp sql-gen.y); \
25762026-03-10T17:55:38.132Z if expr "$ret" : ".*conflicts" >/dev/null; then \
25772026-03-10T17:55:38.132Z echo "$ret"; exit 1; \
25782026-03-10T17:55:38.132Z fi
25792026-03-10T17:55:38.136Z[ 44%] Building CXX object CMakeFiles/geos.dir/src/index/strtree/SIRtree.cpp.o
25802026-03-10T17:55:38.153Z[ 44%] Building CXX object CMakeFiles/geos.dir/src/index/strtree/STRtree.cpp.o
25812026-03-10T17:55:38.182Z[ 44%] Building CXX object CMakeFiles/geos.dir/src/index/sweepline/SweepLineEvent.cpp.o
25822026-03-10T17:55:38.192Z[ 44%] Building CXX object CMakeFiles/geos.dir/src/index/sweepline/SweepLineIndex.cpp.o
25832026-03-10T17:55:38.201Z[ 44%] Building CXX object CMakeFiles/geos.dir/src/index/sweepline/SweepLineInterval.cpp.o
25842026-03-10T17:55:38.206Z[ 45%] Building CXX object CMakeFiles/geos.dir/src/inlines.cpp.o
25852026-03-10T17:55:38.215Z[ 45%] Building CXX object CMakeFiles/geos.dir/src/io/ByteOrderDataInStream.cpp.o
25862026-03-10T17:55:38.226Z[ 45%] Building CXX object CMakeFiles/geos.dir/src/io/ByteOrderValues.cpp.o
25872026-03-10T17:55:38.244Z[ 45%] Building CXX object CMakeFiles/geos.dir/src/io/CLocalizer.cpp.o
25882026-03-10T17:55:38.326Z[ 45%] Building CXX object CMakeFiles/geos.dir/src/io/ParseException.cpp.o
25892026-03-10T17:55:38.331Zmv -f pkg/sql/parser/helpmap_test.go.tmp pkg/sql/parser/helpmap_test.go
25902026-03-10T17:55:38.335Zgofmt -s -w pkg/sql/parser/helpmap_test.go
25912026-03-10T17:55:38.340Zawk -f pkg/sql/parser/help.awk < pkg/sql/parser/sql.y > pkg/sql/parser/help_messages.go.tmp || rm pkg/sql/parser/help_messages.go.tmp
25922026-03-10T17:55:38.345Zmv -f pkg/sql/parser/help_messages.go.tmp pkg/sql/parser/help_messages.go
25932026-03-10T17:55:38.356Zgofmt -s -w pkg/sql/parser/help_messages.go
25942026-03-10T17:55:38.375Zgo run -mod=vendor -tags all-keywords pkg/sql/lexbase/allkeywords/main.go < pkg/sql/parser/sql.y > pkg/sql/lexbase/keywords.go.tmp || rm pkg/sql/lexbase/keywords.go.tmp
25952026-03-10T17:55:38.395Zawk -f pkg/sql/parser/reserved_keywords.awk < pkg/sql/parser/sql.y > pkg/sql/lexbase/reserved_keywords.go.tmp || rm pkg/sql/lexbase/reserved_keywords.go.tmp
25962026-03-10T17:55:38.400Zmv -f pkg/sql/lexbase/reserved_keywords.go.tmp pkg/sql/lexbase/reserved_keywords.go
25972026-03-10T17:55:38.400Zgofmt -s -w pkg/sql/lexbase/reserved_keywords.go
25982026-03-10T17:55:38.400Zstringer -output=pkg/sql/opt/rule_name_string.go -type=RuleName pkg/sql/opt/rule_name.go pkg/sql/opt/rule_name.og.go
25992026-03-10T17:55:38.430Z[ 47%] Building CXX object CMakeFiles/geos.dir/src/io/StringTokenizer.cpp.o
26002026-03-10T17:55:38.496Z[ 47%] Building CXX object CMakeFiles/geos.dir/src/io/Unload.cpp.o
26012026-03-10T17:55:38.506Z[ 47%] Building CXX object CMakeFiles/geos.dir/src/io/WKBReader.cpp.o
26022026-03-10T17:55:38.533Z[ 47%] Building CXX object CMakeFiles/geos.dir/src/io/WKBWriter.cpp.o
26032026-03-10T17:55:38.603Zmv -f pkg/sql/lexbase/keywords.go.tmp pkg/sql/lexbase/keywords.go
26042026-03-10T17:55:38.613Zgofmt -s -w pkg/sql/lexbase/keywords.go
26052026-03-10T17:55:38.620Z[ 47%] Building CXX object CMakeFiles/geos.dir/src/io/WKTReader.cpp.o
26062026-03-10T17:55:38.626Z[ 49%] Building CXX object CMakeFiles/geos.dir/src/io/WKTWriter.cpp.o
26072026-03-10T17:55:38.725Z[ 49%] Building CXX object CMakeFiles/geos.dir/src/io/Writer.cpp.o
26082026-03-10T17:55:38.812Z[ 49%] Building CXX object CMakeFiles/geos.dir/src/linearref/ExtractLineByLocation.cpp.o
26092026-03-10T17:55:38.833Z[ 49%] Building CXX object CMakeFiles/geos.dir/src/linearref/LengthIndexOfPoint.cpp.o
26102026-03-10T17:55:38.837Z[ 49%] Building CXX object CMakeFiles/geos.dir/src/linearref/LengthIndexedLine.cpp.o
26112026-03-10T17:55:39.050Z[ 50%] Building CXX object CMakeFiles/geos.dir/src/linearref/LengthLocationMap.cpp.o
26122026-03-10T17:55:39.073Z[ 50%] Building CXX object CMakeFiles/geos.dir/src/linearref/LinearGeometryBuilder.cpp.o
26132026-03-10T17:55:39.086Z[ 50%] Building CXX object CMakeFiles/geos.dir/src/linearref/LinearLocation.cpp.o
26142026-03-10T17:55:39.103Z[ 50%] Building CXX object CMakeFiles/geos.dir/src/linearref/LinearIterator.cpp.o
26152026-03-10T17:55:39.120Z[ 50%] Building CXX object CMakeFiles/geos.dir/src/linearref/LocationIndexOfLine.cpp.o
26162026-03-10T17:55:39.213Z[ 52%] Building CXX object CMakeFiles/geos.dir/src/linearref/LocationIndexOfPoint.cpp.o
26172026-03-10T17:55:39.269Z[ 52%] Building CXX object CMakeFiles/geos.dir/src/noding/BasicSegmentString.cpp.o
26182026-03-10T17:55:39.345Z[ 52%] Building CXX object CMakeFiles/geos.dir/src/noding/FastNodingValidator.cpp.o
26192026-03-10T17:55:39.350Z[ 52%] Building CXX object CMakeFiles/geos.dir/src/noding/FastSegmentSetIntersectionFinder.cpp.o
26202026-03-10T17:55:39.378Z[ 52%] Building CXX object CMakeFiles/geos.dir/src/noding/GeometryNoder.cpp.o
26212026-03-10T17:55:39.413Z[ 54%] Building CXX object CMakeFiles/geos.dir/src/noding/IntersectionAdder.cpp.o
26222026-03-10T17:55:39.453Z[ 54%] Building CXX object CMakeFiles/geos.dir/src/noding/IntersectionFinderAdder.cpp.o
26232026-03-10T17:55:39.453Z[ 54%] Building CXX object CMakeFiles/geos.dir/src/noding/IteratedNoder.cpp.o
26242026-03-10T17:55:39.578Z[ 54%] Building CXX object CMakeFiles/geos.dir/src/noding/MCIndexNoder.cpp.o
26252026-03-10T17:55:39.599Z[ 54%] Building CXX object CMakeFiles/geos.dir/src/noding/MCIndexSegmentSetMutualIntersector.cpp.o
26262026-03-10T17:55:39.622Z[ 55%] Building CXX object CMakeFiles/geos.dir/src/noding/NodedSegmentString.cpp.o
26272026-03-10T17:55:39.667Z[ 55%] Building CXX object CMakeFiles/geos.dir/src/noding/NodingIntersectionFinder.cpp.o
26282026-03-10T17:55:39.751Z[ 55%] Building CXX object CMakeFiles/geos.dir/src/noding/NodingValidator.cpp.o
26292026-03-10T17:55:39.767Z[ 55%] Building CXX object CMakeFiles/geos.dir/src/noding/Octant.cpp.o
26302026-03-10T17:55:39.857Z[ 55%] Building CXX object CMakeFiles/geos.dir/src/noding/OrientedCoordinateArray.cpp.o
26312026-03-10T17:55:39.865Z[ 57%] Building CXX object CMakeFiles/geos.dir/src/noding/ScaledNoder.cpp.o
26322026-03-10T17:55:39.891Z[ 57%] Building CXX object CMakeFiles/geos.dir/src/noding/SegmentIntersectionDetector.cpp.o
26332026-03-10T17:55:39.905Z[ 57%] Building CXX object CMakeFiles/geos.dir/src/noding/SegmentNode.cpp.o
26342026-03-10T17:55:39.982Z[ 57%] Building CXX object CMakeFiles/geos.dir/src/noding/SegmentNodeList.cpp.o
26352026-03-10T17:55:39.998Z[ 57%] Building CXX object CMakeFiles/geos.dir/src/noding/SegmentString.cpp.o
26362026-03-10T17:55:40.021Z[ 59%] Building CXX object CMakeFiles/geos.dir/src/noding/SegmentStringUtil.cpp.o
26372026-03-10T17:55:40.059Z[ 59%] Building CXX object CMakeFiles/geos.dir/src/noding/SimpleNoder.cpp.o
26382026-03-10T17:55:40.094Z[ 59%] Building CXX object CMakeFiles/geos.dir/src/noding/snapround/HotPixel.cpp.o
26392026-03-10T17:55:40.111Z[ 59%] Building CXX object CMakeFiles/geos.dir/src/noding/snapround/MCIndexPointSnapper.cpp.o
26402026-03-10T17:55:40.177Z[ 59%] Building CXX object CMakeFiles/geos.dir/src/noding/snapround/MCIndexSnapRounder.cpp.o
26412026-03-10T17:55:40.254Z[ 60%] Building CXX object CMakeFiles/geos.dir/src/noding/snapround/SimpleSnapRounder.cpp.o
26422026-03-10T17:55:40.269Z(echo "// Code generated by goyacc. DO NOT EDIT."; \
26432026-03-10T17:55:40.270Z echo "// GENERATED FILE DO NOT EDIT"; \
26442026-03-10T17:55:40.270Z cat pkg/sql/parser/gen/sql.go.tmp | \
26452026-03-10T17:55:40.270Z sed -E 's/^const ([A-Z][_A-Z0-9]*) =.*$/const \1 = lexbase.\1/g') > pkg/sql/parser/sql.go.tmp || rm pkg/sql/parser/sql.go.tmp
26462026-03-10T17:55:40.270Z[ 60%] Building CXX object CMakeFiles/geos.dir/src/operation/GeometryGraphOperation.cpp.o
26472026-03-10T17:55:40.303Zmv -f pkg/sql/parser/sql.go.tmp pkg/sql/parser/sql.go
26482026-03-10T17:55:40.331Zgoimports -w pkg/sql/parser/sql.go
26492026-03-10T17:55:40.331Z[ 60%] Building CXX object CMakeFiles/geos.dir/src/operation/IsSimpleOp.cpp.o
26502026-03-10T17:55:40.331Z(echo "// Code generated by make. DO NOT EDIT."; \
26512026-03-10T17:55:40.331Z echo "// GENERATED FILE DO NOT EDIT"; \
26522026-03-10T17:55:40.331Z echo; \
26532026-03-10T17:55:40.331Z echo "package lexbase"; \
26542026-03-10T17:55:40.332Z echo; \
26552026-03-10T17:55:40.332Z grep '^const [A-Z][_A-Z0-9]* ' pkg/sql/parser/gen/sql.go.tmp) > pkg/sql/lexbase/tokens.go.tmp || rm pkg/sql/lexbase/tokens.go.tmp
26562026-03-10T17:55:40.332Zmv -f pkg/sql/lexbase/tokens.go.tmp pkg/sql/lexbase/tokens.go
26572026-03-10T17:55:40.332Z[ 60%] Building CXX object CMakeFiles/geos.dir/src/operation/buffer/BufferBuilder.cpp.o
26582026-03-10T17:55:40.400Z[ 60%] Building CXX object CMakeFiles/geos.dir/src/operation/buffer/BufferInputLineSimplifier.cpp.o
26592026-03-10T17:55:40.514Z[ 62%] Building CXX object CMakeFiles/geos.dir/src/operation/buffer/BufferOp.cpp.o
26602026-03-10T17:55:40.551Z[ 62%] Building CXX object CMakeFiles/geos.dir/src/operation/buffer/BufferParameters.cpp.o
26612026-03-10T17:55:40.559Z[ 62%] Building CXX object CMakeFiles/geos.dir/src/operation/buffer/BufferSubgraph.cpp.o
26622026-03-10T17:55:40.559Z[ 62%] Building CXX object CMakeFiles/geos.dir/src/operation/buffer/OffsetCurveBuilder.cpp.o
26632026-03-10T17:55:40.599Z[ 62%] Building CXX object CMakeFiles/geos.dir/src/operation/buffer/OffsetCurveSetBuilder.cpp.o
26642026-03-10T17:55:40.841Z[ 63%] Building CXX object CMakeFiles/geos.dir/src/operation/buffer/OffsetSegmentGenerator.cpp.o
26652026-03-10T17:55:40.898Z[ 63%] Building CXX object CMakeFiles/geos.dir/src/operation/buffer/RightmostEdgeFinder.cpp.o
26662026-03-10T17:55:40.911Z[ 63%] Building CXX object CMakeFiles/geos.dir/src/operation/buffer/SubgraphDepthLocater.cpp.o
26672026-03-10T17:55:40.912Z[ 63%] Building CXX object CMakeFiles/geos.dir/src/operation/distance/ConnectedElementLocationFilter.cpp.o
26682026-03-10T17:55:40.961Z[ 63%] Building CXX object CMakeFiles/geos.dir/src/operation/distance/ConnectedElementPointFilter.cpp.o
26692026-03-10T17:55:40.982Z[ 65%] Building CXX object CMakeFiles/geos.dir/src/operation/distance/DistanceOp.cpp.o
26702026-03-10T17:55:41.140Z[ 65%] Building CXX object CMakeFiles/geos.dir/src/operation/distance/FacetSequence.cpp.o
26712026-03-10T17:55:41.160Z[ 65%] Building CXX object CMakeFiles/geos.dir/src/operation/distance/FacetSequenceTreeBuilder.cpp.o
26722026-03-10T17:55:41.160Z[ 65%] Building CXX object CMakeFiles/geos.dir/src/operation/distance/GeometryLocation.cpp.o
26732026-03-10T17:55:41.160Z[ 65%] Building CXX object CMakeFiles/geos.dir/src/operation/distance/IndexedFacetDistance.cpp.o
26742026-03-10T17:55:41.160Z[ 67%] Building CXX object CMakeFiles/geos.dir/src/operation/intersection/Rectangle.cpp.o
26752026-03-10T17:55:41.198Z[ 67%] Building CXX object CMakeFiles/geos.dir/src/operation/intersection/RectangleIntersection.cpp.o
26762026-03-10T17:55:41.348Z[ 67%] Building CXX object CMakeFiles/geos.dir/src/operation/intersection/RectangleIntersectionBuilder.cpp.o
26772026-03-10T17:55:41.363Z[ 67%] Building CXX object CMakeFiles/geos.dir/src/operation/linemerge/EdgeString.cpp.o
26782026-03-10T17:55:41.401Z[ 68%] Building CXX object CMakeFiles/geos.dir/src/operation/linemerge/LineMergeEdge.cpp.o
26792026-03-10T17:55:41.422Z[ 68%] Building CXX object CMakeFiles/geos.dir/src/operation/linemerge/LineMergeDirectedEdge.cpp.o
26802026-03-10T17:55:41.437Z[ 68%] Building CXX object CMakeFiles/geos.dir/src/operation/linemerge/LineMerger.cpp.o
26812026-03-10T17:55:41.437Z[ 68%] Building CXX object CMakeFiles/geos.dir/src/operation/linemerge/LineMergeGraph.cpp.o
26822026-03-10T17:55:41.474Z[ 68%] Building CXX object CMakeFiles/geos.dir/src/operation/linemerge/LineSequencer.cpp.o
26832026-03-10T17:55:41.531Z[ 68%] Building CXX object CMakeFiles/geos.dir/src/operation/overlay/EdgeSetNoder.cpp.o
26842026-03-10T17:55:41.613Z[ 70%] Building CXX object CMakeFiles/geos.dir/src/operation/overlay/ElevationMatrix.cpp.o
26852026-03-10T17:55:41.668Z[ 70%] Building CXX object CMakeFiles/geos.dir/src/operation/overlay/ElevationMatrixCell.cpp.o
26862026-03-10T17:55:41.738Z[ 70%] Building CXX object CMakeFiles/geos.dir/src/operation/overlay/LineBuilder.cpp.o
26872026-03-10T17:55:41.746Z[ 70%] Building CXX object CMakeFiles/geos.dir/src/operation/overlay/MaximalEdgeRing.cpp.o
26882026-03-10T17:55:41.794Z[ 70%] Building CXX object CMakeFiles/geos.dir/src/operation/overlay/MinimalEdgeRing.cpp.o
26892026-03-10T17:55:41.896Z[ 72%] Building CXX object CMakeFiles/geos.dir/src/operation/overlay/OverlayNodeFactory.cpp.o
26902026-03-10T17:55:41.935Z[ 72%] Building CXX object CMakeFiles/geos.dir/src/operation/overlay/OverlayOp.cpp.o
26912026-03-10T17:55:41.940Z[ 72%] Building CXX object CMakeFiles/geos.dir/src/operation/overlay/PointBuilder.cpp.o
26922026-03-10T17:55:41.940Z[ 72%] Building CXX object CMakeFiles/geos.dir/src/operation/overlay/PolygonBuilder.cpp.o
26932026-03-10T17:55:42.043Z[ 72%] Building CXX object CMakeFiles/geos.dir/src/operation/overlay/snap/GeometrySnapper.cpp.o
26942026-03-10T17:55:42.056Z[ 73%] Building CXX object CMakeFiles/geos.dir/src/operation/overlay/snap/LineStringSnapper.cpp.o
26952026-03-10T17:55:42.097Z[ 73%] Building CXX object CMakeFiles/geos.dir/src/operation/overlay/snap/SnapIfNeededOverlayOp.cpp.o
26962026-03-10T17:55:42.138Z[ 73%] Building CXX object CMakeFiles/geos.dir/src/operation/overlay/snap/SnapOverlayOp.cpp.o
26972026-03-10T17:55:42.232Z[ 73%] Building CXX object CMakeFiles/geos.dir/src/operation/overlay/validate/FuzzyPointLocator.cpp.o
26982026-03-10T17:55:42.347Z[ 73%] Building CXX object CMakeFiles/geos.dir/src/operation/overlay/validate/OffsetPointGenerator.cpp.o
26992026-03-10T17:55:42.399Z[ 75%] Building CXX object CMakeFiles/geos.dir/src/operation/overlay/validate/OverlayResultValidator.cpp.o
27002026-03-10T17:55:42.405Z[ 75%] Building CXX object CMakeFiles/geos.dir/src/operation/polygonize/BuildArea.cpp.o
27012026-03-10T17:55:42.481Z[ 75%] Building CXX object CMakeFiles/geos.dir/src/operation/polygonize/EdgeRing.cpp.o
27022026-03-10T17:55:42.503Z[ 75%] Building CXX object CMakeFiles/geos.dir/src/operation/polygonize/HoleAssigner.cpp.o
27032026-03-10T17:55:42.518Z[ 75%] Building CXX object CMakeFiles/geos.dir/src/operation/polygonize/PolygonizeDirectedEdge.cpp.o
27042026-03-10T17:55:42.637Z[ 77%] Building CXX object CMakeFiles/geos.dir/src/operation/polygonize/PolygonizeEdge.cpp.o
27052026-03-10T17:55:42.687Z[ 77%] Building CXX object CMakeFiles/geos.dir/src/operation/polygonize/PolygonizeGraph.cpp.o
27062026-03-10T17:55:42.723Z[ 77%] Building CXX object CMakeFiles/geos.dir/src/operation/polygonize/Polygonizer.cpp.o
27072026-03-10T17:55:42.762Z[ 77%] Building CXX object CMakeFiles/geos.dir/src/operation/predicate/RectangleContains.cpp.o
27082026-03-10T17:55:42.806Z[ 77%] Building CXX object CMakeFiles/geos.dir/src/operation/predicate/RectangleIntersects.cpp.o
27092026-03-10T17:55:42.845Z[ 78%] Building CXX object CMakeFiles/geos.dir/src/operation/predicate/SegmentIntersectionTester.cpp.o
27102026-03-10T17:55:42.897Z[ 78%] Building CXX object CMakeFiles/geos.dir/src/operation/relate/EdgeEndBundle.cpp.o
27112026-03-10T17:55:42.900Z[ 78%] Building CXX object CMakeFiles/geos.dir/src/operation/relate/EdgeEndBuilder.cpp.o
27122026-03-10T17:55:43.064Z[ 78%] Building CXX object CMakeFiles/geos.dir/src/operation/relate/EdgeEndBundleStar.cpp.o
27132026-03-10T17:55:43.069Z[ 80%] Building CXX object CMakeFiles/geos.dir/src/operation/relate/RelateComputer.cpp.o
27142026-03-10T17:55:43.072Z[ 80%] Building CXX object CMakeFiles/geos.dir/src/operation/relate/RelateNode.cpp.o
27152026-03-10T17:55:43.176Z[ 80%] Building CXX object CMakeFiles/geos.dir/src/operation/relate/RelateNodeFactory.cpp.o
27162026-03-10T17:55:43.228Z[ 80%] Building CXX object CMakeFiles/geos.dir/src/operation/relate/RelateNodeGraph.cpp.o
27172026-03-10T17:55:43.236Z[ 80%] Building CXX object CMakeFiles/geos.dir/src/operation/relate/RelateOp.cpp.o
27182026-03-10T17:55:43.252Z[ 81%] Building CXX object CMakeFiles/geos.dir/src/operation/sharedpaths/SharedPathsOp.cpp.o
27192026-03-10T17:55:43.345Z[ 81%] Building CXX object CMakeFiles/geos.dir/src/operation/union/CascadedPolygonUnion.cpp.o
27202026-03-10T17:55:43.364Z[ 81%] Building CXX object CMakeFiles/geos.dir/src/operation/union/CascadedUnion.cpp.o
27212026-03-10T17:55:43.410Z[ 81%] Building CXX object CMakeFiles/geos.dir/src/operation/union/CoverageUnion.cpp.o
27222026-03-10T17:55:43.472Z[ 81%] Building CXX object CMakeFiles/geos.dir/src/operation/union/OverlapUnion.cpp.o
27232026-03-10T17:55:43.525Z[ 83%] Building CXX object CMakeFiles/geos.dir/src/operation/union/PointGeometryUnion.cpp.o
27242026-03-10T17:55:43.618Z[ 83%] Building CXX object CMakeFiles/geos.dir/src/operation/union/UnaryUnionOp.cpp.o
27252026-03-10T17:55:43.650Z[ 83%] Building CXX object CMakeFiles/geos.dir/src/operation/valid/ConnectedInteriorTester.cpp.o
27262026-03-10T17:55:43.818Z[ 83%] Building CXX object CMakeFiles/geos.dir/src/operation/valid/ConsistentAreaTester.cpp.o
27272026-03-10T17:55:43.824Z[ 83%] Building CXX object CMakeFiles/geos.dir/src/operation/valid/IndexedNestedRingTester.cpp.o
27282026-03-10T17:55:43.857Z[ 85%] Building CXX object CMakeFiles/geos.dir/src/operation/valid/IsValidOp.cpp.o
27292026-03-10T17:55:43.885Z[ 85%] Building CXX object CMakeFiles/geos.dir/src/operation/valid/MakeValid.cpp.o
27302026-03-10T17:55:43.925Z[ 85%] Building CXX object CMakeFiles/geos.dir/src/operation/valid/QuadtreeNestedRingTester.cpp.o
27312026-03-10T17:55:44.066Z[ 85%] Building CXX object CMakeFiles/geos.dir/src/operation/valid/RepeatedPointRemover.cpp.o
27322026-03-10T17:55:44.070Z[ 85%] Building CXX object CMakeFiles/geos.dir/src/operation/valid/RepeatedPointTester.cpp.o
27332026-03-10T17:55:44.106Z[ 86%] Building CXX object CMakeFiles/geos.dir/src/operation/valid/SimpleNestedRingTester.cpp.o
27342026-03-10T17:55:44.197Z[ 86%] Building CXX object CMakeFiles/geos.dir/src/operation/valid/SweeplineNestedRingTester.cpp.o
27352026-03-10T17:55:44.295Z[ 86%] Building CXX object CMakeFiles/geos.dir/src/planargraph/DirectedEdge.cpp.o
27362026-03-10T17:55:44.300Z[ 86%] Building CXX object CMakeFiles/geos.dir/src/operation/valid/TopologyValidationError.cpp.o
27372026-03-10T17:55:44.326Z[ 86%] Building CXX object CMakeFiles/geos.dir/src/planargraph/DirectedEdgeStar.cpp.o
27382026-03-10T17:55:44.343Z[ 88%] Building CXX object CMakeFiles/geos.dir/src/planargraph/Edge.cpp.o
27392026-03-10T17:55:44.417Z[ 88%] Building CXX object CMakeFiles/geos.dir/src/planargraph/Node.cpp.o
27402026-03-10T17:55:44.431Z[ 88%] Building CXX object CMakeFiles/geos.dir/src/planargraph/NodeMap.cpp.o
27412026-03-10T17:55:44.451Z[ 88%] Building CXX object CMakeFiles/geos.dir/src/planargraph/PlanarGraph.cpp.o
27422026-03-10T17:55:44.460Z[ 88%] Building CXX object CMakeFiles/geos.dir/src/planargraph/Subgraph.cpp.o
27432026-03-10T17:55:44.522Z[ 90%] Building CXX object CMakeFiles/geos.dir/src/planargraph/algorithm/ConnectedSubgraphFinder.cpp.o
27442026-03-10T17:55:44.544Z[ 90%] Building CXX object CMakeFiles/geos.dir/src/precision/CommonBits.cpp.o
27452026-03-10T17:55:44.554Z[ 90%] Building CXX object CMakeFiles/geos.dir/src/precision/CommonBitsOp.cpp.o
27462026-03-10T17:55:44.591Z[ 90%] Building CXX object CMakeFiles/geos.dir/src/precision/CommonBitsRemover.cpp.o
27472026-03-10T17:55:44.635Z[ 90%] Building CXX object CMakeFiles/geos.dir/src/precision/EnhancedPrecisionOp.cpp.o
27482026-03-10T17:55:44.645Z[ 91%] Building CXX object CMakeFiles/geos.dir/src/precision/GeometryPrecisionReducer.cpp.o
27492026-03-10T17:55:44.650Z[ 91%] Building CXX object CMakeFiles/geos.dir/src/precision/MinimumClearance.cpp.o
27502026-03-10T17:55:44.650Z[ 91%] Building CXX object CMakeFiles/geos.dir/src/precision/PrecisionReducerCoordinateOperation.cpp.o
27512026-03-10T17:55:44.747Z[ 91%] Building CXX object CMakeFiles/geos.dir/src/precision/SimpleGeometryPrecisionReducer.cpp.o
27522026-03-10T17:55:44.787Z[ 91%] Building CXX object CMakeFiles/geos.dir/src/simplify/DouglasPeuckerLineSimplifier.cpp.o
27532026-03-10T17:55:44.804Z[ 93%] Building CXX object CMakeFiles/geos.dir/src/simplify/DouglasPeuckerSimplifier.cpp.o
27542026-03-10T17:55:44.889Z[ 93%] Building CXX object CMakeFiles/geos.dir/src/simplify/LineSegmentIndex.cpp.o
27552026-03-10T17:55:44.899Z[ 93%] Building CXX object CMakeFiles/geos.dir/src/simplify/TaggedLineSegment.cpp.o
27562026-03-10T17:55:44.926Z[ 93%] Building CXX object CMakeFiles/geos.dir/src/simplify/TaggedLineString.cpp.o
27572026-03-10T17:55:44.979Z[ 93%] Building CXX object CMakeFiles/geos.dir/src/simplify/TaggedLineStringSimplifier.cpp.o
27582026-03-10T17:55:45.042Z[ 95%] Building CXX object CMakeFiles/geos.dir/src/simplify/TaggedLinesSimplifier.cpp.o
27592026-03-10T17:55:45.061Z[ 95%] Building CXX object CMakeFiles/geos.dir/src/simplify/TopologyPreservingSimplifier.cpp.o
27602026-03-10T17:55:45.097Z[ 95%] Building CXX object CMakeFiles/geos.dir/src/triangulate/DelaunayTriangulationBuilder.cpp.o
27612026-03-10T17:55:45.107Z[ 95%] Building CXX object CMakeFiles/geos.dir/src/triangulate/IncrementalDelaunayTriangulator.cpp.o
27622026-03-10T17:55:45.161Z[ 95%] Building CXX object CMakeFiles/geos.dir/src/triangulate/VoronoiDiagramBuilder.cpp.o
27632026-03-10T17:55:45.259Z[ 96%] Building CXX object CMakeFiles/geos.dir/src/triangulate/quadedge/LastFoundQuadEdgeLocator.cpp.o
27642026-03-10T17:55:45.259Z[ 96%] Building CXX object CMakeFiles/geos.dir/src/triangulate/quadedge/LocateFailureException.cpp.o
27652026-03-10T17:55:45.262Z[ 96%] Building CXX object CMakeFiles/geos.dir/src/triangulate/quadedge/QuadEdge.cpp.o
27662026-03-10T17:55:45.364Z[ 96%] Building CXX object CMakeFiles/geos.dir/src/triangulate/quadedge/QuadEdgeSubdivision.cpp.o
27672026-03-10T17:55:45.415Z[ 96%] Building CXX object CMakeFiles/geos.dir/src/triangulate/quadedge/TrianglePredicate.cpp.o
27682026-03-10T17:55:45.419Z[ 98%] Building CXX object CMakeFiles/geos.dir/src/triangulate/quadedge/Vertex.cpp.o
27692026-03-10T17:55:45.500Z[ 98%] Building CXX object CMakeFiles/geos.dir/src/util/Assert.cpp.o
27702026-03-10T17:55:45.510Z[ 98%] Building CXX object CMakeFiles/geos.dir/src/util/GeometricShapeFactory.cpp.o
27712026-03-10T17:55:45.537Z[ 98%] Building CXX object CMakeFiles/geos.dir/src/util/Interrupt.cpp.o
27722026-03-10T17:55:45.581Z[ 98%] Building CXX object CMakeFiles/geos.dir/src/util/Profiler.cpp.o
27732026-03-10T17:55:45.599Z[100%] Building CXX object CMakeFiles/geos.dir/src/util/math.cpp.o
27742026-03-10T17:55:45.720Zgofmt -s -w ./pkg/acceptance/cluster/testconfig.pb.go ./pkg/blobs/blobspb/blobs.pb.go ./pkg/build/info.pb.go ./pkg/clusterversion/cluster_version.pb.go ./pkg/config/system.pb.go ./pkg/config/zonepb/zone.pb.go ./pkg/geo/geoindex/config.pb.go ./pkg/geo/geopb/geopb.pb.go ./pkg/gossip/gossip.pb.go ./pkg/jobs/jobspb/jobs.pb.go ./pkg/jobs/jobspb/schedule.pb.go ./pkg/kv/kvnemesis/operations.pb.go ./pkg/kv/kvserver/api.pb.go ./pkg/kv/kvserver/closedts/ctpb/service.pb.go ./pkg/kv/kvserver/concurrency/lock/lock_waiter.pb.go ./pkg/kv/kvserver/concurrency/lock/locking.pb.go ./pkg/kv/kvserver/concurrency/poison/error.pb.go ./pkg/kv/kvserver/concurrency/poison/policy.pb.go ./pkg/kv/kvserver/kvserverpb/lease_status.pb.go ./pkg/kv/kvserver/kvserverpb/proposer_kv.pb.go ./pkg/kv/kvserver/kvserverpb/raft.pb.go ./pkg/kv/kvserver/kvserverpb/range_log.pb.go ./pkg/kv/kvserver/kvserverpb/state.pb.go ./pkg/kv/kvserver/liveness/livenesspb/liveness.pb.go ./pkg/kv/kvserver/loqrecovery/loqrecoverypb/recovery.pb.go ./pkg/kv/kvserver/protectedts/ptpb/protectedts.pb.go ./pkg/kv/kvserver/protectedts/ptstorage/storage.pb.go ./pkg/kv/kvserver/readsummary/rspb/summary.pb.go ./pkg/kv/kvserver/storage_services.pb.go ./pkg/roachpb/api.pb.go ./pkg/roachpb/app_stats.pb.go ./pkg/roachpb/data.pb.go ./pkg/roachpb/errors.pb.go ./pkg/roachpb/index_usage_stats.pb.go ./pkg/roachpb/internal.pb.go ./pkg/roachpb/internal_raft.pb.go ./pkg/roachpb/io-formats.pb.go ./pkg/roachpb/metadata.pb.go ./pkg/roachpb/span_config.pb.go ./pkg/rpc/heartbeat.pb.go ./pkg/server/diagnostics/diagnosticspb/diagnostics.pb.go ./pkg/server/serverpb/admin.pb.go ./pkg/server/serverpb/authentication.pb.go ./pkg/server/serverpb/init.pb.go ./pkg/server/serverpb/migration.pb.go ./pkg/server/serverpb/status.pb.go ./pkg/server/status/statuspb/status.pb.go ./pkg/settings/encoding.pb.go ./pkg/sql/catalog/catpb/catalog.pb.go ./pkg/sql/catalog/catpb/privilege.pb.go ./pkg/sql/catalog/descpb/encoded_datum.pb.go ./pkg/sql/catalog/descpb/index_fetch.pb.go ./pkg/sql/catalog/descpb/join_type.pb.go ./pkg/sql/catalog/descpb/locking.pb.go ./pkg/sql/catalog/descpb/structured.pb.go ./pkg/sql/catalog/descpb/tenant.pb.go ./pkg/sql/contentionpb/contention.pb.go ./pkg/sql/execinfrapb/api.pb.go ./pkg/sql/execinfrapb/component_stats.pb.go ./pkg/sql/execinfrapb/data.pb.go ./pkg/sql/execinfrapb/processors.pb.go ./pkg/sql/execinfrapb/processors_base.pb.go ./pkg/sql/execinfrapb/processors_bulk_io.pb.go ./pkg/sql/execinfrapb/processors_changefeeds.pb.go ./pkg/sql/execinfrapb/processors_sql.pb.go ./pkg/sql/execinfrapb/processors_table_stats.pb.go ./pkg/sql/inverted/span_expression.pb.go ./pkg/sql/lex/encode.pb.go ./pkg/sql/pgwire/pgerror/errors.pb.go ./pkg/sql/protoreflect/test/test.pb.go ./pkg/sql/rowenc/rowencpb/index_encoding.pb.go ./pkg/sql/schemachanger/scpb/elements.pb.go ./pkg/sql/schemachanger/scpb/scpb.pb.go ./pkg/sql/sessiondatapb/local_only_session_data.pb.go ./pkg/sql/sessiondatapb/session_data.pb.go ./pkg/sql/sessiondatapb/session_migration.pb.go ./pkg/sql/sessiondatapb/session_revival_token.pb.go ./pkg/sql/sqlstats/persistedsqlstats/sql_stats_compact.pb.go ./pkg/sql/stats/histogram.pb.go ./pkg/sql/stats/table_statistic.pb.go ./pkg/sql/types/types.pb.go ./pkg/startupmigrations/leasemanager/lease.pb.go ./pkg/storage/enginepb/engine.pb.go ./pkg/storage/enginepb/file_registry.pb.go ./pkg/storage/enginepb/mvcc.pb.go ./pkg/storage/enginepb/mvcc3.pb.go ./pkg/storage/enginepb/rocksdb.pb.go ./pkg/testutils/grpcutils/testservice.pb.go ./pkg/ts/catalog/chart_catalog.pb.go ./pkg/ts/tspb/timeseries.pb.go ./pkg/util/duration/duration.pb.go ./pkg/util/hlc/legacy_timestamp.pb.go ./pkg/util/hlc/timestamp.pb.go ./pkg/util/log/eventpb/cluster_events.pb.go ./pkg/util/log/eventpb/ddl_events.pb.go ./pkg/util/log/eventpb/debug_events.pb.go ./pkg/util/log/eventpb/events.pb.go ./pkg/util/log/eventpb/health_events.pb.go ./pkg/util/log/eventpb/job_events.pb.go ./pkg/util/log/eventpb/misc_sql_events.pb.go ./pkg/util/log/eventpb/privilege_events.pb.go ./pkg/util/log/eventpb/role_events.pb.go ./pkg/util/log/eventpb/session_events.pb.go ./pkg/util/log/eventpb/sql_audit_events.pb.go ./pkg/util/log/eventpb/telemetry.pb.go ./pkg/util/log/eventpb/zone_events.pb.go ./pkg/util/log/logpb/log.pb.go ./pkg/util/metric/metric.pb.go ./pkg/util/optional/optional.pb.go ./pkg/util/protoutil/clone.pb.go ./pkg/util/timeutil/pgdate/pgdate.pb.go ./pkg/util/tracing/tracingpb/recorded_span.pb.go ./pkg/util/tracing/tracingpb/tracing.pb.go ./pkg/util/tracing/tracingservicepb/tracing_service.pb.go ./pkg/util/unresolved_addr.pb.go
27752026-03-10T17:55:46.046Z[100%] Linking CXX shared library lib/libgeos.so
27762026-03-10T17:55:46.153Z[100%] Building CXX object CMakeFiles/geos_c.dir/capi/geos_c.cpp.o
27772026-03-10T17:55:46.153Z[100%] Building CXX object CMakeFiles/geos_c.dir/capi/geos_ts_c.cpp.o
27782026-03-10T17:55:46.560Ztouch bin/.go_protobuf_sources
27792026-03-10T17:55:48.112Z[100%] Linking CXX shared library lib/libgeos_c.so
27802026-03-10T17:55:48.174Zmkdir -p lib
27812026-03-10T17:55:48.187Zrm -f lib/lib{geos,geos_c}.so
27822026-03-10T17:55:48.187Zcp /work/oxidecomputer/cockroach/lib/build/geos/lib/lib{geos,geos_c}.so lib
27832026-03-10T17:55:48.187ZGOFLAGS= go test -mod=vendor -tags 'make x86_64_linux_gnu crdb_test' -ldflags '-X github.com/cockroachdb/cockroach/pkg/build.typ=development -extldflags "" -X "github.com/cockroachdb/cockroach/pkg/build.tag=v22.1.22-53-g21bdff50eb" -X "github.com/cockroachdb/cockroach/pkg/build.rev=21bdff50eb9c7974e74b511a5ae08a75cbc1619c" -X "github.com/cockroachdb/cockroach/pkg/build.cgoTargetTriple=x86_64-linux-gnu" ' -run "." -timeout 60m ./pkg/...
27842026-03-10T17:56:15.393Zok github.com/cockroachdb/cockroach/pkg/acceptance 0.009s
27852026-03-10T17:56:15.393Z? github.com/cockroachdb/cockroach/pkg/acceptance/cluster [no test files]
27862026-03-10T17:56:31.601Z? github.com/cockroachdb/cockroach/pkg/acceptance/localcluster [no test files]
27872026-03-10T17:56:31.604Z? github.com/cockroachdb/cockroach/pkg/acceptance/localcluster/tc [no test files]
27882026-03-10T17:56:32.202Zok github.com/cockroachdb/cockroach/pkg/base 0.028s
27892026-03-10T17:56:59.622Zok github.com/cockroachdb/cockroach/pkg/bench 0.021s [no tests to run]
27902026-03-10T17:56:59.712Z? github.com/cockroachdb/cockroach/pkg/bench/cmd/pgbenchsetup [no test files]
27912026-03-10T17:57:03.450Z? github.com/cockroachdb/cockroach/pkg/blobs/blobspb [no test files]
27922026-03-10T17:57:03.450Z? github.com/cockroachdb/cockroach/pkg/build [no test files]
27932026-03-10T17:57:03.454Z? github.com/cockroachdb/cockroach/pkg/build/bazel [no test files]
27942026-03-10T17:57:11.135Z? github.com/cockroachdb/cockroach/pkg/cli/clicfg [no test files]
27952026-03-10T17:57:11.139Z? github.com/cockroachdb/cockroach/pkg/cli/clierrorplus [no test files]
27962026-03-10T17:57:11.139Z? github.com/cockroachdb/cockroach/pkg/cli/cliflags [no test files]
27972026-03-10T17:57:11.139Z? github.com/cockroachdb/cockroach/pkg/cli/clisqlcfg [no test files]
27982026-03-10T17:57:12.166Zok github.com/cockroachdb/cockroach/pkg/bench/rttanalysis 11.055s
27992026-03-10T17:57:12.166Zok github.com/cockroachdb/cockroach/pkg/bench/tpcc 0.030s
28002026-03-10T17:57:12.166Zok github.com/cockroachdb/cockroach/pkg/blobs 0.191s
28012026-03-10T17:57:12.166Zok github.com/cockroachdb/cockroach/pkg/build/starlarkutil 0.034s
28022026-03-10T17:57:12.166Zok github.com/cockroachdb/cockroach/pkg/build/util 0.035s
28032026-03-10T17:57:16.068Z? github.com/cockroachdb/cockroach/pkg/cli/democluster/api [no test files]
28042026-03-10T17:57:16.071Z? github.com/cockroachdb/cockroach/pkg/cli/syncbench [no test files]
28052026-03-10T17:57:16.071Z? github.com/cockroachdb/cockroach/pkg/cloud [no test files]
28062026-03-10T17:57:17.679Z? github.com/cockroachdb/cockroach/pkg/cloud/cloudtestutils [no test files]
28072026-03-10T17:57:18.934Z? github.com/cockroachdb/cockroach/pkg/cloud/impl [no test files]
28082026-03-10T17:57:21.057Z? github.com/cockroachdb/cockroach/pkg/cloud/userfile/filetable [no test files]
28092026-03-10T17:57:21.326Z? github.com/cockroachdb/cockroach/pkg/cmd/allocsim [no test files]
28102026-03-10T17:57:21.333Z? github.com/cockroachdb/cockroach/pkg/cmd/cmdutil [no test files]
28112026-03-10T17:57:21.513Z? github.com/cockroachdb/cockroach/pkg/cmd/cmp-protocol [no test files]
28122026-03-10T17:57:21.513Z? github.com/cockroachdb/cockroach/pkg/cmd/cmp-protocol/pgconnect [no test files]
28132026-03-10T17:57:21.513Z? github.com/cockroachdb/cockroach/pkg/cmd/cmp-sql [no test files]
28142026-03-10T17:57:23.903Z? github.com/cockroachdb/cockroach/pkg/cmd/cockroach-short [no test files]
28152026-03-10T17:57:23.903Z? github.com/cockroachdb/cockroach/pkg/cmd/cockroach-sql [no test files]
28162026-03-10T17:57:23.904Z? github.com/cockroachdb/cockroach/pkg/cmd/cr2pg [no test files]
28172026-03-10T17:57:23.904Z? github.com/cockroachdb/cockroach/pkg/cmd/cr2pg/sqlstream [no test files]
28182026-03-10T17:57:23.913Z? github.com/cockroachdb/cockroach/pkg/cmd/dev/io/exec [no test files]
28192026-03-10T17:57:23.913Z? github.com/cockroachdb/cockroach/pkg/cmd/dev/io/os [no test files]
28202026-03-10T17:57:26.277Z? github.com/cockroachdb/cockroach/pkg/cmd/docgen [no test files]
28212026-03-10T17:57:26.280Z? github.com/cockroachdb/cockroach/pkg/cmd/fuzz [no test files]
28222026-03-10T17:57:26.280Z? github.com/cockroachdb/cockroach/pkg/cmd/generate-binary [no test files]
28232026-03-10T17:57:26.280Z? github.com/cockroachdb/cockroach/pkg/cmd/generate-distdir [no test files]
28242026-03-10T17:57:26.280Z? github.com/cockroachdb/cockroach/pkg/cmd/generate-metadata-tables [no test files]
28252026-03-10T17:57:26.280Z? github.com/cockroachdb/cockroach/pkg/cmd/generate-metadata-tables/rdbms [no test files]
28262026-03-10T17:57:26.329Z? github.com/cockroachdb/cockroach/pkg/cmd/generate-spatial-ref-sys [no test files]
28272026-03-10T17:57:26.991Z? github.com/cockroachdb/cockroach/pkg/cmd/generate-staticcheck [no test files]
28282026-03-10T17:57:26.991Z? github.com/cockroachdb/cockroach/pkg/cmd/generate-test-suites [no test files]
28292026-03-10T17:57:26.994Z? github.com/cockroachdb/cockroach/pkg/cmd/geoviz [no test files]
28302026-03-10T17:57:27.010Z? github.com/cockroachdb/cockroach/pkg/cmd/gossipsim [no test files]
28312026-03-10T17:57:27.010Z? github.com/cockroachdb/cockroach/pkg/cmd/import-tools [no test files]
28322026-03-10T17:57:27.771Z? github.com/cockroachdb/cockroach/pkg/cmd/protoc-gen-gogoroach [no test files]
28332026-03-10T17:57:28.081Z? github.com/cockroachdb/cockroach/pkg/cmd/reduce [no test files]
28342026-03-10T17:57:31.118Z? github.com/cockroachdb/cockroach/pkg/cmd/returncheck [no test files]
28352026-03-10T17:57:41.941Z? github.com/cockroachdb/cockroach/pkg/cmd/roachprod [no test files]
28362026-03-10T17:57:41.944Z? github.com/cockroachdb/cockroach/pkg/cmd/roachprod-stress [no test files]
28372026-03-10T17:57:46.079Z? github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster [no test files]
28382026-03-10T17:57:46.079Z? github.com/cockroachdb/cockroach/pkg/cmd/roachtest/option [no test files]
28392026-03-10T17:57:46.689Z? github.com/cockroachdb/cockroach/pkg/cmd/roachtest/registry [no test files]
28402026-03-10T17:57:46.689Z? github.com/cockroachdb/cockroach/pkg/cmd/roachtest/spec [no test files]
28412026-03-10T17:57:46.693Z? github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test [no test files]
28422026-03-10T17:57:50.592Z? github.com/cockroachdb/cockroach/pkg/cmd/roachvet [no test files]
28432026-03-10T17:57:50.595Z? github.com/cockroachdb/cockroach/pkg/cmd/skip-test [no test files]
28442026-03-10T17:57:50.595Z? github.com/cockroachdb/cockroach/pkg/cmd/skiperrs [no test files]
28452026-03-10T17:57:50.595Z? github.com/cockroachdb/cockroach/pkg/cmd/skipped-tests [no test files]
28462026-03-10T17:57:50.595Z? github.com/cockroachdb/cockroach/pkg/cmd/smithtest [no test files]
28472026-03-10T17:57:50.595Z? github.com/cockroachdb/cockroach/pkg/cmd/smithcmp [no test files]
28482026-03-10T17:57:51.190Z? github.com/cockroachdb/cockroach/pkg/cmd/uptodate [no test files]
28492026-03-10T17:57:51.190Z? github.com/cockroachdb/cockroach/pkg/cmd/urlcheck [no test files]
28502026-03-10T17:57:51.194Z? github.com/cockroachdb/cockroach/pkg/cmd/urlcheck/lib/urlcheck [no test files]
28512026-03-10T17:57:51.194Z? github.com/cockroachdb/cockroach/pkg/cmd/workload [no test files]
28522026-03-10T17:57:51.194Z? github.com/cockroachdb/cockroach/pkg/cmd/wraprules [no test files]
28532026-03-10T17:57:51.194Z? github.com/cockroachdb/cockroach/pkg/cmd/zerosum [no test files]
28542026-03-10T17:57:52.877Z? github.com/cockroachdb/cockroach/pkg/col/coldatatestutils [no test files]
28552026-03-10T17:57:54.087Z? github.com/cockroachdb/cockroach/pkg/col/colserde/arrowserde [no test files]
28562026-03-10T17:57:54.091Z? github.com/cockroachdb/cockroach/pkg/col/typeconv [no test files]
28572026-03-10T17:57:54.091Z? github.com/cockroachdb/cockroach/pkg/compose [no test files]
28582026-03-10T17:57:54.091Z? github.com/cockroachdb/cockroach/pkg/compose/compare/compare [no test files]
28592026-03-10T17:57:54.282Z? github.com/cockroachdb/cockroach/pkg/docs [no test files]
28602026-03-10T17:57:54.282Z? github.com/cockroachdb/cockroach/pkg/featureflag [no test files]
28612026-03-10T17:57:54.286Z? github.com/cockroachdb/cockroach/pkg/gen/genbzl [no test files]
28622026-03-10T17:57:54.652Z? github.com/cockroachdb/cockroach/pkg/geo/geodist [no test files]
28632026-03-10T17:57:55.589Z? github.com/cockroachdb/cockroach/pkg/geo/geoprojbase/embeddedproj [no test files]
28642026-03-10T17:57:56.084Z? github.com/cockroachdb/cockroach/pkg/geo/geosegmentize [no test files]
28652026-03-10T17:57:57.088Z? github.com/cockroachdb/cockroach/pkg/gossip/simulation [no test files]
28662026-03-10T17:57:57.091Z? github.com/cockroachdb/cockroach/pkg/internal/reporoot [no test files]
28672026-03-10T17:58:18.749Z? github.com/cockroachdb/cockroach/pkg/jobs/jobspb [no test files]
28682026-03-10T17:58:18.752Z? github.com/cockroachdb/cockroach/pkg/jobs/jobstest [no test files]
28692026-03-10T17:58:18.752Z? github.com/cockroachdb/cockroach/pkg/keysbase [no test files]
28702026-03-10T17:58:29.710Zok github.com/cockroachdb/cockroach/pkg/cli 78.451s
28712026-03-10T17:58:29.710Zok github.com/cockroachdb/cockroach/pkg/cli/clierror 0.437s
28722026-03-10T17:58:29.710Zok github.com/cockroachdb/cockroach/pkg/cli/clisqlclient 0.743s
28732026-03-10T17:58:29.710Zok github.com/cockroachdb/cockroach/pkg/cli/clisqlexec 1.999s
28742026-03-10T17:58:29.710Zok github.com/cockroachdb/cockroach/pkg/cli/clisqlshell 2.514s
28752026-03-10T17:58:29.710Zok github.com/cockroachdb/cockroach/pkg/cli/democluster 3.345s
28762026-03-10T17:58:29.710Zok github.com/cockroachdb/cockroach/pkg/cli/exit 0.002s [no tests to run]
28772026-03-10T17:58:29.710Zok github.com/cockroachdb/cockroach/pkg/cloud/amazon 0.553s
28782026-03-10T17:58:29.710Zok github.com/cockroachdb/cockroach/pkg/cloud/azure 0.024s
28792026-03-10T17:58:29.711Zok github.com/cockroachdb/cockroach/pkg/cloud/gcp 0.025s
28802026-03-10T17:58:29.711Zok github.com/cockroachdb/cockroach/pkg/cloud/httpsink 0.265s
28812026-03-10T17:58:29.711Zok github.com/cockroachdb/cockroach/pkg/cloud/impl/cloudimpltests 0.025s [no tests to run]
28822026-03-10T17:58:29.711Zok github.com/cockroachdb/cockroach/pkg/cloud/nodelocal 0.171s
28832026-03-10T17:58:29.711Zok github.com/cockroachdb/cockroach/pkg/cloud/nullsink 0.040s
28842026-03-10T17:58:29.711Zok github.com/cockroachdb/cockroach/pkg/cloud/userfile 1.624s
28852026-03-10T17:58:29.711Zok github.com/cockroachdb/cockroach/pkg/cloud/userfile/filetable/filetabletest 4.525s
28862026-03-10T17:58:29.711Zok github.com/cockroachdb/cockroach/pkg/clusterversion 0.014s
28872026-03-10T17:58:29.711Zok github.com/cockroachdb/cockroach/pkg/cmd/bazci 0.014s
28882026-03-10T17:58:29.711Zok github.com/cockroachdb/cockroach/pkg/cmd/cmpconn 0.075s
28892026-03-10T17:58:29.711Zok github.com/cockroachdb/cockroach/pkg/cmd/cockroach-oss 0.644s
28902026-03-10T17:58:29.711Zok github.com/cockroachdb/cockroach/pkg/cmd/dev 0.029s
28912026-03-10T17:58:29.711Zok github.com/cockroachdb/cockroach/pkg/cmd/docgen/extract 0.005s
28922026-03-10T17:58:29.711Zok github.com/cockroachdb/cockroach/pkg/cmd/docs-issue-generation 0.006s
28932026-03-10T17:58:29.716Zok github.com/cockroachdb/cockroach/pkg/cmd/internal/issues 0.015s
28942026-03-10T17:58:29.716Zok github.com/cockroachdb/cockroach/pkg/cmd/label-merged-pr 0.006s
28952026-03-10T17:58:29.716Zok github.com/cockroachdb/cockroach/pkg/cmd/mirror 0.009s
28962026-03-10T17:58:29.716Zok github.com/cockroachdb/cockroach/pkg/cmd/prereqs 0.355s
28972026-03-10T17:58:29.716Zok github.com/cockroachdb/cockroach/pkg/cmd/publish-artifacts 0.050s
28982026-03-10T17:58:29.716Zok github.com/cockroachdb/cockroach/pkg/cmd/publish-provisional-artifacts 0.040s
28992026-03-10T17:58:29.716Zok github.com/cockroachdb/cockroach/pkg/cmd/reduce/reduce 0.007s
29002026-03-10T17:58:29.716Zok github.com/cockroachdb/cockroach/pkg/cmd/reduce/reduce/reducesql 0.071s
29012026-03-10T17:58:29.716Zok github.com/cockroachdb/cockroach/pkg/cmd/release 0.061s
29022026-03-10T17:58:29.716Zok github.com/cockroachdb/cockroach/pkg/cmd/roachtest 0.113s
29032026-03-10T17:58:29.716Zok github.com/cockroachdb/cockroach/pkg/cmd/roachtest/prometheus 0.006s
29042026-03-10T17:58:29.716Zok github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests 0.114s
29052026-03-10T17:58:29.716Zok github.com/cockroachdb/cockroach/pkg/cmd/testfilter 0.013s
29062026-03-10T17:58:29.716Zok github.com/cockroachdb/cockroach/pkg/col/coldata 0.389s
29072026-03-10T17:58:29.716Zok github.com/cockroachdb/cockroach/pkg/col/coldataext 0.017s
29082026-03-10T17:58:29.716Zok github.com/cockroachdb/cockroach/pkg/col/colserde 0.407s
29092026-03-10T17:58:29.716Zok github.com/cockroachdb/cockroach/pkg/config 0.040s
29102026-03-10T17:58:29.716Zok github.com/cockroachdb/cockroach/pkg/config/zonepb 0.050s
29112026-03-10T17:58:29.716Zok github.com/cockroachdb/cockroach/pkg/geo 0.070s
29122026-03-10T17:58:29.716Zok github.com/cockroachdb/cockroach/pkg/geo/geogen 0.008s
29132026-03-10T17:58:29.716Zok github.com/cockroachdb/cockroach/pkg/geo/geogfn 0.096s
29142026-03-10T17:58:29.716Zok github.com/cockroachdb/cockroach/pkg/geo/geographiclib 0.018s
29152026-03-10T17:58:29.716Zok github.com/cockroachdb/cockroach/pkg/geo/geoindex 0.189s
29162026-03-10T17:58:29.717Zok github.com/cockroachdb/cockroach/pkg/geo/geomfn 0.099s
29172026-03-10T17:58:29.717Zok github.com/cockroachdb/cockroach/pkg/geo/geopb 0.003s
29182026-03-10T17:58:29.717Zok github.com/cockroachdb/cockroach/pkg/geo/geoproj 0.020s
29192026-03-10T17:58:29.717Zok github.com/cockroachdb/cockroach/pkg/geo/geoprojbase 0.113s
29202026-03-10T17:58:29.717Zok github.com/cockroachdb/cockroach/pkg/geo/geos 0.005s
29212026-03-10T17:58:29.717Zok github.com/cockroachdb/cockroach/pkg/geo/geotransform 0.005s
29222026-03-10T17:58:29.717Zok github.com/cockroachdb/cockroach/pkg/geo/twkb 0.007s
29232026-03-10T17:58:29.717Zok github.com/cockroachdb/cockroach/pkg/gossip 6.689s
29242026-03-10T17:58:29.717Zok github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher 0.237s
29252026-03-10T17:58:29.717Zok github.com/cockroachdb/cockroach/pkg/internal/rsg 0.095s
29262026-03-10T17:58:29.717Zok github.com/cockroachdb/cockroach/pkg/internal/rsg/yacc 0.024s
29272026-03-10T17:58:29.717Zok github.com/cockroachdb/cockroach/pkg/internal/sqlsmith 2.320s
29282026-03-10T17:58:29.717Zok github.com/cockroachdb/cockroach/pkg/internal/team 0.024s
29292026-03-10T17:58:54.570Z? github.com/cockroachdb/cockroach/pkg/kv/kvbase [no test files]
29302026-03-10T17:58:54.570Z? github.com/cockroachdb/cockroach/pkg/kv/kvclient [no test files]
29312026-03-10T17:58:55.664Zok github.com/cockroachdb/cockroach/pkg/jobs 36.791s
29322026-03-10T17:58:55.664Zok github.com/cockroachdb/cockroach/pkg/jobs/joberror 0.020s
29332026-03-10T17:58:55.664Zok github.com/cockroachdb/cockroach/pkg/jobs/jobsprotectedts 0.026s [no tests to run]
29342026-03-10T17:58:55.664Zok github.com/cockroachdb/cockroach/pkg/keys 0.022s
29352026-03-10T17:59:06.070Z? github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvtenant [no test files]
29362026-03-10T17:59:06.074Z? github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangecache/rangecachemock [no test files]
29372026-03-10T17:59:14.747Zok github.com/cockroachdb/cockroach/pkg/kv 20.162s
29382026-03-10T17:59:14.747Zok github.com/cockroachdb/cockroach/pkg/kv/bulk 1.526s
29392026-03-10T17:59:58.094Z? github.com/cockroachdb/cockroach/pkg/kv/kvserver/closedts/ctpb [no test files]
29402026-03-10T17:59:59.634Z? github.com/cockroachdb/cockroach/pkg/kv/kvserver/constraint [no test files]
29412026-03-10T17:59:59.634Z? github.com/cockroachdb/cockroach/pkg/kv/kvserver/diskmap [no test files]
29422026-03-10T18:00:14.750Z? github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvserverbase [no test files]
29432026-03-10T18:00:14.753Z? github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvserverpb [no test files]
29442026-03-10T18:00:26.575Zok github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord 80.486s
29452026-03-10T18:00:26.575Zok github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangecache 0.192s
29462026-03-10T18:00:26.575Zok github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed 4.341s
29472026-03-10T18:00:26.575Zok github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedbuffer 0.054s
29482026-03-10T18:00:26.575Zok github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache 0.540s
29492026-03-10T18:00:26.575Zok github.com/cockroachdb/cockroach/pkg/kv/kvnemesis 3.189s
29502026-03-10T18:00:26.575Zok github.com/cockroachdb/cockroach/pkg/kv/kvprober 3.763s
29512026-03-10T18:00:33.267Z? github.com/cockroachdb/cockroach/pkg/kv/kvserver/liveness/livenesspb [no test files]
29522026-03-10T18:00:33.271Z? github.com/cockroachdb/cockroach/pkg/kv/kvserver/loqrecovery/loqrecoverypb [no test files]
29532026-03-10T18:00:33.271Z? github.com/cockroachdb/cockroach/pkg/kv/kvserver/protectedts/ptpb [no test files]
29542026-03-10T18:00:33.271Z? github.com/cockroachdb/cockroach/pkg/kv/kvserver/protectedts/ptprovider [no test files]
29552026-03-10T18:00:37.924Z? github.com/cockroachdb/cockroach/pkg/kv/kvserver/protectedts/ptutil [no test files]
29562026-03-10T18:00:37.928Z? github.com/cockroachdb/cockroach/pkg/kv/kvserver/readsummary [no test files]
29572026-03-10T18:00:37.928Z? github.com/cockroachdb/cockroach/pkg/kv/kvserver/readsummary/rspb [no test files]
29582026-03-10T18:00:44.497Z? github.com/cockroachdb/cockroach/pkg/migration [no test files]
29592026-03-10T18:00:48.654Z? github.com/cockroachdb/cockroach/pkg/migration/migrationjob [no test files]
29602026-03-10T18:00:51.297Z? github.com/cockroachdb/cockroach/pkg/migration/nodelivenesstest [no test files]
29612026-03-10T18:00:51.297Z? github.com/cockroachdb/cockroach/pkg/multitenant [no test files]
29622026-03-10T18:00:51.300Z? github.com/cockroachdb/cockroach/pkg/multitenant/tenantcostmodel [no test files]
29632026-03-10T18:00:51.300Z? github.com/cockroachdb/cockroach/pkg/release [no test files]
29642026-03-10T18:00:54.619Z? github.com/cockroachdb/cockroach/pkg/roachpb/gen [no test files]
29652026-03-10T18:00:54.619Z? github.com/cockroachdb/cockroach/pkg/roachpb/roachpbmock [no test files]
29662026-03-10T18:00:54.619Z? github.com/cockroachdb/cockroach/pkg/roachprod [no test files]
29672026-03-10T18:00:55.019Z? github.com/cockroachdb/cockroach/pkg/roachprod/errors [no test files]
29682026-03-10T18:00:55.417Z? github.com/cockroachdb/cockroach/pkg/roachprod/logger [no test files]
29692026-03-10T18:00:55.420Z? github.com/cockroachdb/cockroach/pkg/roachprod/ui [no test files]
29702026-03-10T18:00:55.433Z? github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws [no test files]
29712026-03-10T18:00:55.439Z? github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/terraformgen [no test files]
29722026-03-10T18:00:55.439Z? github.com/cockroachdb/cockroach/pkg/roachprod/vm/azure [no test files]
29732026-03-10T18:00:55.439Z? github.com/cockroachdb/cockroach/pkg/roachprod/vm/flagstub [no test files]
29742026-03-10T18:00:55.442Z? github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce [no test files]
29752026-03-10T18:00:55.442Z? github.com/cockroachdb/cockroach/pkg/roachprod/vm/local [no test files]
29762026-03-10T18:00:56.885Z? github.com/cockroachdb/cockroach/pkg/scheduledjobs [no test files]
29772026-03-10T18:01:52.124Z? github.com/cockroachdb/cockroach/pkg/security/pprompt [no test files]
29782026-03-10T18:01:52.124Z? github.com/cockroachdb/cockroach/pkg/security/securitytest [no test files]
29792026-03-10T18:01:57.267Z? github.com/cockroachdb/cockroach/pkg/server/diagnostics/diagnosticspb [no test files]
29802026-03-10T18:02:11.871Z? github.com/cockroachdb/cockroach/pkg/server/status/statuspb [no test files]
29812026-03-10T18:02:11.875Z? github.com/cockroachdb/cockroach/pkg/server/systemconfigwatcher/systemconfigwatchertest [no test files]
29822026-03-10T18:02:31.069Z? github.com/cockroachdb/cockroach/pkg/settings/cluster [no test files]
29832026-03-10T18:02:31.072Z? github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigjob [no test files]
29842026-03-10T18:02:36.677Z? github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfiglimiter [no test files]
29852026-03-10T18:02:38.481Z? github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigsplitter [no test files]
29862026-03-10T18:02:42.697Z? github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigtestutils/spanconfigtestcluster [no test files]
29872026-03-10T18:02:54.895Z? github.com/cockroachdb/cockroach/pkg/sql/catalog/catconstants [no test files]
29882026-03-10T18:02:57.467Z? github.com/cockroachdb/cockroach/pkg/sql/catalog/colinfo/colinfotestutils [no test files]
29892026-03-10T18:02:57.575Z? github.com/cockroachdb/cockroach/pkg/sql/catalog/descbuilder [no test files]
29902026-03-10T18:02:57.582Z? github.com/cockroachdb/cockroach/pkg/sql/catalog/descidgen [no test files]
29912026-03-10T18:03:00.122Z? github.com/cockroachdb/cockroach/pkg/sql/catalog/desctestutils [no test files]
29922026-03-10T18:03:00.127Z? github.com/cockroachdb/cockroach/pkg/sql/catalog/ingesting [no test files]
29932026-03-10T18:03:00.127Z? github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/catkv [no test files]
29942026-03-10T18:03:22.104Z? github.com/cockroachdb/cockroach/pkg/sql/catalog/rewrite [no test files]
29952026-03-10T18:04:47.417Z? github.com/cockroachdb/cockroach/pkg/sql/descmetadata [no test files]
29962026-03-10T18:05:28.411Z? github.com/cockroachdb/cockroach/pkg/sql/faketreeeval [no test files]
29972026-03-10T18:05:58.855Z? github.com/cockroachdb/cockroach/pkg/sql/lexbase/allkeywords [no test files]
29982026-03-10T18:06:01.401Z? github.com/cockroachdb/cockroach/pkg/sql/memsize [no test files]
29992026-03-10T18:06:01.404Z? github.com/cockroachdb/cockroach/pkg/sql/mutations [no test files]
30002026-03-10T18:06:10.093Z? github.com/cockroachdb/cockroach/pkg/sql/opt/exec [no test files]
30012026-03-10T18:07:41.211Z? github.com/cockroachdb/cockroach/pkg/sql/opt/optgen/cmd/langgen [no test files]
30022026-03-10T18:07:52.905Z? github.com/cockroachdb/cockroach/pkg/sql/opt/testutils/testexpr [no test files]
30032026-03-10T18:08:01.501Z? github.com/cockroachdb/cockroach/pkg/sql/optionalnodeliveness [no test files]
30042026-03-10T18:08:01.504Z? github.com/cockroachdb/cockroach/pkg/sql/paramparse [no test files]
30052026-03-10T18:08:19.918Z? github.com/cockroachdb/cockroach/pkg/sql/parser/fuzz [no test files]
30062026-03-10T18:08:20.818Z? github.com/cockroachdb/cockroach/pkg/sql/pgwire/pgcode [no test files]
30072026-03-10T18:08:25.939Z? github.com/cockroachdb/cockroach/pkg/sql/pgwire/pgnotice [no test files]
30082026-03-10T18:08:25.942Z? github.com/cockroachdb/cockroach/pkg/sql/pgwire/pgwirebase [no test files]
30092026-03-10T18:09:25.296Z? github.com/cockroachdb/cockroach/pkg/sql/protoreflect/test [no test files]
30102026-03-10T18:09:29.080Z? github.com/cockroachdb/cockroach/pkg/sql/roleoption [no test files]
30112026-03-10T18:09:52.146Z? github.com/cockroachdb/cockroach/pkg/sql/rowenc/rowencpb [no test files]
30122026-03-10T18:10:04.287Z? github.com/cockroachdb/cockroach/pkg/sql/rowinfra [no test files]
30132026-03-10T18:10:18.796Z? github.com/cockroachdb/cockroach/pkg/sql/schemachanger/rel/internal/comparetest [no test files]
30142026-03-10T18:10:18.891Z? github.com/cockroachdb/cockroach/pkg/sql/schemachanger/rel/internal/cyclegraphtest [no test files]
30152026-03-10T18:10:18.995Z? github.com/cockroachdb/cockroach/pkg/sql/schemachanger/rel/internal/entitynodetest [no test files]
30162026-03-10T18:10:18.998Z? github.com/cockroachdb/cockroach/pkg/sql/schemachanger/rel/reltest [no test files]
30172026-03-10T18:10:19.254Z? github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scbackup [no test files]
30182026-03-10T18:10:22.126Z? github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scbuild/internal/scbuildstmt [no test files]
30192026-03-10T18:10:24.831Z? github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scdeps/sctestdeps [no test files]
30202026-03-10T18:10:24.831Z? github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scdeps/sctestutils [no test files]
30212026-03-10T18:10:24.831Z? github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scerrors [no test files]
30222026-03-10T18:10:28.580Z? github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scexec/scmutationexec [no test files]
30232026-03-10T18:10:28.580Z? github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scjob [no test files]
30242026-03-10T18:10:28.585Z? github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scop [no test files]
30252026-03-10T18:10:28.585Z? github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scpb [no test files]
30262026-03-10T18:10:31.721Z? github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scplan/internal/scgraphviz [no test files]
30272026-03-10T18:10:31.724Z? github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scplan/internal/scstage [no test files]
30282026-03-10T18:10:33.319Z? github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scrun [no test files]
30292026-03-10T18:10:33.319Z? github.com/cockroachdb/cockroach/pkg/sql/schemachanger/sctest [no test files]
30302026-03-10T18:10:33.319Z? github.com/cockroachdb/cockroach/pkg/sql/scrub [no test files]
30312026-03-10T18:11:13.468Z? github.com/cockroachdb/cockroach/pkg/sql/sem/catid [no test files]
30322026-03-10T18:11:13.471Z? github.com/cockroachdb/cockroach/pkg/sql/sem/transform [no test files]
30332026-03-10T18:11:41.529Z? github.com/cockroachdb/cockroach/pkg/sql/sem/tree/treebin [no test files]
30342026-03-10T18:11:41.529Z? github.com/cockroachdb/cockroach/pkg/sql/sem/tree/treecmp [no test files]
30352026-03-10T18:11:41.529Z? github.com/cockroachdb/cockroach/pkg/sql/sem/tree/treewindow [no test files]
30362026-03-10T18:11:42.203Z? github.com/cockroachdb/cockroach/pkg/sql/sessiondatapb [no test files]
30372026-03-10T18:11:43.697Z? github.com/cockroachdb/cockroach/pkg/sql/sessionphase [no test files]
30382026-03-10T18:11:44.301Z? github.com/cockroachdb/cockroach/pkg/sql/sqlerrors [no test files]
30392026-03-10T18:11:44.301Z? github.com/cockroachdb/cockroach/pkg/sql/sqlfsm [no test files]
30402026-03-10T18:11:44.304Z? github.com/cockroachdb/cockroach/pkg/sql/sqlinstance [no test files]
30412026-03-10T18:11:46.508Z? github.com/cockroachdb/cockroach/pkg/sql/sqlliveness [no test files]
30422026-03-10T18:11:46.511Z? github.com/cockroachdb/cockroach/pkg/sql/sqlliveness/slprovider [no test files]
30432026-03-10T18:11:47.392Z? github.com/cockroachdb/cockroach/pkg/sql/sqlliveness/sqllivenesstestutils [no test files]
30442026-03-10T18:11:47.392Z? github.com/cockroachdb/cockroach/pkg/sql/sqlstats [no test files]
30452026-03-10T18:11:49.850Z? github.com/cockroachdb/cockroach/pkg/sql/sqlstats/ssmemstorage [no test files]
30462026-03-10T18:11:49.850Z? github.com/cockroachdb/cockroach/pkg/sql/sqltelemetry [no test files]
30472026-03-10T18:11:49.853Z? github.com/cockroachdb/cockroach/pkg/sql/sqltestutils [no test files]
30482026-03-10T18:11:49.853Z? github.com/cockroachdb/cockroach/pkg/sql/sqlutil [no test files]
30492026-03-10T18:12:04.660Z? github.com/cockroachdb/cockroach/pkg/sql/storageparam [no test files]
30502026-03-10T18:12:04.660Z? github.com/cockroachdb/cockroach/pkg/sql/storageparam/indexstorageparam [no test files]
30512026-03-10T18:12:04.663Z? github.com/cockroachdb/cockroach/pkg/sql/storageparam/tablestorageparam [no test files]
30522026-03-10T18:12:08.769Z? github.com/cockroachdb/cockroach/pkg/sql/ttl/ttlschedule [no test files]
30532026-03-10T18:12:13.507Z? github.com/cockroachdb/cockroach/pkg/sql/vtable [no test files]
30542026-03-10T18:12:20.791Z? github.com/cockroachdb/cockroach/pkg/testutils/buildutil [no test files]
30552026-03-10T18:12:20.791Z? github.com/cockroachdb/cockroach/pkg/testutils/colcontainerutils [no test files]
30562026-03-10T18:12:20.791Z? github.com/cockroachdb/cockroach/pkg/testutils/diagutils [no test files]
30572026-03-10T18:12:20.791Z? github.com/cockroachdb/cockroach/pkg/testutils/distsqlutils [no test files]
30582026-03-10T18:12:20.797Z? github.com/cockroachdb/cockroach/pkg/testutils/docker [no test files]
30592026-03-10T18:12:20.843Z? github.com/cockroachdb/cockroach/pkg/testutils/docker/docker-fsnotify [no test files]
30602026-03-10T18:12:20.843Z? github.com/cockroachdb/cockroach/pkg/testutils/echotest [no test files]
30612026-03-10T18:12:20.948Z? github.com/cockroachdb/cockroach/pkg/testutils/gossiputil [no test files]
30622026-03-10T18:12:20.999Z? github.com/cockroachdb/cockroach/pkg/testutils/grpcutils [no test files]
30632026-03-10T18:12:20.999Z? github.com/cockroachdb/cockroach/pkg/testutils/jobutils [no test files]
30642026-03-10T18:12:21.932Z? github.com/cockroachdb/cockroach/pkg/testutils/kvclientutils [no test files]
30652026-03-10T18:12:21.944Z? github.com/cockroachdb/cockroach/pkg/testutils/lint [no test files]
30662026-03-10T18:12:21.960Z? github.com/cockroachdb/cockroach/pkg/testutils/lint/passes/descriptormarshal [no test files]
30672026-03-10T18:12:21.988Z? github.com/cockroachdb/cockroach/pkg/testutils/lint/passes/errcheck [no test files]
30682026-03-10T18:12:21.988Z? github.com/cockroachdb/cockroach/pkg/testutils/lint/passes/errcmp [no test files]
30692026-03-10T18:12:24.342Z? github.com/cockroachdb/cockroach/pkg/testutils/lint/passes/grpcclientconnclose [no test files]
30702026-03-10T18:12:24.354Z? github.com/cockroachdb/cockroach/pkg/testutils/lint/passes/grpcstatuswithdetails [no test files]
30712026-03-10T18:12:26.193Z? github.com/cockroachdb/cockroach/pkg/testutils/lint/passes/returncheck [no test files]
30722026-03-10T18:12:26.884Z? github.com/cockroachdb/cockroach/pkg/testutils/lint/passes/staticcheck [no test files]
30732026-03-10T18:12:27.623Z? github.com/cockroachdb/cockroach/pkg/testutils/localtestcluster [no test files]
30742026-03-10T18:12:27.623Z? github.com/cockroachdb/cockroach/pkg/testutils/metrictestutils [no test files]
30752026-03-10T18:12:27.623Z? github.com/cockroachdb/cockroach/pkg/testutils/pgtest [no test files]
30762026-03-10T18:12:27.623Z? github.com/cockroachdb/cockroach/pkg/testutils/physicalplanutils [no test files]
30772026-03-10T18:12:27.623Z? github.com/cockroachdb/cockroach/pkg/testutils/serverutils [no test files]
30782026-03-10T18:12:27.623Z? github.com/cockroachdb/cockroach/pkg/testutils/skip [no test files]
30792026-03-10T18:12:44.985Z? github.com/cockroachdb/cockroach/pkg/testutils/sstutil [no test files]
30802026-03-10T18:12:44.985Z? github.com/cockroachdb/cockroach/pkg/testutils/storageutils [no test files]
30812026-03-10T18:12:51.641Z? github.com/cockroachdb/cockroach/pkg/ts/catalog [no test files]
30822026-03-10T18:12:51.645Z? github.com/cockroachdb/cockroach/pkg/ts/tspb [no test files]
30832026-03-10T18:12:51.645Z? github.com/cockroachdb/cockroach/pkg/ui [no test files]
30842026-03-10T18:12:51.648Z? github.com/cockroachdb/cockroach/pkg/ui/distoss [no test files]
30852026-03-10T18:12:53.430Z? github.com/cockroachdb/cockroach/pkg/util/arith [no test files]
30862026-03-10T18:12:55.244Z? github.com/cockroachdb/cockroach/pkg/util/bufalloc [no test files]
30872026-03-10T18:12:58.029Z? github.com/cockroachdb/cockroach/pkg/util/cancelchecker [no test files]
30882026-03-10T18:13:04.823Z? github.com/cockroachdb/cockroach/pkg/util/encoding/encodingtype [no test files]
30892026-03-10T18:13:05.527Z? github.com/cockroachdb/cockroach/pkg/util/fileutil [no test files]
30902026-03-10T18:13:06.643Z? github.com/cockroachdb/cockroach/pkg/util/growstack [no test files]
30912026-03-10T18:13:45.138Z? github.com/cockroachdb/cockroach/pkg/util/interval/generic/internal [no test files]
30922026-03-10T18:13:45.138Z? github.com/cockroachdb/cockroach/pkg/util/ioctx [no test files]
30932026-03-10T18:13:45.141Z? github.com/cockroachdb/cockroach/pkg/util/iterutil [no test files]
30942026-03-10T18:13:46.498Z? github.com/cockroachdb/cockroach/pkg/util/jsonbytes [no test files]
30952026-03-10T18:13:46.501Z? github.com/cockroachdb/cockroach/pkg/util/keysutil [no test files]
30962026-03-10T18:13:46.501Z? github.com/cockroachdb/cockroach/pkg/util/leaktest [no test files]
30972026-03-10T18:13:53.587Z? github.com/cockroachdb/cockroach/pkg/util/log/channel [no test files]
30982026-03-10T18:13:53.591Z? github.com/cockroachdb/cockroach/pkg/util/log/gen [no test files]
30992026-03-10T18:15:47.065Z? github.com/cockroachdb/cockroach/pkg/util/log/logflags [no test files]
31002026-03-10T18:15:47.065Z? github.com/cockroachdb/cockroach/pkg/util/log/logpb [no test files]
31012026-03-10T18:15:47.065Z? github.com/cockroachdb/cockroach/pkg/util/log/logtestutils [no test files]
31022026-03-10T18:15:47.065Z? github.com/cockroachdb/cockroach/pkg/util/log/severity [no test files]
31032026-03-10T18:15:48.040Z? github.com/cockroachdb/cockroach/pkg/util/memzipper [no test files]
31042026-03-10T18:15:49.746Z? github.com/cockroachdb/cockroach/pkg/util/pprofutil [no test files]
31052026-03-10T18:15:52.893Z? github.com/cockroachdb/cockroach/pkg/util/sort [no test files]
31062026-03-10T18:15:55.823Z? github.com/cockroachdb/cockroach/pkg/util/stringencoding [no test files]
31072026-03-10T18:15:56.173Z? github.com/cockroachdb/cockroach/pkg/util/system [no test files]
31082026-03-10T18:15:56.531Z? github.com/cockroachdb/cockroach/pkg/util/targz/gen [no test files]
31092026-03-10T18:15:58.511Z? github.com/cockroachdb/cockroach/pkg/util/timeutil/gen [no test files]
31102026-03-10T18:17:03.596Z? github.com/cockroachdb/cockroach/pkg/util/tracing/tracingpb [no test files]
31112026-03-10T18:17:03.596Z? github.com/cockroachdb/cockroach/pkg/util/tracing/tracingservicepb [no test files]
31122026-03-10T18:17:03.596Z? github.com/cockroachdb/cockroach/pkg/util/tracing/tracingui [no test files]
31132026-03-10T18:17:03.604Z? github.com/cockroachdb/cockroach/pkg/util/tracing/zipper [no test files]
31142026-03-10T18:17:03.831Z? github.com/cockroachdb/cockroach/pkg/util/unaccent [no test files]
31152026-03-10T18:17:08.509Z? github.com/cockroachdb/cockroach/pkg/workload/bulkingest [no test files]
31162026-03-10T18:17:08.512Z? github.com/cockroachdb/cockroach/pkg/workload/connectionlatency [no test files]
31172026-03-10T18:17:08.542Z? github.com/cockroachdb/cockroach/pkg/workload/debug [no test files]
31182026-03-10T18:17:08.542Z? github.com/cockroachdb/cockroach/pkg/workload/examples [no test files]
31192026-03-10T18:17:08.793Z? github.com/cockroachdb/cockroach/pkg/workload/geospatial [no test files]
31202026-03-10T18:17:08.797Z? github.com/cockroachdb/cockroach/pkg/workload/indexes [no test files]
31212026-03-10T18:17:08.857Z? github.com/cockroachdb/cockroach/pkg/workload/jsonload [no test files]
31222026-03-10T18:17:08.857Z? github.com/cockroachdb/cockroach/pkg/workload/kv [no test files]
31232026-03-10T18:17:08.893Z? github.com/cockroachdb/cockroach/pkg/workload/ledger [no test files]
31242026-03-10T18:17:09.407Z? github.com/cockroachdb/cockroach/pkg/workload/querybench [no test files]
31252026-03-10T18:17:09.415Z? github.com/cockroachdb/cockroach/pkg/workload/querylog [no test files]
31262026-03-10T18:17:09.415Z? github.com/cockroachdb/cockroach/pkg/workload/queue [no test files]
31272026-03-10T18:17:09.415Z? github.com/cockroachdb/cockroach/pkg/workload/rand [no test files]
31282026-03-10T18:17:09.669Z? github.com/cockroachdb/cockroach/pkg/workload/schemachange [no test files]
31292026-03-10T18:17:09.669Z? github.com/cockroachdb/cockroach/pkg/workload/sqlsmith [no test files]
31302026-03-10T18:17:10.289Z? github.com/cockroachdb/cockroach/pkg/workload/tpccchecks [no test files]
31312026-03-10T18:17:10.292Z? github.com/cockroachdb/cockroach/pkg/workload/tpcds [no test files]
31322026-03-10T18:17:10.320Z? github.com/cockroachdb/cockroach/pkg/workload/ttllogger [no test files]
31332026-03-10T18:18:23.665Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver 1110.081s
31342026-03-10T18:18:23.665Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/abortspan 0.066s
31352026-03-10T18:18:23.665Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply 0.002s
31362026-03-10T18:18:23.665Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval 42.141s
31372026-03-10T18:18:23.665Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval/result 0.046s
31382026-03-10T18:18:23.665Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/closedts 0.019s
31392026-03-10T18:18:23.665Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/closedts/sidetransport 1.830s
31402026-03-10T18:18:23.665Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/closedts/tracker 2.047s
31412026-03-10T18:18:23.665Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency 19.754s
31422026-03-10T18:18:23.665Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock 0.006s
31432026-03-10T18:18:23.665Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/poison 0.009s
31442026-03-10T18:18:23.665Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/gc 1.229s
31452026-03-10T18:18:23.665Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/idalloc 0.150s
31462026-03-10T18:18:23.669Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver 0.137s
31472026-03-10T18:18:23.669Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/liveness 8.960s
31482026-03-10T18:18:23.669Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/loqrecovery 1.473s
31492026-03-10T18:18:23.669Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/protectedts 0.031s
31502026-03-10T18:18:23.669Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/protectedts/ptcache 2.006s
31512026-03-10T18:18:23.669Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/protectedts/ptreconcile 0.403s
31522026-03-10T18:18:23.669Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/protectedts/ptstorage 13.668s
31532026-03-10T18:18:23.669Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/raftentry 2.738s
31542026-03-10T18:18:23.669Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed 0.492s
31552026-03-10T18:18:23.669Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/rditer 0.013s
31562026-03-10T18:18:23.669Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/reports 8.730s
31572026-03-10T18:18:23.669Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch 5.654s
31582026-03-10T18:18:23.669Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset 0.063s
31592026-03-10T18:18:23.669Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/split 0.039s
31602026-03-10T18:18:23.669Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/stateloader 0.061s
31612026-03-10T18:18:23.669Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/tenantrate 0.155s
31622026-03-10T18:18:23.669Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/tscache 2.037s
31632026-03-10T18:18:23.669Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/txnrecovery 0.110s
31642026-03-10T18:18:23.669Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/txnwait 0.015s
31652026-03-10T18:18:23.669Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/uncertainty 0.056s
31662026-03-10T18:18:23.669Zok github.com/cockroachdb/cockroach/pkg/migration/migrationcluster 1.121s
31672026-03-10T19:00:48.823Zinitialized metamorphic constant "max-batch-size" with value 2315
31682026-03-10T19:00:48.823Zinitialized metamorphic constant "max-batch-byte-size" with value 33168512
31692026-03-10T19:00:48.823Zinitialized metamorphic constant "span-reuse-rate" with value 20
31702026-03-10T19:00:48.823Zinitialized metamorphic constant "row-container-rows-per-chunk-shift" with value 1
31712026-03-10T19:00:48.823Zinitialized metamorphic constant "addsst-rewrite-concurrency" with value 15
31722026-03-10T19:00:48.823Zinitialized metamorphic constant "addsst-rewrite-forced" with value true
31732026-03-10T19:00:48.823Zinitialized metamorphic constant "split-scans-right-for-stats-first" with value true
31742026-03-10T19:00:48.823Zinitialized metamorphic constant "kv.rangefeed.use_dedicated_connection_class.enabled" with value true
31752026-03-10T19:00:48.824Zinitialized metamorphic constant "kv.rangefeed.catchup_scan_iterator_optimization.enabled" with value false
31762026-03-10T19:00:48.824Zinitialized metamorphic constant "kv-batch-size" with value 1
31772026-03-10T19:00:48.824Zinitialized metamorphic constant "datum-row-converter-batch-size" with value 1
31782026-03-10T19:00:48.824Zinitialized metamorphic constant "ColIndexJoin-batch-size" with value 1345566
31792026-03-10T19:00:48.824Zinitialized metamorphic constant "inverted-joiner-batch-size" with value 1
31802026-03-10T19:00:48.824Zinitialized metamorphic constant "zig-zag-joiner-batch-size" with value 1
31812026-03-10T19:00:48.824Zinitialized metamorphic constant "async-IE-result-channel-buffer-size" with value 5
31822026-03-10T19:00:48.824Zpanic: test timed out after 1h0m0s
31832026-03-10T19:00:48.824Zrunning tests:
31842026-03-10T19:00:48.826Z TestPauseMigration (59m58s)
31852026-03-10T19:00:48.826Z
31862026-03-10T19:00:48.826Zgoroutine 3660515 [running]:
31872026-03-10T19:00:48.826Ztesting.(*M).startAlarm.func1()
31882026-03-10T19:00:48.826Z /work/go/src/testing/testing.go:2366 +0x385
31892026-03-10T19:00:48.826Zcreated by time.goFunc
31902026-03-10T19:00:48.826Z /work/go/src/time/sleep.go:177 +0x2d
31912026-03-10T19:00:48.826Z
31922026-03-10T19:00:48.826Zgoroutine 1 [chan receive, 59 minutes]:
31932026-03-10T19:00:48.826Ztesting.(*T).Run(0xc000a44820, {0x40ca71c?, 0x0?}, 0x4316308)
31942026-03-10T19:00:48.826Z /work/go/src/testing/testing.go:1750 +0x3ab
31952026-03-10T19:00:48.826Ztesting.runTests.func1(0xc000a44820)
31962026-03-10T19:00:48.826Z /work/go/src/testing/testing.go:2161 +0x37
31972026-03-10T19:00:48.826Ztesting.tRunner(0xc000a44820, 0xc00099fc20)
31982026-03-10T19:00:48.826Z /work/go/src/testing/testing.go:1689 +0xfb
31992026-03-10T19:00:48.826Ztesting.runTests(0xc000a2c528, {0x73e43e0, 0x5, 0x5}, {0x41f225?, 0x100007644e0?, 0x7703de0?})
32002026-03-10T19:00:48.826Z /work/go/src/testing/testing.go:2159 +0x445
32012026-03-10T19:00:48.826Ztesting.(*M).Run(0xc0007819a0)
32022026-03-10T19:00:48.826Z /work/go/src/testing/testing.go:2027 +0x68b
32032026-03-10T19:00:48.826Zgithub.com/cockroachdb/cockroach/pkg/migration/migrationmanager_test.TestMain(0xc0000ae708?)
32042026-03-10T19:00:48.826Z /work/oxidecomputer/cockroach/pkg/migration/migrationmanager/main_test.go:28 +0xe8
32052026-03-10T19:00:48.826Zmain.main()
32062026-03-10T19:00:48.826Z _testmain.go:59 +0x195
32072026-03-10T19:00:48.826Z
32082026-03-10T19:00:48.826Zgoroutine 20 [chan receive]:
32092026-03-10T19:00:48.826Zgithub.com/cockroachdb/cockroach/pkg/util/goschedstats.init.0.func1()
32102026-03-10T19:00:48.826Z /work/oxidecomputer/cockroach/pkg/util/goschedstats/runnable.go:165 +0x159
32112026-03-10T19:00:48.826Zcreated by github.com/cockroachdb/cockroach/pkg/util/goschedstats.init.0 in goroutine 1
32122026-03-10T19:00:48.826Z /work/oxidecomputer/cockroach/pkg/util/goschedstats/runnable.go:157 +0x1a
32132026-03-10T19:00:48.826Z
32142026-03-10T19:00:48.826Zgoroutine 21 [chan receive]:
32152026-03-10T19:00:48.827Zgithub.com/cockroachdb/cockroach/pkg/util/log.flushDaemon()
32162026-03-10T19:00:48.827Z /work/oxidecomputer/cockroach/pkg/util/log/log_flush.go:75 +0x4e
32172026-03-10T19:00:48.827Zcreated by github.com/cockroachdb/cockroach/pkg/util/log.init.5 in goroutine 1
32182026-03-10T19:00:48.827Z /work/oxidecomputer/cockroach/pkg/util/log/log_flush.go:41 +0x1a
32192026-03-10T19:00:48.827Z
32202026-03-10T19:00:48.827Zgoroutine 22 [chan receive, 60 minutes]:
32212026-03-10T19:00:48.827Zgithub.com/cockroachdb/cockroach/pkg/util/log.signalFlusher()
32222026-03-10T19:00:48.827Z /work/oxidecomputer/cockroach/pkg/util/log/log_flush.go:98 +0xd9
32232026-03-10T19:00:48.827Zcreated by github.com/cockroachdb/cockroach/pkg/util/log.init.5 in goroutine 1
32242026-03-10T19:00:48.827Z /work/oxidecomputer/cockroach/pkg/util/log/log_flush.go:42 +0x26
32252026-03-10T19:00:48.827Z
32262026-03-10T19:00:48.827Zgoroutine 34 [syscall, 60 minutes]:
32272026-03-10T19:00:48.827Zos/signal.signal_recv()
32282026-03-10T19:00:48.827Z /work/go/src/runtime/sigqueue.go:152 +0x29
32292026-03-10T19:00:48.827Zos/signal.loop()
32302026-03-10T19:00:48.827Z /work/go/src/os/signal/signal_unix.go:23 +0x13
32312026-03-10T19:00:48.828Zcreated by os/signal.Notify.func1.1 in goroutine 22
32322026-03-10T19:00:48.828Z /work/go/src/os/signal/signal.go:151 +0x1f
32332026-03-10T19:00:48.828Z
32342026-03-10T19:00:48.828Zgoroutine 9099 [chan receive, 59 minutes]:
32352026-03-10T19:00:48.828Zgithub.com/cockroachdb/cockroach/pkg/migration/migrationmanager_test.TestPauseMigration(0xc00130c000)
32362026-03-10T19:00:48.828Z /work/oxidecomputer/cockroach/pkg/migration/migrationmanager/manager_external_test.go:442 +0x506
32372026-03-10T19:00:48.828Ztesting.tRunner(0xc00130c000, 0x4316308)
32382026-03-10T19:00:48.828Z /work/go/src/testing/testing.go:1689 +0xfb
32392026-03-10T19:00:48.832Zcreated by testing.(*T).Run in goroutine 1
32402026-03-10T19:00:48.832Z /work/go/src/testing/testing.go:1742 +0x390
32412026-03-10T19:00:48.832Z
32422026-03-10T19:00:48.832Zgoroutine 9422 [sync.Cond.Wait]:
32432026-03-10T19:00:48.832Zsync.runtime_notifyListWait(0xc0005ac550, 0x3735)
32442026-03-10T19:00:48.832Z /work/go/src/runtime/sema.go:569 +0x159
32452026-03-10T19:00:48.832Zsync.(*Cond).Wait(0x3ae1600?)
32462026-03-10T19:00:48.832Z /work/go/src/sync/cond.go:70 +0x85
32472026-03-10T19:00:48.832Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc0066238f0})
32482026-03-10T19:00:48.832Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
32492026-03-10T19:00:48.832Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
32502026-03-10T19:00:48.832Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
32512026-03-10T19:00:48.832Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
32522026-03-10T19:00:48.832Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
32532026-03-10T19:00:48.832Z
32542026-03-10T19:00:48.832Zgoroutine 9426 [sync.Cond.Wait]:
32552026-03-10T19:00:48.832Zsync.runtime_notifyListWait(0xc0005ac550, 0x373b)
32562026-03-10T19:00:48.832Z /work/go/src/runtime/sema.go:569 +0x159
32572026-03-10T19:00:48.832Zsync.(*Cond).Wait(0x3ae1600?)
32582026-03-10T19:00:48.832Z /work/go/src/sync/cond.go:70 +0x85
32592026-03-10T19:00:48.832Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc0066239b0})
32602026-03-10T19:00:48.832Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
32612026-03-10T19:00:48.832Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
32622026-03-10T19:00:48.832Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
32632026-03-10T19:00:48.832Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
32642026-03-10T19:00:48.833Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
32652026-03-10T19:00:48.833Z
32662026-03-10T19:00:48.833Zgoroutine 10326 [select]:
32672026-03-10T19:00:48.833Zgithub.com/cockroachdb/cockroach/pkg/kv/kvprober.(*Prober).Start.func1.1({0x50e60e0, 0xc002e68900})
32682026-03-10T19:00:48.833Z /work/oxidecomputer/cockroach/pkg/kv/kvprober/kvprober.go:293 +0x314
32692026-03-10T19:00:48.833Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
32702026-03-10T19:00:48.833Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
32712026-03-10T19:00:48.833Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
32722026-03-10T19:00:48.833Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
32732026-03-10T19:00:48.833Z
32742026-03-10T19:00:48.833Zgoroutine 9380 [sync.Cond.Wait]:
32752026-03-10T19:00:48.833Zsync.runtime_notifyListWait(0xc0005ac550, 0x3762)
32762026-03-10T19:00:48.833Z /work/go/src/runtime/sema.go:569 +0x159
32772026-03-10T19:00:48.833Zsync.(*Cond).Wait(0x3ae1600?)
32782026-03-10T19:00:48.833Z /work/go/src/sync/cond.go:70 +0x85
32792026-03-10T19:00:48.833Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006622ed0})
32802026-03-10T19:00:48.833Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
32812026-03-10T19:00:48.833Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
32822026-03-10T19:00:48.833Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
32832026-03-10T19:00:48.833Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
32842026-03-10T19:00:48.833Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
32852026-03-10T19:00:48.833Z
32862026-03-10T19:00:48.833Zgoroutine 9381 [sync.Cond.Wait]:
32872026-03-10T19:00:48.833Zsync.runtime_notifyListWait(0xc0005ac550, 0x376c)
32882026-03-10T19:00:48.835Z /work/go/src/runtime/sema.go:569 +0x159
32892026-03-10T19:00:48.835Zsync.(*Cond).Wait(0x3ae1600?)
32902026-03-10T19:00:48.835Z /work/go/src/sync/cond.go:70 +0x85
32912026-03-10T19:00:48.835Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006622f00})
32922026-03-10T19:00:48.835Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
32932026-03-10T19:00:48.835Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
32942026-03-10T19:00:48.835Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
32952026-03-10T19:00:48.835Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
32962026-03-10T19:00:48.835Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
32972026-03-10T19:00:48.835Z
32982026-03-10T19:00:48.835Zgoroutine 10345 [select, 59 minutes]:
32992026-03-10T19:00:48.835Zdatabase/sql.(*DB).connectionOpener(0xc001a57e10, {0x50e6118, 0xc006c05bd0})
33002026-03-10T19:00:48.835Z /work/go/src/database/sql/sql.go:1246 +0x87
33012026-03-10T19:00:48.835Zcreated by database/sql.OpenDB in goroutine 9099
33022026-03-10T19:00:48.835Z /work/go/src/database/sql/sql.go:824 +0x14c
33032026-03-10T19:00:48.835Z
33042026-03-10T19:00:48.835Zgoroutine 10084 [sync.Cond.Wait, 59 minutes]:
33052026-03-10T19:00:48.835Zsync.runtime_notifyListWait(0xc00676e610, 0x33)
33062026-03-10T19:00:48.835Z /work/go/src/runtime/sema.go:569 +0x159
33072026-03-10T19:00:48.835Zsync.(*Cond).Wait(0x0?)
33082026-03-10T19:00:48.835Z /work/go/src/sync/cond.go:70 +0x85
33092026-03-10T19:00:48.835Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0072ac0c0})
33102026-03-10T19:00:48.835Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
33112026-03-10T19:00:48.835Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
33122026-03-10T19:00:48.835Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
33132026-03-10T19:00:48.835Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
33142026-03-10T19:00:48.835Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
33152026-03-10T19:00:48.835Z
33162026-03-10T19:00:48.835Zgoroutine 9425 [sync.Cond.Wait]:
33172026-03-10T19:00:48.835Zsync.runtime_notifyListWait(0xc0005ac550, 0x3742)
33182026-03-10T19:00:48.835Z /work/go/src/runtime/sema.go:569 +0x159
33192026-03-10T19:00:48.835Zsync.(*Cond).Wait(0x3ae1600?)
33202026-03-10T19:00:48.836Z /work/go/src/sync/cond.go:70 +0x85
33212026-03-10T19:00:48.836Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006623980})
33222026-03-10T19:00:48.836Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
33232026-03-10T19:00:48.836Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
33242026-03-10T19:00:48.836Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
33252026-03-10T19:00:48.836Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
33262026-03-10T19:00:48.836Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
33272026-03-10T19:00:48.836Z
33282026-03-10T19:00:48.836Zgoroutine 9429 [sync.Cond.Wait]:
33292026-03-10T19:00:48.836Zsync.runtime_notifyListWait(0xc0005ac550, 0x3745)
33302026-03-10T19:00:48.836Z /work/go/src/runtime/sema.go:569 +0x159
33312026-03-10T19:00:48.836Zsync.(*Cond).Wait(0x3ae1600?)
33322026-03-10T19:00:48.836Z /work/go/src/sync/cond.go:70 +0x85
33332026-03-10T19:00:48.836Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006623a40})
33342026-03-10T19:00:48.836Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
33352026-03-10T19:00:48.836Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
33362026-03-10T19:00:48.836Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
33372026-03-10T19:00:48.836Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
33382026-03-10T19:00:48.836Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
33392026-03-10T19:00:48.840Z
33402026-03-10T19:00:48.840Zgoroutine 9427 [sync.Cond.Wait]:
33412026-03-10T19:00:48.840Zsync.runtime_notifyListWait(0xc0005ac550, 0x373d)
33422026-03-10T19:00:48.840Z /work/go/src/runtime/sema.go:569 +0x159
33432026-03-10T19:00:48.841Zsync.(*Cond).Wait(0x3ae1600?)
33442026-03-10T19:00:48.841Z /work/go/src/sync/cond.go:70 +0x85
33452026-03-10T19:00:48.841Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc0066239e0})
33462026-03-10T19:00:48.841Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
33472026-03-10T19:00:48.841Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
33482026-03-10T19:00:48.841Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
33492026-03-10T19:00:48.841Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
33502026-03-10T19:00:48.841Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
33512026-03-10T19:00:48.841Z
33522026-03-10T19:00:48.841Zgoroutine 10138 [select, 59 minutes]:
33532026-03-10T19:00:48.841Zgoogle.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc006517080)
33542026-03-10T19:00:48.841Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:112 +0x6d
33552026-03-10T19:00:48.841Zcreated by google.golang.org/grpc.newCCBalancerWrapper in goroutine 9099
33562026-03-10T19:00:48.841Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:73 +0x22b
33572026-03-10T19:00:48.841Z
33582026-03-10T19:00:48.841Zgoroutine 9441 [select]:
33592026-03-10T19:00:48.841Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).startComputePeriodicMetrics.func1({0x50e60e0, 0xc006810090})
33602026-03-10T19:00:48.842Z /work/oxidecomputer/cockroach/pkg/server/node.go:748 +0xff
33612026-03-10T19:00:48.842Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
33622026-03-10T19:00:48.842Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
33632026-03-10T19:00:48.842Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
33642026-03-10T19:00:48.842Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
33652026-03-10T19:00:48.842Z
33662026-03-10T19:00:48.842Zgoroutine 10093 [sync.Cond.Wait, 59 minutes]:
33672026-03-10T19:00:48.842Zsync.runtime_notifyListWait(0xc00676e610, 0x3b)
33682026-03-10T19:00:48.842Z /work/go/src/runtime/sema.go:569 +0x159
33692026-03-10T19:00:48.842Zsync.(*Cond).Wait(0x0?)
33702026-03-10T19:00:48.842Z /work/go/src/sync/cond.go:70 +0x85
33712026-03-10T19:00:48.842Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0072ac270})
33722026-03-10T19:00:48.842Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
33732026-03-10T19:00:48.842Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
33742026-03-10T19:00:48.842Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
33752026-03-10T19:00:48.842Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
33762026-03-10T19:00:48.842Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
33772026-03-10T19:00:48.842Z
33782026-03-10T19:00:48.842Zgoroutine 9417 [sync.Cond.Wait]:
33792026-03-10T19:00:48.842Zsync.runtime_notifyListWait(0xc0005ac550, 0x3774)
33802026-03-10T19:00:48.842Z /work/go/src/runtime/sema.go:569 +0x159
33812026-03-10T19:00:48.842Zsync.(*Cond).Wait(0x3ae1600?)
33822026-03-10T19:00:48.842Z /work/go/src/sync/cond.go:70 +0x85
33832026-03-10T19:00:48.842Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006623770})
33842026-03-10T19:00:48.843Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
33852026-03-10T19:00:48.843Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
33862026-03-10T19:00:48.843Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
33872026-03-10T19:00:48.843Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
33882026-03-10T19:00:48.843Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
33892026-03-10T19:00:48.843Z
33902026-03-10T19:00:48.843Zgoroutine 9320 [chan receive, 59 minutes]:
33912026-03-10T19:00:48.843Zgithub.com/cockroachdb/cockroach/pkg/server.(*httpServer).start.func1({0x50e60e0, 0xc00634df20})
33922026-03-10T19:00:48.843Z /work/oxidecomputer/cockroach/pkg/server/server_http.go:226 +0x45
33932026-03-10T19:00:48.843Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
33942026-03-10T19:00:48.843Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
33952026-03-10T19:00:48.843Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
33962026-03-10T19:00:48.843Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
33972026-03-10T19:00:48.843Z
33982026-03-10T19:00:48.843Zgoroutine 10026 [select]:
33992026-03-10T19:00:48.843Zgithub.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc0036bf400, {0x50e60e0, 0xc006f68900}, 0xc006f46be0, {0xc001982680, 0xf}, 0xc0069698c0)
34002026-03-10T19:00:48.843Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:1462 +0x27b
34012026-03-10T19:00:48.843Zgithub.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1({0x50e60e0, 0xc006f68900})
34022026-03-10T19:00:48.843Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:1380 +0x85
34032026-03-10T19:00:48.843Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
34042026-03-10T19:00:48.843Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
34052026-03-10T19:00:48.843Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10023
34062026-03-10T19:00:48.845Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
34072026-03-10T19:00:48.845Z
34082026-03-10T19:00:48.845Zgoroutine 9222 [chan receive, 59 minutes]:
34092026-03-10T19:00:48.845Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50e60e0?, 0xc005cee2a0?})
34102026-03-10T19:00:48.845Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
34112026-03-10T19:00:48.846Zruntime/pprof.Do({0x50e6038?, 0x778e7c0?}, {{0xc00030e2c0?, 0xc002dfec60?, 0xc002c30120?}}, 0xc0000497a8)
34122026-03-10T19:00:48.846Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
34132026-03-10T19:00:48.846Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc0000497b8?)
34142026-03-10T19:00:48.846Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
34152026-03-10T19:00:48.846Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9099
34162026-03-10T19:00:48.846Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
34172026-03-10T19:00:48.846Z
34182026-03-10T19:00:48.846Zgoroutine 9315 [select, 59 minutes]:
34192026-03-10T19:00:48.846Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initCancelingWorkers.func1({0x50e60e0, 0xc00634c900})
34202026-03-10T19:00:48.846Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:139 +0xc5
34212026-03-10T19:00:48.846Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
34222026-03-10T19:00:48.846Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
34232026-03-10T19:00:48.846Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
34242026-03-10T19:00:48.846Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
34252026-03-10T19:00:48.846Z
34262026-03-10T19:00:48.846Zgoroutine 9311 [select, 59 minutes]:
34272026-03-10T19:00:48.846Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc00088e008?, 0xc0016e5590?})
34282026-03-10T19:00:48.846Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
34292026-03-10T19:00:48.846Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
34302026-03-10T19:00:48.846Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
34312026-03-10T19:00:48.846Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
34322026-03-10T19:00:48.846Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
34332026-03-10T19:00:48.846Z
34342026-03-10T19:00:48.846Zgoroutine 9985 [select]:
34352026-03-10T19:00:48.846Zgithub.com/cockroachdb/cockroach/pkg/sql/sqlliveness/slinstance.(*Instance).heartbeatLoop(0xc001651c80, {0x50e6118, 0xc00137a0f0})
34362026-03-10T19:00:48.846Z /work/oxidecomputer/cockroach/pkg/sql/sqlliveness/slinstance/slinstance.go:244 +0x187
34372026-03-10T19:00:48.846Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
34382026-03-10T19:00:48.846Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
34392026-03-10T19:00:48.850Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
34402026-03-10T19:00:48.850Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
34412026-03-10T19:00:48.850Z
34422026-03-10T19:00:48.850Zgoroutine 10098 [semacquire, 59 minutes]:
34432026-03-10T19:00:48.850Zsync.runtime_Semacquire(0x0?)
34442026-03-10T19:00:48.850Z /work/go/src/runtime/sema.go:62 +0x25
34452026-03-10T19:00:48.850Zsync.(*WaitGroup).Wait(0x0?)
34462026-03-10T19:00:48.850Z /work/go/src/sync/waitgroup.go:116 +0x48
34472026-03-10T19:00:48.850Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Wait(...)
34482026-03-10T19:00:48.850Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:236
34492026-03-10T19:00:48.850Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
34502026-03-10T19:00:48.850Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
34512026-03-10T19:00:48.850Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
34522026-03-10T19:00:48.850Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
34532026-03-10T19:00:48.850Z
34542026-03-10T19:00:48.851Zgoroutine 9310 [select, 59 minutes]:
34552026-03-10T19:00:48.851Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc004377508?, 0xc001b307e0?})
34562026-03-10T19:00:48.851Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
34572026-03-10T19:00:48.851Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
34582026-03-10T19:00:48.851Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
34592026-03-10T19:00:48.851Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
34602026-03-10T19:00:48.851Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
34612026-03-10T19:00:48.851Z
34622026-03-10T19:00:48.851Zgoroutine 9197 [select]:
34632026-03-10T19:00:48.851Zgithub.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1()
34642026-03-10T19:00:48.851Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:71 +0xcb
34652026-03-10T19:00:48.851Zcreated by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 9099
34662026-03-10T19:00:48.851Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:66 +0x58
34672026-03-10T19:00:48.851Z
34682026-03-10T19:00:48.851Zgoroutine 10496 [select]:
34692026-03-10T19:00:48.851Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).outputLoop(0xc002d2d6c0, {0x50e6118, 0xc003c021e0})
34702026-03-10T19:00:48.851Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:310 +0x247
34712026-03-10T19:00:48.851Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).runOutputLoop(0xc002d2d6c0, {0x50e60e0, 0xc006fe6cf0}, 0x1b?)
34722026-03-10T19:00:48.851Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:335 +0xba
34732026-03-10T19:00:48.851Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run.func1({0x50e60e0?, 0xc006fe6cf0?})
34742026-03-10T19:00:48.851Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:297 +0x46
34752026-03-10T19:00:48.851Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
34762026-03-10T19:00:48.851Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
34772026-03-10T19:00:48.851Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10420
34782026-03-10T19:00:48.851Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
34792026-03-10T19:00:48.851Z
34802026-03-10T19:00:48.851Zgoroutine 9790 [select]:
34812026-03-10T19:00:48.851Zgithub.com/cockroachdb/cockroach/pkg/kv/kvprober.(*Prober).Start.func1.1({0x50e60e0, 0xc00457b680})
34822026-03-10T19:00:48.851Z /work/oxidecomputer/cockroach/pkg/kv/kvprober/kvprober.go:293 +0x314
34832026-03-10T19:00:48.851Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
34842026-03-10T19:00:48.851Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
34852026-03-10T19:00:48.851Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
34862026-03-10T19:00:48.851Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
34872026-03-10T19:00:48.851Z
34882026-03-10T19:00:48.851Zgoroutine 10310 [select]:
34892026-03-10T19:00:48.851Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/closedts/sidetransport.(*Sender).Run.func2({0x50e60e0, 0xc0036c0060})
34902026-03-10T19:00:48.851Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/closedts/sidetransport/sender.go:251 +0x1ab
34912026-03-10T19:00:48.851Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
34922026-03-10T19:00:48.851Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
34932026-03-10T19:00:48.851Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
34942026-03-10T19:00:48.852Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
34952026-03-10T19:00:48.853Z
34962026-03-10T19:00:48.853Zgoroutine 9953 [select]:
34972026-03-10T19:00:48.853Zgoogle.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc006685e50, 0x1)
34982026-03-10T19:00:48.853Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:407 +0x113
34992026-03-10T19:00:48.853Zgoogle.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc004b2e2a0)
35002026-03-10T19:00:48.853Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:534 +0x7b
35012026-03-10T19:00:48.853Zgoogle.golang.org/grpc/internal/transport.NewServerTransport.func2()
35022026-03-10T19:00:48.853Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:332 +0xc6
35032026-03-10T19:00:48.853Zcreated by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 9627
35042026-03-10T19:00:48.853Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:329 +0x184c
35052026-03-10T19:00:48.853Z
35062026-03-10T19:00:48.853Zgoroutine 9289 [chan receive, 59 minutes]:
35072026-03-10T19:00:48.853Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50e60e0?, 0xc005cee810?})
35082026-03-10T19:00:48.853Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
35092026-03-10T19:00:48.853Zruntime/pprof.Do({0x50e6038?, 0x778e7c0?}, {{0xc00030e2c0?, 0x10?, 0xc0032d07d0?}}, 0xc0032d07a8)
35102026-03-10T19:00:48.853Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
35112026-03-10T19:00:48.854Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x3?)
35122026-03-10T19:00:48.854Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
35132026-03-10T19:00:48.854Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9099
35142026-03-10T19:00:48.854Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
35152026-03-10T19:00:48.854Z
35162026-03-10T19:00:48.854Zgoroutine 10371 [select]:
35172026-03-10T19:00:48.854Zgoogle.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0053c96d0, 0x1)
35182026-03-10T19:00:48.854Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:407 +0x113
35192026-03-10T19:00:48.854Zgoogle.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc003dab5c0)
35202026-03-10T19:00:48.854Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:534 +0x7b
35212026-03-10T19:00:48.854Zgoogle.golang.org/grpc/internal/transport.NewServerTransport.func2()
35222026-03-10T19:00:48.854Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:332 +0xc6
35232026-03-10T19:00:48.854Zcreated by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 10304
35242026-03-10T19:00:48.854Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:329 +0x184c
35252026-03-10T19:00:48.854Z
35262026-03-10T19:00:48.854Zgoroutine 9597 [select, 59 minutes]:
35272026-03-10T19:00:48.854Zgithub.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowScheduler).Start.func1({0xc001f13b08?, 0xc006037260?})
35282026-03-10T19:00:48.854Z /work/oxidecomputer/cockroach/pkg/sql/flowinfra/flow_scheduler.go:331 +0x25c
35292026-03-10T19:00:48.854Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
35302026-03-10T19:00:48.854Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
35312026-03-10T19:00:48.854Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
35322026-03-10T19:00:48.854Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
35332026-03-10T19:00:48.854Z
35342026-03-10T19:00:48.854Zgoroutine 10104 [select]:
35352026-03-10T19:00:48.854Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).startGossip.func3({0x50e60e0, 0xc0072ac4e0})
35362026-03-10T19:00:48.854Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:2152 +0x225
35372026-03-10T19:00:48.854Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
35382026-03-10T19:00:48.854Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
35392026-03-10T19:00:48.859Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
35402026-03-10T19:00:48.859Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
35412026-03-10T19:00:48.859Z
35422026-03-10T19:00:48.859Zgoroutine 9691 [semacquire, 59 minutes]:
35432026-03-10T19:00:48.859Zsync.runtime_Semacquire(0xc001ea2498?)
35442026-03-10T19:00:48.859Z /work/go/src/runtime/sema.go:62 +0x25
35452026-03-10T19:00:48.859Zsync.(*WaitGroup).Wait(0x44f7e0?)
35462026-03-10T19:00:48.859Z /work/go/src/sync/waitgroup.go:116 +0x48
35472026-03-10T19:00:48.859Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc004afde00)
35482026-03-10T19:00:48.859Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
35492026-03-10T19:00:48.859Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc004afde00?, {0x50e6118?, 0xc0046a69b0?}})
35502026-03-10T19:00:48.859Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
35512026-03-10T19:00:48.860Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed(0xc00432cf08, {0x50e6118?, 0xc0046a68c0?}, {0xc005f94240, 0x1, 0x33f4ac0?}, {0xc002745f00?, 0xb985f9?, 0x0?}, 0x1, ...)
35522026-03-10T19:00:48.860Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:139 +0x6c5
35532026-03-10T19:00:48.860Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*dbAdapter).RangeFeed(0x1d0ae7ee9624f6?, {0x50e6118?, 0xc0046a68c0?}, {0xc005f94240?, 0x0?, 0xc001e9ea08?}, {0x50e60e0?, 0x2c08cf0?, 0xc0?}, 0x1, ...)
35542026-03-10T19:00:48.860Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/db_adapter.go:78 +0x3e
35552026-03-10T19:00:48.860Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func1({0x50e6118?, 0xc0046a68c0?})
35562026-03-10T19:00:48.860Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:285 +0x6c
35572026-03-10T19:00:48.860Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
35582026-03-10T19:00:48.860Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
35592026-03-10T19:00:48.860Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
35602026-03-10T19:00:48.861Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
35612026-03-10T19:00:48.861Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9671
35622026-03-10T19:00:48.861Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
35632026-03-10T19:00:48.861Z
35642026-03-10T19:00:48.861Zgoroutine 10132 [select]:
35652026-03-10T19:00:48.861Zgoogle.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc006dd6140, 0x1)
35662026-03-10T19:00:48.861Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:407 +0x113
35672026-03-10T19:00:48.861Zgoogle.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc004d20960)
35682026-03-10T19:00:48.861Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:534 +0x7b
35692026-03-10T19:00:48.861Zgoogle.golang.org/grpc/internal/transport.NewServerTransport.func2()
35702026-03-10T19:00:48.862Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:332 +0xc6
35712026-03-10T19:00:48.862Zcreated by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 10149
35722026-03-10T19:00:48.862Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:329 +0x184c
35732026-03-10T19:00:48.862Z
35742026-03-10T19:00:48.862Zgoroutine 10414 [chan receive, 59 minutes]:
35752026-03-10T19:00:48.862Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).rangeFeedWithRangeID(0xc0019cc808, 0x3dbc860?, 0xc002de1e60, {0x7f784cb862a8, 0xc002773c20})
35762026-03-10T19:00:48.862Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:249 +0x76b
35772026-03-10T19:00:48.862Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed(...)
35782026-03-10T19:00:48.862Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:149
35792026-03-10T19:00:48.862Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).RangeFeed(0xc001668008, 0xc002de1e60, {0x5119e20, 0xc002773c20})
35802026-03-10T19:00:48.862Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:3082 +0x16f
35812026-03-10T19:00:48.863Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).RangeFeed(0xc006849e00, 0xc002de1e60, {0x5119e20, 0xc002773c20})
35822026-03-10T19:00:48.863Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/stores.go:216 +0x19a
35832026-03-10T19:00:48.863Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed(0x7366f90?, 0x403a5e0?, {0x5119e20, 0xc002773c20})
35842026-03-10T19:00:48.863Z /work/oxidecomputer/cockroach/pkg/server/node.go:1232 +0x28
35852026-03-10T19:00:48.863Zgithub.com/cockroachdb/cockroach/pkg/roachpb._Internal_RangeFeed_Handler({0x403a5e0, 0xc001668908}, {0x5114010, 0xc000f107e0})
35862026-03-10T19:00:48.863Z /work/oxidecomputer/cockroach/pkg/roachpb/api.pb.go:9523 +0x107
35872026-03-10T19:00:48.863Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.StreamServerInterceptor.func14({0x403a5e0, 0xc001668908}, {0x51147c0, 0xc0044988f0}, 0xc0016dde18, 0x4319298)
35882026-03-10T19:00:48.863Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:205 +0x35b
35892026-03-10T19:00:48.863Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x403a5e0?, 0xc001668908?}, {0x51147c0?, 0xc0044988f0?})
35902026-03-10T19:00:48.863Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1408 +0x59
35912026-03-10T19:00:48.863Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func4({0x403a5e0, 0xc001668908}, {0x51147c0, 0xc0044988f0}, 0xc000946918?, 0xc0007911c0)
35922026-03-10T19:00:48.863Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:247 +0x6c
35932026-03-10T19:00:48.863Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x403a5e0?, 0xc001668908?}, {0x51147c0?, 0xc0044988f0?})
35942026-03-10T19:00:48.863Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
35952026-03-10T19:00:48.863Zgithub.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({{{0x2a?}}}, {0x403a5e0, 0xc001668908}, {0x51147c0, 0xc0044988f0}, 0xc0016dde18, 0xc0007911c0)
35962026-03-10T19:00:48.863Z /work/oxidecomputer/cockroach/pkg/rpc/auth.go:98 +0x19b
35972026-03-10T19:00:48.863Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x403a5e0?, 0xc001668908?}, {0x51147c0?, 0xc0044988f0?})
35982026-03-10T19:00:48.863Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
35992026-03-10T19:00:48.864Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func2.1({0x66?, 0x40?})
36002026-03-10T19:00:48.864Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:216 +0x27
36012026-03-10T19:00:48.864Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc006848400, {0x50e60e0, 0xc002a50480}, {0xc000946b28?, 0x416dfb?}, 0xc000946ac0)
36022026-03-10T19:00:48.864Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:344 +0xcb
36032026-03-10T19:00:48.865Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func2({0x403a5e0, 0xc001668908}, {0x51147c0, 0xc0044988f0}, 0xc0016dde18, 0xc0007911c0)
36042026-03-10T19:00:48.865Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:215 +0xdf
36052026-03-10T19:00:48.865Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x403a5e0?, 0xc001668908?}, {0x51147c0?, 0xc0044988f0?})
36062026-03-10T19:00:48.865Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
36072026-03-10T19:00:48.865Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x403a5e0, 0xc001668908}, {0x51147c0, 0xc0044988f0}, 0xc0016dde18, 0x4319298)
36082026-03-10T19:00:48.866Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1413 +0x108
36092026-03-10T19:00:48.866Zgoogle.golang.org/grpc.(*Server).processStreamingRPC(0xc005fee1c0, {0x5126b60, 0xc00130c340}, 0xc002de17a0, 0xc0077eb920, 0x73e2b80, 0x0)
36102026-03-10T19:00:48.866Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1549 +0xe51
36112026-03-10T19:00:48.866Zgoogle.golang.org/grpc.(*Server).handleStream(0xc005fee1c0, {0x5126b60, 0xc00130c340}, 0xc002de17a0, 0x0)
36122026-03-10T19:00:48.866Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1624 +0x999
36132026-03-10T19:00:48.866Zgoogle.golang.org/grpc.(*Server).serveStreams.func1.2()
36142026-03-10T19:00:48.866Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:922 +0x8d
36152026-03-10T19:00:48.866Zcreated by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 10187
36162026-03-10T19:00:48.866Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:920 +0x226
36172026-03-10T19:00:48.866Z
36182026-03-10T19:00:48.866Zgoroutine 9478 [select]:
36192026-03-10T19:00:48.867Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).startWriteNodeStatus.func2({0x50e60e0, 0xc002de6960})
36202026-03-10T19:00:48.867Z /work/oxidecomputer/cockroach/pkg/server/node.go:852 +0xf3
36212026-03-10T19:00:48.867Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
36222026-03-10T19:00:48.867Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
36232026-03-10T19:00:48.867Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
36242026-03-10T19:00:48.867Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
36252026-03-10T19:00:48.867Z
36262026-03-10T19:00:48.867Zgoroutine 9224 [chan receive, 59 minutes]:
36272026-03-10T19:00:48.867Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50e60e0?, 0xc005cee240?})
36282026-03-10T19:00:48.867Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
36292026-03-10T19:00:48.867Zruntime/pprof.Do({0x50e6038?, 0x778e7c0?}, {{0xc00030e2c0?, 0x10000c000d3e080?, 0xc0015aa4d0?}}, 0xc002885fa8)
36302026-03-10T19:00:48.867Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
36312026-03-10T19:00:48.867Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc0061b6de0?)
36322026-03-10T19:00:48.867Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
36332026-03-10T19:00:48.867Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9099
36342026-03-10T19:00:48.867Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
36352026-03-10T19:00:48.867Z
36362026-03-10T19:00:48.868Zgoroutine 10475 [chan receive, 59 minutes]:
36372026-03-10T19:00:48.868Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).rangeFeedWithRangeID(0xc004e66008, 0x3dbc860?, 0xc00383c900, {0x7f784cb862a8, 0xc004688ee0})
36382026-03-10T19:00:48.868Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:249 +0x76b
36392026-03-10T19:00:48.873Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed(...)
36402026-03-10T19:00:48.873Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:149
36412026-03-10T19:00:48.873Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).RangeFeed(0xc001668008, 0xc00383c900, {0x5119e20, 0xc004688ee0})
36422026-03-10T19:00:48.873Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:3082 +0x16f
36432026-03-10T19:00:48.873Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).RangeFeed(0xc006849e00, 0xc00383c900, {0x5119e20, 0xc004688ee0})
36442026-03-10T19:00:48.873Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/stores.go:216 +0x19a
36452026-03-10T19:00:48.873Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed(0x7366f90?, 0x403a5e0?, {0x5119e20, 0xc004688ee0})
36462026-03-10T19:00:48.873Z /work/oxidecomputer/cockroach/pkg/server/node.go:1232 +0x28
36472026-03-10T19:00:48.873Zgithub.com/cockroachdb/cockroach/pkg/roachpb._Internal_RangeFeed_Handler({0x403a5e0, 0xc001668908}, {0x5114010, 0xc001f0e880})
36482026-03-10T19:00:48.873Z /work/oxidecomputer/cockroach/pkg/roachpb/api.pb.go:9523 +0x107
36492026-03-10T19:00:48.873Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.StreamServerInterceptor.func14({0x403a5e0, 0xc001668908}, {0x51147c0, 0xc006615c70}, 0xc001ed1a40, 0x4319298)
36502026-03-10T19:00:48.873Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:205 +0x35b
36512026-03-10T19:00:48.873Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x403a5e0?, 0xc001668908?}, {0x51147c0?, 0xc006615c70?})
36522026-03-10T19:00:48.873Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1408 +0x59
36532026-03-10T19:00:48.873Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func4({0x403a5e0, 0xc001668908}, {0x51147c0, 0xc006615c70}, 0x3dc2920?, 0xc000e6e380)
36542026-03-10T19:00:48.873Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:247 +0x6c
36552026-03-10T19:00:48.873Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x403a5e0?, 0xc001668908?}, {0x51147c0?, 0xc006615c70?})
36562026-03-10T19:00:48.874Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
36572026-03-10T19:00:48.874Zgithub.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({{{0x6cd58e?}}}, {0x403a5e0, 0xc001668908}, {0x51147c0, 0xc006615c70}, 0xc001ed1a40, 0xc000e6e380)
36582026-03-10T19:00:48.874Z /work/oxidecomputer/cockroach/pkg/rpc/auth.go:98 +0x19b
36592026-03-10T19:00:48.874Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x403a5e0?, 0xc001668908?}, {0x51147c0?, 0xc006615c70?})
36602026-03-10T19:00:48.874Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
36612026-03-10T19:00:48.874Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func2.1({0x66?, 0x40?})
36622026-03-10T19:00:48.874Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:216 +0x27
36632026-03-10T19:00:48.874Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc006848400, {0x50e60e0, 0xc0064b1e90}, {0xc005e75b28?, 0x416dfb?}, 0xc005e75ac0)
36642026-03-10T19:00:48.874Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:344 +0xcb
36652026-03-10T19:00:48.874Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func2({0x403a5e0, 0xc001668908}, {0x51147c0, 0xc006615c70}, 0xc001ed1a40, 0xc000e6e380)
36662026-03-10T19:00:48.874Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:215 +0xdf
36672026-03-10T19:00:48.874Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x403a5e0?, 0xc001668908?}, {0x51147c0?, 0xc006615c70?})
36682026-03-10T19:00:48.874Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
36692026-03-10T19:00:48.875Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x403a5e0, 0xc001668908}, {0x51147c0, 0xc006615c70}, 0xc001ed1a40, 0x4319298)
36702026-03-10T19:00:48.875Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1413 +0x108
36712026-03-10T19:00:48.875Zgoogle.golang.org/grpc.(*Server).processStreamingRPC(0xc005fee1c0, {0x5126b60, 0xc00130c340}, 0xc00383c7e0, 0xc0077eb920, 0x73e2b80, 0x0)
36722026-03-10T19:00:48.875Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1549 +0xe51
36732026-03-10T19:00:48.875Zgoogle.golang.org/grpc.(*Server).handleStream(0xc005fee1c0, {0x5126b60, 0xc00130c340}, 0xc00383c7e0, 0x0)
36742026-03-10T19:00:48.875Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1624 +0x999
36752026-03-10T19:00:48.875Zgoogle.golang.org/grpc.(*Server).serveStreams.func1.2()
36762026-03-10T19:00:48.875Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:922 +0x8d
36772026-03-10T19:00:48.875Zcreated by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 10187
36782026-03-10T19:00:48.875Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:920 +0x226
36792026-03-10T19:00:48.875Z
36802026-03-10T19:00:48.875Zgoroutine 10179 [select]:
36812026-03-10T19:00:48.875Zgoogle.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc0013f6780)
36822026-03-10T19:00:48.875Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1572 +0x137
36832026-03-10T19:00:48.875Zcreated by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 10139
36842026-03-10T19:00:48.875Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:360 +0x180b
36852026-03-10T19:00:48.876Z
36862026-03-10T19:00:48.876Zgoroutine 10193 [chan receive, 59 minutes]:
36872026-03-10T19:00:48.876Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).rangeFeedWithRangeID(0xc004e66008, 0x3dbc860?, 0xc000c2a360, {0x7f784cb862a8, 0xc006877c20})
36882026-03-10T19:00:48.876Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:249 +0x76b
36892026-03-10T19:00:48.876Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed(...)
36902026-03-10T19:00:48.876Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:149
36912026-03-10T19:00:48.876Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).RangeFeed(0xc001668008, 0xc000c2a360, {0x5119e20, 0xc006877c20})
36922026-03-10T19:00:48.876Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:3082 +0x16f
36932026-03-10T19:00:48.876Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).RangeFeed(0xc006849e00, 0xc000c2a360, {0x5119e20, 0xc006877c20})
36942026-03-10T19:00:48.876Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/stores.go:216 +0x19a
36952026-03-10T19:00:48.876Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed(0x7366f90?, 0x403a5e0?, {0x5119e20, 0xc006877c20})
36962026-03-10T19:00:48.876Z /work/oxidecomputer/cockroach/pkg/server/node.go:1232 +0x28
36972026-03-10T19:00:48.876Zgithub.com/cockroachdb/cockroach/pkg/roachpb._Internal_RangeFeed_Handler({0x403a5e0, 0xc001668908}, {0x5114010, 0xc0074294a0})
36982026-03-10T19:00:48.877Z /work/oxidecomputer/cockroach/pkg/roachpb/api.pb.go:9523 +0x107
36992026-03-10T19:00:48.877Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.StreamServerInterceptor.func14({0x403a5e0, 0xc001668908}, {0x51147c0, 0xc006606d00}, 0xc00054b740, 0x4319298)
37002026-03-10T19:00:48.878Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:205 +0x35b
37012026-03-10T19:00:48.878Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x403a5e0?, 0xc001668908?}, {0x51147c0?, 0xc006606d00?})
37022026-03-10T19:00:48.878Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1408 +0x59
37032026-03-10T19:00:48.878Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func4({0x403a5e0, 0xc001668908}, {0x51147c0, 0xc006606d00}, 0xcd?, 0xc004dbaec0)
37042026-03-10T19:00:48.878Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:247 +0x6c
37052026-03-10T19:00:48.878Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x403a5e0?, 0xc001668908?}, {0x51147c0?, 0xc006606d00?})
37062026-03-10T19:00:48.878Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
37072026-03-10T19:00:48.878Zgithub.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({{{0x41f225?}}}, {0x403a5e0, 0xc001668908}, {0x51147c0, 0xc006606d00}, 0xc00054b740, 0xc004dbaec0)
37082026-03-10T19:00:48.878Z /work/oxidecomputer/cockroach/pkg/rpc/auth.go:98 +0x19b
37092026-03-10T19:00:48.878Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x403a5e0?, 0xc001668908?}, {0x51147c0?, 0xc006606d00?})
37102026-03-10T19:00:48.878Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
37112026-03-10T19:00:48.878Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func2.1({0x66?, 0x40?})
37122026-03-10T19:00:48.878Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:216 +0x27
37132026-03-10T19:00:48.878Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc006848400, {0x50e60e0, 0xc001b3ccf0}, {0xc002e5bb28?, 0x416dfb?}, 0xc002e5bac0)
37142026-03-10T19:00:48.878Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:344 +0xcb
37152026-03-10T19:00:48.878Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func2({0x403a5e0, 0xc001668908}, {0x51147c0, 0xc006606d00}, 0xc00054b740, 0xc004dbaec0)
37162026-03-10T19:00:48.878Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:215 +0xdf
37172026-03-10T19:00:48.878Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x403a5e0?, 0xc001668908?}, {0x51147c0?, 0xc006606d00?})
37182026-03-10T19:00:48.878Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
37192026-03-10T19:00:48.878Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x403a5e0, 0xc001668908}, {0x51147c0, 0xc006606d00}, 0xc00054b740, 0x4319298)
37202026-03-10T19:00:48.878Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1413 +0x108
37212026-03-10T19:00:48.878Zgoogle.golang.org/grpc.(*Server).processStreamingRPC(0xc005fee1c0, {0x5126b60, 0xc00130c340}, 0xc0016e9c20, 0xc0077eb920, 0x73e2b80, 0x0)
37222026-03-10T19:00:48.878Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1549 +0xe51
37232026-03-10T19:00:48.878Zgoogle.golang.org/grpc.(*Server).handleStream(0xc005fee1c0, {0x5126b60, 0xc00130c340}, 0xc0016e9c20, 0x0)
37242026-03-10T19:00:48.878Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1624 +0x999
37252026-03-10T19:00:48.878Zgoogle.golang.org/grpc.(*Server).serveStreams.func1.2()
37262026-03-10T19:00:48.878Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:922 +0x8d
37272026-03-10T19:00:48.878Zcreated by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 10187
37282026-03-10T19:00:48.878Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:920 +0x226
37292026-03-10T19:00:48.878Z
37302026-03-10T19:00:48.878Zgoroutine 9379 [sync.Cond.Wait]:
37312026-03-10T19:00:48.878Zsync.runtime_notifyListWait(0xc0005ac550, 0x375d)
37322026-03-10T19:00:48.878Z /work/go/src/runtime/sema.go:569 +0x159
37332026-03-10T19:00:48.878Zsync.(*Cond).Wait(0x3ae1600?)
37342026-03-10T19:00:48.879Z /work/go/src/sync/cond.go:70 +0x85
37352026-03-10T19:00:48.879Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006622ea0})
37362026-03-10T19:00:48.879Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
37372026-03-10T19:00:48.879Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
37382026-03-10T19:00:48.879Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
37392026-03-10T19:00:48.883Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
37402026-03-10T19:00:48.883Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
37412026-03-10T19:00:48.883Z
37422026-03-10T19:00:48.883Zgoroutine 10075 [sync.Cond.Wait, 59 minutes]:
37432026-03-10T19:00:48.883Zsync.runtime_notifyListWait(0xc00676e610, 0x27)
37442026-03-10T19:00:48.883Z /work/go/src/runtime/sema.go:569 +0x159
37452026-03-10T19:00:48.884Zsync.(*Cond).Wait(0x0?)
37462026-03-10T19:00:48.884Z /work/go/src/sync/cond.go:70 +0x85
37472026-03-10T19:00:48.884Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c9e90})
37482026-03-10T19:00:48.884Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
37492026-03-10T19:00:48.884Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
37502026-03-10T19:00:48.884Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
37512026-03-10T19:00:48.884Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
37522026-03-10T19:00:48.884Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
37532026-03-10T19:00:48.884Z
37542026-03-10T19:00:48.884Zgoroutine 9103 [select]:
37552026-03-10T19:00:48.884Zgithub.com/cockroachdb/cockroach/pkg/sql/contention.(*eventStore).startResolver.func1({0x50e60e0, 0xc002f6be90})
37562026-03-10T19:00:48.884Z /work/oxidecomputer/cockroach/pkg/sql/contention/event_store.go:219 +0x1f4
37572026-03-10T19:00:48.884Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
37582026-03-10T19:00:48.884Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
37592026-03-10T19:00:48.884Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
37602026-03-10T19:00:48.884Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
37612026-03-10T19:00:48.884Z
37622026-03-10T19:00:48.885Zgoroutine 9400 [sync.Cond.Wait]:
37632026-03-10T19:00:48.885Zsync.runtime_notifyListWait(0xc0005ac550, 0x3736)
37642026-03-10T19:00:48.885Z /work/go/src/runtime/sema.go:569 +0x159
37652026-03-10T19:00:48.885Zsync.(*Cond).Wait(0x3ae1600?)
37662026-03-10T19:00:48.885Z /work/go/src/sync/cond.go:70 +0x85
37672026-03-10T19:00:48.885Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc0066233b0})
37682026-03-10T19:00:48.885Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
37692026-03-10T19:00:48.885Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
37702026-03-10T19:00:48.885Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
37712026-03-10T19:00:48.885Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
37722026-03-10T19:00:48.885Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
37732026-03-10T19:00:48.885Z
37742026-03-10T19:00:48.885Zgoroutine 9231 [select]:
37752026-03-10T19:00:48.885Zgithub.com/cockroachdb/cockroach/pkg/sql/contention.(*eventStore).startEventIntake.func3({0xc0067f6008?, 0x0?})
37762026-03-10T19:00:48.885Z /work/oxidecomputer/cockroach/pkg/sql/contention/event_store.go:188 +0xa5
37772026-03-10T19:00:48.885Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
37782026-03-10T19:00:48.885Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
37792026-03-10T19:00:48.885Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
37802026-03-10T19:00:48.885Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
37812026-03-10T19:00:48.885Z
37822026-03-10T19:00:48.885Zgoroutine 9367 [sync.Cond.Wait]:
37832026-03-10T19:00:48.885Zsync.runtime_notifyListWait(0xc0005ac550, 0x3740)
37842026-03-10T19:00:48.885Z /work/go/src/runtime/sema.go:569 +0x159
37852026-03-10T19:00:48.885Zsync.(*Cond).Wait(0x3ae1600?)
37862026-03-10T19:00:48.886Z /work/go/src/sync/cond.go:70 +0x85
37872026-03-10T19:00:48.886Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006622c30})
37882026-03-10T19:00:48.886Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
37892026-03-10T19:00:48.886Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
37902026-03-10T19:00:48.886Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
37912026-03-10T19:00:48.887Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
37922026-03-10T19:00:48.887Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
37932026-03-10T19:00:48.887Z
37942026-03-10T19:00:48.887Zgoroutine 10208 [select, 1 minutes]:
37952026-03-10T19:00:48.887Zgithub.com/cockroachdb/cockroach/pkg/sql/catalog/lease.(*Manager).PeriodicallyRefreshSomeLeases.func1({0x50e60e0, 0xc005c71b60})
37962026-03-10T19:00:48.887Z /work/oxidecomputer/cockroach/pkg/sql/catalog/lease/lease.go:1151 +0x2df
37972026-03-10T19:00:48.887Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
37982026-03-10T19:00:48.887Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
37992026-03-10T19:00:48.887Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
38002026-03-10T19:00:48.887Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
38012026-03-10T19:00:48.887Z
38022026-03-10T19:00:48.887Zgoroutine 9193 [chan receive, 59 minutes]:
38032026-03-10T19:00:48.887Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50e60e0?, 0xc00634c0f0?})
38042026-03-10T19:00:48.887Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
38052026-03-10T19:00:48.887Zruntime/pprof.Do({0x50e6038?, 0x778e7c0?}, {{0xc00030e2c0?, 0x10000c002388e00?, 0x5126b60?}}, 0xc00004d7a8)
38062026-03-10T19:00:48.887Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
38072026-03-10T19:00:48.887Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc00004d7b8?)
38082026-03-10T19:00:48.888Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
38092026-03-10T19:00:48.888Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9099
38102026-03-10T19:00:48.888Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
38112026-03-10T19:00:48.888Z
38122026-03-10T19:00:48.888Zgoroutine 9234 [select, 59 minutes]:
38132026-03-10T19:00:48.888Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc0067f6908?, 0x0?})
38142026-03-10T19:00:48.888Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
38152026-03-10T19:00:48.888Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
38162026-03-10T19:00:48.888Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
38172026-03-10T19:00:48.888Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
38182026-03-10T19:00:48.888Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
38192026-03-10T19:00:48.888Z
38202026-03-10T19:00:48.888Zgoroutine 9059 [select, 59 minutes]:
38212026-03-10T19:00:48.888Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1({0x50e6118, 0xc000ff8320})
38222026-03-10T19:00:48.889Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:116 +0x157
38232026-03-10T19:00:48.889Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.Group.GoCtx.func3()
38242026-03-10T19:00:48.889Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
38252026-03-10T19:00:48.889Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
38262026-03-10T19:00:48.889Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
38272026-03-10T19:00:48.889Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9100
38282026-03-10T19:00:48.889Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
38292026-03-10T19:00:48.889Z
38302026-03-10T19:00:48.889Zgoroutine 9278 [select]:
38312026-03-10T19:00:48.889Zgithub.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
38322026-03-10T19:00:48.889Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:439 +0x1c9
38332026-03-10T19:00:48.889Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 9099
38342026-03-10T19:00:48.889Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:412 +0x4f
38352026-03-10T19:00:48.890Z
38362026-03-10T19:00:48.890Zgoroutine 9164 [select, 59 minutes]:
38372026-03-10T19:00:48.890Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2({0x50e60e0, 0xc001eac2d0})
38382026-03-10T19:00:48.890Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/queue.go:823 +0xff
38392026-03-10T19:00:48.894Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
38402026-03-10T19:00:48.894Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
38412026-03-10T19:00:48.894Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9437
38422026-03-10T19:00:48.894Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
38432026-03-10T19:00:48.894Z
38442026-03-10T19:00:48.894Zgoroutine 10336 [select, 59 minutes]:
38452026-03-10T19:00:48.894Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1({0x50e6118, 0xc0053af680})
38462026-03-10T19:00:48.894Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:116 +0x157
38472026-03-10T19:00:48.895Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.Group.GoCtx.func3()
38482026-03-10T19:00:48.895Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
38492026-03-10T19:00:48.895Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
38502026-03-10T19:00:48.895Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
38512026-03-10T19:00:48.895Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10334
38522026-03-10T19:00:48.895Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
38532026-03-10T19:00:48.895Z
38542026-03-10T19:00:48.895Zgoroutine 9480 [select]:
38552026-03-10T19:00:48.895Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.(*Watcher).Run(0xc0045cfa70, {0x50e6118, 0xc00408b270})
38562026-03-10T19:00:48.895Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:319 +0x955
38572026-03-10T19:00:48.895Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.Start.func1({0x50e60e0?, 0xc002de6ae0?})
38582026-03-10T19:00:48.895Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:189 +0x2d1
38592026-03-10T19:00:48.895Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
38602026-03-10T19:00:48.895Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
38612026-03-10T19:00:48.895Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
38622026-03-10T19:00:48.895Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
38632026-03-10T19:00:48.895Z
38642026-03-10T19:00:48.895Zgoroutine 9286 [chan receive, 59 minutes]:
38652026-03-10T19:00:48.895Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50e60e0?, 0xc005cee690?})
38662026-03-10T19:00:48.895Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
38672026-03-10T19:00:48.895Zruntime/pprof.Do({0x50e6038?, 0x778e7c0?}, {{0xc00030e2c0?, 0x12?, 0x0?}}, 0xc003bb6fa8)
38682026-03-10T19:00:48.895Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
38692026-03-10T19:00:48.895Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x1?)
38702026-03-10T19:00:48.895Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
38712026-03-10T19:00:48.895Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9099
38722026-03-10T19:00:48.895Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
38732026-03-10T19:00:48.898Z
38742026-03-10T19:00:48.898Zgoroutine 9283 [select]:
38752026-03-10T19:00:48.898Zgithub.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).startTickerLocked.func1()
38762026-03-10T19:00:48.898Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:365 +0xd2
38772026-03-10T19:00:48.898Zcreated by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).startTickerLocked in goroutine 9099
38782026-03-10T19:00:48.898Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:360 +0x6e
38792026-03-10T19:00:48.898Z
38802026-03-10T19:00:48.898Zgoroutine 9661 [chan receive, 59 minutes]:
38812026-03-10T19:00:48.898Zgithub.com/cockroachdb/cmux.muxListener.Accept(...)
38822026-03-10T19:00:48.899Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184
38832026-03-10T19:00:48.899Znet/http.(*Server).Serve(0xc001a42870, {0x50d12a0, 0xc003438558})
38842026-03-10T19:00:48.899Z /work/go/src/net/http/server.go:3260 +0x33e
38852026-03-10T19:00:48.899Zgithub.com/cockroachdb/cockroach/pkg/server.(*httpServer).start.func3({0xc00670cc08?, 0xc002f5e1e0?})
38862026-03-10T19:00:48.899Z /work/oxidecomputer/cockroach/pkg/server/server_http.go:264 +0x126
38872026-03-10T19:00:48.899Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
38882026-03-10T19:00:48.900Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
38892026-03-10T19:00:48.900Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
38902026-03-10T19:00:48.900Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
38912026-03-10T19:00:48.900Z
38922026-03-10T19:00:48.900Zgoroutine 9239 [select, 59 minutes]:
38932026-03-10T19:00:48.900Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc004bcc008?, 0x741274?})
38942026-03-10T19:00:48.900Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
38952026-03-10T19:00:48.900Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
38962026-03-10T19:00:48.900Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
38972026-03-10T19:00:48.900Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
38982026-03-10T19:00:48.900Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
38992026-03-10T19:00:48.900Z
39002026-03-10T19:00:48.900Zgoroutine 9273 [select]:
39012026-03-10T19:00:48.900Zgithub.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue.func2()
39022026-03-10T19:00:48.900Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:366 +0x85
39032026-03-10T19:00:48.901Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue in goroutine 9099
39042026-03-10T19:00:48.901Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:363 +0x3ed
39052026-03-10T19:00:48.901Z
39062026-03-10T19:00:48.901Zgoroutine 9573 [semacquire, 59 minutes]:
39072026-03-10T19:00:48.901Zsync.runtime_Semacquire(0xc006ed2c18?)
39082026-03-10T19:00:48.901Z /work/go/src/runtime/sema.go:62 +0x25
39092026-03-10T19:00:48.901Zsync.(*WaitGroup).Wait(0x44f7e0?)
39102026-03-10T19:00:48.901Z /work/go/src/sync/waitgroup.go:116 +0x48
39112026-03-10T19:00:48.901Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc000cde1c0)
39122026-03-10T19:00:48.901Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
39132026-03-10T19:00:48.901Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc000cde1c0?, {0x50e6118?, 0xc003c22aa0?}})
39142026-03-10T19:00:48.901Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
39152026-03-10T19:00:48.901Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed(0xc00432cf08, {0x50e6118?, 0xc003c22a00?}, {0xc006037e60, 0x1, 0xc00401bd40?}, {0xc005e6ef00?, 0xb985f9?, 0x0?}, 0x0, ...)
39162026-03-10T19:00:48.901Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:139 +0x6c5
39172026-03-10T19:00:48.901Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*dbAdapter).RangeFeed(0x50e60e0?, {0x50e6118?, 0xc003c22a00?}, {0xc006037e60?, 0x50e60e0?, 0xc00398d080?}, {0x1000000000001?, 0x50e60e0?, 0x0?}, 0x0, ...)
39182026-03-10T19:00:48.901Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/db_adapter.go:78 +0x3e
39192026-03-10T19:00:48.901Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func1({0x50e6118?, 0xc003c22a00?})
39202026-03-10T19:00:48.901Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:285 +0x6c
39212026-03-10T19:00:48.901Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
39222026-03-10T19:00:48.901Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
39232026-03-10T19:00:48.901Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
39242026-03-10T19:00:48.901Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
39252026-03-10T19:00:48.901Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9636
39262026-03-10T19:00:48.901Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
39272026-03-10T19:00:48.901Z
39282026-03-10T19:00:48.901Zgoroutine 9943 [select]:
39292026-03-10T19:00:48.901Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.(*Watcher).Run(0xc003c85c20, {0x50e6118, 0xc006685680})
39302026-03-10T19:00:48.901Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:319 +0x955
39312026-03-10T19:00:48.901Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.Start.func1({0x50e60e0?, 0xc00674c420?})
39322026-03-10T19:00:48.901Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:189 +0x2d1
39332026-03-10T19:00:48.901Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
39342026-03-10T19:00:48.901Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
39352026-03-10T19:00:48.901Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
39362026-03-10T19:00:48.901Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
39372026-03-10T19:00:48.901Z
39382026-03-10T19:00:48.901Zgoroutine 9416 [sync.Cond.Wait]:
39392026-03-10T19:00:48.907Zsync.runtime_notifyListWait(0xc0005ac550, 0x3761)
39402026-03-10T19:00:48.907Z /work/go/src/runtime/sema.go:569 +0x159
39412026-03-10T19:00:48.907Zsync.(*Cond).Wait(0x3ae1600?)
39422026-03-10T19:00:48.907Z /work/go/src/sync/cond.go:70 +0x85
39432026-03-10T19:00:48.907Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006623740})
39442026-03-10T19:00:48.907Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
39452026-03-10T19:00:48.907Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
39462026-03-10T19:00:48.907Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
39472026-03-10T19:00:48.907Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
39482026-03-10T19:00:48.907Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
39492026-03-10T19:00:48.907Z
39502026-03-10T19:00:48.907Zgoroutine 9946 [chan receive, 59 minutes]:
39512026-03-10T19:00:48.907Zgithub.com/cockroachdb/cockroach/pkg/util/netutil.MakeServer.func2({0xc006525208?, 0xc002f5e1e0?})
39522026-03-10T19:00:48.907Z /work/oxidecomputer/cockroach/pkg/util/netutil/net.go:113 +0x49
39532026-03-10T19:00:48.907Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
39542026-03-10T19:00:48.907Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
39552026-03-10T19:00:48.907Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9661
39562026-03-10T19:00:48.907Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
39572026-03-10T19:00:48.907Z
39582026-03-10T19:00:48.907Zgoroutine 9479 [select]:
39592026-03-10T19:00:48.907Zgithub.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigkvsubscriber.(*KVSubscriber).Start.func1({0x50e60e0, 0xc002de6ab0})
39602026-03-10T19:00:48.907Z /work/oxidecomputer/cockroach/pkg/spanconfig/spanconfigkvsubscriber/kvsubscriber.go:268 +0x227
39612026-03-10T19:00:48.907Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
39622026-03-10T19:00:48.907Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
39632026-03-10T19:00:48.907Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
39642026-03-10T19:00:48.907Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
39652026-03-10T19:00:48.907Z
39662026-03-10T19:00:48.907Zgoroutine 9303 [select, 59 minutes]:
39672026-03-10T19:00:48.907Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc002c06f08?, 0x0?})
39682026-03-10T19:00:48.907Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
39692026-03-10T19:00:48.907Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
39702026-03-10T19:00:48.907Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
39712026-03-10T19:00:48.907Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
39722026-03-10T19:00:48.907Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
39732026-03-10T19:00:48.907Z
39742026-03-10T19:00:48.908Zgoroutine 9264 [select]:
39752026-03-10T19:00:48.908Zgithub.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1()
39762026-03-10T19:00:48.908Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:71 +0xcb
39772026-03-10T19:00:48.908Zcreated by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 9099
39782026-03-10T19:00:48.908Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:66 +0x58
39792026-03-10T19:00:48.908Z
39802026-03-10T19:00:48.908Zgoroutine 3660533 [runnable]:
39812026-03-10T19:00:48.908Zgithub.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).initConnEx.func1()
39822026-03-10T19:00:48.908Z /work/oxidecomputer/cockroach/pkg/sql/internal.go:211
39832026-03-10T19:00:48.908Zcreated by github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).initConnEx in goroutine 10209
39842026-03-10T19:00:48.908Z /work/oxidecomputer/cockroach/pkg/sql/internal.go:211 +0x5a5
39852026-03-10T19:00:48.909Z
39862026-03-10T19:00:48.909Zgoroutine 9579 [select]:
39872026-03-10T19:00:48.909Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run(0xc002d3e000, {0x50e60e0?, 0xc0036c10e0?}, 0xb991a6?, 0xc006ed2cf0, 0xc006848400)
39882026-03-10T19:00:48.910Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:269 +0x4dc
39892026-03-10T19:00:48.910Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).Start.func1({0x50e60e0?, 0xc0036c10e0?})
39902026-03-10T19:00:48.910Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:222 +0x36
39912026-03-10T19:00:48.910Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
39922026-03-10T19:00:48.910Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
39932026-03-10T19:00:48.910Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9578
39942026-03-10T19:00:48.910Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
39952026-03-10T19:00:48.910Z
39962026-03-10T19:00:48.910Zgoroutine 9263 [chan receive, 59 minutes]:
39972026-03-10T19:00:48.910Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50e60e0?, 0xc005cee570?})
39982026-03-10T19:00:48.910Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
39992026-03-10T19:00:48.910Zruntime/pprof.Do({0x50e6038?, 0x778e7c0?}, {{0xc00030e2c0?, 0x0?, 0x0?}}, 0xc0022c77a8)
40002026-03-10T19:00:48.910Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
40012026-03-10T19:00:48.910Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
40022026-03-10T19:00:48.910Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
40032026-03-10T19:00:48.910Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9099
40042026-03-10T19:00:48.910Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
40052026-03-10T19:00:48.910Z
40062026-03-10T19:00:48.910Zgoroutine 9227 [select]:
40072026-03-10T19:00:48.911Zgithub.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1()
40082026-03-10T19:00:48.911Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:71 +0xcb
40092026-03-10T19:00:48.911Zcreated by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 9099
40102026-03-10T19:00:48.911Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:66 +0x58
40112026-03-10T19:00:48.911Z
40122026-03-10T19:00:48.911Zgoroutine 10023 [select]:
40132026-03-10T19:00:48.911Zgithub.com/cockroachdb/cockroach/pkg/gossip.(*client).gossip(0xc006a95790, {0x50e60e0, 0xc006f68540}, 0xc0037169c0, {0x51208e0, 0xc006d31570}, 0xc003659b80, 0xc006458380)
40142026-03-10T19:00:48.911Z /work/oxidecomputer/cockroach/pkg/gossip/client.go:364 +0x415
40152026-03-10T19:00:48.911Zgithub.com/cockroachdb/cockroach/pkg/gossip.(*client).startLocked.func1({0x50e60e0, 0xc006f68540})
40162026-03-10T19:00:48.911Z /work/oxidecomputer/cockroach/pkg/gossip/client.go:125 +0x3d2
40172026-03-10T19:00:48.911Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
40182026-03-10T19:00:48.911Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
40192026-03-10T19:00:48.911Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9951
40202026-03-10T19:00:48.911Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
40212026-03-10T19:00:48.911Z
40222026-03-10T19:00:48.911Zgoroutine 9280 [select]:
40232026-03-10T19:00:48.911Zgithub.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
40242026-03-10T19:00:48.911Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:439 +0x1c9
40252026-03-10T19:00:48.911Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 9099
40262026-03-10T19:00:48.911Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:412 +0x4f
40272026-03-10T19:00:48.911Z
40282026-03-10T19:00:48.911Zgoroutine 10476 [select]:
40292026-03-10T19:00:48.911Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).outputLoop(0xc003c317a0, {0x50e6118, 0xc000f3b4f0})
40302026-03-10T19:00:48.911Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:310 +0x247
40312026-03-10T19:00:48.911Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).runOutputLoop(0xc003c317a0, {0x50e60e0, 0xc0069361b0}, 0x1b?)
40322026-03-10T19:00:48.911Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:335 +0xba
40332026-03-10T19:00:48.911Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run.func1({0x50e60e0?, 0xc0069361b0?})
40342026-03-10T19:00:48.911Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:297 +0x46
40352026-03-10T19:00:48.911Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
40362026-03-10T19:00:48.911Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
40372026-03-10T19:00:48.911Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9579
40382026-03-10T19:00:48.911Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
40392026-03-10T19:00:48.919Z
40402026-03-10T19:00:48.919Zgoroutine 9443 [select, 1 minutes]:
40412026-03-10T19:00:48.919Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).startGossiping.func1({0x50e60e0, 0xc0068100f0})
40422026-03-10T19:00:48.919Z /work/oxidecomputer/cockroach/pkg/server/node.go:713 +0x1be
40432026-03-10T19:00:48.919Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
40442026-03-10T19:00:48.919Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
40452026-03-10T19:00:48.919Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
40462026-03-10T19:00:48.919Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
40472026-03-10T19:00:48.919Z
40482026-03-10T19:00:48.919Zgoroutine 9374 [sync.Cond.Wait]:
40492026-03-10T19:00:48.919Zsync.runtime_notifyListWait(0xc0005ac550, 0x374b)
40502026-03-10T19:00:48.919Z /work/go/src/runtime/sema.go:569 +0x159
40512026-03-10T19:00:48.920Zsync.(*Cond).Wait(0x3ae1600?)
40522026-03-10T19:00:48.920Z /work/go/src/sync/cond.go:70 +0x85
40532026-03-10T19:00:48.920Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006622db0})
40542026-03-10T19:00:48.920Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
40552026-03-10T19:00:48.920Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
40562026-03-10T19:00:48.920Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
40572026-03-10T19:00:48.920Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
40582026-03-10T19:00:48.920Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
40592026-03-10T19:00:48.920Z
40602026-03-10T19:00:48.920Zgoroutine 10143 [select]:
40612026-03-10T19:00:48.920Zgoogle.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000bde500, 0x1)
40622026-03-10T19:00:48.920Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:407 +0x113
40632026-03-10T19:00:48.920Zgoogle.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc004c46f00)
40642026-03-10T19:00:48.920Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:534 +0x7b
40652026-03-10T19:00:48.920Zgoogle.golang.org/grpc/internal/transport.NewServerTransport.func2()
40662026-03-10T19:00:48.920Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:332 +0xc6
40672026-03-10T19:00:48.920Zcreated by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 10169
40682026-03-10T19:00:48.921Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:329 +0x184c
40692026-03-10T19:00:48.921Z
40702026-03-10T19:00:48.921Zgoroutine 9220 [chan receive, 59 minutes]:
40712026-03-10T19:00:48.921Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50e60e0?, 0xc00634c240?})
40722026-03-10T19:00:48.921Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
40732026-03-10T19:00:48.922Zruntime/pprof.Do({0x50e6038?, 0x778e7c0?}, {{0xc00030e2c0?, 0x124cf8148314340?, 0xb3c81e8af9205f45?}}, 0xc0028847a8)
40742026-03-10T19:00:48.922Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
40752026-03-10T19:00:48.922Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x67982168ff36d3a7?)
40762026-03-10T19:00:48.922Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
40772026-03-10T19:00:48.922Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9099
40782026-03-10T19:00:48.922Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
40792026-03-10T19:00:48.922Z
40802026-03-10T19:00:48.922Zgoroutine 9037 [IO wait, 59 minutes]:
40812026-03-10T19:00:48.922Zinternal/poll.runtime_pollWait(0x7f784abc3130, 0x72)
40822026-03-10T19:00:48.922Z /work/go/src/runtime/netpoll.go:345 +0x85
40832026-03-10T19:00:48.922Zinternal/poll.(*pollDesc).wait(0x8?, 0x1?, 0x0)
40842026-03-10T19:00:48.922Z /work/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
40852026-03-10T19:00:48.922Zinternal/poll.(*pollDesc).waitRead(...)
40862026-03-10T19:00:48.922Z /work/go/src/internal/poll/fd_poll_runtime.go:89
40872026-03-10T19:00:48.922Zinternal/poll.(*FD).Accept(0xc006848380)
40882026-03-10T19:00:48.922Z /work/go/src/internal/poll/fd_unix.go:611 +0x2ac
40892026-03-10T19:00:48.922Znet.(*netFD).accept(0xc006848380)
40902026-03-10T19:00:48.922Z /work/go/src/net/fd_unix.go:172 +0x29
40912026-03-10T19:00:48.922Znet.(*TCPListener).accept(0xc004cb6e20)
40922026-03-10T19:00:48.922Z /work/go/src/net/tcpsock_posix.go:159 +0x1e
40932026-03-10T19:00:48.922Znet.(*TCPListener).Accept(0xc004cb6e20)
40942026-03-10T19:00:48.922Z /work/go/src/net/tcpsock.go:327 +0x30
40952026-03-10T19:00:48.922Zgithub.com/cockroachdb/cmux.(*cMux).Serve(0xc001203d80)
40962026-03-10T19:00:48.922Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:124 +0x97
40972026-03-10T19:00:48.922Zgithub.com/cockroachdb/cockroach/pkg/server.startListenRPCAndSQL.func5.2.1()
40982026-03-10T19:00:48.922Z /work/oxidecomputer/cockroach/pkg/server/start_listen.go:139 +0x27
40992026-03-10T19:00:48.922Zsync.(*Once).doSlow(0xc004331ed8?, 0xc004331f18?)
41002026-03-10T19:00:48.922Z /work/go/src/sync/once.go:74 +0xc2
41012026-03-10T19:00:48.922Zsync.(*Once).Do(...)
41022026-03-10T19:00:48.922Z /work/go/src/sync/once.go:65
41032026-03-10T19:00:48.922Zgithub.com/cockroachdb/cockroach/pkg/server.startListenRPCAndSQL.func5.2({0xc0016ee308?, 0xc000c65090?})
41042026-03-10T19:00:48.922Z /work/oxidecomputer/cockroach/pkg/server/start_listen.go:138 +0x4f
41052026-03-10T19:00:48.922Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
41062026-03-10T19:00:48.922Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
41072026-03-10T19:00:48.922Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
41082026-03-10T19:00:48.923Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
41092026-03-10T19:00:48.923Z
41102026-03-10T19:00:48.923Zgoroutine 9384 [sync.Cond.Wait]:
41112026-03-10T19:00:48.923Zsync.runtime_notifyListWait(0xc0005ac550, 0x374e)
41122026-03-10T19:00:48.923Z /work/go/src/runtime/sema.go:569 +0x159
41132026-03-10T19:00:48.923Zsync.(*Cond).Wait(0x3ae1600?)
41142026-03-10T19:00:48.923Z /work/go/src/sync/cond.go:70 +0x85
41152026-03-10T19:00:48.923Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006623020})
41162026-03-10T19:00:48.923Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
41172026-03-10T19:00:48.923Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
41182026-03-10T19:00:48.923Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
41192026-03-10T19:00:48.923Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
41202026-03-10T19:00:48.923Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
41212026-03-10T19:00:48.923Z
41222026-03-10T19:00:48.923Zgoroutine 9402 [sync.Cond.Wait]:
41232026-03-10T19:00:48.923Zsync.runtime_notifyListWait(0xc0005ac550, 0x3765)
41242026-03-10T19:00:48.924Z /work/go/src/runtime/sema.go:569 +0x159
41252026-03-10T19:00:48.924Zsync.(*Cond).Wait(0x3ae1600?)
41262026-03-10T19:00:48.924Z /work/go/src/sync/cond.go:70 +0x85
41272026-03-10T19:00:48.924Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006623410})
41282026-03-10T19:00:48.924Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
41292026-03-10T19:00:48.924Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
41302026-03-10T19:00:48.924Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
41312026-03-10T19:00:48.924Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
41322026-03-10T19:00:48.924Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
41332026-03-10T19:00:48.924Z
41342026-03-10T19:00:48.924Zgoroutine 10415 [select]:
41352026-03-10T19:00:48.924Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).outputLoop(0xc002d2cd20, {0x50e6118, 0xc002cfbe50})
41362026-03-10T19:00:48.924Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:310 +0x247
41372026-03-10T19:00:48.924Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).runOutputLoop(0xc002d2cd20, {0x50e60e0, 0xc002a50750}, 0x1b?)
41382026-03-10T19:00:48.924Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:335 +0xba
41392026-03-10T19:00:48.928Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run.func1({0x50e60e0?, 0xc002a50750?})
41402026-03-10T19:00:48.928Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:297 +0x46
41412026-03-10T19:00:48.928Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
41422026-03-10T19:00:48.929Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
41432026-03-10T19:00:48.929Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9729
41442026-03-10T19:00:48.929Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
41452026-03-10T19:00:48.929Z
41462026-03-10T19:00:48.929Zgoroutine 10097 [sync.Cond.Wait, 59 minutes]:
41472026-03-10T19:00:48.929Zsync.runtime_notifyListWait(0xc00676e610, 0x3f)
41482026-03-10T19:00:48.929Z /work/go/src/runtime/sema.go:569 +0x159
41492026-03-10T19:00:48.929Zsync.(*Cond).Wait(0x0?)
41502026-03-10T19:00:48.929Z /work/go/src/sync/cond.go:70 +0x85
41512026-03-10T19:00:48.929Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0072ac330})
41522026-03-10T19:00:48.929Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
41532026-03-10T19:00:48.929Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
41542026-03-10T19:00:48.929Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
41552026-03-10T19:00:48.929Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
41562026-03-10T19:00:48.929Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
41572026-03-10T19:00:48.930Z
41582026-03-10T19:00:48.930Zgoroutine 10078 [sync.Cond.Wait, 59 minutes]:
41592026-03-10T19:00:48.930Zsync.runtime_notifyListWait(0xc00676e610, 0x2c)
41602026-03-10T19:00:48.930Z /work/go/src/runtime/sema.go:569 +0x159
41612026-03-10T19:00:48.930Zsync.(*Cond).Wait(0x0?)
41622026-03-10T19:00:48.930Z /work/go/src/sync/cond.go:70 +0x85
41632026-03-10T19:00:48.930Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c9f20})
41642026-03-10T19:00:48.930Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
41652026-03-10T19:00:48.930Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
41662026-03-10T19:00:48.930Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
41672026-03-10T19:00:48.930Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
41682026-03-10T19:00:48.930Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
41692026-03-10T19:00:48.930Z
41702026-03-10T19:00:48.930Zgoroutine 9383 [sync.Cond.Wait]:
41712026-03-10T19:00:48.930Zsync.runtime_notifyListWait(0xc0005ac550, 0x373a)
41722026-03-10T19:00:48.930Z /work/go/src/runtime/sema.go:569 +0x159
41732026-03-10T19:00:48.930Zsync.(*Cond).Wait(0x3ae1600?)
41742026-03-10T19:00:48.931Z /work/go/src/sync/cond.go:70 +0x85
41752026-03-10T19:00:48.931Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006622ff0})
41762026-03-10T19:00:48.931Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
41772026-03-10T19:00:48.931Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
41782026-03-10T19:00:48.931Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
41792026-03-10T19:00:48.931Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
41802026-03-10T19:00:48.931Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
41812026-03-10T19:00:48.931Z
41822026-03-10T19:00:48.931Zgoroutine 9261 [chan receive, 59 minutes]:
41832026-03-10T19:00:48.931Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50e60e0?, 0xc005c70150?})
41842026-03-10T19:00:48.931Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
41852026-03-10T19:00:48.931Zruntime/pprof.Do({0x50e6038?, 0x778e7c0?}, {{0xc00030e2c0?, 0x0?, 0x0?}}, 0xc0022c67a8)
41862026-03-10T19:00:48.931Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
41872026-03-10T19:00:48.931Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
41882026-03-10T19:00:48.931Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
41892026-03-10T19:00:48.931Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9099
41902026-03-10T19:00:48.931Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
41912026-03-10T19:00:48.932Z
41922026-03-10T19:00:48.932Zgoroutine 9304 [select, 59 minutes]:
41932026-03-10T19:00:48.932Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc002c07208?, 0x0?})
41942026-03-10T19:00:48.932Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
41952026-03-10T19:00:48.932Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
41962026-03-10T19:00:48.932Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
41972026-03-10T19:00:48.932Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
41982026-03-10T19:00:48.932Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
41992026-03-10T19:00:48.932Z
42002026-03-10T19:00:48.932Zgoroutine 10096 [sync.Cond.Wait, 59 minutes]:
42012026-03-10T19:00:48.932Zsync.runtime_notifyListWait(0xc00676e610, 0x3e)
42022026-03-10T19:00:48.932Z /work/go/src/runtime/sema.go:569 +0x159
42032026-03-10T19:00:48.932Zsync.(*Cond).Wait(0x0?)
42042026-03-10T19:00:48.932Z /work/go/src/sync/cond.go:70 +0x85
42052026-03-10T19:00:48.932Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0072ac300})
42062026-03-10T19:00:48.932Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
42072026-03-10T19:00:48.932Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
42082026-03-10T19:00:48.932Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
42092026-03-10T19:00:48.932Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
42102026-03-10T19:00:48.932Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
42112026-03-10T19:00:48.932Z
42122026-03-10T19:00:48.932Zgoroutine 9940 [select, 59 minutes]:
42132026-03-10T19:00:48.932Zgithub.com/cockroachdb/cockroach/pkg/sql/scheduledlogging.(*CaptureIndexUsageStatsLoggingScheduler).start.func1({0x50e60e0, 0xc00674c060})
42142026-03-10T19:00:48.932Z /work/oxidecomputer/cockroach/pkg/sql/scheduledlogging/captured_index_usage_stats.go:144 +0xeb
42152026-03-10T19:00:48.932Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
42162026-03-10T19:00:48.932Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
42172026-03-10T19:00:48.932Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
42182026-03-10T19:00:48.932Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
42192026-03-10T19:00:48.932Z
42202026-03-10T19:00:48.932Zgoroutine 9159 [select, 59 minutes]:
42212026-03-10T19:00:48.932Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2({0x50e60e0, 0xc001eac120})
42222026-03-10T19:00:48.932Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/queue.go:823 +0xff
42232026-03-10T19:00:48.932Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
42242026-03-10T19:00:48.932Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
42252026-03-10T19:00:48.932Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9437
42262026-03-10T19:00:48.932Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
42272026-03-10T19:00:48.932Z
42282026-03-10T19:00:48.932Zgoroutine 10495 [chan receive, 59 minutes]:
42292026-03-10T19:00:48.933Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).rangeFeedWithRangeID(0xc001b0a008, 0x3dbc860?, 0xc0021ed200, {0x7f784cb862a8, 0xc005efeab0})
42302026-03-10T19:00:48.933Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:249 +0x76b
42312026-03-10T19:00:48.933Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed(...)
42322026-03-10T19:00:48.933Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:149
42332026-03-10T19:00:48.933Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).RangeFeed(0xc001668008, 0xc0021ed200, {0x5119e20, 0xc005efeab0})
42342026-03-10T19:00:48.933Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:3082 +0x16f
42352026-03-10T19:00:48.933Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).RangeFeed(0xc006849e00, 0xc0021ed200, {0x5119e20, 0xc005efeab0})
42362026-03-10T19:00:48.933Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/stores.go:216 +0x19a
42372026-03-10T19:00:48.933Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed(0x7366f90?, 0x403a5e0?, {0x5119e20, 0xc005efeab0})
42382026-03-10T19:00:48.933Z /work/oxidecomputer/cockroach/pkg/server/node.go:1232 +0x28
42392026-03-10T19:00:48.937Zgithub.com/cockroachdb/cockroach/pkg/roachpb._Internal_RangeFeed_Handler({0x403a5e0, 0xc001668908}, {0x5114010, 0xc004c15300})
42402026-03-10T19:00:48.937Z /work/oxidecomputer/cockroach/pkg/roachpb/api.pb.go:9523 +0x107
42412026-03-10T19:00:48.937Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.StreamServerInterceptor.func14({0x403a5e0, 0xc001668908}, {0x51147c0, 0xc004499c70}, 0xc004b21260, 0x4319298)
42422026-03-10T19:00:48.937Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:205 +0x35b
42432026-03-10T19:00:48.937Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x403a5e0?, 0xc001668908?}, {0x51147c0?, 0xc004499c70?})
42442026-03-10T19:00:48.937Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1408 +0x59
42452026-03-10T19:00:48.937Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func4({0x403a5e0, 0xc001668908}, {0x51147c0, 0xc004499c70}, 0xc000bd6918?, 0xc000f9aac0)
42462026-03-10T19:00:48.937Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:247 +0x6c
42472026-03-10T19:00:48.937Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x403a5e0?, 0xc001668908?}, {0x51147c0?, 0xc004499c70?})
42482026-03-10T19:00:48.937Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
42492026-03-10T19:00:48.937Zgithub.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({{{0x2c?}}}, {0x403a5e0, 0xc001668908}, {0x51147c0, 0xc004499c70}, 0xc004b21260, 0xc000f9aac0)
42502026-03-10T19:00:48.937Z /work/oxidecomputer/cockroach/pkg/rpc/auth.go:98 +0x19b
42512026-03-10T19:00:48.937Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x403a5e0?, 0xc001668908?}, {0x51147c0?, 0xc004499c70?})
42522026-03-10T19:00:48.938Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
42532026-03-10T19:00:48.938Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func2.1({0x66?, 0x40?})
42542026-03-10T19:00:48.939Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:216 +0x27
42552026-03-10T19:00:48.939Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc006848400, {0x50e60e0, 0xc006fe6a50}, {0xc000bd6b28?, 0x416dfb?}, 0xc000bd6ac0)
42562026-03-10T19:00:48.939Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:344 +0xcb
42572026-03-10T19:00:48.939Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func2({0x403a5e0, 0xc001668908}, {0x51147c0, 0xc004499c70}, 0xc004b21260, 0xc000f9aac0)
42582026-03-10T19:00:48.939Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:215 +0xdf
42592026-03-10T19:00:48.939Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x403a5e0?, 0xc001668908?}, {0x51147c0?, 0xc004499c70?})
42602026-03-10T19:00:48.939Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
42612026-03-10T19:00:48.939Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x403a5e0, 0xc001668908}, {0x51147c0, 0xc004499c70}, 0xc004b21260, 0x4319298)
42622026-03-10T19:00:48.939Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1413 +0x108
42632026-03-10T19:00:48.939Zgoogle.golang.org/grpc.(*Server).processStreamingRPC(0xc005fee1c0, {0x5126b60, 0xc00130c340}, 0xc0021ecfc0, 0xc0077eb920, 0x73e2b80, 0x0)
42642026-03-10T19:00:48.939Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1549 +0xe51
42652026-03-10T19:00:48.939Zgoogle.golang.org/grpc.(*Server).handleStream(0xc005fee1c0, {0x5126b60, 0xc00130c340}, 0xc0021ecfc0, 0x0)
42662026-03-10T19:00:48.939Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1624 +0x999
42672026-03-10T19:00:48.939Zgoogle.golang.org/grpc.(*Server).serveStreams.func1.2()
42682026-03-10T19:00:48.939Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:922 +0x8d
42692026-03-10T19:00:48.939Zcreated by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 10187
42702026-03-10T19:00:48.939Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:920 +0x226
42712026-03-10T19:00:48.939Z
42722026-03-10T19:00:48.939Zgoroutine 9246 [select, 59 minutes]:
42732026-03-10T19:00:48.939Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc0015e8608?, 0xc002ef2150?})
42742026-03-10T19:00:48.939Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
42752026-03-10T19:00:48.939Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
42762026-03-10T19:00:48.939Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
42772026-03-10T19:00:48.939Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
42782026-03-10T19:00:48.939Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
42792026-03-10T19:00:48.939Z
42802026-03-10T19:00:48.939Zgoroutine 10085 [sync.Cond.Wait, 59 minutes]:
42812026-03-10T19:00:48.939Zsync.runtime_notifyListWait(0xc00676e610, 0x34)
42822026-03-10T19:00:48.939Z /work/go/src/runtime/sema.go:569 +0x159
42832026-03-10T19:00:48.939Zsync.(*Cond).Wait(0x0?)
42842026-03-10T19:00:48.939Z /work/go/src/sync/cond.go:70 +0x85
42852026-03-10T19:00:48.939Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0072ac0f0})
42862026-03-10T19:00:48.940Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
42872026-03-10T19:00:48.940Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
42882026-03-10T19:00:48.940Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
42892026-03-10T19:00:48.940Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
42902026-03-10T19:00:48.940Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
42912026-03-10T19:00:48.940Z
42922026-03-10T19:00:48.940Zgoroutine 9158 [select, 9 minutes]:
42932026-03-10T19:00:48.940Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2({0x50e60e0, 0xc001eac090})
42942026-03-10T19:00:48.940Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/queue.go:823 +0xff
42952026-03-10T19:00:48.940Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
42962026-03-10T19:00:48.940Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
42972026-03-10T19:00:48.940Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9437
42982026-03-10T19:00:48.940Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
42992026-03-10T19:00:48.940Z
43002026-03-10T19:00:48.940Zgoroutine 10308 [select]:
43012026-03-10T19:00:48.940Zgithub.com/cockroachdb/cockroach/pkg/jobs.(*jobScheduler).runDaemon.func1({0x50e60e0?, 0xc00398de90?})
43022026-03-10T19:00:48.941Z /work/oxidecomputer/cockroach/pkg/jobs/job_scheduler.go:443 +0x293
43032026-03-10T19:00:48.941Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
43042026-03-10T19:00:48.941Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
43052026-03-10T19:00:48.941Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
43062026-03-10T19:00:48.941Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
43072026-03-10T19:00:48.941Z
43082026-03-10T19:00:48.941Zgoroutine 9189 [chan receive, 59 minutes]:
43092026-03-10T19:00:48.941Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50e60e0?, 0xc005cee030?})
43102026-03-10T19:00:48.941Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
43112026-03-10T19:00:48.941Zruntime/pprof.Do({0x50e6038?, 0x778e7c0?}, {{0xc00030e2c0?, 0x10000c004602b01?, 0x1000000000001?}}, 0xc0000ac7a8)
43122026-03-10T19:00:48.941Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
43132026-03-10T19:00:48.941Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
43142026-03-10T19:00:48.941Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
43152026-03-10T19:00:48.941Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9099
43162026-03-10T19:00:48.942Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
43172026-03-10T19:00:48.942Z
43182026-03-10T19:00:48.942Zgoroutine 9871 [select, 59 minutes]:
43192026-03-10T19:00:48.942Zgithub.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run(0xc0013c6420, {0x50e60e0, 0xc0065c9440})
43202026-03-10T19:00:48.942Z /work/oxidecomputer/cockroach/pkg/internal/client/requestbatcher/batcher.go:475 +0x326
43212026-03-10T19:00:48.942Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
43222026-03-10T19:00:48.942Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
43232026-03-10T19:00:48.942Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
43242026-03-10T19:00:48.942Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
43252026-03-10T19:00:48.942Z
43262026-03-10T19:00:48.942Zgoroutine 9398 [sync.Cond.Wait]:
43272026-03-10T19:00:48.942Zsync.runtime_notifyListWait(0xc0005ac550, 0x3764)
43282026-03-10T19:00:48.942Z /work/go/src/runtime/sema.go:569 +0x159
43292026-03-10T19:00:48.942Zsync.(*Cond).Wait(0x3ae1600?)
43302026-03-10T19:00:48.942Z /work/go/src/sync/cond.go:70 +0x85
43312026-03-10T19:00:48.942Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006623350})
43322026-03-10T19:00:48.942Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
43332026-03-10T19:00:48.942Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
43342026-03-10T19:00:48.942Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
43352026-03-10T19:00:48.942Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
43362026-03-10T19:00:48.942Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
43372026-03-10T19:00:48.942Z
43382026-03-10T19:00:48.942Zgoroutine 10400 [select]:
43392026-03-10T19:00:49.031Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).outputLoop(0xc004daa7e0, {0x50e6118, 0xc007a22230})
43402026-03-10T19:00:49.031Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:310 +0x247
43412026-03-10T19:00:49.031Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).runOutputLoop(0xc004daa7e0, {0x50e60e0, 0xc006213770}, 0x1b?)
43422026-03-10T19:00:49.031Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:335 +0xba
43432026-03-10T19:00:49.031Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run.func1({0x50e60e0?, 0xc006213770?})
43442026-03-10T19:00:49.031Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:297 +0x46
43452026-03-10T19:00:49.031Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
43462026-03-10T19:00:49.031Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
43472026-03-10T19:00:49.032Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9579
43482026-03-10T19:00:49.032Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
43492026-03-10T19:00:49.032Z
43502026-03-10T19:00:49.032Zgoroutine 10309 [select, 59 minutes]:
43512026-03-10T19:00:49.032Zgithub.com/cockroachdb/cockroach/pkg/sql/scheduledlogging.(*CaptureIndexUsageStatsLoggingScheduler).start.func1({0x50e60e0, 0xc00398dec0})
43522026-03-10T19:00:49.032Z /work/oxidecomputer/cockroach/pkg/sql/scheduledlogging/captured_index_usage_stats.go:144 +0xeb
43532026-03-10T19:00:49.032Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
43542026-03-10T19:00:49.032Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
43552026-03-10T19:00:49.032Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
43562026-03-10T19:00:49.032Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
43572026-03-10T19:00:49.032Z
43582026-03-10T19:00:49.032Zgoroutine 10192 [select, 59 minutes]:
43592026-03-10T19:00:49.032Zgithub.com/cockroachdb/cockroach/pkg/util/tracing.newTracingClientStream.func2()
43602026-03-10T19:00:49.033Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:386 +0x67
43612026-03-10T19:00:49.033Zcreated by github.com/cockroachdb/cockroach/pkg/util/tracing.newTracingClientStream in goroutine 10156
43622026-03-10T19:00:49.033Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:385 +0x125
43632026-03-10T19:00:49.033Z
43642026-03-10T19:00:49.033Zgoroutine 9161 [select, 59 minutes]:
43652026-03-10T19:00:49.033Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2({0x50e60e0, 0xc001eac180})
43662026-03-10T19:00:49.033Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/queue.go:823 +0xff
43672026-03-10T19:00:49.033Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
43682026-03-10T19:00:49.033Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
43692026-03-10T19:00:49.033Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9437
43702026-03-10T19:00:49.033Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
43712026-03-10T19:00:49.033Z
43722026-03-10T19:00:49.033Zgoroutine 9039 [chan receive, 59 minutes]:
43732026-03-10T19:00:49.033Zgithub.com/cockroachdb/cockroach/pkg/gossip.(*server).start.func3({0xc0016ee908?, 0xc00417eab0?})
43742026-03-10T19:00:49.033Z /work/oxidecomputer/cockroach/pkg/gossip/server.go:380 +0x48
43752026-03-10T19:00:49.033Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
43762026-03-10T19:00:49.033Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
43772026-03-10T19:00:49.033Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
43782026-03-10T19:00:49.033Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
43792026-03-10T19:00:49.033Z
43802026-03-10T19:00:49.033Zgoroutine 9656 [select, 59 minutes]:
43812026-03-10T19:00:49.033Zdatabase/sql.(*DB).connectionOpener(0xc006a94f70, {0x50e6118, 0xc006f46050})
43822026-03-10T19:00:49.033Z /work/go/src/database/sql/sql.go:1246 +0x87
43832026-03-10T19:00:49.034Zcreated by database/sql.OpenDB in goroutine 9099
43842026-03-10T19:00:49.034Z /work/go/src/database/sql/sql.go:824 +0x14c
43852026-03-10T19:00:49.034Z
43862026-03-10T19:00:49.034Zgoroutine 9729 [select]:
43872026-03-10T19:00:49.034Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run(0xc00676be00, {0x50e60e0?, 0xc004bb3710?}, 0xb991a6?, 0xc006a555a8, 0xc006848400)
43882026-03-10T19:00:49.034Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:269 +0x4dc
43892026-03-10T19:00:49.034Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).Start.func1({0x50e60e0?, 0xc004bb3710?})
43902026-03-10T19:00:49.034Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:222 +0x36
43912026-03-10T19:00:49.034Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
43922026-03-10T19:00:49.034Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
43932026-03-10T19:00:49.034Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9728
43942026-03-10T19:00:49.034Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
43952026-03-10T19:00:49.034Z
43962026-03-10T19:00:49.034Zgoroutine 9942 [select, 59 minutes]:
43972026-03-10T19:00:49.034Zgithub.com/cockroachdb/cockroach/pkg/server.(*Server).startAttemptUpgrade.func1({0x50e60e0?, 0xc00674c180?})
43982026-03-10T19:00:49.034Z /work/oxidecomputer/cockroach/pkg/server/auto_upgrade.go:45 +0x1cf
43992026-03-10T19:00:49.034Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
44002026-03-10T19:00:49.034Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
44012026-03-10T19:00:49.034Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
44022026-03-10T19:00:49.034Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
44032026-03-10T19:00:49.034Z
44042026-03-10T19:00:49.034Zgoroutine 9330 [chan receive, 59 minutes]:
44052026-03-10T19:00:49.034Zgithub.com/cockroachdb/cockroach/pkg/util/netutil.MakeServer.func2({0xc00431f208?, 0x17ef7bc?})
44062026-03-10T19:00:49.034Z /work/oxidecomputer/cockroach/pkg/util/netutil/net.go:113 +0x49
44072026-03-10T19:00:49.034Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
44082026-03-10T19:00:49.034Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
44092026-03-10T19:00:49.034Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9322
44102026-03-10T19:00:49.035Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
44112026-03-10T19:00:49.035Z
44122026-03-10T19:00:49.035Zgoroutine 9376 [sync.Cond.Wait]:
44132026-03-10T19:00:49.035Zsync.runtime_notifyListWait(0xc0005ac550, 0x3752)
44142026-03-10T19:00:49.035Z /work/go/src/runtime/sema.go:569 +0x159
44152026-03-10T19:00:49.035Zsync.(*Cond).Wait(0x3ae1600?)
44162026-03-10T19:00:49.035Z /work/go/src/sync/cond.go:70 +0x85
44172026-03-10T19:00:49.035Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006622e10})
44182026-03-10T19:00:49.035Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
44192026-03-10T19:00:49.035Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
44202026-03-10T19:00:49.035Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
44212026-03-10T19:00:49.035Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
44222026-03-10T19:00:49.035Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
44232026-03-10T19:00:49.035Z
44242026-03-10T19:00:49.035Zgoroutine 9317 [select, 7 minutes]:
44252026-03-10T19:00:49.035Zgithub.com/cockroachdb/cockroach/pkg/server.startPurgeOldSessions.func1({0xc00088f808?, 0xc00398d170?})
44262026-03-10T19:00:49.035Z /work/oxidecomputer/cockroach/pkg/server/purge_auth_session.go:69 +0x205
44272026-03-10T19:00:49.035Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
44282026-03-10T19:00:49.035Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
44292026-03-10T19:00:49.036Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
44302026-03-10T19:00:49.036Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
44312026-03-10T19:00:49.036Z
44322026-03-10T19:00:49.036Zgoroutine 10207 [select, 59 minutes]:
44332026-03-10T19:00:49.036Zgithub.com/cockroachdb/cockroach/pkg/sql/catalog/lease.(*Manager).RefreshLeases.func1({0x50e60e0, 0xc005c71b30})
44342026-03-10T19:00:49.036Z /work/oxidecomputer/cockroach/pkg/sql/catalog/lease/lease.go:1041 +0x105
44352026-03-10T19:00:49.036Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
44362026-03-10T19:00:49.036Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
44372026-03-10T19:00:49.036Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
44382026-03-10T19:00:49.036Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
44392026-03-10T19:00:49.042Z
44402026-03-10T19:00:49.042Zgoroutine 9368 [sync.Cond.Wait]:
44412026-03-10T19:00:49.042Zsync.runtime_notifyListWait(0xc0005ac550, 0x3747)
44422026-03-10T19:00:49.042Z /work/go/src/runtime/sema.go:569 +0x159
44432026-03-10T19:00:49.042Zsync.(*Cond).Wait(0x3ae1600?)
44442026-03-10T19:00:49.042Z /work/go/src/sync/cond.go:70 +0x85
44452026-03-10T19:00:49.042Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006622c60})
44462026-03-10T19:00:49.042Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
44472026-03-10T19:00:49.042Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
44482026-03-10T19:00:49.042Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
44492026-03-10T19:00:49.042Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
44502026-03-10T19:00:49.042Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
44512026-03-10T19:00:49.042Z
44522026-03-10T19:00:49.043Zgoroutine 10113 [select]:
44532026-03-10T19:00:49.043Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/reports.(*Reporter).Start.func2({0x50e60e0?, 0xc0072ac900?})
44542026-03-10T19:00:49.043Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/reports/reporter.go:165 +0x305
44552026-03-10T19:00:49.043Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
44562026-03-10T19:00:49.043Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
44572026-03-10T19:00:49.043Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
44582026-03-10T19:00:49.043Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
44592026-03-10T19:00:49.043Z
44602026-03-10T19:00:49.043Zgoroutine 9394 [sync.Cond.Wait]:
44612026-03-10T19:00:49.043Zsync.runtime_notifyListWait(0xc0005ac550, 0x3756)
44622026-03-10T19:00:49.043Z /work/go/src/runtime/sema.go:569 +0x159
44632026-03-10T19:00:49.043Zsync.(*Cond).Wait(0x3ae1600?)
44642026-03-10T19:00:49.043Z /work/go/src/sync/cond.go:70 +0x85
44652026-03-10T19:00:49.043Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006623200})
44662026-03-10T19:00:49.043Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
44672026-03-10T19:00:49.043Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
44682026-03-10T19:00:49.043Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
44692026-03-10T19:00:49.043Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
44702026-03-10T19:00:49.043Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
44712026-03-10T19:00:49.043Z
44722026-03-10T19:00:49.044Zgoroutine 3451789 [sync.Cond.Wait]:
44732026-03-10T19:00:49.044Zsync.runtime_notifyListWait(0xc0028ec5e8, 0x3d1)
44742026-03-10T19:00:49.044Z /work/go/src/runtime/sema.go:569 +0x159
44752026-03-10T19:00:49.044Zsync.(*Cond).Wait(0xc0028ec508?)
44762026-03-10T19:00:49.044Z /work/go/src/sync/cond.go:70 +0x85
44772026-03-10T19:00:49.044Zgithub.com/cockroachdb/pebble/record.(*flusherCond).Wait(...)
44782026-03-10T19:00:49.044Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/record/log_writer.go:193
44792026-03-10T19:00:49.044Zgithub.com/cockroachdb/pebble/record.(*LogWriter).flushLoop(0xc0028ec508, {0x3afca60?, 0x778e7c0?})
44802026-03-10T19:00:49.044Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/record/log_writer.go:396 +0x475
44812026-03-10T19:00:49.044Zruntime/pprof.Do({0x50e6038?, 0x778e7c0?}, {{0xc00030e1c0?, 0xc002063d40?, 0xc005fee1c0?}}, 0xc00051afc0)
44822026-03-10T19:00:49.044Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
44832026-03-10T19:00:49.044Zgithub.com/cockroachdb/pebble/record.NewLogWriter.func2()
44842026-03-10T19:00:49.044Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/record/log_writer.go:314 +0x56
44852026-03-10T19:00:49.044Zcreated by github.com/cockroachdb/pebble/record.NewLogWriter in goroutine 9409
44862026-03-10T19:00:49.044Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/record/log_writer.go:313 +0x2c9
44872026-03-10T19:00:49.044Z
44882026-03-10T19:00:49.044Zgoroutine 9351 [select, 59 minutes]:
44892026-03-10T19:00:49.044Zgithub.com/cockroachdb/cockroach/pkg/server.(*loopbackListener).Accept(0xc0064d49c0)
44902026-03-10T19:00:49.044Z /work/oxidecomputer/cockroach/pkg/server/loopback.go:49 +0x90
44912026-03-10T19:00:49.044Zgoogle.golang.org/grpc.(*Server).Serve(0xc005fee1c0, {0x50d1030, 0xc0064d49c0})
44922026-03-10T19:00:49.044Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:780 +0x469
44932026-03-10T19:00:49.044Zgithub.com/cockroachdb/cockroach/pkg/server.configureGRPCGateway.func2({0xc00169e008?, 0xc00417e9c0?})
44942026-03-10T19:00:49.044Z /work/oxidecomputer/cockroach/pkg/server/grpc_gateway.go:94 +0x25
44952026-03-10T19:00:49.045Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
44962026-03-10T19:00:49.045Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
44972026-03-10T19:00:49.045Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
44982026-03-10T19:00:49.045Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
44992026-03-10T19:00:49.045Z
45002026-03-10T19:00:49.045Zgoroutine 9275 [select]:
45012026-03-10T19:00:49.045Zgithub.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue.func2()
45022026-03-10T19:00:49.045Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:366 +0x85
45032026-03-10T19:00:49.045Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue in goroutine 9099
45042026-03-10T19:00:49.045Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:363 +0x3ed
45052026-03-10T19:00:49.045Z
45062026-03-10T19:00:49.045Zgoroutine 9308 [select, 59 minutes]:
45072026-03-10T19:00:49.045Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc004376308?, 0xc00398d560?})
45082026-03-10T19:00:49.045Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
45092026-03-10T19:00:49.045Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
45102026-03-10T19:00:49.045Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
45112026-03-10T19:00:49.045Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
45122026-03-10T19:00:49.045Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
45132026-03-10T19:00:49.045Z
45142026-03-10T19:00:49.045Zgoroutine 10088 [sync.Cond.Wait, 59 minutes]:
45152026-03-10T19:00:49.045Zsync.runtime_notifyListWait(0xc00676e610, 0x37)
45162026-03-10T19:00:49.045Z /work/go/src/runtime/sema.go:569 +0x159
45172026-03-10T19:00:49.045Zsync.(*Cond).Wait(0x0?)
45182026-03-10T19:00:49.045Z /work/go/src/sync/cond.go:70 +0x85
45192026-03-10T19:00:49.045Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0072ac180})
45202026-03-10T19:00:49.045Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
45212026-03-10T19:00:49.045Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
45222026-03-10T19:00:49.045Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
45232026-03-10T19:00:49.045Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
45242026-03-10T19:00:49.045Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
45252026-03-10T19:00:49.045Z
45262026-03-10T19:00:49.045Zgoroutine 9274 [select]:
45272026-03-10T19:00:49.045Zgithub.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
45282026-03-10T19:00:49.045Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:439 +0x1c9
45292026-03-10T19:00:49.045Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 9099
45302026-03-10T19:00:49.045Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:412 +0x4f
45312026-03-10T19:00:49.045Z
45322026-03-10T19:00:49.045Zgoroutine 9190 [chan receive, 59 minutes]:
45332026-03-10T19:00:49.045Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50e60e0?, 0xc00634c090?})
45342026-03-10T19:00:49.046Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
45352026-03-10T19:00:49.046Zruntime/pprof.Do({0x50e6038?, 0x778e7c0?}, {{0xc00030e2c0?, 0x10000c0013ca820?, 0x5126b60?}}, 0xc0000ad7a8)
45362026-03-10T19:00:49.046Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
45372026-03-10T19:00:49.046Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
45382026-03-10T19:00:49.046Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
45392026-03-10T19:00:49.050Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9099
45402026-03-10T19:00:49.050Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
45412026-03-10T19:00:49.050Z
45422026-03-10T19:00:49.050Zgoroutine 10372 [select]:
45432026-03-10T19:00:49.050Zgoogle.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc004937a00)
45442026-03-10T19:00:49.050Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1131 +0x205
45452026-03-10T19:00:49.050Zcreated by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 10304
45462026-03-10T19:00:49.050Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:341 +0x188e
45472026-03-10T19:00:49.051Z
45482026-03-10T19:00:49.051Zgoroutine 9242 [select, 59 minutes]:
45492026-03-10T19:00:49.051Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc004bcc908?, 0xc003221560?})
45502026-03-10T19:00:49.051Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
45512026-03-10T19:00:49.051Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
45522026-03-10T19:00:49.051Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
45532026-03-10T19:00:49.051Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
45542026-03-10T19:00:49.051Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
45552026-03-10T19:00:49.051Z
45562026-03-10T19:00:49.051Zgoroutine 9270 [chan receive, 59 minutes]:
45572026-03-10T19:00:49.051Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewContext.func2({0xc002a80008?, 0x0?})
45582026-03-10T19:00:49.051Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:541 +0x3f
45592026-03-10T19:00:49.051Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
45602026-03-10T19:00:49.051Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
45612026-03-10T19:00:49.051Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
45622026-03-10T19:00:49.051Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
45632026-03-10T19:00:49.051Z
45642026-03-10T19:00:49.051Zgoroutine 9243 [select, 59 minutes]:
45652026-03-10T19:00:49.051Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc004bcd208?, 0x4171a5?})
45662026-03-10T19:00:49.052Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
45672026-03-10T19:00:49.052Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
45682026-03-10T19:00:49.052Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
45692026-03-10T19:00:49.052Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
45702026-03-10T19:00:49.052Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
45712026-03-10T19:00:49.052Z
45722026-03-10T19:00:49.052Zgoroutine 9781 [select]:
45732026-03-10T19:00:49.052Zgithub.com/cockroachdb/cockroach/pkg/rpc.respStreamClientAdapter.recvInternal(...)
45742026-03-10T19:00:49.052Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:694
45752026-03-10T19:00:49.052Zgithub.com/cockroachdb/cockroach/pkg/rpc.rangeFeedClientAdapter.Recv(...)
45762026-03-10T19:00:49.052Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:723
45772026-03-10T19:00:49.052Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed.func3()
45782026-03-10T19:00:49.052Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:516 +0x25
45792026-03-10T19:00:49.052Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*stuckRangeFeedCanceler).do(0xc004129a90, 0xc001b91a50)
45802026-03-10T19:00:49.052Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed_canceler.go:100 +0x1b3
45812026-03-10T19:00:49.052Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed(0xc00432cf08, {0x50e6118?, 0xc004129950?}, {{0xc00357fb78, 0x1, 0x8}, {0xc00357fb80, 0x1, 0x8}}, {0x189b8cc3e00145ab, ...}, ...)
45822026-03-10T19:00:49.052Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:515 +0xc98
45832026-03-10T19:00:49.052Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).partialRangeFeed(0xc00432cf08, {0x50e6118, 0xc004129950}, 0xc0041297c0, {{0xc00357fb78, 0x1, 0x8}, {0xc00357fb80, 0x1, 0x8}}, ...)
45842026-03-10T19:00:49.052Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:336 +0x745
45852026-03-10T19:00:49.053Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.1({0x50e6118?, 0xc004129950?})
45862026-03-10T19:00:49.053Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:120 +0xa9
45872026-03-10T19:00:49.053Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.Group.GoCtx.2()
45882026-03-10T19:00:49.053Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
45892026-03-10T19:00:49.053Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
45902026-03-10T19:00:49.053Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
45912026-03-10T19:00:49.053Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9779
45922026-03-10T19:00:49.053Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
45932026-03-10T19:00:49.053Z
45942026-03-10T19:00:49.053Zgoroutine 9418 [sync.Cond.Wait]:
45952026-03-10T19:00:49.053Zsync.runtime_notifyListWait(0xc0005ac550, 0x3773)
45962026-03-10T19:00:49.053Z /work/go/src/runtime/sema.go:569 +0x159
45972026-03-10T19:00:49.053Zsync.(*Cond).Wait(0x3ae1600?)
45982026-03-10T19:00:49.053Z /work/go/src/sync/cond.go:70 +0x85
45992026-03-10T19:00:49.053Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc0066237a0})
46002026-03-10T19:00:49.054Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
46012026-03-10T19:00:49.054Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
46022026-03-10T19:00:49.054Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
46032026-03-10T19:00:49.054Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
46042026-03-10T19:00:49.054Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
46052026-03-10T19:00:49.054Z
46062026-03-10T19:00:49.054Zgoroutine 9481 [select, 10 minutes]:
46072026-03-10T19:00:49.054Zgithub.com/cockroachdb/cockroach/pkg/server.(*Server).startSystemLogsGC.func1({0x50e60e0, 0xc002de6b70})
46082026-03-10T19:00:49.054Z /work/oxidecomputer/cockroach/pkg/server/server_systemlog_gc.go:179 +0x153
46092026-03-10T19:00:49.054Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
46102026-03-10T19:00:49.054Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
46112026-03-10T19:00:49.054Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
46122026-03-10T19:00:49.054Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
46132026-03-10T19:00:49.054Z
46142026-03-10T19:00:49.054Zgoroutine 9671 [semacquire, 59 minutes]:
46152026-03-10T19:00:49.054Zsync.runtime_Semacquire(0xc001ea2438?)
46162026-03-10T19:00:49.054Z /work/go/src/runtime/sema.go:62 +0x25
46172026-03-10T19:00:49.054Zsync.(*WaitGroup).Wait(0x44f7e0?)
46182026-03-10T19:00:49.054Z /work/go/src/sync/waitgroup.go:116 +0x48
46192026-03-10T19:00:49.054Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc004afddc0)
46202026-03-10T19:00:49.054Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
46212026-03-10T19:00:49.054Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc004afddc0?, {0x50e6118?, 0xc0046a68c0?}})
46222026-03-10T19:00:49.054Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
46232026-03-10T19:00:49.054Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait({0x50e60e0?, 0xc0023665d0?}, {0xc004c64d78, 0x2, 0x4000000000000000?})
46242026-03-10T19:00:49.054Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:193 +0xff
46252026-03-10T19:00:49.054Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0xc00133c580, {0x50e60e0, 0xc0023665d0}, 0xc005f941b0)
46262026-03-10T19:00:49.055Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:291 +0x59c
46272026-03-10T19:00:49.055Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).Start.func2({0x50e60e0, 0xc005f942a0})
46282026-03-10T19:00:49.055Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:215 +0x2f3
46292026-03-10T19:00:49.055Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
46302026-03-10T19:00:49.055Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
46312026-03-10T19:00:49.056Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9741
46322026-03-10T19:00:49.056Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
46332026-03-10T19:00:49.056Z
46342026-03-10T19:00:49.056Zgoroutine 10420 [select]:
46352026-03-10T19:00:49.056Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run(0xc001443500, {0x50e60e0?, 0xc00244bdd0?}, 0x2c?, 0xc000012840, 0xc006848400)
46362026-03-10T19:00:49.056Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:269 +0x4dc
46372026-03-10T19:00:49.056Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).Start.func1({0x50e60e0?, 0xc00244bdd0?})
46382026-03-10T19:00:49.056Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:222 +0x36
46392026-03-10T19:00:49.060Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
46402026-03-10T19:00:49.060Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
46412026-03-10T19:00:49.060Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10419
46422026-03-10T19:00:49.060Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
46432026-03-10T19:00:49.060Z
46442026-03-10T19:00:49.060Zgoroutine 10232 [IO wait]:
46452026-03-10T19:00:49.060Zinternal/poll.runtime_pollWait(0x7f784b7f0f80, 0x72)
46462026-03-10T19:00:49.060Z /work/go/src/runtime/netpoll.go:345 +0x85
46472026-03-10T19:00:49.060Zinternal/poll.(*pollDesc).wait(0xc003092f00?, 0xc004014000?, 0x0)
46482026-03-10T19:00:49.060Z /work/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
46492026-03-10T19:00:49.060Zinternal/poll.(*pollDesc).waitRead(...)
46502026-03-10T19:00:49.060Z /work/go/src/internal/poll/fd_poll_runtime.go:89
46512026-03-10T19:00:49.060Zinternal/poll.(*FD).Read(0xc003092f00, {0xc004014000, 0x700, 0x700})
46522026-03-10T19:00:49.060Z /work/go/src/internal/poll/fd_unix.go:164 +0x27a
46532026-03-10T19:00:49.060Znet.(*netFD).Read(0xc003092f00, {0xc004014000?, 0x7f7835399a18?, 0xc02c3ec2e8?})
46542026-03-10T19:00:49.060Z /work/go/src/net/fd_posix.go:55 +0x25
46552026-03-10T19:00:49.060Znet.(*conn).Read(0xc000b1e1d8, {0xc004014000?, 0xc0059838e0?, 0x416dfb?})
46562026-03-10T19:00:49.060Z /work/go/src/net/net.go:185 +0x45
46572026-03-10T19:00:49.060Zcrypto/tls.(*atLeastReader).Read(0xc02c3ec2e8, {0xc004014000?, 0x0?, 0xc02c3ec2e8?})
46582026-03-10T19:00:49.060Z /work/go/src/crypto/tls/conn.go:806 +0x3b
46592026-03-10T19:00:49.060Zbytes.(*Buffer).ReadFrom(0xc004b109b0, {0x50b6dc0, 0xc02c3ec2e8})
46602026-03-10T19:00:49.060Z /work/go/src/bytes/buffer.go:211 +0x98
46612026-03-10T19:00:49.060Zcrypto/tls.(*Conn).readFromUntil(0xc004b10708, {0x50b6fa0, 0xc000b1e1d8}, 0xc005983928?)
46622026-03-10T19:00:49.061Z /work/go/src/crypto/tls/conn.go:828 +0xde
46632026-03-10T19:00:49.061Zcrypto/tls.(*Conn).readRecordOrCCS(0xc004b10708, 0x0)
46642026-03-10T19:00:49.061Z /work/go/src/crypto/tls/conn.go:626 +0x3cf
46652026-03-10T19:00:49.061Zcrypto/tls.(*Conn).readRecord(...)
46662026-03-10T19:00:49.061Z /work/go/src/crypto/tls/conn.go:588
46672026-03-10T19:00:49.061Zcrypto/tls.(*Conn).Read(0xc004b10708, {0xc002aca000, 0x8000, 0xc005983c88?})
46682026-03-10T19:00:49.061Z /work/go/src/crypto/tls/conn.go:1370 +0x156
46692026-03-10T19:00:49.061Zbufio.(*Reader).Read(0xc0044c7260, {0xc0049091c0, 0x9, 0x0?})
46702026-03-10T19:00:49.061Z /work/go/src/bufio/bufio.go:241 +0x197
46712026-03-10T19:00:49.061Zio.ReadAtLeast({0x50b1100, 0xc0044c7260}, {0xc0049091c0, 0x9, 0x9}, 0x9)
46722026-03-10T19:00:49.061Z /work/go/src/io/io.go:335 +0x90
46732026-03-10T19:00:49.061Zio.ReadFull(...)
46742026-03-10T19:00:49.061Z /work/go/src/io/io.go:354
46752026-03-10T19:00:49.061Zgolang.org/x/net/http2.readFrameHeader({0xc0049091c0, 0x9, 0xc0066229c0?}, {0x50b1100?, 0xc0044c7260?})
46762026-03-10T19:00:49.061Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x65
46772026-03-10T19:00:49.061Zgolang.org/x/net/http2.(*Framer).ReadFrame(0xc004909180)
46782026-03-10T19:00:49.062Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:501 +0x85
46792026-03-10T19:00:49.062Zgoogle.golang.org/grpc/internal/transport.(*http2Client).reader(0xc000076960)
46802026-03-10T19:00:49.062Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1498 +0x41d
46812026-03-10T19:00:49.062Zcreated by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 10158
46822026-03-10T19:00:49.062Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:365 +0x184d
46832026-03-10T19:00:49.062Z
46842026-03-10T19:00:49.062Zgoroutine 10068 [sync.Cond.Wait, 59 minutes]:
46852026-03-10T19:00:49.062Zsync.runtime_notifyListWait(0xc00676e610, 0x23)
46862026-03-10T19:00:49.062Z /work/go/src/runtime/sema.go:569 +0x159
46872026-03-10T19:00:49.062Zsync.(*Cond).Wait(0x0?)
46882026-03-10T19:00:49.062Z /work/go/src/sync/cond.go:70 +0x85
46892026-03-10T19:00:49.062Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c9d40})
46902026-03-10T19:00:49.062Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
46912026-03-10T19:00:49.062Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
46922026-03-10T19:00:49.062Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
46932026-03-10T19:00:49.062Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
46942026-03-10T19:00:49.062Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
46952026-03-10T19:00:49.062Z
46962026-03-10T19:00:49.062Zgoroutine 9470 [semacquire, 59 minutes]:
46972026-03-10T19:00:49.063Zsync.runtime_Semacquire(0xc001ea3ce0?)
46982026-03-10T19:00:49.063Z /work/go/src/runtime/sema.go:62 +0x25
46992026-03-10T19:00:49.063Zsync.(*WaitGroup).Wait(0x44f7e0?)
47002026-03-10T19:00:49.063Z /work/go/src/sync/waitgroup.go:116 +0x48
47012026-03-10T19:00:49.063Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc006102800)
47022026-03-10T19:00:49.063Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
47032026-03-10T19:00:49.063Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc006102800?, {0x50e6118?, 0xc0017499f0?}})
47042026-03-10T19:00:49.063Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
47052026-03-10T19:00:49.063Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait({0x50e60e0?, 0xc0025de540?}, {0xc005b37d78, 0x2, 0xc005b37c70?})
47062026-03-10T19:00:49.063Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:193 +0xff
47072026-03-10T19:00:49.063Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0xc0019f0000, {0x50e60e0, 0xc0025de540}, 0xc0067bbb30)
47082026-03-10T19:00:49.063Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:291 +0x59c
47092026-03-10T19:00:49.063Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).Start.func2({0x50e60e0, 0xc0025de090})
47102026-03-10T19:00:49.063Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:215 +0x2f3
47112026-03-10T19:00:49.063Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
47122026-03-10T19:00:49.063Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
47132026-03-10T19:00:49.063Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9480
47142026-03-10T19:00:49.063Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
47152026-03-10T19:00:49.063Z
47162026-03-10T19:00:49.063Zgoroutine 9314 [select, 59 minutes]:
47172026-03-10T19:00:49.063Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initCancelingWorkers.func1({0x50e60e0, 0xc00634c7b0})
47182026-03-10T19:00:49.063Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:139 +0xc5
47192026-03-10T19:00:49.064Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
47202026-03-10T19:00:49.064Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
47212026-03-10T19:00:49.064Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
47222026-03-10T19:00:49.064Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
47232026-03-10T19:00:49.065Z
47242026-03-10T19:00:49.065Zgoroutine 9102 [select]:
47252026-03-10T19:00:49.065Zgithub.com/cockroachdb/cockroach/pkg/sql/contention.(*eventStore).startEventIntake.func3({0xc0034f4f08?, 0xc001b30480?})
47262026-03-10T19:00:49.065Z /work/oxidecomputer/cockroach/pkg/sql/contention/event_store.go:188 +0xa5
47272026-03-10T19:00:49.065Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
47282026-03-10T19:00:49.065Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
47292026-03-10T19:00:49.065Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
47302026-03-10T19:00:49.065Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
47312026-03-10T19:00:49.065Z
47322026-03-10T19:00:49.065Zgoroutine 9223 [chan receive, 59 minutes]:
47332026-03-10T19:00:49.065Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50e60e0?, 0xc00634c330?})
47342026-03-10T19:00:49.065Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
47352026-03-10T19:00:49.065Zruntime/pprof.Do({0x50e6038?, 0x778e7c0?}, {{0xc00030e2c0?, 0x4217492?, 0xb4?}}, 0xc0028857a8)
47362026-03-10T19:00:49.065Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
47372026-03-10T19:00:49.065Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc000f9ae40?)
47382026-03-10T19:00:49.065Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
47392026-03-10T19:00:49.069Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9099
47402026-03-10T19:00:49.069Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
47412026-03-10T19:00:49.069Z
47422026-03-10T19:00:49.069Zgoroutine 9660 [IO wait, 59 minutes]:
47432026-03-10T19:00:49.069Zinternal/poll.runtime_pollWait(0x7f784abc3cd0, 0x72)
47442026-03-10T19:00:49.069Z /work/go/src/runtime/netpoll.go:345 +0x85
47452026-03-10T19:00:49.069Zinternal/poll.(*pollDesc).wait(0xd?, 0x1?, 0x0)
47462026-03-10T19:00:49.071Z /work/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
47472026-03-10T19:00:49.071Zinternal/poll.(*pollDesc).waitRead(...)
47482026-03-10T19:00:49.071Z /work/go/src/internal/poll/fd_poll_runtime.go:89
47492026-03-10T19:00:49.071Zinternal/poll.(*FD).Accept(0xc001287380)
47502026-03-10T19:00:49.071Z /work/go/src/internal/poll/fd_unix.go:611 +0x2ac
47512026-03-10T19:00:49.071Znet.(*netFD).accept(0xc001287380)
47522026-03-10T19:00:49.071Z /work/go/src/net/fd_unix.go:172 +0x29
47532026-03-10T19:00:49.071Znet.(*TCPListener).accept(0xc0075f2f60)
47542026-03-10T19:00:49.071Z /work/go/src/net/tcpsock_posix.go:159 +0x1e
47552026-03-10T19:00:49.071Znet.(*TCPListener).Accept(0xc0075f2f60)
47562026-03-10T19:00:49.071Z /work/go/src/net/tcpsock.go:327 +0x30
47572026-03-10T19:00:49.071Zgithub.com/cockroachdb/cmux.(*cMux).Serve(0xc00670a400)
47582026-03-10T19:00:49.071Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:124 +0x97
47592026-03-10T19:00:49.071Zgithub.com/cockroachdb/cockroach/pkg/server.(*httpServer).start.func2({0xc00670c908?, 0xc006623110?})
47602026-03-10T19:00:49.071Z /work/oxidecomputer/cockroach/pkg/server/server_http.go:243 +0x27
47612026-03-10T19:00:49.071Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
47622026-03-10T19:00:49.071Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
47632026-03-10T19:00:49.071Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
47642026-03-10T19:00:49.071Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
47652026-03-10T19:00:49.071Z
47662026-03-10T19:00:49.071Zgoroutine 9319 [chan receive, 59 minutes]:
47672026-03-10T19:00:49.071Zgithub.com/cockroachdb/cockroach/pkg/util/netutil.MakeServer.func2({0xc00088fb08?, 0xc0016e4ae0?})
47682026-03-10T19:00:49.071Z /work/oxidecomputer/cockroach/pkg/util/netutil/net.go:113 +0x49
47692026-03-10T19:00:49.071Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
47702026-03-10T19:00:49.072Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
47712026-03-10T19:00:49.072Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
47722026-03-10T19:00:49.072Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
47732026-03-10T19:00:49.072Z
47742026-03-10T19:00:49.072Zgoroutine 10302 [IO wait]:
47752026-03-10T19:00:49.072Zinternal/poll.runtime_pollWait(0x7f784b7f1d10, 0x72)
47762026-03-10T19:00:49.072Z /work/go/src/runtime/netpoll.go:345 +0x85
47772026-03-10T19:00:49.072Zinternal/poll.(*pollDesc).wait(0xc000907e00?, 0xc003616000?, 0x0)
47782026-03-10T19:00:49.072Z /work/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
47792026-03-10T19:00:49.072Zinternal/poll.(*pollDesc).waitRead(...)
47802026-03-10T19:00:49.072Z /work/go/src/internal/poll/fd_poll_runtime.go:89
47812026-03-10T19:00:49.072Zinternal/poll.(*FD).Read(0xc000907e00, {0xc003616000, 0x600, 0x600})
47822026-03-10T19:00:49.072Z /work/go/src/internal/poll/fd_unix.go:164 +0x27a
47832026-03-10T19:00:49.072Znet.(*netFD).Read(0xc000907e00, {0xc003616000?, 0x7f78483fc4d0?, 0xc012e1b878?})
47842026-03-10T19:00:49.072Z /work/go/src/net/fd_posix.go:55 +0x25
47852026-03-10T19:00:49.072Znet.(*conn).Read(0xc0020f2a08, {0xc003616000?, 0xc005e9ce08?, 0x416dfb?})
47862026-03-10T19:00:49.072Z /work/go/src/net/net.go:185 +0x45
47872026-03-10T19:00:49.072Zcrypto/tls.(*atLeastReader).Read(0xc012e1b878, {0xc003616000?, 0x0?, 0xc012e1b878?})
47882026-03-10T19:00:49.072Z /work/go/src/crypto/tls/conn.go:806 +0x3b
47892026-03-10T19:00:49.072Zbytes.(*Buffer).ReadFrom(0xc002069b30, {0x50b6dc0, 0xc012e1b878})
47902026-03-10T19:00:49.072Z /work/go/src/bytes/buffer.go:211 +0x98
47912026-03-10T19:00:49.072Zcrypto/tls.(*Conn).readFromUntil(0xc002069888, {0x50b6fa0, 0xc0020f2a08}, 0xc005e9ce50?)
47922026-03-10T19:00:49.072Z /work/go/src/crypto/tls/conn.go:828 +0xde
47932026-03-10T19:00:49.072Zcrypto/tls.(*Conn).readRecordOrCCS(0xc002069888, 0x0)
47942026-03-10T19:00:49.073Z /work/go/src/crypto/tls/conn.go:626 +0x3cf
47952026-03-10T19:00:49.073Zcrypto/tls.(*Conn).readRecord(...)
47962026-03-10T19:00:49.073Z /work/go/src/crypto/tls/conn.go:588
47972026-03-10T19:00:49.073Zcrypto/tls.(*Conn).Read(0xc002069888, {0xc002794000, 0x1000, 0x0?})
47982026-03-10T19:00:49.073Z /work/go/src/crypto/tls/conn.go:1370 +0x156
47992026-03-10T19:00:49.073Zgithub.com/cockroachdb/cockroach/pkg/sql/pgwire.(*readTimeoutConn).Read(0xc004e7bd58, {0xc002794000, 0x1000, 0x1000})
48002026-03-10T19:00:49.073Z /work/oxidecomputer/cockroach/pkg/sql/pgwire/conn.go:1840 +0xfe
48012026-03-10T19:00:49.073Zbufio.(*Reader).fill(0xc00483cb10)
48022026-03-10T19:00:49.073Z /work/go/src/bufio/bufio.go:110 +0x103
48032026-03-10T19:00:49.073Zbufio.(*Reader).ReadByte(0xc00483cb10)
48042026-03-10T19:00:49.073Z /work/go/src/bufio/bufio.go:269 +0x27
48052026-03-10T19:00:49.073Zgithub.com/cockroachdb/cockroach/pkg/sql/pgwire/pgwirebase.(*ReadBuffer).ReadTypedMsg(0xc00483d2e0, {0x50dfcc0, 0xc00483cb10})
48062026-03-10T19:00:49.073Z /work/oxidecomputer/cockroach/pkg/sql/pgwire/pgwirebase/encoding.go:206 +0x2a
48072026-03-10T19:00:49.073Zgithub.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).serveImpl.func5(0xc00483ca88, 0xc003fcccc0, 0xc005e9d48b, 0xc005e9d48d, {0x50dfc90, 0xc00479cb40}, 0xc003073508, 0xc000a103e0, 0xc005e9d489)
48082026-03-10T19:00:49.073Z /work/oxidecomputer/cockroach/pkg/sql/pgwire/conn.go:400 +0x97
48092026-03-10T19:00:49.073Zgithub.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).serveImpl(0xc00483ca88, {0x50e60e0, 0xc002e69a40}, 0xc003fcccb0, 0xc003073508, {0x5400, 0x15000, 0xc002c5c0f0, 0x0, 0x0}, ...)
48102026-03-10T19:00:49.073Z /work/oxidecomputer/cockroach/pkg/sql/pgwire/conn.go:540 +0xb2a
48112026-03-10T19:00:49.073Zgithub.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).serveConn(0xc005280580, {0x50e60e0, _}, {_, _}, {{{0xc00243401e, 0x4}}, 0x1, 0xc002e699e0, 0xc002e69a10, ...}, ...)
48122026-03-10T19:00:49.074Z /work/oxidecomputer/cockroach/pkg/sql/pgwire/conn.go:173 +0x2b6
48132026-03-10T19:00:49.074Zgithub.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).ServeConn(0xc005280580, {0x50e60e0, 0xc002e69a40}, {0x5120298, 0xc0020f2a08}, 0x1)
48142026-03-10T19:00:49.074Z /work/oxidecomputer/cockroach/pkg/sql/pgwire/server.go:874 +0xd49
48152026-03-10T19:00:49.075Zgithub.com/cockroachdb/cockroach/pkg/server.(*SQLServer).startServeSQL.func1.1({0x50e60e0?, 0xc0037fcc30?}, {0x5120298, 0xc0020f2a08})
48162026-03-10T19:00:49.075Z /work/oxidecomputer/cockroach/pkg/server/server_sql.go:1378 +0xbd
48172026-03-10T19:00:49.075Zgithub.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith.func1({0x50e60e0, 0xc0037fcc30})
48182026-03-10T19:00:49.075Z /work/oxidecomputer/cockroach/pkg/util/netutil/net.go:162 +0x11a
48192026-03-10T19:00:49.075Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
48202026-03-10T19:00:49.075Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
48212026-03-10T19:00:49.075Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9655
48222026-03-10T19:00:49.075Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
48232026-03-10T19:00:49.075Z
48242026-03-10T19:00:49.075Zgoroutine 10343 [select]:
48252026-03-10T19:00:49.075Zgithub.com/cockroachdb/cockroach/pkg/ts.(*poller).start.func1({0x50e60e0, 0xc0030c2510})
48262026-03-10T19:00:49.075Z /work/oxidecomputer/cockroach/pkg/ts/db.go:176 +0xea
48272026-03-10T19:00:49.075Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
48282026-03-10T19:00:49.075Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
48292026-03-10T19:00:49.075Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
48302026-03-10T19:00:49.075Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
48312026-03-10T19:00:49.075Z
48322026-03-10T19:00:49.075Zgoroutine 9639 [select]:
48332026-03-10T19:00:49.075Zgithub.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func8({0x50e60e0?, 0xc0035b40f0?})
48342026-03-10T19:00:49.075Z /work/oxidecomputer/cockroach/pkg/jobs/registry.go:811 +0x192
48352026-03-10T19:00:49.075Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
48362026-03-10T19:00:49.075Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
48372026-03-10T19:00:49.076Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
48382026-03-10T19:00:49.076Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
48392026-03-10T19:00:49.080Z
48402026-03-10T19:00:49.080Zgoroutine 10493 [select, 59 minutes]:
48412026-03-10T19:00:49.080Zgoogle.golang.org/grpc.newClientStreamWithParams.func4()
48422026-03-10T19:00:49.080Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:341 +0x8c
48432026-03-10T19:00:49.080Zcreated by google.golang.org/grpc.newClientStreamWithParams in goroutine 10492
48442026-03-10T19:00:49.080Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:340 +0xb2a
48452026-03-10T19:00:49.080Z
48462026-03-10T19:00:49.080Zgoroutine 10366 [select]:
48472026-03-10T19:00:49.080Zgoogle.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc001296b90, {0xc004150ef8, 0x5, 0x5})
48482026-03-10T19:00:49.080Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:191 +0x95
48492026-03-10T19:00:49.080Zgoogle.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc001296b90, {0xc004150ef8?, 0xc0042030b0?, 0xc003695048?})
48502026-03-10T19:00:49.080Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:171 +0x12d
48512026-03-10T19:00:49.080Zgoogle.golang.org/grpc/internal/transport.(*transportReader).Read(0xc004e36cf0, {0xc004150ef8?, 0xc0036950c0?, 0xa418a5?})
48522026-03-10T19:00:49.080Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:485 +0x2c
48532026-03-10T19:00:49.080Zio.ReadAtLeast({0x50bb160, 0xc004e36cf0}, {0xc004150ef8, 0x5, 0x5}, 0x5)
48542026-03-10T19:00:49.080Z /work/go/src/io/io.go:335 +0x90
48552026-03-10T19:00:49.080Zio.ReadFull(...)
48562026-03-10T19:00:49.080Z /work/go/src/io/io.go:354
48572026-03-10T19:00:49.080Zgoogle.golang.org/grpc/internal/transport.(*Stream).Read(0xc00328cd80, {0xc004150ef8, 0x5, 0x5})
48582026-03-10T19:00:49.081Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:469 +0x96
48592026-03-10T19:00:49.081Zgoogle.golang.org/grpc.(*parser).recvMsg(0xc004150ee8, 0x7fffffff)
48602026-03-10T19:00:49.081Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:559 +0x42
48612026-03-10T19:00:49.081Zgoogle.golang.org/grpc.recvAndDecompress(0x60?, 0xc00328cd80, {0x0, 0x0}, 0x7fffffff, 0xc003695378, {0x50ddc20, 0x778e7c0})
48622026-03-10T19:00:49.081Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:690 +0x5a
48632026-03-10T19:00:49.081Zgoogle.golang.org/grpc.recv(0x50c4e80?, {0x7f784b7a48e8, 0xc000567440}, 0x2b?, {0x0?, 0x0?}, {0x3fa0a40, 0xc0263768a0}, 0x10?, 0xc003695378, ...)
48642026-03-10T19:00:49.081Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:756 +0x68
48652026-03-10T19:00:49.081Zgoogle.golang.org/grpc.(*csAttempt).recvMsg(0xc0064cea50, {0x3fa0a40, 0xc0263768a0}, 0x10?)
48662026-03-10T19:00:49.081Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:983 +0x27e
48672026-03-10T19:00:49.081Zgoogle.golang.org/grpc.(*clientStream).RecvMsg.func1(0x41f225?)
48682026-03-10T19:00:49.081Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:834 +0x1f
48692026-03-10T19:00:49.081Zgoogle.golang.org/grpc.(*clientStream).withRetry(0xc00328c7e0, 0xc0036954a8, 0xc003695498)
48702026-03-10T19:00:49.081Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:688 +0x2ee
48712026-03-10T19:00:49.081Zgoogle.golang.org/grpc.(*clientStream).RecvMsg(0xc00328c7e0, {0x3fa0a40?, 0xc0263768a0?})
48722026-03-10T19:00:49.081Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:833 +0x107
48732026-03-10T19:00:49.081Zgithub.com/cockroachdb/cockroach/pkg/util/tracing.(*tracingClientStream).RecvMsg(0xc0002d8d00, {0x3fa0a40?, 0xc0263768a0?})
48742026-03-10T19:00:49.081Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:441 +0x31
48752026-03-10T19:00:49.081Zgithub.com/cockroachdb/cockroach/pkg/roachpb.(*internalRangeFeedClient).Recv(0xc006949b80)
48762026-03-10T19:00:49.081Z /work/oxidecomputer/cockroach/pkg/roachpb/api.pb.go:9289 +0x46
48772026-03-10T19:00:49.081Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed.func3()
48782026-03-10T19:00:49.081Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:516 +0x25
48792026-03-10T19:00:49.082Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*stuckRangeFeedCanceler).do(0xc001296a00, 0xc003695a50)
48802026-03-10T19:00:49.082Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed_canceler.go:100 +0x1b3
48812026-03-10T19:00:49.082Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed(0xc0021b6f08, {0x50e6118?, 0xc001296910?}, {{0xc007bc2700, 0x1, 0x8}, {0xc007bc2708, 0x1, 0x8}}, {0x189b8cc3e6aab445, ...}, ...)
48822026-03-10T19:00:49.082Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:515 +0xc98
48832026-03-10T19:00:49.082Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).partialRangeFeed(0xc0021b6f08, {0x50e6118, 0xc001296910}, 0xc0012968c0, {{0xc007bc2700, 0x1, 0x8}, {0xc007bc2708, 0x1, 0x8}}, ...)
48842026-03-10T19:00:49.082Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:336 +0x745
48852026-03-10T19:00:49.082Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.1({0x50e6118?, 0xc001296910?})
48862026-03-10T19:00:49.082Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:120 +0xa9
48872026-03-10T19:00:49.082Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.Group.GoCtx.2()
48882026-03-10T19:00:49.083Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
48892026-03-10T19:00:49.083Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
48902026-03-10T19:00:49.083Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
48912026-03-10T19:00:49.083Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10364
48922026-03-10T19:00:49.083Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
48932026-03-10T19:00:49.083Z
48942026-03-10T19:00:49.083Zgoroutine 10483 [select, 59 minutes]:
48952026-03-10T19:00:49.083Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1({0x50e6118, 0xc004f9b8b0})
48962026-03-10T19:00:49.083Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:116 +0x157
48972026-03-10T19:00:49.083Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.Group.GoCtx.func3()
48982026-03-10T19:00:49.083Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
48992026-03-10T19:00:49.083Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
49002026-03-10T19:00:49.083Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
49012026-03-10T19:00:49.083Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10397
49022026-03-10T19:00:49.083Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
49032026-03-10T19:00:49.084Z
49042026-03-10T19:00:49.084Zgoroutine 9165 [select, 59 minutes]:
49052026-03-10T19:00:49.084Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2({0x50e60e0, 0xc001eac300})
49062026-03-10T19:00:49.084Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/queue.go:823 +0xff
49072026-03-10T19:00:49.084Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
49082026-03-10T19:00:49.084Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
49092026-03-10T19:00:49.084Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9437
49102026-03-10T19:00:49.084Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
49112026-03-10T19:00:49.084Z
49122026-03-10T19:00:49.084Zgoroutine 9411 [sync.Cond.Wait]:
49132026-03-10T19:00:49.084Zsync.runtime_notifyListWait(0xc0005ac550, 0x3760)
49142026-03-10T19:00:49.084Z /work/go/src/runtime/sema.go:569 +0x159
49152026-03-10T19:00:49.084Zsync.(*Cond).Wait(0x3ae1600?)
49162026-03-10T19:00:49.084Z /work/go/src/sync/cond.go:70 +0x85
49172026-03-10T19:00:49.084Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006623650})
49182026-03-10T19:00:49.084Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
49192026-03-10T19:00:49.084Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
49202026-03-10T19:00:49.084Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
49212026-03-10T19:00:49.085Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
49222026-03-10T19:00:49.085Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
49232026-03-10T19:00:49.085Z
49242026-03-10T19:00:49.085Zgoroutine 9276 [select]:
49252026-03-10T19:00:49.085Zgithub.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
49262026-03-10T19:00:49.085Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:439 +0x1c9
49272026-03-10T19:00:49.085Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 9099
49282026-03-10T19:00:49.085Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:412 +0x4f
49292026-03-10T19:00:49.085Z
49302026-03-10T19:00:49.085Zgoroutine 10324 [select]:
49312026-03-10T19:00:49.085Zgithub.com/cockroachdb/cockroach/pkg/kv/kvprober.(*Prober).Start.func1.1({0x50e60e0, 0xc002e688a0})
49322026-03-10T19:00:49.085Z /work/oxidecomputer/cockroach/pkg/kv/kvprober/kvprober.go:293 +0x314
49332026-03-10T19:00:49.085Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
49342026-03-10T19:00:49.085Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
49352026-03-10T19:00:49.085Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
49362026-03-10T19:00:49.085Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
49372026-03-10T19:00:49.085Z
49382026-03-10T19:00:49.085Zgoroutine 9299 [select, 59 minutes]:
49392026-03-10T19:00:49.089Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc002c06008?, 0x0?})
49402026-03-10T19:00:49.089Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
49412026-03-10T19:00:49.089Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
49422026-03-10T19:00:49.089Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
49432026-03-10T19:00:49.089Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
49442026-03-10T19:00:49.090Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
49452026-03-10T19:00:49.090Z
49462026-03-10T19:00:49.090Zgoroutine 9301 [select, 59 minutes]:
49472026-03-10T19:00:49.090Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc002c06608?, 0x0?})
49482026-03-10T19:00:49.090Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
49492026-03-10T19:00:49.090Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
49502026-03-10T19:00:49.091Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
49512026-03-10T19:00:49.091Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
49522026-03-10T19:00:49.091Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
49532026-03-10T19:00:49.091Z
49542026-03-10T19:00:49.091Zgoroutine 10363 [select]:
49552026-03-10T19:00:49.091Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents(0xc0035129a0, {0x50e6118, 0xc001296870}, 0xc002cb1d70, 0xc000e36660)
49562026-03-10T19:00:49.091Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:331 +0xc5
49572026-03-10T19:00:49.091Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func2({0x50e6118?, 0xc001296870?})
49582026-03-10T19:00:49.091Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:288 +0x32
49592026-03-10T19:00:49.091Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
49602026-03-10T19:00:49.091Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
49612026-03-10T19:00:49.091Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
49622026-03-10T19:00:49.091Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
49632026-03-10T19:00:49.091Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10276
49642026-03-10T19:00:49.091Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
49652026-03-10T19:00:49.091Z
49662026-03-10T19:00:49.091Zgoroutine 9365 [chan receive, 59 minutes]:
49672026-03-10T19:00:49.091Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func1({0xc0016d4008?, 0xc004278060?})
49682026-03-10T19:00:49.091Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:203 +0x37
49692026-03-10T19:00:49.091Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
49702026-03-10T19:00:49.091Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
49712026-03-10T19:00:49.091Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
49722026-03-10T19:00:49.091Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
49732026-03-10T19:00:49.091Z
49742026-03-10T19:00:49.091Zgoroutine 9982 [select, 10 minutes]:
49752026-03-10T19:00:49.091Zgithub.com/cockroachdb/cockroach/pkg/server.(*Server).startSystemLogsGC.func1({0x50e60e0, 0xc006f82570})
49762026-03-10T19:00:49.091Z /work/oxidecomputer/cockroach/pkg/server/server_systemlog_gc.go:179 +0x153
49772026-03-10T19:00:49.091Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
49782026-03-10T19:00:49.091Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
49792026-03-10T19:00:49.091Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
49802026-03-10T19:00:49.091Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
49812026-03-10T19:00:49.091Z
49822026-03-10T19:00:49.091Zgoroutine 9600 [select, 59 minutes]:
49832026-03-10T19:00:49.091Zgithub.com/cockroachdb/cockroach/pkg/sql/sqlstats/sslocal.(*SQLStats).Start.func1({0x50e60e0, 0xc006037cb0})
49842026-03-10T19:00:49.091Z /work/oxidecomputer/cockroach/pkg/sql/sqlstats/sslocal/sslocal_provider.go:80 +0x306
49852026-03-10T19:00:49.091Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
49862026-03-10T19:00:49.091Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
49872026-03-10T19:00:49.091Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
49882026-03-10T19:00:49.093Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
49892026-03-10T19:00:49.093Z
49902026-03-10T19:00:49.093Zgoroutine 10492 [select]:
49912026-03-10T19:00:49.093Zgoogle.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc000286230, {0xc004b20b80, 0x5, 0x5})
49922026-03-10T19:00:49.093Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:191 +0x95
49932026-03-10T19:00:49.093Zgoogle.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc000286230, {0xc004b20b80?, 0xc004046db0?, 0xc003651048?})
49942026-03-10T19:00:49.093Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:171 +0x12d
49952026-03-10T19:00:49.093Zgoogle.golang.org/grpc/internal/transport.(*transportReader).Read(0xc006fe6720, {0xc004b20b80?, 0xc0036510c0?, 0xa418a5?})
49962026-03-10T19:00:49.093Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:485 +0x2c
49972026-03-10T19:00:49.093Zio.ReadAtLeast({0x50bb160, 0xc006fe6720}, {0xc004b20b80, 0x5, 0x5}, 0x5)
49982026-03-10T19:00:49.093Z /work/go/src/io/io.go:335 +0x90
49992026-03-10T19:00:49.093Zio.ReadFull(...)
50002026-03-10T19:00:49.093Z /work/go/src/io/io.go:354
50012026-03-10T19:00:49.093Zgoogle.golang.org/grpc/internal/transport.(*Stream).Read(0xc0021ec900, {0xc004b20b80, 0x5, 0x5})
50022026-03-10T19:00:49.093Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:469 +0x96
50032026-03-10T19:00:49.093Zgoogle.golang.org/grpc.(*parser).recvMsg(0xc004b20b70, 0x7fffffff)
50042026-03-10T19:00:49.093Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:559 +0x42
50052026-03-10T19:00:49.093Zgoogle.golang.org/grpc.recvAndDecompress(0x60?, 0xc0021ec900, {0x0, 0x0}, 0x7fffffff, 0xc003651378, {0x50ddc20, 0x778e7c0})
50062026-03-10T19:00:49.093Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:690 +0x5a
50072026-03-10T19:00:49.093Zgoogle.golang.org/grpc.recv(0x50c4e80?, {0x7f784b7a48e8, 0xc000567440}, 0xc003651300?, {0x0?, 0x0?}, {0x3fa0a40, 0xc018678ee0}, 0x10?, 0xc003651378, ...)
50082026-03-10T19:00:49.093Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:756 +0x68
50092026-03-10T19:00:49.094Zgoogle.golang.org/grpc.(*csAttempt).recvMsg(0xc004c18b00, {0x3fa0a40, 0xc018678ee0}, 0x10?)
50102026-03-10T19:00:49.094Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:983 +0x27e
50112026-03-10T19:00:49.094Zgoogle.golang.org/grpc.(*clientStream).RecvMsg.func1(0x41f225?)
50122026-03-10T19:00:49.094Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:834 +0x1f
50132026-03-10T19:00:49.094Zgoogle.golang.org/grpc.(*clientStream).withRetry(0xc0021ec5a0, 0xc0036514a8, 0xc003651498)
50142026-03-10T19:00:49.094Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:688 +0x2ee
50152026-03-10T19:00:49.094Zgoogle.golang.org/grpc.(*clientStream).RecvMsg(0xc0021ec5a0, {0x3fa0a40?, 0xc018678ee0?})
50162026-03-10T19:00:49.094Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:833 +0x107
50172026-03-10T19:00:49.094Zgithub.com/cockroachdb/cockroach/pkg/util/tracing.(*tracingClientStream).RecvMsg(0xc004c15160, {0x3fa0a40?, 0xc018678ee0?})
50182026-03-10T19:00:49.094Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:441 +0x31
50192026-03-10T19:00:49.094Zgithub.com/cockroachdb/cockroach/pkg/roachpb.(*internalRangeFeedClient).Recv(0xc005efe8b0)
50202026-03-10T19:00:49.094Z /work/oxidecomputer/cockroach/pkg/roachpb/api.pb.go:9289 +0x46
50212026-03-10T19:00:49.094Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed.func3()
50222026-03-10T19:00:49.094Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:516 +0x25
50232026-03-10T19:00:49.094Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*stuckRangeFeedCanceler).do(0xc004f9bea0, 0xc003651a50)
50242026-03-10T19:00:49.094Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed_canceler.go:100 +0x1b3
50252026-03-10T19:00:49.094Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed(0xc0021b6f08, {0x50e6118?, 0xc004f9bd10?}, {{0xc005b26360, 0x1, 0x8}, {0xc005b26368, 0x1, 0x8}}, {0x189b8cc3e6be20c1, ...}, ...)
50262026-03-10T19:00:49.094Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:515 +0xc98
50272026-03-10T19:00:49.094Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).partialRangeFeed(0xc0021b6f08, {0x50e6118, 0xc004f9bd10}, 0xc004f9bcc0, {{0xc005b26360, 0x1, 0x8}, {0xc005b26368, 0x1, 0x8}}, ...)
50282026-03-10T19:00:49.094Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:336 +0x745
50292026-03-10T19:00:49.094Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.1({0x50e6118?, 0xc004f9bd10?})
50302026-03-10T19:00:49.095Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:120 +0xa9
50312026-03-10T19:00:49.095Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.Group.GoCtx.2()
50322026-03-10T19:00:49.095Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
50332026-03-10T19:00:49.095Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
50342026-03-10T19:00:49.095Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
50352026-03-10T19:00:49.095Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10490
50362026-03-10T19:00:49.095Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
50372026-03-10T19:00:49.095Z
50382026-03-10T19:00:49.095Zgoroutine 10490 [select, 59 minutes]:
50392026-03-10T19:00:49.099Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1({0x50e6118, 0xc004f9bd10})
50402026-03-10T19:00:49.099Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:116 +0x157
50412026-03-10T19:00:49.099Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.Group.GoCtx.func3()
50422026-03-10T19:00:49.099Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
50432026-03-10T19:00:49.099Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
50442026-03-10T19:00:49.099Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
50452026-03-10T19:00:49.099Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10488
50462026-03-10T19:00:49.099Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
50472026-03-10T19:00:49.099Z
50482026-03-10T19:00:49.099Zgoroutine 9233 [select, 59 minutes]:
50492026-03-10T19:00:49.099Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc0067f6608?, 0x934cd177ccf7c163?})
50502026-03-10T19:00:49.099Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
50512026-03-10T19:00:49.099Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
50522026-03-10T19:00:49.099Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
50532026-03-10T19:00:49.099Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
50542026-03-10T19:00:49.099Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
50552026-03-10T19:00:49.099Z
50562026-03-10T19:00:49.099Zgoroutine 9638 [select, 2 minutes]:
50572026-03-10T19:00:49.099Zgithub.com/cockroachdb/cockroach/pkg/sql/catalog/lease.(*Manager).PeriodicallyRefreshSomeLeases.func1({0x50e60e0, 0xc0035b4030})
50582026-03-10T19:00:49.099Z /work/oxidecomputer/cockroach/pkg/sql/catalog/lease/lease.go:1151 +0x2df
50592026-03-10T19:00:49.099Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
50602026-03-10T19:00:49.099Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
50612026-03-10T19:00:49.099Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
50622026-03-10T19:00:49.099Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
50632026-03-10T19:00:49.099Z
50642026-03-10T19:00:49.099Zgoroutine 10433 [select, 59 minutes]:
50652026-03-10T19:00:49.099Zgoogle.golang.org/grpc.newClientStreamWithParams.func4()
50662026-03-10T19:00:49.100Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:341 +0x8c
50672026-03-10T19:00:49.100Zcreated by google.golang.org/grpc.newClientStreamWithParams in goroutine 10432
50682026-03-10T19:00:49.100Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:340 +0xb2a
50692026-03-10T19:00:49.100Z
50702026-03-10T19:00:49.100Zgoroutine 9240 [select, 59 minutes]:
50712026-03-10T19:00:49.100Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc004bcc308?, 0x40aaea6?})
50722026-03-10T19:00:49.100Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
50732026-03-10T19:00:49.100Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
50742026-03-10T19:00:49.100Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
50752026-03-10T19:00:49.100Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
50762026-03-10T19:00:49.100Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
50772026-03-10T19:00:49.100Z
50782026-03-10T19:00:49.100Zgoroutine 9241 [select, 59 minutes]:
50792026-03-10T19:00:49.100Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc004bcc608?, 0x0?})
50802026-03-10T19:00:49.100Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
50812026-03-10T19:00:49.100Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
50822026-03-10T19:00:49.100Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
50832026-03-10T19:00:49.101Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
50842026-03-10T19:00:49.101Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
50852026-03-10T19:00:49.101Z
50862026-03-10T19:00:49.101Zgoroutine 9260 [chan receive, 59 minutes]:
50872026-03-10T19:00:49.101Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50e60e0?, 0xc005cee4b0?})
50882026-03-10T19:00:49.101Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
50892026-03-10T19:00:49.101Zruntime/pprof.Do({0x50e6038?, 0x778e7c0?}, {{0xc00030e2c0?, 0x0?, 0x0?}}, 0xc0022c5fa8)
50902026-03-10T19:00:49.101Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
50912026-03-10T19:00:49.101Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
50922026-03-10T19:00:49.101Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
50932026-03-10T19:00:49.101Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9099
50942026-03-10T19:00:49.102Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
50952026-03-10T19:00:49.102Z
50962026-03-10T19:00:49.102Zgoroutine 9225 [select]:
50972026-03-10T19:00:49.102Zgithub.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1()
50982026-03-10T19:00:49.102Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:71 +0xcb
50992026-03-10T19:00:49.102Zcreated by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 9099
51002026-03-10T19:00:49.102Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:66 +0x58
51012026-03-10T19:00:49.102Z
51022026-03-10T19:00:49.102Zgoroutine 9628 [select]:
51032026-03-10T19:00:49.102Zgoogle.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc0016874a0)
51042026-03-10T19:00:49.102Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1572 +0x137
51052026-03-10T19:00:49.102Zcreated by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 10025
51062026-03-10T19:00:49.102Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:360 +0x180b
51072026-03-10T19:00:49.102Z
51082026-03-10T19:00:49.102Zgoroutine 9527 [select]:
51092026-03-10T19:00:49.102Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.(*Watcher).Run(0xc00143c240, {0x50e6118, 0xc0031546e0})
51102026-03-10T19:00:49.102Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:319 +0x955
51112026-03-10T19:00:49.102Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.Start.func1({0x50e60e0?, 0xc00398d620?})
51122026-03-10T19:00:49.103Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:189 +0x2d1
51132026-03-10T19:00:49.103Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
51142026-03-10T19:00:49.103Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
51152026-03-10T19:00:49.103Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
51162026-03-10T19:00:49.103Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
51172026-03-10T19:00:49.103Z
51182026-03-10T19:00:49.103Zgoroutine 9287 [chan receive, 59 minutes]:
51192026-03-10T19:00:49.103Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50e60e0?, 0xc005cee7b0?})
51202026-03-10T19:00:49.103Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
51212026-03-10T19:00:49.103Zruntime/pprof.Do({0x50e6038?, 0x778e7c0?}, {{0xc00030e2c0?, 0x12?, 0x40b1340?}}, 0xc003bb77a8)
51222026-03-10T19:00:49.103Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
51232026-03-10T19:00:49.103Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
51242026-03-10T19:00:49.103Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
51252026-03-10T19:00:49.103Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9099
51262026-03-10T19:00:49.103Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
51272026-03-10T19:00:49.103Z
51282026-03-10T19:00:49.103Zgoroutine 10494 [select, 59 minutes]:
51292026-03-10T19:00:49.103Zgithub.com/cockroachdb/cockroach/pkg/util/tracing.newTracingClientStream.func2()
51302026-03-10T19:00:49.103Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:386 +0x67
51312026-03-10T19:00:49.103Zcreated by github.com/cockroachdb/cockroach/pkg/util/tracing.newTracingClientStream in goroutine 10492
51322026-03-10T19:00:49.103Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:385 +0x125
51332026-03-10T19:00:49.103Z
51342026-03-10T19:00:49.103Zgoroutine 9981 [select]:
51352026-03-10T19:00:49.103Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.(*Watcher).Run(0xc002e2c360, {0x50e6118, 0xc00661b900})
51362026-03-10T19:00:49.103Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:319 +0x955
51372026-03-10T19:00:49.103Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.Start.func1({0x50e60e0?, 0xc006f82510?})
51382026-03-10T19:00:49.104Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:189 +0x2d1
51392026-03-10T19:00:49.108Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
51402026-03-10T19:00:49.108Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
51412026-03-10T19:00:49.108Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
51422026-03-10T19:00:49.108Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
51432026-03-10T19:00:49.108Z
51442026-03-10T19:00:49.108Zgoroutine 9040 [select, 59 minutes]:
51452026-03-10T19:00:49.108Zgithub.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).bootstrap.func1({0x50e60e0?, 0xc007a03740?})
51462026-03-10T19:00:49.108Z /work/oxidecomputer/cockroach/pkg/gossip/gossip.go:1385 +0x253
51472026-03-10T19:00:49.108Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
51482026-03-10T19:00:49.108Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
51492026-03-10T19:00:49.108Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
51502026-03-10T19:00:49.108Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
51512026-03-10T19:00:49.108Z
51522026-03-10T19:00:49.108Zgoroutine 10386 [select]:
51532026-03-10T19:00:49.108Zgoogle.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0052749b0, 0x1)
51542026-03-10T19:00:49.108Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:407 +0x113
51552026-03-10T19:00:49.108Zgoogle.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00461ac00)
51562026-03-10T19:00:49.108Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:534 +0x7b
51572026-03-10T19:00:49.108Zgoogle.golang.org/grpc/internal/transport.newHTTP2Client.func3()
51582026-03-10T19:00:49.108Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:415 +0x67
51592026-03-10T19:00:49.108Zcreated by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 10354
51602026-03-10T19:00:49.108Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:413 +0x1dcb
51612026-03-10T19:00:49.108Z
51622026-03-10T19:00:49.108Zgoroutine 9266 [select]:
51632026-03-10T19:00:49.108Zgithub.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1()
51642026-03-10T19:00:49.108Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:71 +0xcb
51652026-03-10T19:00:49.108Zcreated by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 9099
51662026-03-10T19:00:49.108Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:66 +0x58
51672026-03-10T19:00:49.108Z
51682026-03-10T19:00:49.109Zgoroutine 9163 [select]:
51692026-03-10T19:00:49.109Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2({0x50e60e0, 0xc001eac240})
51702026-03-10T19:00:49.109Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/queue.go:823 +0xff
51712026-03-10T19:00:49.109Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
51722026-03-10T19:00:49.109Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
51732026-03-10T19:00:49.109Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9437
51742026-03-10T19:00:49.109Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
51752026-03-10T19:00:49.109Z
51762026-03-10T19:00:49.109Zgoroutine 10080 [sync.Cond.Wait, 59 minutes]:
51772026-03-10T19:00:49.109Zsync.runtime_notifyListWait(0xc00676e610, 0x2f)
51782026-03-10T19:00:49.109Z /work/go/src/runtime/sema.go:569 +0x159
51792026-03-10T19:00:49.109Zsync.(*Cond).Wait(0x0?)
51802026-03-10T19:00:49.109Z /work/go/src/sync/cond.go:70 +0x85
51812026-03-10T19:00:49.109Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0072ac000})
51822026-03-10T19:00:49.109Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
51832026-03-10T19:00:49.109Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
51842026-03-10T19:00:49.110Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
51852026-03-10T19:00:49.110Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
51862026-03-10T19:00:49.110Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
51872026-03-10T19:00:49.110Z
51882026-03-10T19:00:49.110Zgoroutine 9216 [select]:
51892026-03-10T19:00:49.110Zgithub.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).startTickerLocked.func1()
51902026-03-10T19:00:49.110Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:365 +0xd2
51912026-03-10T19:00:49.110Zcreated by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).startTickerLocked in goroutine 9099
51922026-03-10T19:00:49.110Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:360 +0x6e
51932026-03-10T19:00:49.110Z
51942026-03-10T19:00:49.110Zgoroutine 9253 [semacquire, 59 minutes]:
51952026-03-10T19:00:49.110Zsync.runtime_Semacquire(0xc0032f40f0?)
51962026-03-10T19:00:49.110Z /work/go/src/runtime/sema.go:62 +0x25
51972026-03-10T19:00:49.110Zsync.(*WaitGroup).Wait(0x44f7e0?)
51982026-03-10T19:00:49.110Z /work/go/src/sync/waitgroup.go:116 +0x48
51992026-03-10T19:00:49.110Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc0020a4040)
52002026-03-10T19:00:49.110Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
52012026-03-10T19:00:49.111Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc0020a4040?, {0x50e6118?, 0xc002de2190?}})
52022026-03-10T19:00:49.111Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
52032026-03-10T19:00:49.111Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait({0x50e60e0?, 0xc005cee300?}, {0xc0034b2d78, 0x2, 0x4000000000000000?})
52042026-03-10T19:00:49.111Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:193 +0xff
52052026-03-10T19:00:49.111Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0xc005280420, {0x50e60e0, 0xc005cee300}, 0xc001ead830)
52062026-03-10T19:00:49.111Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:291 +0x59c
52072026-03-10T19:00:49.111Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).Start.func2({0x50e60e0, 0xc001eada40})
52082026-03-10T19:00:49.111Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:215 +0x2f3
52092026-03-10T19:00:49.111Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
52102026-03-10T19:00:49.111Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
52112026-03-10T19:00:49.111Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
52122026-03-10T19:00:49.111Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
52132026-03-10T19:00:49.111Z
52142026-03-10T19:00:49.111Zgoroutine 9282 [select]:
52152026-03-10T19:00:49.111Zgithub.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
52162026-03-10T19:00:49.111Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:439 +0x1c9
52172026-03-10T19:00:49.111Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 9099
52182026-03-10T19:00:49.112Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:412 +0x4f
52192026-03-10T19:00:49.112Z
52202026-03-10T19:00:49.112Zgoroutine 9451 [select]:
52212026-03-10T19:00:49.112Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/liveness.(*NodeLiveness).Start.func1({0xc001654008?, 0xc002ff0600?})
52222026-03-10T19:00:49.112Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/liveness/liveness.go:789 +0x37f
52232026-03-10T19:00:49.112Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
52242026-03-10T19:00:49.112Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
52252026-03-10T19:00:49.112Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
52262026-03-10T19:00:49.112Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
52272026-03-10T19:00:49.112Z
52282026-03-10T19:00:49.112Zgoroutine 9100 [semacquire, 59 minutes]:
52292026-03-10T19:00:49.112Zsync.runtime_Semacquire(0xc00396a198?)
52302026-03-10T19:00:49.112Z /work/go/src/runtime/sema.go:62 +0x25
52312026-03-10T19:00:49.112Zsync.(*WaitGroup).Wait(0x44f7e0?)
52322026-03-10T19:00:49.112Z /work/go/src/sync/waitgroup.go:116 +0x48
52332026-03-10T19:00:49.112Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc0030900c0)
52342026-03-10T19:00:49.112Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
52352026-03-10T19:00:49.112Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc0030900c0?, {0x50e6118?, 0xc000ff8320?}})
52362026-03-10T19:00:49.112Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
52372026-03-10T19:00:49.112Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed(0xc00432cf08, {0x50e6118?, 0xc002de2190?}, {0xc001ead950, 0x1, 0xc002880708?}, {0xc0028806f0?, 0x47afd3?, 0x0?}, 0x0, ...)
52382026-03-10T19:00:49.112Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:139 +0x6c5
52392026-03-10T19:00:49.116Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*dbAdapter).RangeFeed(0xc0028807d0?, {0x50e6118?, 0xc002de2190?}, {0xc001ead950?, 0x0?, 0x0?}, {0xc0025c85a0?, 0x4b7bc01?, 0xc0?}, 0x0, ...)
52402026-03-10T19:00:49.116Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/db_adapter.go:78 +0x3e
52412026-03-10T19:00:49.116Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func1({0x50e6118?, 0xc002de2190?})
52422026-03-10T19:00:49.116Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:285 +0x6c
52432026-03-10T19:00:49.116Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
52442026-03-10T19:00:49.116Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
52452026-03-10T19:00:49.116Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
52462026-03-10T19:00:49.116Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
52472026-03-10T19:00:49.116Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9253
52482026-03-10T19:00:49.116Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
52492026-03-10T19:00:49.116Z
52502026-03-10T19:00:49.116Zgoroutine 10086 [sync.Cond.Wait, 59 minutes]:
52512026-03-10T19:00:49.116Zsync.runtime_notifyListWait(0xc00676e610, 0x35)
52522026-03-10T19:00:49.116Z /work/go/src/runtime/sema.go:569 +0x159
52532026-03-10T19:00:49.117Zsync.(*Cond).Wait(0x0?)
52542026-03-10T19:00:49.117Z /work/go/src/sync/cond.go:70 +0x85
52552026-03-10T19:00:49.117Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0072ac120})
52562026-03-10T19:00:49.117Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
52572026-03-10T19:00:49.117Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
52582026-03-10T19:00:49.117Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
52592026-03-10T19:00:49.117Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
52602026-03-10T19:00:49.117Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
52612026-03-10T19:00:49.117Z
52622026-03-10T19:00:49.117Zgoroutine 10054 [sync.Cond.Wait, 59 minutes]:
52632026-03-10T19:00:49.117Zsync.runtime_notifyListWait(0xc00676e610, 0x15)
52642026-03-10T19:00:49.117Z /work/go/src/runtime/sema.go:569 +0x159
52652026-03-10T19:00:49.117Zsync.(*Cond).Wait(0x48a03c?)
52662026-03-10T19:00:49.117Z /work/go/src/sync/cond.go:70 +0x85
52672026-03-10T19:00:49.117Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c9aa0})
52682026-03-10T19:00:49.117Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
52692026-03-10T19:00:49.117Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
52702026-03-10T19:00:49.117Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
52712026-03-10T19:00:49.117Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
52722026-03-10T19:00:49.117Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
52732026-03-10T19:00:49.117Z
52742026-03-10T19:00:49.117Zgoroutine 10364 [select, 59 minutes]:
52752026-03-10T19:00:49.117Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1({0x50e6118, 0xc001296910})
52762026-03-10T19:00:49.117Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:116 +0x157
52772026-03-10T19:00:49.117Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.Group.GoCtx.func3()
52782026-03-10T19:00:49.117Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
52792026-03-10T19:00:49.117Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
52802026-03-10T19:00:49.117Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
52812026-03-10T19:00:49.117Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10362
52822026-03-10T19:00:49.117Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
52832026-03-10T19:00:49.117Z
52842026-03-10T19:00:49.117Zgoroutine 9574 [select]:
52852026-03-10T19:00:49.117Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents(0xc0007aeb00, {0x50e6118, 0xc003c22a00}, 0xc006037dd0, 0xc0011d60c0)
52862026-03-10T19:00:49.117Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:331 +0xc5
52872026-03-10T19:00:49.117Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func2({0x50e6118?, 0xc003c22a00?})
52882026-03-10T19:00:49.117Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:288 +0x32
52892026-03-10T19:00:49.117Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
52902026-03-10T19:00:49.117Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
52912026-03-10T19:00:49.118Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
52922026-03-10T19:00:49.118Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
52932026-03-10T19:00:49.118Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9636
52942026-03-10T19:00:49.118Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
52952026-03-10T19:00:49.118Z
52962026-03-10T19:00:49.118Zgoroutine 9322 [chan receive, 59 minutes]:
52972026-03-10T19:00:49.118Zgithub.com/cockroachdb/cmux.muxListener.Accept(...)
52982026-03-10T19:00:49.118Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184
52992026-03-10T19:00:49.118Znet/http.(*Server).Serve(0xc00144e000, {0x50d12a0, 0xc0036a74e8})
53002026-03-10T19:00:49.118Z /work/go/src/net/http/server.go:3260 +0x33e
53012026-03-10T19:00:49.118Zgithub.com/cockroachdb/cockroach/pkg/server.(*httpServer).start.func3({0xc0015ea608?, 0xc0016e4ab0?})
53022026-03-10T19:00:49.118Z /work/oxidecomputer/cockroach/pkg/server/server_http.go:264 +0x126
53032026-03-10T19:00:49.118Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
53042026-03-10T19:00:49.118Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
53052026-03-10T19:00:49.118Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
53062026-03-10T19:00:49.118Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
53072026-03-10T19:00:49.118Z
53082026-03-10T19:00:49.118Zgoroutine 9944 [semacquire, 59 minutes]:
53092026-03-10T19:00:49.118Zsync.runtime_Semacquire(0xc0077cdf68?)
53102026-03-10T19:00:49.118Z /work/go/src/runtime/sema.go:62 +0x25
53112026-03-10T19:00:49.118Zsync.(*WaitGroup).Wait(0x44f7e0?)
53122026-03-10T19:00:49.118Z /work/go/src/sync/waitgroup.go:116 +0x48
53132026-03-10T19:00:49.118Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc003348500)
53142026-03-10T19:00:49.118Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
53152026-03-10T19:00:49.118Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc003348500?, {0x50e6118?, 0xc0053af5e0?}})
53162026-03-10T19:00:49.118Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
53172026-03-10T19:00:49.119Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait({0x50e60e0?, 0xc00674c600?}, {0xc0063fbd78, 0x2, 0xc006723c70?})
53182026-03-10T19:00:49.119Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:193 +0xff
53192026-03-10T19:00:49.119Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0xc0015f49a0, {0x50e60e0, 0xc00674c600}, 0xc00674c450)
53202026-03-10T19:00:49.119Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:291 +0x59c
53212026-03-10T19:00:49.119Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).Start.func2({0x50e60e0, 0xc00674c570})
53222026-03-10T19:00:49.119Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:215 +0x2f3
53232026-03-10T19:00:49.119Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
53242026-03-10T19:00:49.119Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
53252026-03-10T19:00:49.119Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9943
53262026-03-10T19:00:49.119Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
53272026-03-10T19:00:49.119Z
53282026-03-10T19:00:49.119Zgoroutine 9188 [select]:
53292026-03-10T19:00:49.119Zgithub.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).startTickerLocked.func1()
53302026-03-10T19:00:49.119Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:365 +0xd2
53312026-03-10T19:00:49.119Zcreated by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).startTickerLocked in goroutine 9099
53322026-03-10T19:00:49.119Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:360 +0x6e
53332026-03-10T19:00:49.119Z
53342026-03-10T19:00:49.119Zgoroutine 10083 [sync.Cond.Wait, 59 minutes]:
53352026-03-10T19:00:49.119Zsync.runtime_notifyListWait(0xc00676e610, 0x32)
53362026-03-10T19:00:49.119Z /work/go/src/runtime/sema.go:569 +0x159
53372026-03-10T19:00:49.119Zsync.(*Cond).Wait(0x0?)
53382026-03-10T19:00:49.119Z /work/go/src/sync/cond.go:70 +0x85
53392026-03-10T19:00:49.124Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0072ac090})
53402026-03-10T19:00:49.124Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
53412026-03-10T19:00:49.124Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
53422026-03-10T19:00:49.124Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
53432026-03-10T19:00:49.124Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
53442026-03-10T19:00:49.124Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
53452026-03-10T19:00:49.124Z
53462026-03-10T19:00:49.124Zgoroutine 9285 [chan receive, 59 minutes]:
53472026-03-10T19:00:49.124Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50e60e0?, 0xc005cee750?})
53482026-03-10T19:00:49.124Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
53492026-03-10T19:00:49.124Zruntime/pprof.Do({0x50e6038?, 0x778e7c0?}, {{0xc00030e2c0?, 0x0?, 0x0?}}, 0xc003bb67a8)
53502026-03-10T19:00:49.124Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
53512026-03-10T19:00:49.124Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
53522026-03-10T19:00:49.124Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
53532026-03-10T19:00:49.124Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9099
53542026-03-10T19:00:49.124Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
53552026-03-10T19:00:49.124Z
53562026-03-10T19:00:49.124Zgoroutine 10181 [select]:
53572026-03-10T19:00:49.124Zgoogle.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc006c40410, 0x1)
53582026-03-10T19:00:49.124Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:407 +0x113
53592026-03-10T19:00:49.124Zgoogle.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc001bd03c0)
53602026-03-10T19:00:49.124Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:534 +0x7b
53612026-03-10T19:00:49.125Zgoogle.golang.org/grpc/internal/transport.newHTTP2Client.func3()
53622026-03-10T19:00:49.125Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:415 +0x67
53632026-03-10T19:00:49.125Zcreated by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 10139
53642026-03-10T19:00:49.125Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:413 +0x1dcb
53652026-03-10T19:00:49.125Z
53662026-03-10T19:00:49.125Zgoroutine 9237 [select, 59 minutes]:
53672026-03-10T19:00:49.125Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc0067f7208?, 0x0?})
53682026-03-10T19:00:49.125Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
53692026-03-10T19:00:49.125Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
53702026-03-10T19:00:49.126Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
53712026-03-10T19:00:49.126Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
53722026-03-10T19:00:49.126Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
53732026-03-10T19:00:49.126Z
53742026-03-10T19:00:49.126Zgoroutine 9817 [select]:
53752026-03-10T19:00:49.126Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run(0xc0060c2600, {0x50e60e0?, 0xc006295da0?}, 0xb991a6?, 0xc001ea3e30, 0xc006848400)
53762026-03-10T19:00:49.126Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:269 +0x4dc
53772026-03-10T19:00:49.126Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).Start.func1({0x50e60e0?, 0xc006295da0?})
53782026-03-10T19:00:49.126Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:222 +0x36
53792026-03-10T19:00:49.126Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
53802026-03-10T19:00:49.126Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
53812026-03-10T19:00:49.126Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9816
53822026-03-10T19:00:49.126Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
53832026-03-10T19:00:49.126Z
53842026-03-10T19:00:49.126Zgoroutine 9447 [select, 2 minutes]:
53852026-03-10T19:00:49.126Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/protectedts/ptcache.(*Cache).periodicallyRefreshProtectedtsCache(0xc006832d10, {0x50e60e0, 0xc006810270})
53862026-03-10T19:00:49.126Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/protectedts/ptcache/cache.go:175 +0x216
53872026-03-10T19:00:49.126Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
53882026-03-10T19:00:49.126Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
53892026-03-10T19:00:49.126Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
53902026-03-10T19:00:49.126Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
53912026-03-10T19:00:49.126Z
53922026-03-10T19:00:49.126Zgoroutine 9228 [sync.Cond.Wait, 59 minutes]:
53932026-03-10T19:00:49.126Zsync.runtime_notifyListWait(0xc00195afe8, 0x0)
53942026-03-10T19:00:49.126Z /work/go/src/runtime/sema.go:569 +0x159
53952026-03-10T19:00:49.126Zsync.(*Cond).Wait(0x8?)
53962026-03-10T19:00:49.126Z /work/go/src/sync/cond.go:70 +0x85
53972026-03-10T19:00:49.126Zgithub.com/cockroachdb/pebble/record.(*flusherCond).Wait(...)
53982026-03-10T19:00:49.126Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/record/log_writer.go:193
53992026-03-10T19:00:49.126Zgithub.com/cockroachdb/pebble/record.(*LogWriter).flushLoop(0xc00195af08, {0x3afca60?, 0x778e7c0?})
54002026-03-10T19:00:49.126Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/record/log_writer.go:396 +0x475
54012026-03-10T19:00:49.126Zruntime/pprof.Do({0x50e6038?, 0x778e7c0?}, {{0xc00030e1c0?, 0x1?, 0x1?}}, 0xc00287ffc0)
54022026-03-10T19:00:49.126Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
54032026-03-10T19:00:49.126Zgithub.com/cockroachdb/pebble/record.NewLogWriter.func2()
54042026-03-10T19:00:49.126Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/record/log_writer.go:314 +0x56
54052026-03-10T19:00:49.127Zcreated by github.com/cockroachdb/pebble/record.NewLogWriter in goroutine 9099
54062026-03-10T19:00:49.127Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/record/log_writer.go:313 +0x2c9
54072026-03-10T19:00:49.127Z
54082026-03-10T19:00:49.127Zgoroutine 9450 [select]:
54092026-03-10T19:00:49.127Zgithub.com/cockroachdb/cockroach/pkg/util/admission.(*StoreGrantCoordinators).SetPebbleMetricsProvider.func1()
54102026-03-10T19:00:49.127Z /work/oxidecomputer/cockroach/pkg/util/admission/granter.go:1261 +0xd5
54112026-03-10T19:00:49.127Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.(*StoreGrantCoordinators).SetPebbleMetricsProvider in goroutine 9099
54122026-03-10T19:00:49.127Z /work/oxidecomputer/cockroach/pkg/util/admission/granter.go:1256 +0x28d
54132026-03-10T19:00:49.127Z
54142026-03-10T19:00:49.127Zgoroutine 9448 [select]:
54152026-03-10T19:00:49.127Zgithub.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue.func2()
54162026-03-10T19:00:49.127Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:366 +0x85
54172026-03-10T19:00:49.127Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue in goroutine 9099
54182026-03-10T19:00:49.127Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:363 +0x3ed
54192026-03-10T19:00:49.127Z
54202026-03-10T19:00:49.127Zgoroutine 9249 [select, 59 minutes]:
54212026-03-10T19:00:49.127Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initCancelingWorkers.func1({0x50e60e0, 0xc001ead650})
54222026-03-10T19:00:49.127Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:139 +0xc5
54232026-03-10T19:00:49.128Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
54242026-03-10T19:00:49.128Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
54252026-03-10T19:00:49.128Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
54262026-03-10T19:00:49.128Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
54272026-03-10T19:00:49.128Z
54282026-03-10T19:00:49.128Zgoroutine 10373 [IO wait]:
54292026-03-10T19:00:49.128Zinternal/poll.runtime_pollWait(0x7f784b7f1f00, 0x72)
54302026-03-10T19:00:49.128Z /work/go/src/runtime/netpoll.go:345 +0x85
54312026-03-10T19:00:49.128Zinternal/poll.(*pollDesc).wait(0xc002d66200?, 0xc001212600?, 0x0)
54322026-03-10T19:00:49.128Z /work/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
54332026-03-10T19:00:49.128Zinternal/poll.(*pollDesc).waitRead(...)
54342026-03-10T19:00:49.128Z /work/go/src/internal/poll/fd_poll_runtime.go:89
54352026-03-10T19:00:49.128Zinternal/poll.(*FD).Read(0xc002d66200, {0xc001212600, 0x600, 0x600})
54362026-03-10T19:00:49.128Z /work/go/src/internal/poll/fd_unix.go:164 +0x27a
54372026-03-10T19:00:49.128Znet.(*netFD).Read(0xc002d66200, {0xc001212600?, 0xc000941868?, 0x41f225?})
54382026-03-10T19:00:49.128Z /work/go/src/net/fd_posix.go:55 +0x25
54392026-03-10T19:00:49.132Znet.(*conn).Read(0xc0020f2a50, {0xc001212600?, 0x7f78353f6b68?, 0xc02dbec798?})
54402026-03-10T19:00:49.132Z /work/go/src/net/net.go:185 +0x45
54412026-03-10T19:00:49.132Zgithub.com/cockroachdb/cmux.(*MuxConn).Read(0xc00342f500, {0xc001212600, 0x600, 0x600})
54422026-03-10T19:00:49.132Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x14f
54432026-03-10T19:00:49.132Zcrypto/tls.(*atLeastReader).Read(0xc02dbec798, {0xc001212600?, 0x0?, 0xc02dbec798?})
54442026-03-10T19:00:49.132Z /work/go/src/crypto/tls/conn.go:806 +0x3b
54452026-03-10T19:00:49.132Zbytes.(*Buffer).ReadFrom(0xc0033562b0, {0x50b6dc0, 0xc02dbec798})
54462026-03-10T19:00:49.132Z /work/go/src/bytes/buffer.go:211 +0x98
54472026-03-10T19:00:49.132Zcrypto/tls.(*Conn).readFromUntil(0xc003356008, {0x7f784ca77e58, 0xc00342f500}, 0xc000941990?)
54482026-03-10T19:00:49.133Z /work/go/src/crypto/tls/conn.go:828 +0xde
54492026-03-10T19:00:49.133Zcrypto/tls.(*Conn).readRecordOrCCS(0xc003356008, 0x0)
54502026-03-10T19:00:49.133Z /work/go/src/crypto/tls/conn.go:626 +0x3cf
54512026-03-10T19:00:49.133Zcrypto/tls.(*Conn).readRecord(...)
54522026-03-10T19:00:49.133Z /work/go/src/crypto/tls/conn.go:588
54532026-03-10T19:00:49.133Zcrypto/tls.(*Conn).Read(0xc003356008, {0xc0051de000, 0x8000, 0xc000941ce8?})
54542026-03-10T19:00:49.133Z /work/go/src/crypto/tls/conn.go:1370 +0x156
54552026-03-10T19:00:49.133Zbufio.(*Reader).Read(0xc0021bf740, {0xc004d0c660, 0x9, 0xc00056e008?})
54562026-03-10T19:00:49.133Z /work/go/src/bufio/bufio.go:241 +0x197
54572026-03-10T19:00:49.133Zio.ReadAtLeast({0x50b1100, 0xc0021bf740}, {0xc004d0c660, 0x9, 0x9}, 0x9)
54582026-03-10T19:00:49.133Z /work/go/src/io/io.go:335 +0x90
54592026-03-10T19:00:49.134Zio.ReadFull(...)
54602026-03-10T19:00:49.134Z /work/go/src/io/io.go:354
54612026-03-10T19:00:49.134Zgolang.org/x/net/http2.readFrameHeader({0xc004d0c660, 0x9, 0xc02d373bc0?}, {0x50b1100?, 0xc0021bf740?})
54622026-03-10T19:00:49.134Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x65
54632026-03-10T19:00:49.134Zgolang.org/x/net/http2.(*Framer).ReadFrame(0xc004d0c620)
54642026-03-10T19:00:49.134Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:501 +0x85
54652026-03-10T19:00:49.134Zgoogle.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc004937a00, 0xc0016e4690, 0x431bf68)
54662026-03-10T19:00:49.134Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:611 +0x14e
54672026-03-10T19:00:49.134Zgoogle.golang.org/grpc.(*Server).serveStreams(0xc005fee1c0, {0x5126b60, 0xc004937a00})
54682026-03-10T19:00:49.134Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:906 +0x114
54692026-03-10T19:00:49.134Zgoogle.golang.org/grpc.(*Server).handleRawConn.func1()
54702026-03-10T19:00:49.134Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:848 +0x45
54712026-03-10T19:00:49.134Zcreated by google.golang.org/grpc.(*Server).handleRawConn in goroutine 10304
54722026-03-10T19:00:49.134Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:847 +0x15b
54732026-03-10T19:00:49.135Z
54742026-03-10T19:00:49.136Zgoroutine 9221 [chan receive, 59 minutes]:
54752026-03-10T19:00:49.136Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50e60e0?, 0xc00634c390?})
54762026-03-10T19:00:49.136Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
54772026-03-10T19:00:49.136Zruntime/pprof.Do({0x50e6038?, 0x778e7c0?}, {{0xc00030e2c0?, 0x1?, 0x1?}}, 0xc002884fa8)
54782026-03-10T19:00:49.136Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
54792026-03-10T19:00:49.136Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc0022b9a20?)
54802026-03-10T19:00:49.136Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
54812026-03-10T19:00:49.136Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9099
54822026-03-10T19:00:49.136Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
54832026-03-10T19:00:49.136Z
54842026-03-10T19:00:49.136Zgoroutine 10052 [sync.Cond.Wait, 59 minutes]:
54852026-03-10T19:00:49.136Zsync.runtime_notifyListWait(0xc00676e610, 0x13)
54862026-03-10T19:00:49.136Z /work/go/src/runtime/sema.go:569 +0x159
54872026-03-10T19:00:49.136Zsync.(*Cond).Wait(0xc0000d8c40?)
54882026-03-10T19:00:49.136Z /work/go/src/sync/cond.go:70 +0x85
54892026-03-10T19:00:49.136Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c9a40})
54902026-03-10T19:00:49.136Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
54912026-03-10T19:00:49.136Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
54922026-03-10T19:00:49.136Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
54932026-03-10T19:00:49.136Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
54942026-03-10T19:00:49.136Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
54952026-03-10T19:00:49.136Z
54962026-03-10T19:00:49.136Zgoroutine 9105 [select, 59 minutes]:
54972026-03-10T19:00:49.137Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc0034f5808?, 0x0?})
54982026-03-10T19:00:49.137Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
54992026-03-10T19:00:49.137Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
55002026-03-10T19:00:49.137Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
55012026-03-10T19:00:49.137Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
55022026-03-10T19:00:49.137Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
55032026-03-10T19:00:49.137Z
55042026-03-10T19:00:49.137Zgoroutine 10053 [sync.Cond.Wait, 59 minutes]:
55052026-03-10T19:00:49.137Zsync.runtime_notifyListWait(0xc00676e610, 0x14)
55062026-03-10T19:00:49.137Z /work/go/src/runtime/sema.go:569 +0x159
55072026-03-10T19:00:49.137Zsync.(*Cond).Wait(0xc003f6ecb0?)
55082026-03-10T19:00:49.137Z /work/go/src/sync/cond.go:70 +0x85
55092026-03-10T19:00:49.137Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c9a70})
55102026-03-10T19:00:49.137Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
55112026-03-10T19:00:49.137Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
55122026-03-10T19:00:49.137Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
55132026-03-10T19:00:49.137Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
55142026-03-10T19:00:49.137Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
55152026-03-10T19:00:49.137Z
55162026-03-10T19:00:49.137Zgoroutine 10055 [sync.Cond.Wait, 59 minutes]:
55172026-03-10T19:00:49.138Zsync.runtime_notifyListWait(0xc00676e610, 0x16)
55182026-03-10T19:00:49.138Z /work/go/src/runtime/sema.go:569 +0x159
55192026-03-10T19:00:49.138Zsync.(*Cond).Wait(0x3f1b920?)
55202026-03-10T19:00:49.138Z /work/go/src/sync/cond.go:70 +0x85
55212026-03-10T19:00:49.139Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c9ad0})
55222026-03-10T19:00:49.139Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
55232026-03-10T19:00:49.139Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
55242026-03-10T19:00:49.139Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
55252026-03-10T19:00:49.139Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
55262026-03-10T19:00:49.139Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
55272026-03-10T19:00:49.139Z
55282026-03-10T19:00:49.139Zgoroutine 10487 [select, 59 minutes]:
55292026-03-10T19:00:49.139Zgithub.com/cockroachdb/cockroach/pkg/util/tracing.newTracingClientStream.func2()
55302026-03-10T19:00:49.139Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:386 +0x67
55312026-03-10T19:00:49.139Zcreated by github.com/cockroachdb/cockroach/pkg/util/tracing.newTracingClientStream in goroutine 10485
55322026-03-10T19:00:49.139Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:385 +0x125
55332026-03-10T19:00:49.139Z
55342026-03-10T19:00:49.139Zgoroutine 9444 [select, 59 minutes]:
55352026-03-10T19:00:49.139Zgithub.com/cockroachdb/cockroach/pkg/server.periodicallyPersistHLCUpperBound(0xc004860910, 0xc000a204d0, 0xc006a211d0, 0x4318fb0, 0xc005cf6660, 0x0)
55362026-03-10T19:00:49.139Z /work/oxidecomputer/cockroach/pkg/server/clock_monotonicity.go:193 +0x14c
55372026-03-10T19:00:49.139Zgithub.com/cockroachdb/cockroach/pkg/server.(*Server).startPersistingHLCUpperBound.func3({0xc002a56908?, 0xc003b0a4b0?})
55382026-03-10T19:00:49.139Z /work/oxidecomputer/cockroach/pkg/server/clock_monotonicity.go:268 +0x45
55392026-03-10T19:00:49.144Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
55402026-03-10T19:00:49.144Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
55412026-03-10T19:00:49.144Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
55422026-03-10T19:00:49.144Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
55432026-03-10T19:00:49.144Z
55442026-03-10T19:00:49.144Zgoroutine 10203 [select]:
55452026-03-10T19:00:49.144Zgithub.com/cockroachdb/cockroach/pkg/sql/contention/txnidcache.(*Cache).Start.func1({0xc001953508?, 0xc004c46f00?})
55462026-03-10T19:00:49.144Z /work/oxidecomputer/cockroach/pkg/sql/contention/txnidcache/txn_id_cache.go:144 +0x9b
55472026-03-10T19:00:49.144Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
55482026-03-10T19:00:49.144Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
55492026-03-10T19:00:49.145Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
55502026-03-10T19:00:49.145Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
55512026-03-10T19:00:49.145Z
55522026-03-10T19:00:49.145Zgoroutine 10112 [select, 59 minutes]:
55532026-03-10T19:00:49.145Zgithub.com/cockroachdb/cockroach/pkg/server.periodicallyPersistHLCUpperBound(0xc00472fa90, 0xc00107e7e0, 0xc0072393d0, 0x4318fb0, 0xc003241980, 0x0)
55542026-03-10T19:00:49.145Z /work/oxidecomputer/cockroach/pkg/server/clock_monotonicity.go:193 +0x14c
55552026-03-10T19:00:49.145Zgithub.com/cockroachdb/cockroach/pkg/server.(*Server).startPersistingHLCUpperBound.func3({0xc007378008?, 0x0?})
55562026-03-10T19:00:49.145Z /work/oxidecomputer/cockroach/pkg/server/clock_monotonicity.go:268 +0x45
55572026-03-10T19:00:49.145Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
55582026-03-10T19:00:49.145Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
55592026-03-10T19:00:49.145Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
55602026-03-10T19:00:49.145Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
55612026-03-10T19:00:49.145Z
55622026-03-10T19:00:49.145Zgoroutine 9245 [select, 59 minutes]:
55632026-03-10T19:00:49.146Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc0015e8308?, 0x0?})
55642026-03-10T19:00:49.146Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
55652026-03-10T19:00:49.147Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
55662026-03-10T19:00:49.147Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
55672026-03-10T19:00:49.147Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
55682026-03-10T19:00:49.147Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
55692026-03-10T19:00:49.147Z
55702026-03-10T19:00:49.147Zgoroutine 10051 [sync.Cond.Wait, 59 minutes]:
55712026-03-10T19:00:49.147Zsync.runtime_notifyListWait(0xc00676e610, 0x12)
55722026-03-10T19:00:49.147Z /work/go/src/runtime/sema.go:569 +0x159
55732026-03-10T19:00:49.147Zsync.(*Cond).Wait(0x0?)
55742026-03-10T19:00:49.147Z /work/go/src/sync/cond.go:70 +0x85
55752026-03-10T19:00:49.147Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c9a10})
55762026-03-10T19:00:49.147Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
55772026-03-10T19:00:49.147Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
55782026-03-10T19:00:49.147Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
55792026-03-10T19:00:49.147Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
55802026-03-10T19:00:49.147Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
55812026-03-10T19:00:49.147Z
55822026-03-10T19:00:49.147Zgoroutine 10110 [select, 10 minutes]:
55832026-03-10T19:00:49.147Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.KVAdmissionControllerImpl.SetTenantWeightProvider.func1()
55842026-03-10T19:00:49.147Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:3809 +0xd5
55852026-03-10T19:00:49.147Zcreated by github.com/cockroachdb/cockroach/pkg/kv/kvserver.KVAdmissionControllerImpl.SetTenantWeightProvider in goroutine 9099
55862026-03-10T19:00:49.147Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:3802 +0xcc
55872026-03-10T19:00:49.147Z
55882026-03-10T19:00:49.148Zgoroutine 9160 [select, 59 minutes]:
55892026-03-10T19:00:49.148Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2({0x50e60e0, 0xc001eac150})
55902026-03-10T19:00:49.148Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/queue.go:823 +0xff
55912026-03-10T19:00:49.148Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
55922026-03-10T19:00:49.148Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
55932026-03-10T19:00:49.148Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9437
55942026-03-10T19:00:49.148Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
55952026-03-10T19:00:49.148Z
55962026-03-10T19:00:49.148Zgoroutine 9392 [sync.Cond.Wait]:
55972026-03-10T19:00:49.148Zsync.runtime_notifyListWait(0xc0005ac550, 0x3750)
55982026-03-10T19:00:49.148Z /work/go/src/runtime/sema.go:569 +0x159
55992026-03-10T19:00:49.148Zsync.(*Cond).Wait(0x3ae1600?)
56002026-03-10T19:00:49.148Z /work/go/src/sync/cond.go:70 +0x85
56012026-03-10T19:00:49.148Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc0066231a0})
56022026-03-10T19:00:49.148Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
56032026-03-10T19:00:49.148Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
56042026-03-10T19:00:49.148Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
56052026-03-10T19:00:49.148Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
56062026-03-10T19:00:49.149Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
56072026-03-10T19:00:49.149Z
56082026-03-10T19:00:49.149Zgoroutine 10014 [chan receive, 59 minutes]:
56092026-03-10T19:00:49.149Zgithub.com/cockroachdb/cmux.muxListener.Accept(...)
56102026-03-10T19:00:49.149Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184
56112026-03-10T19:00:49.149Zgoogle.golang.org/grpc.(*Server).Serve(0xc003999340, {0x50d12a0, 0xc003438738})
56122026-03-10T19:00:49.149Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:780 +0x469
56132026-03-10T19:00:49.149Zgithub.com/cockroachdb/cockroach/pkg/server.startListenRPCAndSQL.func5.1({0xc006f4a308?, 0xc007a03680?})
56142026-03-10T19:00:49.149Z /work/oxidecomputer/cockroach/pkg/server/start_listen.go:134 +0x25
56152026-03-10T19:00:49.149Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
56162026-03-10T19:00:49.149Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
56172026-03-10T19:00:49.149Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
56182026-03-10T19:00:49.149Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
56192026-03-10T19:00:49.149Z
56202026-03-10T19:00:49.149Zgoroutine 9445 [select]:
56212026-03-10T19:00:49.149Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/reports.(*Reporter).Start.func2({0x50e60e0?, 0xc006810210?})
56222026-03-10T19:00:49.149Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/reports/reporter.go:165 +0x305
56232026-03-10T19:00:49.149Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
56242026-03-10T19:00:49.149Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
56252026-03-10T19:00:49.149Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
56262026-03-10T19:00:49.149Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
56272026-03-10T19:00:49.149Z
56282026-03-10T19:00:49.149Zgoroutine 10153 [select]:
56292026-03-10T19:00:49.149Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents(0xc0009d6b00, {0x50e6118, 0xc00717cc80}, 0xc005c718f0, 0xc006968420)
56302026-03-10T19:00:49.149Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:331 +0xc5
56312026-03-10T19:00:49.149Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func2({0x50e6118?, 0xc00717cc80?})
56322026-03-10T19:00:49.149Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:288 +0x32
56332026-03-10T19:00:49.149Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
56342026-03-10T19:00:49.149Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
56352026-03-10T19:00:49.149Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
56362026-03-10T19:00:49.149Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
56372026-03-10T19:00:49.149Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10206
56382026-03-10T19:00:49.149Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
56392026-03-10T19:00:49.154Z
56402026-03-10T19:00:49.154Zgoroutine 9321 [IO wait, 59 minutes]:
56412026-03-10T19:00:49.154Zinternal/poll.runtime_pollWait(0x7f784abc3510, 0x72)
56422026-03-10T19:00:49.154Z /work/go/src/runtime/netpoll.go:345 +0x85
56432026-03-10T19:00:49.154Zinternal/poll.(*pollDesc).wait(0xa?, 0xc004330de0?, 0x0)
56442026-03-10T19:00:49.154Z /work/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
56452026-03-10T19:00:49.154Zinternal/poll.(*pollDesc).waitRead(...)
56462026-03-10T19:00:49.154Z /work/go/src/internal/poll/fd_poll_runtime.go:89
56472026-03-10T19:00:49.154Zinternal/poll.(*FD).Accept(0xc003008c80)
56482026-03-10T19:00:49.154Z /work/go/src/internal/poll/fd_unix.go:611 +0x2ac
56492026-03-10T19:00:49.155Znet.(*netFD).accept(0xc003008c80)
56502026-03-10T19:00:49.155Z /work/go/src/net/fd_unix.go:172 +0x29
56512026-03-10T19:00:49.155Znet.(*TCPListener).accept(0xc006f5d700)
56522026-03-10T19:00:49.155Z /work/go/src/net/tcpsock_posix.go:159 +0x1e
56532026-03-10T19:00:49.155Znet.(*TCPListener).Accept(0xc006f5d700)
56542026-03-10T19:00:49.155Z /work/go/src/net/tcpsock.go:327 +0x30
56552026-03-10T19:00:49.155Zgithub.com/cockroachdb/cmux.(*cMux).Serve(0xc001203740)
56562026-03-10T19:00:49.155Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:124 +0x97
56572026-03-10T19:00:49.155Zgithub.com/cockroachdb/cockroach/pkg/server.(*httpServer).start.func2({0xc0015ea308?, 0xc002458bd0?})
56582026-03-10T19:00:49.155Z /work/oxidecomputer/cockroach/pkg/server/server_http.go:243 +0x27
56592026-03-10T19:00:49.155Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
56602026-03-10T19:00:49.155Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
56612026-03-10T19:00:49.155Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
56622026-03-10T19:00:49.155Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
56632026-03-10T19:00:49.155Z
56642026-03-10T19:00:49.155Zgoroutine 10275 [select]:
56652026-03-10T19:00:49.155Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.(*Watcher).Run(0xc002e2c2d0, {0x50e6118, 0xc0013a6c30})
56662026-03-10T19:00:49.155Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:319 +0x955
56672026-03-10T19:00:49.155Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.Start.func1({0x50e60e0?, 0xc002cb1470?})
56682026-03-10T19:00:49.155Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:189 +0x2d1
56692026-03-10T19:00:49.155Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
56702026-03-10T19:00:49.155Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
56712026-03-10T19:00:49.155Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
56722026-03-10T19:00:49.155Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
56732026-03-10T19:00:49.155Z
56742026-03-10T19:00:49.155Zgoroutine 9395 [sync.Cond.Wait]:
56752026-03-10T19:00:49.155Zsync.runtime_notifyListWait(0xc0005ac550, 0x3754)
56762026-03-10T19:00:49.155Z /work/go/src/runtime/sema.go:569 +0x159
56772026-03-10T19:00:49.155Zsync.(*Cond).Wait(0x3ae1600?)
56782026-03-10T19:00:49.155Z /work/go/src/sync/cond.go:70 +0x85
56792026-03-10T19:00:49.155Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006623260})
56802026-03-10T19:00:49.155Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
56812026-03-10T19:00:49.155Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
56822026-03-10T19:00:49.155Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
56832026-03-10T19:00:49.155Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
56842026-03-10T19:00:49.155Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
56852026-03-10T19:00:49.155Z
56862026-03-10T19:00:49.155Zgoroutine 3631583 [select]:
56872026-03-10T19:00:49.156Zgithub.com/cockroachdb/cockroach/pkg/sql/stats.(*Refresher).Start.func1.1({0x50e60e0, 0xc00398d7a0})
56882026-03-10T19:00:49.156Z /work/oxidecomputer/cockroach/pkg/sql/stats/automatic_stats.go:410 +0x145
56892026-03-10T19:00:49.156Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
56902026-03-10T19:00:49.156Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
56912026-03-10T19:00:49.156Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10204
56922026-03-10T19:00:49.156Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
56932026-03-10T19:00:49.156Z
56942026-03-10T19:00:49.156Zgoroutine 9294 [select]:
56952026-03-10T19:00:49.156Zgithub.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1()
56962026-03-10T19:00:49.156Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:71 +0xcb
56972026-03-10T19:00:49.156Zcreated by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 9099
56982026-03-10T19:00:49.156Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:66 +0x58
56992026-03-10T19:00:49.156Z
57002026-03-10T19:00:49.156Zgoroutine 10069 [sync.Cond.Wait, 59 minutes]:
57012026-03-10T19:00:49.156Zsync.runtime_notifyListWait(0xc00676e610, 0x24)
57022026-03-10T19:00:49.157Z /work/go/src/runtime/sema.go:569 +0x159
57032026-03-10T19:00:49.157Zsync.(*Cond).Wait(0x0?)
57042026-03-10T19:00:49.157Z /work/go/src/sync/cond.go:70 +0x85
57052026-03-10T19:00:49.157Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c9d70})
57062026-03-10T19:00:49.157Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
57072026-03-10T19:00:49.157Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
57082026-03-10T19:00:49.157Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
57092026-03-10T19:00:49.157Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
57102026-03-10T19:00:49.157Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
57112026-03-10T19:00:49.157Z
57122026-03-10T19:00:49.157Zgoroutine 10335 [select]:
57132026-03-10T19:00:49.157Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents(0xc0015f49a0, {0x50e6118, 0xc0053af5e0}, 0xc00674c450, 0xc001bbf740)
57142026-03-10T19:00:49.157Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:331 +0xc5
57152026-03-10T19:00:49.157Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func2({0x50e6118?, 0xc0053af5e0?})
57162026-03-10T19:00:49.157Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:288 +0x32
57172026-03-10T19:00:49.157Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
57182026-03-10T19:00:49.157Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
57192026-03-10T19:00:49.157Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
57202026-03-10T19:00:49.157Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
57212026-03-10T19:00:49.157Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9944
57222026-03-10T19:00:49.157Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
57232026-03-10T19:00:49.157Z
57242026-03-10T19:00:49.158Zgoroutine 9101 [select]:
57252026-03-10T19:00:49.158Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents(0xc005280420, {0x50e6118, 0xc002de2190}, 0xc001ead830, 0xc0012b4240)
57262026-03-10T19:00:49.158Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:331 +0xc5
57272026-03-10T19:00:49.158Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func2({0x50e6118?, 0xc002de2190?})
57282026-03-10T19:00:49.158Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:288 +0x32
57292026-03-10T19:00:49.158Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
57302026-03-10T19:00:49.158Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
57312026-03-10T19:00:49.158Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
57322026-03-10T19:00:49.158Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
57332026-03-10T19:00:49.158Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9253
57342026-03-10T19:00:49.158Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
57352026-03-10T19:00:49.158Z
57362026-03-10T19:00:49.158Zgoroutine 10276 [semacquire, 59 minutes]:
57372026-03-10T19:00:49.158Zsync.runtime_Semacquire(0xc004150d08?)
57382026-03-10T19:00:49.158Z /work/go/src/runtime/sema.go:62 +0x25
57392026-03-10T19:00:49.163Zsync.(*WaitGroup).Wait(0x44f7e0?)
57402026-03-10T19:00:49.163Z /work/go/src/sync/waitgroup.go:116 +0x48
57412026-03-10T19:00:49.163Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc001494d80)
57422026-03-10T19:00:49.165Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
57432026-03-10T19:00:49.165Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc001494d80?, {0x50e6118?, 0xc001296870?}})
57442026-03-10T19:00:49.166Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
57452026-03-10T19:00:49.166Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait({0x50e60e0?, 0xc0016a41b0?}, {0xc006ba1d78, 0x2, 0xc006ba1c70?})
57462026-03-10T19:00:49.166Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:193 +0xff
57472026-03-10T19:00:49.166Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0xc0035129a0, {0x50e60e0, 0xc0016a41b0}, 0xc002cb1d70)
57482026-03-10T19:00:49.166Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:291 +0x59c
57492026-03-10T19:00:49.166Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).Start.func2({0x50e60e0, 0xc0016a4030})
57502026-03-10T19:00:49.166Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:215 +0x2f3
57512026-03-10T19:00:49.166Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
57522026-03-10T19:00:49.166Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
57532026-03-10T19:00:49.166Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10275
57542026-03-10T19:00:49.166Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
57552026-03-10T19:00:49.166Z
57562026-03-10T19:00:49.166Zgoroutine 9979 [select]:
57572026-03-10T19:00:49.166Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).startWriteNodeStatus.func2({0x50e60e0, 0xc006f824b0})
57582026-03-10T19:00:49.166Z /work/oxidecomputer/cockroach/pkg/server/node.go:852 +0xf3
57592026-03-10T19:00:49.166Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
57602026-03-10T19:00:49.166Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
57612026-03-10T19:00:49.166Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
57622026-03-10T19:00:49.166Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
57632026-03-10T19:00:49.166Z
57642026-03-10T19:00:49.166Zgoroutine 9255 [select]:
57652026-03-10T19:00:49.166Zgithub.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).startTickerLocked.func1()
57662026-03-10T19:00:49.166Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:365 +0xd2
57672026-03-10T19:00:49.166Zcreated by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).startTickerLocked in goroutine 9099
57682026-03-10T19:00:49.166Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:360 +0x6e
57692026-03-10T19:00:49.166Z
57702026-03-10T19:00:49.166Zgoroutine 9196 [chan receive, 59 minutes]:
57712026-03-10T19:00:49.166Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50e60e0?, 0xc005c70030?})
57722026-03-10T19:00:49.166Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
57732026-03-10T19:00:49.166Zruntime/pprof.Do({0x50e6038?, 0x778e7c0?}, {{0xc00030e2c0?, 0xc001783b00?, 0x1029d00?}}, 0xc00004bfa8)
57742026-03-10T19:00:49.166Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
57752026-03-10T19:00:49.166Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
57762026-03-10T19:00:49.166Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
57772026-03-10T19:00:49.166Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9099
57782026-03-10T19:00:49.166Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
57792026-03-10T19:00:49.166Z
57802026-03-10T19:00:49.166Zgoroutine 10198 [select, 29 minutes]:
57812026-03-10T19:00:49.166Zgithub.com/cockroachdb/cockroach/pkg/sql.(*TemporaryObjectCleaner).Start.func1({0x50e60e0, 0xc005c716e0})
57822026-03-10T19:00:49.166Z /work/oxidecomputer/cockroach/pkg/sql/temporary_schema.go:632 +0x22e
57832026-03-10T19:00:49.167Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
57842026-03-10T19:00:49.167Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
57852026-03-10T19:00:49.167Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
57862026-03-10T19:00:49.167Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
57872026-03-10T19:00:49.167Z
57882026-03-10T19:00:49.167Zgoroutine 10210 [semacquire, 59 minutes]:
57892026-03-10T19:00:49.167Zsync.runtime_Semacquire(0xc000013668?)
57902026-03-10T19:00:49.167Z /work/go/src/runtime/sema.go:62 +0x25
57912026-03-10T19:00:49.167Zsync.(*WaitGroup).Wait(0x44f7e0?)
57922026-03-10T19:00:49.167Z /work/go/src/sync/waitgroup.go:116 +0x48
57932026-03-10T19:00:49.167Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc003349580)
57942026-03-10T19:00:49.167Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
57952026-03-10T19:00:49.167Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc003349580?, {0x50e6118?, 0xc002d6a780?}})
57962026-03-10T19:00:49.167Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
57972026-03-10T19:00:49.167Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait({0x50e60e0?, 0xc003288c00?}, {0xc005b81d78, 0x2, 0xc005b81c70?})
57982026-03-10T19:00:49.167Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:193 +0xff
57992026-03-10T19:00:49.167Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0xc001085a20, {0x50e60e0, 0xc003288c00}, 0xc003288a20)
58002026-03-10T19:00:49.167Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:291 +0x59c
58012026-03-10T19:00:49.167Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).Start.func2({0x50e60e0, 0xc003288ba0})
58022026-03-10T19:00:49.167Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:215 +0x2f3
58032026-03-10T19:00:49.167Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
58042026-03-10T19:00:49.167Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
58052026-03-10T19:00:49.167Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
58062026-03-10T19:00:49.167Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
58072026-03-10T19:00:49.167Z
58082026-03-10T19:00:49.167Zgoroutine 9941 [select]:
58092026-03-10T19:00:49.167Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/closedts/sidetransport.(*Sender).Run.func2({0x50e60e0, 0xc00674c150})
58102026-03-10T19:00:49.167Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/closedts/sidetransport/sender.go:251 +0x1ab
58112026-03-10T19:00:49.168Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
58122026-03-10T19:00:49.168Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
58132026-03-10T19:00:49.168Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
58142026-03-10T19:00:49.168Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
58152026-03-10T19:00:49.168Z
58162026-03-10T19:00:49.168Zgoroutine 10095 [sync.Cond.Wait, 59 minutes]:
58172026-03-10T19:00:49.168Zsync.runtime_notifyListWait(0xc00676e610, 0x3d)
58182026-03-10T19:00:49.168Z /work/go/src/runtime/sema.go:569 +0x159
58192026-03-10T19:00:49.168Zsync.(*Cond).Wait(0x0?)
58202026-03-10T19:00:49.168Z /work/go/src/sync/cond.go:70 +0x85
58212026-03-10T19:00:49.168Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0072ac2d0})
58222026-03-10T19:00:49.168Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
58232026-03-10T19:00:49.168Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
58242026-03-10T19:00:49.168Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
58252026-03-10T19:00:49.168Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
58262026-03-10T19:00:49.168Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
58272026-03-10T19:00:49.168Z
58282026-03-10T19:00:49.168Zgoroutine 10130 [select]:
58292026-03-10T19:00:49.168Zgoogle.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc000a449c0)
58302026-03-10T19:00:49.168Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1131 +0x205
58312026-03-10T19:00:49.168Zcreated by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 9627
58322026-03-10T19:00:49.168Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:341 +0x188e
58332026-03-10T19:00:49.168Z
58342026-03-10T19:00:49.169Zgoroutine 9788 [select, 1 minutes]:
58352026-03-10T19:00:49.170Zgithub.com/cockroachdb/cockroach/pkg/kv/kvprober.(*Prober).Start.func1.1({0x50e60e0, 0xc00457b5c0})
58362026-03-10T19:00:49.170Z /work/oxidecomputer/cockroach/pkg/kv/kvprober/kvprober.go:293 +0x314
58372026-03-10T19:00:49.170Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
58382026-03-10T19:00:49.170Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
58392026-03-10T19:00:49.175Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
58402026-03-10T19:00:49.175Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
58412026-03-10T19:00:49.175Z
58422026-03-10T19:00:49.175Zgoroutine 10111 [select]:
58432026-03-10T19:00:49.175Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).startGossiping.func1({0x50e60e0, 0xc0072ac810})
58442026-03-10T19:00:49.175Z /work/oxidecomputer/cockroach/pkg/server/node.go:713 +0x1be
58452026-03-10T19:00:49.175Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
58462026-03-10T19:00:49.175Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
58472026-03-10T19:00:49.175Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
58482026-03-10T19:00:49.175Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
58492026-03-10T19:00:49.175Z
58502026-03-10T19:00:49.175Zgoroutine 10079 [sync.Cond.Wait, 59 minutes]:
58512026-03-10T19:00:49.175Zsync.runtime_notifyListWait(0xc00676e610, 0x2e)
58522026-03-10T19:00:49.175Z /work/go/src/runtime/sema.go:569 +0x159
58532026-03-10T19:00:49.175Zsync.(*Cond).Wait(0x0?)
58542026-03-10T19:00:49.175Z /work/go/src/sync/cond.go:70 +0x85
58552026-03-10T19:00:49.175Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c9f50})
58562026-03-10T19:00:49.175Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
58572026-03-10T19:00:49.175Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
58582026-03-10T19:00:49.175Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
58592026-03-10T19:00:49.175Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
58602026-03-10T19:00:49.175Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
58612026-03-10T19:00:49.175Z
58622026-03-10T19:00:49.175Zgoroutine 9382 [sync.Cond.Wait]:
58632026-03-10T19:00:49.175Zsync.runtime_notifyListWait(0xc0005ac550, 0x376a)
58642026-03-10T19:00:49.175Z /work/go/src/runtime/sema.go:569 +0x159
58652026-03-10T19:00:49.175Zsync.(*Cond).Wait(0x3ae1600?)
58662026-03-10T19:00:49.175Z /work/go/src/sync/cond.go:70 +0x85
58672026-03-10T19:00:49.175Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006622f60})
58682026-03-10T19:00:49.175Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
58692026-03-10T19:00:49.175Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
58702026-03-10T19:00:49.175Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
58712026-03-10T19:00:49.175Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
58722026-03-10T19:00:49.176Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
58732026-03-10T19:00:49.176Z
58742026-03-10T19:00:49.176Zgoroutine 10430 [select, 59 minutes]:
58752026-03-10T19:00:49.176Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1({0x50e6118, 0xc002d6a140})
58762026-03-10T19:00:49.176Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:116 +0x157
58772026-03-10T19:00:49.176Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.Group.GoCtx.func3()
58782026-03-10T19:00:49.176Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
58792026-03-10T19:00:49.176Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
58802026-03-10T19:00:49.176Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
58812026-03-10T19:00:49.176Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10428
58822026-03-10T19:00:49.176Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
58832026-03-10T19:00:49.176Z
58842026-03-10T19:00:49.176Zgoroutine 10466 [select, 59 minutes]:
58852026-03-10T19:00:49.176Zgithub.com/cockroachdb/cockroach/pkg/util/tracing.newTracingClientStream.func2()
58862026-03-10T19:00:49.176Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:386 +0x67
58872026-03-10T19:00:49.176Zcreated by github.com/cockroachdb/cockroach/pkg/util/tracing.newTracingClientStream in goroutine 10432
58882026-03-10T19:00:49.176Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:385 +0x125
58892026-03-10T19:00:49.176Z
58902026-03-10T19:00:49.176Zgoroutine 9630 [select]:
58912026-03-10T19:00:49.176Zgoogle.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc004583e50, 0x1)
58922026-03-10T19:00:49.176Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:407 +0x113
58932026-03-10T19:00:49.176Zgoogle.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc005e389c0)
58942026-03-10T19:00:49.176Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:534 +0x7b
58952026-03-10T19:00:49.176Zgoogle.golang.org/grpc/internal/transport.newHTTP2Client.func3()
58962026-03-10T19:00:49.176Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:415 +0x67
58972026-03-10T19:00:49.176Zcreated by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 10025
58982026-03-10T19:00:49.176Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:413 +0x1dcb
58992026-03-10T19:00:49.176Z
59002026-03-10T19:00:49.176Zgoroutine 9581 [select]:
59012026-03-10T19:00:49.176Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).outputLoop(0xc001ee8000, {0x50e6118, 0xc003c22dc0})
59022026-03-10T19:00:49.176Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:310 +0x247
59032026-03-10T19:00:49.176Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).runOutputLoop(0xc001ee8000, {0x50e60e0, 0xc0036c1410}, 0xc0016a5dd0?)
59042026-03-10T19:00:49.176Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:335 +0xba
59052026-03-10T19:00:49.176Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run.func1({0x50e60e0?, 0xc0036c1410?})
59062026-03-10T19:00:49.176Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:297 +0x46
59072026-03-10T19:00:49.176Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
59082026-03-10T19:00:49.176Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
59092026-03-10T19:00:49.176Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9579
59102026-03-10T19:00:49.176Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
59112026-03-10T19:00:49.177Z
59122026-03-10T19:00:49.177Zgoroutine 9312 [select, 59 minutes]:
59132026-03-10T19:00:49.177Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initCancelingWorkers.func1({0x50e60e0, 0xc00634c4e0})
59142026-03-10T19:00:49.177Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:139 +0xc5
59152026-03-10T19:00:49.177Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
59162026-03-10T19:00:49.177Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
59172026-03-10T19:00:49.177Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
59182026-03-10T19:00:49.177Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
59192026-03-10T19:00:49.177Z
59202026-03-10T19:00:49.177Zgoroutine 9250 [select, 59 minutes]:
59212026-03-10T19:00:49.177Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initCancelingWorkers.func1({0x50e60e0, 0xc001ead6b0})
59222026-03-10T19:00:49.177Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:139 +0xc5
59232026-03-10T19:00:49.177Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
59242026-03-10T19:00:49.179Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
59252026-03-10T19:00:49.179Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
59262026-03-10T19:00:49.179Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
59272026-03-10T19:00:49.179Z
59282026-03-10T19:00:49.179Zgoroutine 9950 [chan receive, 59 minutes]:
59292026-03-10T19:00:49.179Zgithub.com/cockroachdb/cockroach/pkg/gossip.(*server).start.func3({0xc006768c08?, 0xc003082fb8?})
59302026-03-10T19:00:49.179Z /work/oxidecomputer/cockroach/pkg/gossip/server.go:380 +0x48
59312026-03-10T19:00:49.179Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
59322026-03-10T19:00:49.179Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
59332026-03-10T19:00:49.179Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
59342026-03-10T19:00:49.179Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
59352026-03-10T19:00:49.179Z
59362026-03-10T19:00:49.179Zgoroutine 9251 [select, 59 minutes]:
59372026-03-10T19:00:49.179Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initCancelingWorkers.func1({0x50e60e0, 0xc001ead6e0})
59382026-03-10T19:00:49.179Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:139 +0xc5
59392026-03-10T19:00:49.183Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
59402026-03-10T19:00:49.183Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
59412026-03-10T19:00:49.183Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
59422026-03-10T19:00:49.183Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
59432026-03-10T19:00:49.183Z
59442026-03-10T19:00:49.183Zgoroutine 9636 [semacquire, 59 minutes]:
59452026-03-10T19:00:49.183Zsync.runtime_Semacquire(0xc006ed2bb8?)
59462026-03-10T19:00:49.183Z /work/go/src/runtime/sema.go:62 +0x25
59472026-03-10T19:00:49.183Zsync.(*WaitGroup).Wait(0x44f7e0?)
59482026-03-10T19:00:49.183Z /work/go/src/sync/waitgroup.go:116 +0x48
59492026-03-10T19:00:49.183Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc000cde140)
59502026-03-10T19:00:49.183Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
59512026-03-10T19:00:49.183Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc000cde140?, {0x50e6118?, 0xc003c22a00?}})
59522026-03-10T19:00:49.183Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
59532026-03-10T19:00:49.184Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait({0x50e60e0?, 0xc0036c04b0?}, {0xc004c65d78, 0x2, 0x4000000000000000?})
59542026-03-10T19:00:49.184Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:193 +0xff
59552026-03-10T19:00:49.184Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0xc0007aeb00, {0x50e60e0, 0xc0036c04b0}, 0xc006037dd0)
59562026-03-10T19:00:49.184Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:291 +0x59c
59572026-03-10T19:00:49.184Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).Start.func2({0x50e60e0, 0xc006037ef0})
59582026-03-10T19:00:49.184Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:215 +0x2f3
59592026-03-10T19:00:49.184Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
59602026-03-10T19:00:49.184Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
59612026-03-10T19:00:49.184Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
59622026-03-10T19:00:49.184Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
59632026-03-10T19:00:49.184Z
59642026-03-10T19:00:49.184Zgoroutine 9484 [select]:
59652026-03-10T19:00:49.184Zgithub.com/cockroachdb/cockroach/pkg/sql/sqlliveness/slinstance.(*Instance).heartbeatLoop(0xc003d04c60, {0x50e6118, 0xc003154230})
59662026-03-10T19:00:49.184Z /work/oxidecomputer/cockroach/pkg/sql/sqlliveness/slinstance/slinstance.go:244 +0x187
59672026-03-10T19:00:49.184Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
59682026-03-10T19:00:49.185Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
59692026-03-10T19:00:49.185Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
59702026-03-10T19:00:49.186Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
59712026-03-10T19:00:49.186Z
59722026-03-10T19:00:49.186Zgoroutine 9267 [sync.Cond.Wait]:
59732026-03-10T19:00:49.186Zsync.runtime_notifyListWait(0xc00195f9e8, 0x646)
59742026-03-10T19:00:49.186Z /work/go/src/runtime/sema.go:569 +0x159
59752026-03-10T19:00:49.186Zsync.(*Cond).Wait(0xc00195f908?)
59762026-03-10T19:00:49.186Z /work/go/src/sync/cond.go:70 +0x85
59772026-03-10T19:00:49.186Zgithub.com/cockroachdb/pebble/record.(*flusherCond).Wait(...)
59782026-03-10T19:00:49.186Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/record/log_writer.go:193
59792026-03-10T19:00:49.186Zgithub.com/cockroachdb/pebble/record.(*LogWriter).flushLoop(0xc00195f908, {0x3afca60?, 0x778e7c0?})
59802026-03-10T19:00:49.186Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/record/log_writer.go:396 +0x475
59812026-03-10T19:00:49.186Zruntime/pprof.Do({0x50e6038?, 0x778e7c0?}, {{0xc00030e1c0?, 0xc0023a8fe0?, 0xc000000000?}}, 0xc0032d57c0)
59822026-03-10T19:00:49.186Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
59832026-03-10T19:00:49.186Zgithub.com/cockroachdb/pebble/record.NewLogWriter.func2()
59842026-03-10T19:00:49.186Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/record/log_writer.go:314 +0x56
59852026-03-10T19:00:49.186Zcreated by github.com/cockroachdb/pebble/record.NewLogWriter in goroutine 9099
59862026-03-10T19:00:49.186Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/record/log_writer.go:313 +0x2c9
59872026-03-10T19:00:49.186Z
59882026-03-10T19:00:49.186Zgoroutine 9302 [select, 59 minutes]:
59892026-03-10T19:00:49.186Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc002c06c08?, 0x0?})
59902026-03-10T19:00:49.186Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
59912026-03-10T19:00:49.187Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
59922026-03-10T19:00:49.187Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
59932026-03-10T19:00:49.187Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
59942026-03-10T19:00:49.187Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
59952026-03-10T19:00:49.187Z
59962026-03-10T19:00:49.187Zgoroutine 9254 [select, 59 minutes]:
59972026-03-10T19:00:49.187Zgithub.com/cockroachdb/cockroach/pkg/server.startPurgeOldSessions.func1({0xc000499b08?, 0xffffffffffffffff?})
59982026-03-10T19:00:49.187Z /work/oxidecomputer/cockroach/pkg/server/purge_auth_session.go:69 +0x205
59992026-03-10T19:00:49.187Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
60002026-03-10T19:00:49.187Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
60012026-03-10T19:00:49.187Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
60022026-03-10T19:00:49.187Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
60032026-03-10T19:00:49.188Z
60042026-03-10T19:00:49.188Zgoroutine 10209 [runnable]:
60052026-03-10T19:00:49.188Zgithub.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).execInternal(0xc0032c7d50, {0x50e6118, 0xc000fb03c0}, {0x40bf96b, 0xf}, 0xc0117bcdc0, 0xc0117cac60, {{{0x40a2464, 0x4}}, {0x0, ...}, ...}, ...)
60062026-03-10T19:00:49.188Z /work/oxidecomputer/cockroach/pkg/sql/internal.go:790 +0xb57
60072026-03-10T19:00:49.188Zgithub.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).ExecEx(0xc0032c7d50, {0x50e6118, 0xc000fb03c0}, {0x40bf96b, 0xf}, 0xc0117cac60, {{{0x40a2464, 0x4}}, {0x0, 0x0}, ...}, ...)
60082026-03-10T19:00:49.188Z /work/oxidecomputer/cockroach/pkg/sql/internal.go:544 +0x188
60092026-03-10T19:00:49.188Zgithub.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func2.1({0x50e6118, 0xc000fb03c0}, 0xc0117cac60)
60102026-03-10T19:00:49.188Z /work/oxidecomputer/cockroach/pkg/jobs/registry.go:729 +0x309
60112026-03-10T19:00:49.188Zgithub.com/cockroachdb/cockroach/pkg/kv.runTxn.func1({0x50e6118?, 0xc000fb03c0?}, 0xc0054f5940?)
60122026-03-10T19:00:49.188Z /work/oxidecomputer/cockroach/pkg/kv/db.go:949 +0x22
60132026-03-10T19:00:49.188Zgithub.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec(0xc0117cac60, {0x50e6118, 0xc000fb03c0}, 0xc0054f5bc8)
60142026-03-10T19:00:49.188Z /work/oxidecomputer/cockroach/pkg/kv/txn.go:993 +0x9b
60152026-03-10T19:00:49.188Zgithub.com/cockroachdb/cockroach/pkg/kv.runTxn({0x50e6118, 0xc000fb03c0}, 0xc0117cac60, 0x2?)
60162026-03-10T19:00:49.188Z /work/oxidecomputer/cockroach/pkg/kv/db.go:948 +0x54
60172026-03-10T19:00:49.188Zgithub.com/cockroachdb/cockroach/pkg/kv.(*DB).TxnWithAdmissionControl(0xc0054f5c10?, {0x50e6118, 0xc000fb03c0}, 0x54f5cc8?, 0xc0?, 0xc0054f5c88)
60182026-03-10T19:00:49.188Z /work/oxidecomputer/cockroach/pkg/kv/db.go:911 +0x7f
60192026-03-10T19:00:49.188Zgithub.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn(...)
60202026-03-10T19:00:49.188Z /work/oxidecomputer/cockroach/pkg/kv/db.go:890
60212026-03-10T19:00:49.189Zgithub.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func2({0x50e6118, 0xc000fb03c0}, {0x50ef178?, 0xc000f5f8c0?})
60222026-03-10T19:00:49.189Z /work/oxidecomputer/cockroach/pkg/jobs/registry.go:722 +0x7d
60232026-03-10T19:00:49.190Zgithub.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func4({0x50e6118, 0xc000fb03c0}, {0x50ef178, 0xc000f5f8c0})
60242026-03-10T19:00:49.190Z /work/oxidecomputer/cockroach/pkg/jobs/registry.go:750 +0x3e
60252026-03-10T19:00:49.190Zgithub.com/cockroachdb/cockroach/pkg/jobs.(*Registry).withSession(0xc002d8a508, {0x50e6118, 0xc000fb03c0}, 0xc006765080)
60262026-03-10T19:00:49.190Z /work/oxidecomputer/cockroach/pkg/jobs/registry.go:700 +0x25b
60272026-03-10T19:00:49.190Zgithub.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.(*Registry).Start.func1.func11({0x50e6118?, 0xc000fb03c0?})
60282026-03-10T19:00:49.190Z /work/oxidecomputer/cockroach/pkg/jobs/registry.go:716 +0x2e
60292026-03-10T19:00:49.190Zgithub.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func8({0x50e60e0?, 0xc005c71c50?})
60302026-03-10T19:00:49.190Z /work/oxidecomputer/cockroach/pkg/jobs/registry.go:820 +0x1cf
60312026-03-10T19:00:49.190Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
60322026-03-10T19:00:49.190Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
60332026-03-10T19:00:49.191Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
60342026-03-10T19:00:49.191Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
60352026-03-10T19:00:49.191Z
60362026-03-10T19:00:49.191Zgoroutine 10428 [semacquire, 59 minutes]:
60372026-03-10T19:00:49.191Zsync.runtime_Semacquire(0xc0000130e0?)
60382026-03-10T19:00:49.191Z /work/go/src/runtime/sema.go:62 +0x25
60392026-03-10T19:00:49.196Zsync.(*WaitGroup).Wait(0x44f7e0?)
60402026-03-10T19:00:49.196Z /work/go/src/sync/waitgroup.go:116 +0x48
60412026-03-10T19:00:49.196Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc003348e80)
60422026-03-10T19:00:49.196Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
60432026-03-10T19:00:49.196Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc003348e80?, {0x50e6118?, 0xc002d6a140?}})
60442026-03-10T19:00:49.196Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
60452026-03-10T19:00:49.196Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed(0xc0021b6f08, {0x50e6118?, 0xc002d6a0a0?}, {0xc002d7d470, 0x1, 0x10000c004c802d0?}, {0xc0077eb920?, 0x357c421?, 0xc0?}, 0x1, ...)
60462026-03-10T19:00:49.196Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:139 +0x6c5
60472026-03-10T19:00:49.196Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*dbAdapter).RangeFeed(0x1eaaf60?, {0x50e6118?, 0xc002d6a0a0?}, {0xc002d7d470?, 0xc0022b2798?, 0x17ef6b5?}, {0x0?, 0x0?, 0x0?}, 0x1, ...)
60482026-03-10T19:00:49.196Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/db_adapter.go:78 +0x3e
60492026-03-10T19:00:49.196Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func1({0x50e6118?, 0xc002d6a0a0?})
60502026-03-10T19:00:49.196Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:285 +0x6c
60512026-03-10T19:00:49.196Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
60522026-03-10T19:00:49.196Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
60532026-03-10T19:00:49.196Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
60542026-03-10T19:00:49.196Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
60552026-03-10T19:00:49.196Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10172
60562026-03-10T19:00:49.196Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
60572026-03-10T19:00:49.196Z
60582026-03-10T19:00:49.196Zgoroutine 10226 [select, 59 minutes]:
60592026-03-10T19:00:49.196Zgithub.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func9({0x50e60e0?, 0xc005c71c80?})
60602026-03-10T19:00:49.196Z /work/oxidecomputer/cockroach/pkg/jobs/registry.go:843 +0x1be
60612026-03-10T19:00:49.196Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
60622026-03-10T19:00:49.196Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
60632026-03-10T19:00:49.196Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
60642026-03-10T19:00:49.196Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
60652026-03-10T19:00:49.196Z
60662026-03-10T19:00:49.196Zgoroutine 9295 [sync.Cond.Wait, 59 minutes]:
60672026-03-10T19:00:49.196Zsync.runtime_notifyListWait(0xc003a0e5e8, 0x0)
60682026-03-10T19:00:49.196Z /work/go/src/runtime/sema.go:569 +0x159
60692026-03-10T19:00:49.196Zsync.(*Cond).Wait(0x8?)
60702026-03-10T19:00:49.196Z /work/go/src/sync/cond.go:70 +0x85
60712026-03-10T19:00:49.197Zgithub.com/cockroachdb/pebble/record.(*flusherCond).Wait(...)
60722026-03-10T19:00:49.197Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/record/log_writer.go:193
60732026-03-10T19:00:49.197Zgithub.com/cockroachdb/pebble/record.(*LogWriter).flushLoop(0xc003a0e508, {0x3afca60?, 0x778e7c0?})
60742026-03-10T19:00:49.197Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/record/log_writer.go:396 +0x475
60752026-03-10T19:00:49.197Zruntime/pprof.Do({0x50e6038?, 0x778e7c0?}, {{0xc00030e1c0?, 0xc006558b10?, 0x14?}}, 0xc003bb37c0)
60762026-03-10T19:00:49.197Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
60772026-03-10T19:00:49.197Zgithub.com/cockroachdb/pebble/record.NewLogWriter.func2()
60782026-03-10T19:00:49.197Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/record/log_writer.go:314 +0x56
60792026-03-10T19:00:49.197Zcreated by github.com/cockroachdb/pebble/record.NewLogWriter in goroutine 9099
60802026-03-10T19:00:49.197Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/record/log_writer.go:313 +0x2c9
60812026-03-10T19:00:49.197Z
60822026-03-10T19:00:49.197Zgoroutine 10488 [semacquire, 59 minutes]:
60832026-03-10T19:00:49.197Zsync.runtime_Semacquire(0xc004b209c0?)
60842026-03-10T19:00:49.197Z /work/go/src/runtime/sema.go:62 +0x25
60852026-03-10T19:00:49.197Zsync.(*WaitGroup).Wait(0x44f7e0?)
60862026-03-10T19:00:49.197Z /work/go/src/sync/waitgroup.go:116 +0x48
60872026-03-10T19:00:49.197Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc000f9a700)
60882026-03-10T19:00:49.197Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
60892026-03-10T19:00:49.197Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc000f9a700?, {0x50e6118?, 0xc004f9bd10?}})
60902026-03-10T19:00:49.197Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
60912026-03-10T19:00:49.197Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed(0xc0021b6f08, {0x50e6118?, 0xc004f9bc70?}, {0xc00674edb0, 0x1, 0x17f4700?}, {0xb92f80?, 0x676be00?, 0xc0?}, 0x0, ...)
60922026-03-10T19:00:49.197Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:139 +0x6c5
60932026-03-10T19:00:49.197Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*dbAdapter).RangeFeed(0x28af?, {0x50e6118?, 0xc004f9bc70?}, {0xc00674edb0?, 0x50e60e0?, 0xc002a50750?}, {0x7070000050e60e0?, 0x50e60e0?, 0x0?}, 0x0, ...)
60942026-03-10T19:00:49.197Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/db_adapter.go:78 +0x3e
60952026-03-10T19:00:49.197Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func1({0x50e6118?, 0xc004f9bc70?})
60962026-03-10T19:00:49.197Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:285 +0x6c
60972026-03-10T19:00:49.197Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
60982026-03-10T19:00:49.198Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
60992026-03-10T19:00:49.198Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
61002026-03-10T19:00:49.198Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
61012026-03-10T19:00:49.198Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9993
61022026-03-10T19:00:49.198Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
61032026-03-10T19:00:49.198Z
61042026-03-10T19:00:49.198Zgoroutine 9305 [select, 59 minutes]:
61052026-03-10T19:00:49.198Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc002c07508?, 0x0?})
61062026-03-10T19:00:49.198Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
61072026-03-10T19:00:49.198Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
61082026-03-10T19:00:49.198Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
61092026-03-10T19:00:49.198Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
61102026-03-10T19:00:49.198Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
61112026-03-10T19:00:49.198Z
61122026-03-10T19:00:49.198Zgoroutine 9271 [select, 59 minutes]:
61132026-03-10T19:00:49.198Zgithub.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler.func1()
61142026-03-10T19:00:49.199Z /work/oxidecomputer/cockroach/pkg/security/certificate_manager.go:241 +0x113
61152026-03-10T19:00:49.199Zcreated by github.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler in goroutine 9099
61162026-03-10T19:00:49.199Z /work/oxidecomputer/cockroach/pkg/security/certificate_manager.go:238 +0x7d
61172026-03-10T19:00:49.199Z
61182026-03-10T19:00:49.200Zgoroutine 9248 [select, 59 minutes]:
61192026-03-10T19:00:49.200Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc0015e8c08?, 0xc0030cbfb8?})
61202026-03-10T19:00:49.200Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
61212026-03-10T19:00:49.201Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
61222026-03-10T19:00:49.201Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
61232026-03-10T19:00:49.201Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
61242026-03-10T19:00:49.201Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
61252026-03-10T19:00:49.201Z
61262026-03-10T19:00:49.201Zgoroutine 9277 [select]:
61272026-03-10T19:00:49.201Zgithub.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue.func2()
61282026-03-10T19:00:49.201Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:366 +0x85
61292026-03-10T19:00:49.201Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue in goroutine 9099
61302026-03-10T19:00:49.201Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:363 +0x3ed
61312026-03-10T19:00:49.201Z
61322026-03-10T19:00:49.201Zgoroutine 9313 [select, 59 minutes]:
61332026-03-10T19:00:49.201Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initCancelingWorkers.func1({0x50e60e0, 0xc00634c5d0})
61342026-03-10T19:00:49.201Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:139 +0xc5
61352026-03-10T19:00:49.201Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
61362026-03-10T19:00:49.201Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
61372026-03-10T19:00:49.201Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
61382026-03-10T19:00:49.201Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
61392026-03-10T19:00:49.205Z
61402026-03-10T19:00:49.205Zgoroutine 9309 [select, 59 minutes]:
61412026-03-10T19:00:49.205Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc004376908?, 0x357e5e0?})
61422026-03-10T19:00:49.205Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
61432026-03-10T19:00:49.205Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
61442026-03-10T19:00:49.205Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
61452026-03-10T19:00:49.205Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
61462026-03-10T19:00:49.205Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
61472026-03-10T19:00:49.205Z
61482026-03-10T19:00:49.205Zgoroutine 9244 [select, 59 minutes]:
61492026-03-10T19:00:49.205Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc004bcd808?, 0xc0032216e0?})
61502026-03-10T19:00:49.205Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
61512026-03-10T19:00:49.206Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
61522026-03-10T19:00:49.206Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
61532026-03-10T19:00:49.206Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
61542026-03-10T19:00:49.206Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
61552026-03-10T19:00:49.206Z
61562026-03-10T19:00:49.206Zgoroutine 9192 [chan receive, 59 minutes]:
61572026-03-10T19:00:49.206Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50e60e0?, 0xc00634c150?})
61582026-03-10T19:00:49.206Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
61592026-03-10T19:00:49.206Zruntime/pprof.Do({0x50e6038?, 0x778e7c0?}, {{0xc00030e2c0?, 0xa54b00?, 0xc001ec3720?}}, 0xc0000acfa8)
61602026-03-10T19:00:49.206Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
61612026-03-10T19:00:49.206Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc0013ca820?)
61622026-03-10T19:00:49.206Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
61632026-03-10T19:00:49.206Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9099
61642026-03-10T19:00:49.206Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
61652026-03-10T19:00:49.206Z
61662026-03-10T19:00:49.207Zgoroutine 10001 [select, 59 minutes]:
61672026-03-10T19:00:49.207Zgoogle.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc001652680)
61682026-03-10T19:00:49.207Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:112 +0x6d
61692026-03-10T19:00:49.207Zcreated by google.golang.org/grpc.newCCBalancerWrapper in goroutine 9099
61702026-03-10T19:00:49.207Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:73 +0x22b
61712026-03-10T19:00:49.207Z
61722026-03-10T19:00:49.207Zgoroutine 9640 [select, 6 minutes]:
61732026-03-10T19:00:49.207Zgithub.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func9({0x50e60e0?, 0xc0035b4120?})
61742026-03-10T19:00:49.207Z /work/oxidecomputer/cockroach/pkg/jobs/registry.go:843 +0x1be
61752026-03-10T19:00:49.207Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
61762026-03-10T19:00:49.207Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
61772026-03-10T19:00:49.207Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
61782026-03-10T19:00:49.207Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
61792026-03-10T19:00:49.207Z
61802026-03-10T19:00:49.207Zgoroutine 10087 [sync.Cond.Wait, 59 minutes]:
61812026-03-10T19:00:49.207Zsync.runtime_notifyListWait(0xc00676e610, 0x36)
61822026-03-10T19:00:49.207Z /work/go/src/runtime/sema.go:569 +0x159
61832026-03-10T19:00:49.207Zsync.(*Cond).Wait(0x0?)
61842026-03-10T19:00:49.208Z /work/go/src/sync/cond.go:70 +0x85
61852026-03-10T19:00:49.208Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0072ac150})
61862026-03-10T19:00:49.208Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
61872026-03-10T19:00:49.208Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
61882026-03-10T19:00:49.208Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
61892026-03-10T19:00:49.208Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
61902026-03-10T19:00:49.208Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
61912026-03-10T19:00:49.208Z
61922026-03-10T19:00:49.208Zgoroutine 9291 [chan receive, 59 minutes]:
61932026-03-10T19:00:49.208Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50e60e0?, 0xc005cee870?})
61942026-03-10T19:00:49.208Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
61952026-03-10T19:00:49.208Zruntime/pprof.Do({0x50e6038?, 0x778e7c0?}, {{0xc00030e2c0?, 0x0?, 0x0?}}, 0xc003bb17a8)
61962026-03-10T19:00:49.208Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
61972026-03-10T19:00:49.208Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
61982026-03-10T19:00:49.208Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
61992026-03-10T19:00:49.208Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9099
62002026-03-10T19:00:49.208Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
62012026-03-10T19:00:49.208Z
62022026-03-10T19:00:49.208Zgoroutine 9262 [chan receive, 59 minutes]:
62032026-03-10T19:00:49.208Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50e60e0?, 0xc005cee450?})
62042026-03-10T19:00:49.208Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
62052026-03-10T19:00:49.208Zruntime/pprof.Do({0x50e6038?, 0x778e7c0?}, {{0xc00030e2c0?, 0x100000001?, 0xc00383aaf0?}}, 0xc0022c6fa8)
62062026-03-10T19:00:49.208Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
62072026-03-10T19:00:49.208Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
62082026-03-10T19:00:49.208Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
62092026-03-10T19:00:49.208Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9099
62102026-03-10T19:00:49.208Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
62112026-03-10T19:00:49.208Z
62122026-03-10T19:00:49.209Zgoroutine 10133 [select]:
62132026-03-10T19:00:49.209Zgoogle.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc002062d00)
62142026-03-10T19:00:49.210Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1131 +0x205
62152026-03-10T19:00:49.210Zcreated by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 10149
62162026-03-10T19:00:49.210Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:341 +0x188e
62172026-03-10T19:00:49.210Z
62182026-03-10T19:00:49.210Zgoroutine 9827 [select]:
62192026-03-10T19:00:49.210Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).outputLoop(0xc00458eee0, {0x50e6118, 0xc0012db4f0})
62202026-03-10T19:00:49.210Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:310 +0x247
62212026-03-10T19:00:49.210Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).runOutputLoop(0xc00458eee0, {0x50e60e0, 0xc004bb37d0}, 0xc0060373b0?)
62222026-03-10T19:00:49.210Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:335 +0xba
62232026-03-10T19:00:49.210Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run.func1({0x50e60e0?, 0xc004bb37d0?})
62242026-03-10T19:00:49.210Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:297 +0x46
62252026-03-10T19:00:49.210Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
62262026-03-10T19:00:49.210Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
62272026-03-10T19:00:49.210Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9729
62282026-03-10T19:00:49.210Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
62292026-03-10T19:00:49.210Z
62302026-03-10T19:00:49.210Zgoroutine 9252 [select, 59 minutes]:
62312026-03-10T19:00:49.210Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initCancelingWorkers.func1({0x50e60e0, 0xc001ead740})
62322026-03-10T19:00:49.210Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:139 +0xc5
62332026-03-10T19:00:49.210Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
62342026-03-10T19:00:49.211Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
62352026-03-10T19:00:49.212Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
62362026-03-10T19:00:49.212Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
62372026-03-10T19:00:49.212Z
62382026-03-10T19:00:49.212Zgoroutine 9984 [select, 59 minutes]:
62392026-03-10T19:00:49.217Zgithub.com/cockroachdb/cockroach/pkg/sql/sqlliveness/slstorage.(*Storage).deleteSessionsLoop(0xc002c5cf00, {0x50e60e0?, 0xc003288960?})
62402026-03-10T19:00:49.217Z /work/oxidecomputer/cockroach/pkg/sql/sqlliveness/slstorage/slstorage.go:354 +0x165
62412026-03-10T19:00:49.217Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
62422026-03-10T19:00:49.217Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
62432026-03-10T19:00:49.217Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
62442026-03-10T19:00:49.217Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
62452026-03-10T19:00:49.217Z
62462026-03-10T19:00:49.217Zgoroutine 9637 [select, 59 minutes]:
62472026-03-10T19:00:49.217Zgithub.com/cockroachdb/cockroach/pkg/sql/catalog/lease.(*Manager).RefreshLeases.func1({0x50e60e0, 0xc0035b4000})
62482026-03-10T19:00:49.217Z /work/oxidecomputer/cockroach/pkg/sql/catalog/lease/lease.go:1041 +0x105
62492026-03-10T19:00:49.217Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
62502026-03-10T19:00:49.218Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
62512026-03-10T19:00:49.218Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
62522026-03-10T19:00:49.218Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
62532026-03-10T19:00:49.218Z
62542026-03-10T19:00:49.218Zgoroutine 9485 [semacquire, 59 minutes]:
62552026-03-10T19:00:49.218Zsync.runtime_Semacquire(0xc006a55458?)
62562026-03-10T19:00:49.218Z /work/go/src/runtime/sema.go:62 +0x25
62572026-03-10T19:00:49.218Zsync.(*WaitGroup).Wait(0x44f7e0?)
62582026-03-10T19:00:49.218Z /work/go/src/sync/waitgroup.go:116 +0x48
62592026-03-10T19:00:49.218Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc0021f5bc0)
62602026-03-10T19:00:49.218Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
62612026-03-10T19:00:49.218Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc0021f5bc0?, {0x50e6118?, 0xc0012db0e0?}})
62622026-03-10T19:00:49.218Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
62632026-03-10T19:00:49.218Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait({0x50e60e0?, 0xc000c58e40?}, {0xc005bafd78, 0x2, 0xc005bafc70?})
62642026-03-10T19:00:49.218Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:193 +0xff
62652026-03-10T19:00:49.218Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0xc006caf080, {0x50e60e0, 0xc000c58e40}, 0xc000c58c60)
62662026-03-10T19:00:49.218Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:291 +0x59c
62672026-03-10T19:00:49.218Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).Start.func2({0x50e60e0, 0xc000c58d80})
62682026-03-10T19:00:49.218Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:215 +0x2f3
62692026-03-10T19:00:49.218Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
62702026-03-10T19:00:49.218Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
62712026-03-10T19:00:49.218Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
62722026-03-10T19:00:49.218Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
62732026-03-10T19:00:49.218Z
62742026-03-10T19:00:49.218Zgoroutine 9435 [select, 1 minutes]:
62752026-03-10T19:00:49.219Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).startGossip.func3({0x50e60e0, 0xc006623c50})
62762026-03-10T19:00:49.219Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:2152 +0x225
62772026-03-10T19:00:49.219Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
62782026-03-10T19:00:49.219Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
62792026-03-10T19:00:49.220Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
62802026-03-10T19:00:49.220Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
62812026-03-10T19:00:49.220Z
62822026-03-10T19:00:49.220Zgoroutine 10131 [IO wait]:
62832026-03-10T19:00:49.220Zinternal/poll.runtime_pollWait(0x7f784b7f0ba0, 0x72)
62842026-03-10T19:00:49.220Z /work/go/src/runtime/netpoll.go:345 +0x85
62852026-03-10T19:00:49.220Zinternal/poll.(*pollDesc).wait(0xc001287c80?, 0xc004368c00?, 0x0)
62862026-03-10T19:00:49.220Z /work/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
62872026-03-10T19:00:49.220Zinternal/poll.(*pollDesc).waitRead(...)
62882026-03-10T19:00:49.220Z /work/go/src/internal/poll/fd_poll_runtime.go:89
62892026-03-10T19:00:49.220Zinternal/poll.(*FD).Read(0xc001287c80, {0xc004368c00, 0x600, 0x600})
62902026-03-10T19:00:49.220Z /work/go/src/internal/poll/fd_unix.go:164 +0x27a
62912026-03-10T19:00:49.220Znet.(*netFD).Read(0xc001287c80, {0xc004368c00?, 0xc0074a2868?, 0x41f225?})
62922026-03-10T19:00:49.220Z /work/go/src/net/fd_posix.go:55 +0x25
62932026-03-10T19:00:49.220Znet.(*conn).Read(0xc00077a948, {0xc004368c00?, 0x7f78482450d8?, 0xc0134473c8?})
62942026-03-10T19:00:49.220Z /work/go/src/net/net.go:185 +0x45
62952026-03-10T19:00:49.220Zgithub.com/cockroachdb/cmux.(*MuxConn).Read(0xc001070d20, {0xc004368c00, 0x600, 0x600})
62962026-03-10T19:00:49.220Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x14f
62972026-03-10T19:00:49.220Zcrypto/tls.(*atLeastReader).Read(0xc0134473c8, {0xc004368c00?, 0x0?, 0xc0134473c8?})
62982026-03-10T19:00:49.220Z /work/go/src/crypto/tls/conn.go:806 +0x3b
62992026-03-10T19:00:49.220Zbytes.(*Buffer).ReadFrom(0xc002dd6630, {0x50b6dc0, 0xc0134473c8})
63002026-03-10T19:00:49.221Z /work/go/src/bytes/buffer.go:211 +0x98
63012026-03-10T19:00:49.221Zcrypto/tls.(*Conn).readFromUntil(0xc002dd6388, {0x7f784ca77e58, 0xc001070d20}, 0xc0074a2990?)
63022026-03-10T19:00:49.223Z /work/go/src/crypto/tls/conn.go:828 +0xde
63032026-03-10T19:00:49.223Zcrypto/tls.(*Conn).readRecordOrCCS(0xc002dd6388, 0x0)
63042026-03-10T19:00:49.223Z /work/go/src/crypto/tls/conn.go:626 +0x3cf
63052026-03-10T19:00:49.223Zcrypto/tls.(*Conn).readRecord(...)
63062026-03-10T19:00:49.223Z /work/go/src/crypto/tls/conn.go:588
63072026-03-10T19:00:49.223Zcrypto/tls.(*Conn).Read(0xc002dd6388, {0xc007482000, 0x8000, 0x40ec60?})
63082026-03-10T19:00:49.223Z /work/go/src/crypto/tls/conn.go:1370 +0x156
63092026-03-10T19:00:49.223Zbufio.(*Reader).Read(0xc004b2e240, {0xc003054d60, 0x9, 0x0?})
63102026-03-10T19:00:49.223Z /work/go/src/bufio/bufio.go:241 +0x197
63112026-03-10T19:00:49.223Zio.ReadAtLeast({0x50b1100, 0xc004b2e240}, {0xc003054d60, 0x9, 0x9}, 0x9)
63122026-03-10T19:00:49.223Z /work/go/src/io/io.go:335 +0x90
63132026-03-10T19:00:49.223Zio.ReadFull(...)
63142026-03-10T19:00:49.223Z /work/go/src/io/io.go:354
63152026-03-10T19:00:49.223Zgolang.org/x/net/http2.readFrameHeader({0xc003054d60, 0x9, 0xc0074a2e10?}, {0x50b1100?, 0xc004b2e240?})
63162026-03-10T19:00:49.223Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x65
63172026-03-10T19:00:49.223Zgolang.org/x/net/http2.(*Framer).ReadFrame(0xc003054d20)
63182026-03-10T19:00:49.223Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:501 +0x85
63192026-03-10T19:00:49.223Zgoogle.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc000a449c0, 0xc00677b080, 0x431bf68)
63202026-03-10T19:00:49.223Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:611 +0x14e
63212026-03-10T19:00:49.223Zgoogle.golang.org/grpc.(*Server).serveStreams(0xc005fee1c0, {0x5126b60, 0xc000a449c0})
63222026-03-10T19:00:49.223Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:906 +0x114
63232026-03-10T19:00:49.223Zgoogle.golang.org/grpc.(*Server).handleRawConn.func1()
63242026-03-10T19:00:49.223Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:848 +0x45
63252026-03-10T19:00:49.225Zcreated by google.golang.org/grpc.(*Server).handleRawConn in goroutine 9627
63262026-03-10T19:00:49.226Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:847 +0x15b
63272026-03-10T19:00:49.226Z
63282026-03-10T19:00:49.226Zgoroutine 10273 [IO wait]:
63292026-03-10T19:00:49.226Zinternal/poll.runtime_pollWait(0x7f784df33780, 0x72)
63302026-03-10T19:00:49.226Z /work/go/src/runtime/netpoll.go:345 +0x85
63312026-03-10T19:00:49.226Zinternal/poll.(*pollDesc).wait(0xc001f0a800?, 0xc003e84000?, 0x0)
63322026-03-10T19:00:49.226Z /work/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
63332026-03-10T19:00:49.226Zinternal/poll.(*pollDesc).waitRead(...)
63342026-03-10T19:00:49.226Z /work/go/src/internal/poll/fd_poll_runtime.go:89
63352026-03-10T19:00:49.226Zinternal/poll.(*FD).Read(0xc001f0a800, {0xc003e84000, 0x4000, 0x4000})
63362026-03-10T19:00:49.226Z /work/go/src/internal/poll/fd_unix.go:164 +0x27a
63372026-03-10T19:00:49.226Znet.(*netFD).Read(0xc001f0a800, {0xc003e84000?, 0x7f78353f6b68?, 0xc02dbec8e8?})
63382026-03-10T19:00:49.226Z /work/go/src/net/fd_posix.go:55 +0x25
63392026-03-10T19:00:49.230Znet.(*conn).Read(0xc0031d5c60, {0xc003e84000?, 0xc0009448e0?, 0x416dfb?})
63402026-03-10T19:00:49.230Z /work/go/src/net/net.go:185 +0x45
63412026-03-10T19:00:49.230Zcrypto/tls.(*atLeastReader).Read(0xc02dbec8e8, {0xc003e84000?, 0x0?, 0xc02dbec8e8?})
63422026-03-10T19:00:49.230Z /work/go/src/crypto/tls/conn.go:806 +0x3b
63432026-03-10T19:00:49.230Zbytes.(*Buffer).ReadFrom(0xc0046950b0, {0x50b6dc0, 0xc02dbec8e8})
63442026-03-10T19:00:49.230Z /work/go/src/bytes/buffer.go:211 +0x98
63452026-03-10T19:00:49.230Zcrypto/tls.(*Conn).readFromUntil(0xc004694e08, {0x50b6fa0, 0xc0031d5c60}, 0xc000944928?)
63462026-03-10T19:00:49.231Z /work/go/src/crypto/tls/conn.go:828 +0xde
63472026-03-10T19:00:49.231Zcrypto/tls.(*Conn).readRecordOrCCS(0xc004694e08, 0x0)
63482026-03-10T19:00:49.231Z /work/go/src/crypto/tls/conn.go:626 +0x3cf
63492026-03-10T19:00:49.232Zcrypto/tls.(*Conn).readRecord(...)
63502026-03-10T19:00:49.232Z /work/go/src/crypto/tls/conn.go:588
63512026-03-10T19:00:49.232Zcrypto/tls.(*Conn).Read(0xc004694e08, {0xc00507e000, 0x8000, 0xc02dbb7c20?})
63522026-03-10T19:00:49.232Z /work/go/src/crypto/tls/conn.go:1370 +0x156
63532026-03-10T19:00:49.232Zbufio.(*Reader).Read(0xc00461aba0, {0xc002cdd540, 0x9, 0x0?})
63542026-03-10T19:00:49.232Z /work/go/src/bufio/bufio.go:241 +0x197
63552026-03-10T19:00:49.232Zio.ReadAtLeast({0x50b1100, 0xc00461aba0}, {0xc002cdd540, 0x9, 0x9}, 0x9)
63562026-03-10T19:00:49.232Z /work/go/src/io/io.go:335 +0x90
63572026-03-10T19:00:49.232Zio.ReadFull(...)
63582026-03-10T19:00:49.232Z /work/go/src/io/io.go:354
63592026-03-10T19:00:49.232Zgolang.org/x/net/http2.readFrameHeader({0xc002cdd540, 0x9, 0x0?}, {0x50b1100?, 0xc00461aba0?})
63602026-03-10T19:00:49.232Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x65
63612026-03-10T19:00:49.232Zgolang.org/x/net/http2.(*Framer).ReadFrame(0xc002cdd500)
63622026-03-10T19:00:49.232Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:501 +0x85
63632026-03-10T19:00:49.232Zgoogle.golang.org/grpc/internal/transport.(*http2Client).reader(0xc00274a960)
63642026-03-10T19:00:49.233Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1498 +0x41d
63652026-03-10T19:00:49.233Zcreated by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 10354
63662026-03-10T19:00:49.233Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:365 +0x184d
63672026-03-10T19:00:49.233Z
63682026-03-10T19:00:49.233Zgoroutine 9635 [select]:
63692026-03-10T19:00:49.233Zgithub.com/cockroachdb/cockroach/pkg/sql/stmtdiagnostics.(*Registry).poll(0xc005051810, {0x50e60e0, 0xc006037d70})
63702026-03-10T19:00:49.233Z /work/oxidecomputer/cockroach/pkg/sql/stmtdiagnostics/statement_diagnostics.go:181 +0x2b5
63712026-03-10T19:00:49.233Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
63722026-03-10T19:00:49.233Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
63732026-03-10T19:00:49.233Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
63742026-03-10T19:00:49.233Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
63752026-03-10T19:00:49.233Z
63762026-03-10T19:00:49.233Zgoroutine 9104 [select, 59 minutes]:
63772026-03-10T19:00:49.233Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc0034f5508?, 0xc00242fe30?})
63782026-03-10T19:00:49.234Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
63792026-03-10T19:00:49.234Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
63802026-03-10T19:00:49.234Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
63812026-03-10T19:00:49.234Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
63822026-03-10T19:00:49.234Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
63832026-03-10T19:00:49.234Z
63842026-03-10T19:00:49.234Zgoroutine 10205 [select]:
63852026-03-10T19:00:49.234Zgithub.com/cockroachdb/cockroach/pkg/sql/stmtdiagnostics.(*Registry).poll(0xc00449ae00, {0x50e60e0, 0xc005c71890})
63862026-03-10T19:00:49.234Z /work/oxidecomputer/cockroach/pkg/sql/stmtdiagnostics/statement_diagnostics.go:181 +0x2b5
63872026-03-10T19:00:49.234Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
63882026-03-10T19:00:49.234Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
63892026-03-10T19:00:49.234Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
63902026-03-10T19:00:49.234Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
63912026-03-10T19:00:49.234Z
63922026-03-10T19:00:49.235Zgoroutine 10152 [semacquire, 59 minutes]:
63932026-03-10T19:00:49.235Zsync.runtime_Semacquire(0xc006e92a98?)
63942026-03-10T19:00:49.235Z /work/go/src/runtime/sema.go:62 +0x25
63952026-03-10T19:00:49.236Zsync.(*WaitGroup).Wait(0x44f7e0?)
63962026-03-10T19:00:49.236Z /work/go/src/sync/waitgroup.go:116 +0x48
63972026-03-10T19:00:49.236Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc0043fa740)
63982026-03-10T19:00:49.236Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
63992026-03-10T19:00:49.236Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc0043fa740?, {0x50e6118?, 0xc00717cd20?}})
64002026-03-10T19:00:49.236Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
64012026-03-10T19:00:49.236Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed(0xc0021b6f08, {0x50e6118?, 0xc00717cc80?}, {0xc005c71a40, 0x1, 0xc005c718f0?}, {0xc0009d6b00?, 0x6f507c0?, 0xc0?}, 0x0, ...)
64022026-03-10T19:00:49.236Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:139 +0x6c5
64032026-03-10T19:00:49.236Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*dbAdapter).RangeFeed(0xc006527590?, {0x50e6118?, 0xc00717cc80?}, {0xc005c71a40?, 0x50e60e0?, 0xc005c71aa0?}, {0x707294d25b60344?, 0x50e60e0?, 0x0?}, 0x0, ...)
64042026-03-10T19:00:49.236Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/db_adapter.go:78 +0x3e
64052026-03-10T19:00:49.236Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func1({0x50e6118?, 0xc00717cc80?})
64062026-03-10T19:00:49.236Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:285 +0x6c
64072026-03-10T19:00:49.236Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
64082026-03-10T19:00:49.236Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
64092026-03-10T19:00:49.236Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
64102026-03-10T19:00:49.237Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
64112026-03-10T19:00:49.237Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10206
64122026-03-10T19:00:49.237Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
64132026-03-10T19:00:49.237Z
64142026-03-10T19:00:49.237Zgoroutine 10489 [select]:
64152026-03-10T19:00:49.237Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents(0xc00146cdc0, {0x50e6118, 0xc004f9bc70}, 0xc00674ed20, 0xc004ff9f80)
64162026-03-10T19:00:49.238Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:331 +0xc5
64172026-03-10T19:00:49.238Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func2({0x50e6118?, 0xc004f9bc70?})
64182026-03-10T19:00:49.238Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:288 +0x32
64192026-03-10T19:00:49.238Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
64202026-03-10T19:00:49.239Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
64212026-03-10T19:00:49.239Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
64222026-03-10T19:00:49.239Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
64232026-03-10T19:00:49.239Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9993
64242026-03-10T19:00:49.239Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
64252026-03-10T19:00:49.239Z
64262026-03-10T19:00:49.239Zgoroutine 9483 [select, 6 minutes]:
64272026-03-10T19:00:49.239Zgithub.com/cockroachdb/cockroach/pkg/sql/sqlliveness/slstorage.(*Storage).deleteSessionsLoop(0xc00148bb30, {0x50e60e0?, 0xc000c58b10?})
64282026-03-10T19:00:49.239Z /work/oxidecomputer/cockroach/pkg/sql/sqlliveness/slstorage/slstorage.go:354 +0x165
64292026-03-10T19:00:49.239Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
64302026-03-10T19:00:49.239Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
64312026-03-10T19:00:49.239Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
64322026-03-10T19:00:49.239Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
64332026-03-10T19:00:49.239Z
64342026-03-10T19:00:49.239Zgoroutine 9449 [select]:
64352026-03-10T19:00:49.239Zgithub.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
64362026-03-10T19:00:49.239Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:439 +0x1c9
64372026-03-10T19:00:49.240Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 9099
64382026-03-10T19:00:49.240Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:412 +0x4f
64392026-03-10T19:00:49.245Z
64402026-03-10T19:00:49.245Zgoroutine 9577 [select]:
64412026-03-10T19:00:49.245Zgithub.com/cockroachdb/cockroach/pkg/rpc.respStreamClientAdapter.recvInternal(...)
64422026-03-10T19:00:49.245Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:694
64432026-03-10T19:00:49.245Zgithub.com/cockroachdb/cockroach/pkg/rpc.rangeFeedClientAdapter.Recv(...)
64442026-03-10T19:00:49.245Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:723
64452026-03-10T19:00:49.245Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed.func3()
64462026-03-10T19:00:49.245Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:516 +0x25
64472026-03-10T19:00:49.245Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*stuckRangeFeedCanceler).do(0xc003c22be0, 0xc0028c1a50)
64482026-03-10T19:00:49.245Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed_canceler.go:100 +0x1b3
64492026-03-10T19:00:49.245Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed(0xc00432cf08, {0x50e6118?, 0xc003c22aa0?}, {{0xc006770728, 0x1, 0x8}, {0xc006770730, 0x1, 0x8}}, {0x0, ...}, ...)
64502026-03-10T19:00:49.245Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:515 +0xc98
64512026-03-10T19:00:49.245Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).partialRangeFeed(0xc00432cf08, {0x50e6118, 0xc003c22aa0}, 0xc003c22a50, {{0xc006770728, 0x1, 0x8}, {0xc006770730, 0x1, 0x8}}, ...)
64522026-03-10T19:00:49.245Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:336 +0x745
64532026-03-10T19:00:49.245Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.1({0x50e6118?, 0xc003c22aa0?})
64542026-03-10T19:00:49.245Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:120 +0xa9
64552026-03-10T19:00:49.245Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.Group.GoCtx.2()
64562026-03-10T19:00:49.245Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
64572026-03-10T19:00:49.246Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
64582026-03-10T19:00:49.246Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
64592026-03-10T19:00:49.246Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9575
64602026-03-10T19:00:49.246Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
64612026-03-10T19:00:49.246Z
64622026-03-10T19:00:49.246Zgoroutine 9162 [select, 59 minutes]:
64632026-03-10T19:00:49.246Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2({0x50e60e0, 0xc001eac1e0})
64642026-03-10T19:00:49.246Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/queue.go:823 +0xff
64652026-03-10T19:00:49.246Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
64662026-03-10T19:00:49.246Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
64672026-03-10T19:00:49.246Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9437
64682026-03-10T19:00:49.246Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
64692026-03-10T19:00:49.246Z
64702026-03-10T19:00:49.246Zgoroutine 9306 [select, 59 minutes]:
64712026-03-10T19:00:49.246Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc002c07808?, 0x0?})
64722026-03-10T19:00:49.246Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
64732026-03-10T19:00:49.246Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
64742026-03-10T19:00:49.246Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
64752026-03-10T19:00:49.246Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
64762026-03-10T19:00:49.246Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
64772026-03-10T19:00:49.246Z
64782026-03-10T19:00:49.246Zgoroutine 9238 [select, 59 minutes]:
64792026-03-10T19:00:49.246Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc0067f7508?, 0x0?})
64802026-03-10T19:00:49.247Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
64812026-03-10T19:00:49.247Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
64822026-03-10T19:00:49.247Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
64832026-03-10T19:00:49.247Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
64842026-03-10T19:00:49.247Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
64852026-03-10T19:00:49.247Z
64862026-03-10T19:00:49.248Zgoroutine 10418 [select]:
64872026-03-10T19:00:49.248Zgithub.com/cockroachdb/cockroach/pkg/rpc.respStreamClientAdapter.recvInternal(...)
64882026-03-10T19:00:49.248Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:694
64892026-03-10T19:00:49.248Zgithub.com/cockroachdb/cockroach/pkg/rpc.rangeFeedClientAdapter.Recv(...)
64902026-03-10T19:00:49.248Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:723
64912026-03-10T19:00:49.248Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed.func3()
64922026-03-10T19:00:49.248Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:516 +0x25
64932026-03-10T19:00:49.248Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*stuckRangeFeedCanceler).do(0xc0053af770, 0xc001b95a50)
64942026-03-10T19:00:49.248Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed_canceler.go:100 +0x1b3
64952026-03-10T19:00:49.248Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed(0xc00432cf08, {0x50e6118?, 0xc0053af680?}, {{0xc002ffa158, 0x1, 0x8}, {0xc002ffa170, 0x1, 0x8}}, {0x189b8cc3e4dcd085, ...}, ...)
64962026-03-10T19:00:49.248Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:515 +0xc98
64972026-03-10T19:00:49.248Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).partialRangeFeed(0xc00432cf08, {0x50e6118, 0xc0053af680}, 0xc0053af630, {{0xc002ffa158, 0x1, 0x8}, {0xc002ffa170, 0x1, 0x8}}, ...)
64982026-03-10T19:00:49.248Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:336 +0x745
64992026-03-10T19:00:49.248Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.1({0x50e6118?, 0xc0053af680?})
65002026-03-10T19:00:49.249Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:120 +0xa9
65012026-03-10T19:00:49.249Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.Group.GoCtx.2()
65022026-03-10T19:00:49.249Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
65032026-03-10T19:00:49.249Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
65042026-03-10T19:00:49.249Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
65052026-03-10T19:00:49.249Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10336
65062026-03-10T19:00:49.249Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
65072026-03-10T19:00:49.249Z
65082026-03-10T19:00:49.250Zgoroutine 10089 [sync.Cond.Wait, 59 minutes]:
65092026-03-10T19:00:49.250Zsync.runtime_notifyListWait(0xc00676e610, 0x38)
65102026-03-10T19:00:49.250Z /work/go/src/runtime/sema.go:569 +0x159
65112026-03-10T19:00:49.250Zsync.(*Cond).Wait(0x0?)
65122026-03-10T19:00:49.250Z /work/go/src/sync/cond.go:70 +0x85
65132026-03-10T19:00:49.250Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0072ac1b0})
65142026-03-10T19:00:49.250Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
65152026-03-10T19:00:49.250Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
65162026-03-10T19:00:49.250Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
65172026-03-10T19:00:49.250Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
65182026-03-10T19:00:49.250Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
65192026-03-10T19:00:49.250Z
65202026-03-10T19:00:49.250Zgoroutine 9256 [chan receive, 59 minutes]:
65212026-03-10T19:00:49.250Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50e60e0?, 0xc00634cae0?})
65222026-03-10T19:00:49.250Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
65232026-03-10T19:00:49.250Zruntime/pprof.Do({0x50e6038?, 0x778e7c0?}, {{0xc00030e2c0?, 0xc001a2afc0?, 0xc001a2b000?}}, 0xc0022cbfa8)
65242026-03-10T19:00:49.250Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
65252026-03-10T19:00:49.250Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
65262026-03-10T19:00:49.250Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
65272026-03-10T19:00:49.250Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9099
65282026-03-10T19:00:49.250Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
65292026-03-10T19:00:49.250Z
65302026-03-10T19:00:49.250Zgoroutine 9247 [select, 59 minutes]:
65312026-03-10T19:00:49.250Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc0015e8908?, 0xc0030cb8f0?})
65322026-03-10T19:00:49.250Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
65332026-03-10T19:00:49.250Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
65342026-03-10T19:00:49.250Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
65352026-03-10T19:00:49.250Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
65362026-03-10T19:00:49.250Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
65372026-03-10T19:00:49.250Z
65382026-03-10T19:00:49.250Zgoroutine 10432 [select]:
65392026-03-10T19:00:49.255Zgoogle.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc002d6a370, {0xc000013270, 0x5, 0x5})
65402026-03-10T19:00:49.255Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:191 +0x95
65412026-03-10T19:00:49.255Zgoogle.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc002d6a370, {0xc000013270?, 0xc004bc32a8?, 0xc006351048?})
65422026-03-10T19:00:49.255Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:171 +0x12d
65432026-03-10T19:00:49.255Zgoogle.golang.org/grpc/internal/transport.(*transportReader).Read(0xc003267620, {0xc000013270?, 0xc0063510c0?, 0xa418a5?})
65442026-03-10T19:00:49.256Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:485 +0x2c
65452026-03-10T19:00:49.256Zio.ReadAtLeast({0x50bb160, 0xc003267620}, {0xc000013270, 0x5, 0x5}, 0x5)
65462026-03-10T19:00:49.256Z /work/go/src/io/io.go:335 +0x90
65472026-03-10T19:00:49.256Zio.ReadFull(...)
65482026-03-10T19:00:49.256Z /work/go/src/io/io.go:354
65492026-03-10T19:00:49.257Zgoogle.golang.org/grpc/internal/transport.(*Stream).Read(0xc00523f9e0, {0xc000013270, 0x5, 0x5})
65502026-03-10T19:00:49.257Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:469 +0x96
65512026-03-10T19:00:49.257Zgoogle.golang.org/grpc.(*parser).recvMsg(0xc000013260, 0x7fffffff)
65522026-03-10T19:00:49.257Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:559 +0x42
65532026-03-10T19:00:49.257Zgoogle.golang.org/grpc.recvAndDecompress(0x60?, 0xc00523f9e0, {0x0, 0x0}, 0x7fffffff, 0xc006351378, {0x50ddc20, 0x778e7c0})
65542026-03-10T19:00:49.257Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:690 +0x5a
65552026-03-10T19:00:49.257Zgoogle.golang.org/grpc.recv(0x50c4e80?, {0x7f784b7a48e8, 0xc000567440}, 0xc006351300?, {0x0?, 0x0?}, {0x3fa0a40, 0xc00e95c8e0}, 0x10?, 0xc006351378, ...)
65562026-03-10T19:00:49.257Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:756 +0x68
65572026-03-10T19:00:49.257Zgoogle.golang.org/grpc.(*csAttempt).recvMsg(0xc0065a0f20, {0x3fa0a40, 0xc00e95c8e0}, 0x10?)
65582026-03-10T19:00:49.257Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:983 +0x27e
65592026-03-10T19:00:49.257Zgoogle.golang.org/grpc.(*clientStream).RecvMsg.func1(0x41f225?)
65602026-03-10T19:00:49.257Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:834 +0x1f
65612026-03-10T19:00:49.257Zgoogle.golang.org/grpc.(*clientStream).withRetry(0xc00523f7a0, 0xc0063514a8, 0xc006351498)
65622026-03-10T19:00:49.257Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:688 +0x2ee
65632026-03-10T19:00:49.257Zgoogle.golang.org/grpc.(*clientStream).RecvMsg(0xc00523f7a0, {0x3fa0a40?, 0xc00e95c8e0?})
65642026-03-10T19:00:49.257Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:833 +0x107
65652026-03-10T19:00:49.257Zgithub.com/cockroachdb/cockroach/pkg/util/tracing.(*tracingClientStream).RecvMsg(0xc001207de0, {0x3fa0a40?, 0xc00e95c8e0?})
65662026-03-10T19:00:49.257Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:441 +0x31
65672026-03-10T19:00:49.257Zgithub.com/cockroachdb/cockroach/pkg/roachpb.(*internalRangeFeedClient).Recv(0xc0067c1ad0)
65682026-03-10T19:00:49.257Z /work/oxidecomputer/cockroach/pkg/roachpb/api.pb.go:9289 +0x46
65692026-03-10T19:00:49.257Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed.func3()
65702026-03-10T19:00:49.257Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:516 +0x25
65712026-03-10T19:00:49.257Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*stuckRangeFeedCanceler).do(0xc002d6a230, 0xc006351a50)
65722026-03-10T19:00:49.257Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed_canceler.go:100 +0x1b3
65732026-03-10T19:00:49.257Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed(0xc0021b6f08, {0x50e6118?, 0xc002d6a140?}, {{0xc00322bcb0, 0x2, 0x8}, {0xc00322bcb8, 0x2, 0x8}}, {0x189b8cc3e6368a45, ...}, ...)
65742026-03-10T19:00:49.257Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:515 +0xc98
65752026-03-10T19:00:49.257Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).partialRangeFeed(0xc0021b6f08, {0x50e6118, 0xc002d6a140}, 0xc002d6a0f0, {{0xc00322bcb0, 0x2, 0x8}, {0xc00322bcb8, 0x2, 0x8}}, ...)
65762026-03-10T19:00:49.257Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:336 +0x745
65772026-03-10T19:00:49.257Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.1({0x50e6118?, 0xc002d6a140?})
65782026-03-10T19:00:49.257Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:120 +0xa9
65792026-03-10T19:00:49.257Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.Group.GoCtx.2()
65802026-03-10T19:00:49.257Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
65812026-03-10T19:00:49.257Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
65822026-03-10T19:00:49.257Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
65832026-03-10T19:00:49.258Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10430
65842026-03-10T19:00:49.258Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
65852026-03-10T19:00:49.258Z
65862026-03-10T19:00:49.258Zgoroutine 10311 [select, 59 minutes]:
65872026-03-10T19:00:49.258Zgithub.com/cockroachdb/cockroach/pkg/server.(*Server).startAttemptUpgrade.func1({0x50e60e0?, 0xc0036c0090?})
65882026-03-10T19:00:49.258Z /work/oxidecomputer/cockroach/pkg/server/auto_upgrade.go:45 +0x1cf
65892026-03-10T19:00:49.258Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
65902026-03-10T19:00:49.258Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
65912026-03-10T19:00:49.258Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
65922026-03-10T19:00:49.258Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
65932026-03-10T19:00:49.258Z
65942026-03-10T19:00:49.258Zgoroutine 9298 [select, 59 minutes]:
65952026-03-10T19:00:49.258Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc0034f5b08?, 0x0?})
65962026-03-10T19:00:49.258Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
65972026-03-10T19:00:49.258Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
65982026-03-10T19:00:49.258Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
65992026-03-10T19:00:49.258Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
66002026-03-10T19:00:49.258Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
66012026-03-10T19:00:49.258Z
66022026-03-10T19:00:49.259Zgoroutine 10409 [select, 59 minutes]:
66032026-03-10T19:00:49.259Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1({0x50e6118, 0xc002cfb9a0})
66042026-03-10T19:00:49.259Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:116 +0x157
66052026-03-10T19:00:49.260Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.Group.GoCtx.func3()
66062026-03-10T19:00:49.260Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
66072026-03-10T19:00:49.260Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
66082026-03-10T19:00:49.260Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
66092026-03-10T19:00:49.260Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10469
66102026-03-10T19:00:49.260Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
66112026-03-10T19:00:49.260Z
66122026-03-10T19:00:49.260Zgoroutine 9366 [sync.Cond.Wait]:
66132026-03-10T19:00:49.260Zsync.runtime_notifyListWait(0xc0005ac550, 0x3743)
66142026-03-10T19:00:49.260Z /work/go/src/runtime/sema.go:569 +0x159
66152026-03-10T19:00:49.260Zsync.(*Cond).Wait(0x3ae1600?)
66162026-03-10T19:00:49.260Z /work/go/src/sync/cond.go:70 +0x85
66172026-03-10T19:00:49.260Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006622c00})
66182026-03-10T19:00:49.260Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
66192026-03-10T19:00:49.261Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
66202026-03-10T19:00:49.261Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
66212026-03-10T19:00:49.261Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
66222026-03-10T19:00:49.261Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
66232026-03-10T19:00:49.261Z
66242026-03-10T19:00:49.261Zgoroutine 9288 [chan receive, 59 minutes]:
66252026-03-10T19:00:49.261Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50e60e0?, 0xc00634cbd0?})
66262026-03-10T19:00:49.261Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
66272026-03-10T19:00:49.261Zruntime/pprof.Do({0x50e6038?, 0x778e7c0?}, {{0xc00030e2c0?, 0xaaaa2492492aaaa2?, 0x92492aaaa2492492?}}, 0xc003bb7fa8)
66282026-03-10T19:00:49.261Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
66292026-03-10T19:00:49.261Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x3c4f262492492aaa?)
66302026-03-10T19:00:49.261Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
66312026-03-10T19:00:49.261Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9099
66322026-03-10T19:00:49.261Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
66332026-03-10T19:00:49.261Z
66342026-03-10T19:00:49.261Zgoroutine 9980 [select]:
66352026-03-10T19:00:49.261Zgithub.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigkvsubscriber.(*KVSubscriber).Start.func1({0x50e60e0, 0xc006f824e0})
66362026-03-10T19:00:49.261Z /work/oxidecomputer/cockroach/pkg/spanconfig/spanconfigkvsubscriber/kvsubscriber.go:268 +0x227
66372026-03-10T19:00:49.261Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
66382026-03-10T19:00:49.261Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
66392026-03-10T19:00:49.265Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
66402026-03-10T19:00:49.265Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
66412026-03-10T19:00:49.265Z
66422026-03-10T19:00:49.265Zgoroutine 9393 [sync.Cond.Wait]:
66432026-03-10T19:00:49.265Zsync.runtime_notifyListWait(0xc0005ac550, 0x3758)
66442026-03-10T19:00:49.266Z /work/go/src/runtime/sema.go:569 +0x159
66452026-03-10T19:00:49.266Zsync.(*Cond).Wait(0x3ae1600?)
66462026-03-10T19:00:49.266Z /work/go/src/sync/cond.go:70 +0x85
66472026-03-10T19:00:49.266Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc0066231d0})
66482026-03-10T19:00:49.266Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
66492026-03-10T19:00:49.266Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
66502026-03-10T19:00:49.266Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
66512026-03-10T19:00:49.266Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
66522026-03-10T19:00:49.266Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
66532026-03-10T19:00:49.266Z
66542026-03-10T19:00:49.266Zgoroutine 9236 [select, 59 minutes]:
66552026-03-10T19:00:49.267Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc0067f6f08?, 0x50e6038?})
66562026-03-10T19:00:49.267Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
66572026-03-10T19:00:49.267Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
66582026-03-10T19:00:49.267Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
66592026-03-10T19:00:49.267Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
66602026-03-10T19:00:49.267Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
66612026-03-10T19:00:49.267Z
66622026-03-10T19:00:49.267Zgoroutine 9279 [select]:
66632026-03-10T19:00:49.267Zgithub.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue.func2()
66642026-03-10T19:00:49.267Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:366 +0x85
66652026-03-10T19:00:49.267Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue in goroutine 9099
66662026-03-10T19:00:49.267Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:363 +0x3ed
66672026-03-10T19:00:49.267Z
66682026-03-10T19:00:49.267Zgoroutine 9601 [select]:
66692026-03-10T19:00:49.267Zgithub.com/cockroachdb/cockroach/pkg/sql/contention/txnidcache.(*Cache).Start.func1({0xc001a08908?, 0xc0060373e0?})
66702026-03-10T19:00:49.267Z /work/oxidecomputer/cockroach/pkg/sql/contention/txnidcache/txn_id_cache.go:144 +0x9b
66712026-03-10T19:00:49.267Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
66722026-03-10T19:00:49.267Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
66732026-03-10T19:00:49.269Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
66742026-03-10T19:00:49.269Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
66752026-03-10T19:00:49.269Z
66762026-03-10T19:00:49.269Zgoroutine 9634 [select]:
66772026-03-10T19:00:49.269Zgithub.com/cockroachdb/cockroach/pkg/sql/stats.(*Refresher).Start.func1({0x50e60e0, 0xc006037d10})
66782026-03-10T19:00:49.269Z /work/oxidecomputer/cockroach/pkg/sql/stats/automatic_stats.go:374 +0x1b9
66792026-03-10T19:00:49.269Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
66802026-03-10T19:00:49.269Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
66812026-03-10T19:00:49.269Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
66822026-03-10T19:00:49.269Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
66832026-03-10T19:00:49.269Z
66842026-03-10T19:00:49.269Zgoroutine 10070 [sync.Cond.Wait, 59 minutes]:
66852026-03-10T19:00:49.269Zsync.runtime_notifyListWait(0xc00676e610, 0x25)
66862026-03-10T19:00:49.269Z /work/go/src/runtime/sema.go:569 +0x159
66872026-03-10T19:00:49.269Zsync.(*Cond).Wait(0x8?)
66882026-03-10T19:00:49.270Z /work/go/src/sync/cond.go:70 +0x85
66892026-03-10T19:00:49.270Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c9da0})
66902026-03-10T19:00:49.270Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
66912026-03-10T19:00:49.270Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
66922026-03-10T19:00:49.270Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
66932026-03-10T19:00:49.270Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
66942026-03-10T19:00:49.270Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
66952026-03-10T19:00:49.270Z
66962026-03-10T19:00:49.270Zgoroutine 9373 [sync.Cond.Wait]:
66972026-03-10T19:00:49.270Zsync.runtime_notifyListWait(0xc0005ac550, 0x3741)
66982026-03-10T19:00:49.270Z /work/go/src/runtime/sema.go:569 +0x159
66992026-03-10T19:00:49.270Zsync.(*Cond).Wait(0x3ae1600?)
67002026-03-10T19:00:49.270Z /work/go/src/sync/cond.go:70 +0x85
67012026-03-10T19:00:49.270Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006622d80})
67022026-03-10T19:00:49.270Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
67032026-03-10T19:00:49.270Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
67042026-03-10T19:00:49.270Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
67052026-03-10T19:00:49.270Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
67062026-03-10T19:00:49.270Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
67072026-03-10T19:00:49.270Z
67082026-03-10T19:00:49.270Zgoroutine 9578 [chan receive, 59 minutes]:
67092026-03-10T19:00:49.270Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).rangeFeedWithRangeID(0xc004e66008, 0x3f72e40?, 0xc0048aa480, {0x7f784b79a928, 0xc000f10da0})
67102026-03-10T19:00:49.270Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:249 +0x76b
67112026-03-10T19:00:49.270Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed(...)
67122026-03-10T19:00:49.270Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:149
67132026-03-10T19:00:49.270Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).RangeFeed(0xc001668008, 0xc0048aa480, {0x511a910, 0xc000f10da0})
67142026-03-10T19:00:49.270Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:3082 +0x16f
67152026-03-10T19:00:49.270Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).RangeFeed(0xc006849e00, 0xc0048aa480, {0x511a910, 0xc000f10da0})
67162026-03-10T19:00:49.270Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/stores.go:216 +0x19a
67172026-03-10T19:00:49.270Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed(0x3f72e40?, 0xc003c22cb0?, {0x511a910, 0xc000f10da0})
67182026-03-10T19:00:49.271Z /work/oxidecomputer/cockroach/pkg/server/node.go:1232 +0x28
67192026-03-10T19:00:49.271Zgithub.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed.func1()
67202026-03-10T19:00:49.271Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:753 +0xa7
67212026-03-10T19:00:49.271Zcreated by github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed in goroutine 9577
67222026-03-10T19:00:49.271Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:750 +0x17c
67232026-03-10T19:00:49.271Z
67242026-03-10T19:00:49.271Zgoroutine 9041 [select]:
67252026-03-10T19:00:49.271Zgithub.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).manage.func1({0x50e60e0, 0xc007a037a0})
67262026-03-10T19:00:49.271Z /work/oxidecomputer/cockroach/pkg/gossip/gossip.go:1420 +0x2d2
67272026-03-10T19:00:49.271Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
67282026-03-10T19:00:49.271Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
67292026-03-10T19:00:49.271Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
67302026-03-10T19:00:49.271Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
67312026-03-10T19:00:49.271Z
67322026-03-10T19:00:49.271Zgoroutine 10197 [select, 59 minutes]:
67332026-03-10T19:00:49.271Zgithub.com/cockroachdb/cockroach/pkg/sql/gcjob/gcjobnotifier.(*Notifier).run(0xc0031ac9c0, {0x27d5?, 0xc0004c8f70?})
67342026-03-10T19:00:49.271Z /work/oxidecomputer/cockroach/pkg/sql/gcjob/gcjobnotifier/notifier.go:145 +0xc8
67352026-03-10T19:00:49.271Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
67362026-03-10T19:00:49.271Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
67372026-03-10T19:00:49.271Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
67382026-03-10T19:00:49.271Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
67392026-03-10T19:00:49.275Z
67402026-03-10T19:00:49.277Zgoroutine 10015 [IO wait, 59 minutes]:
67412026-03-10T19:00:49.277Zinternal/poll.runtime_pollWait(0x7f784b7f20f0, 0x72)
67422026-03-10T19:00:49.277Z /work/go/src/runtime/netpoll.go:345 +0x85
67432026-03-10T19:00:49.277Zinternal/poll.(*pollDesc).wait(0xe?, 0x9672a5b8?, 0x0)
67442026-03-10T19:00:49.277Z /work/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
67452026-03-10T19:00:49.277Zinternal/poll.(*pollDesc).waitRead(...)
67462026-03-10T19:00:49.277Z /work/go/src/internal/poll/fd_poll_runtime.go:89
67472026-03-10T19:00:49.277Zinternal/poll.(*FD).Accept(0xc001287980)
67482026-03-10T19:00:49.277Z /work/go/src/internal/poll/fd_unix.go:611 +0x2ac
67492026-03-10T19:00:49.277Znet.(*netFD).accept(0xc001287980)
67502026-03-10T19:00:49.277Z /work/go/src/net/fd_unix.go:172 +0x29
67512026-03-10T19:00:49.277Znet.(*TCPListener).accept(0xc0075f3500)
67522026-03-10T19:00:49.278Z /work/go/src/net/tcpsock_posix.go:159 +0x1e
67532026-03-10T19:00:49.278Znet.(*TCPListener).Accept(0xc0075f3500)
67542026-03-10T19:00:49.278Z /work/go/src/net/tcpsock.go:327 +0x30
67552026-03-10T19:00:49.278Zgithub.com/cockroachdb/cmux.(*cMux).Serve(0xc00670a780)
67562026-03-10T19:00:49.278Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:124 +0x97
67572026-03-10T19:00:49.278Zgithub.com/cockroachdb/cockroach/pkg/server.startListenRPCAndSQL.func5.2.1()
67582026-03-10T19:00:49.278Z /work/oxidecomputer/cockroach/pkg/server/start_listen.go:139 +0x27
67592026-03-10T19:00:49.278Zsync.(*Once).doSlow(0xc0028aced8?, 0xc0028acf18?)
67602026-03-10T19:00:49.278Z /work/go/src/sync/once.go:74 +0xc2
67612026-03-10T19:00:49.278Zsync.(*Once).Do(...)
67622026-03-10T19:00:49.278Z /work/go/src/sync/once.go:65
67632026-03-10T19:00:49.278Zgithub.com/cockroachdb/cockroach/pkg/server.startListenRPCAndSQL.func5.2({0xc006f4a608?, 0xc006037d70?})
67642026-03-10T19:00:49.278Z /work/oxidecomputer/cockroach/pkg/server/start_listen.go:138 +0x4f
67652026-03-10T19:00:49.279Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
67662026-03-10T19:00:49.279Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
67672026-03-10T19:00:49.279Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
67682026-03-10T19:00:49.279Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
67692026-03-10T19:00:49.279Z
67702026-03-10T19:00:49.279Zgoroutine 10144 [select]:
67712026-03-10T19:00:49.279Zgoogle.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc002063d40)
67722026-03-10T19:00:49.279Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1131 +0x205
67732026-03-10T19:00:49.279Zcreated by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 10169
67742026-03-10T19:00:49.279Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:341 +0x188e
67752026-03-10T19:00:49.279Z
67762026-03-10T19:00:49.279Zgoroutine 10134 [IO wait]:
67772026-03-10T19:00:49.279Zinternal/poll.runtime_pollWait(0x7f784abc3700, 0x72)
67782026-03-10T19:00:49.279Z /work/go/src/runtime/netpoll.go:345 +0x85
67792026-03-10T19:00:49.279Zinternal/poll.(*pollDesc).wait(0xc002052e80?, 0xc0071ec600?, 0x0)
67802026-03-10T19:00:49.279Z /work/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
67812026-03-10T19:00:49.279Zinternal/poll.(*pollDesc).waitRead(...)
67822026-03-10T19:00:49.279Z /work/go/src/internal/poll/fd_poll_runtime.go:89
67832026-03-10T19:00:49.279Zinternal/poll.(*FD).Read(0xc002052e80, {0xc0071ec600, 0x600, 0x600})
67842026-03-10T19:00:49.279Z /work/go/src/internal/poll/fd_unix.go:164 +0x27a
67852026-03-10T19:00:49.279Znet.(*netFD).Read(0xc002052e80, {0xc0071ec600?, 0xc0025c5868?, 0x41f225?})
67862026-03-10T19:00:49.279Z /work/go/src/net/fd_posix.go:55 +0x25
67872026-03-10T19:00:49.279Znet.(*conn).Read(0xc0032909d8, {0xc0071ec600?, 0x7f7848260190?, 0xc03154a8b8?})
67882026-03-10T19:00:49.280Z /work/go/src/net/net.go:185 +0x45
67892026-03-10T19:00:49.280Zgithub.com/cockroachdb/cmux.(*MuxConn).Read(0xc005dfc480, {0xc0071ec600, 0x600, 0x600})
67902026-03-10T19:00:49.280Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x14f
67912026-03-10T19:00:49.280Zcrypto/tls.(*atLeastReader).Read(0xc03154a8b8, {0xc0071ec600?, 0x0?, 0xc03154a8b8?})
67922026-03-10T19:00:49.280Z /work/go/src/crypto/tls/conn.go:806 +0x3b
67932026-03-10T19:00:49.280Zbytes.(*Buffer).ReadFrom(0xc002dd70b0, {0x50b6dc0, 0xc03154a8b8})
67942026-03-10T19:00:49.280Z /work/go/src/bytes/buffer.go:211 +0x98
67952026-03-10T19:00:49.280Zcrypto/tls.(*Conn).readFromUntil(0xc002dd6e08, {0x7f784ca77e58, 0xc005dfc480}, 0xc0025c5990?)
67962026-03-10T19:00:49.280Z /work/go/src/crypto/tls/conn.go:828 +0xde
67972026-03-10T19:00:49.280Zcrypto/tls.(*Conn).readRecordOrCCS(0xc002dd6e08, 0x0)
67982026-03-10T19:00:49.280Z /work/go/src/crypto/tls/conn.go:626 +0x3cf
67992026-03-10T19:00:49.280Zcrypto/tls.(*Conn).readRecord(...)
68002026-03-10T19:00:49.280Z /work/go/src/crypto/tls/conn.go:588
68012026-03-10T19:00:49.280Zcrypto/tls.(*Conn).Read(0xc002dd6e08, {0xc000916000, 0x8000, 0xc0025c5ce8?})
68022026-03-10T19:00:49.280Z /work/go/src/crypto/tls/conn.go:1370 +0x156
68032026-03-10T19:00:49.280Zbufio.(*Reader).Read(0xc004d208a0, {0xc0049084a0, 0x9, 0xc03154a8a0?})
68042026-03-10T19:00:49.280Z /work/go/src/bufio/bufio.go:241 +0x197
68052026-03-10T19:00:49.280Zio.ReadAtLeast({0x50b1100, 0xc004d208a0}, {0xc0049084a0, 0x9, 0x9}, 0x9)
68062026-03-10T19:00:49.280Z /work/go/src/io/io.go:335 +0x90
68072026-03-10T19:00:49.280Zio.ReadFull(...)
68082026-03-10T19:00:49.280Z /work/go/src/io/io.go:354
68092026-03-10T19:00:49.280Zgolang.org/x/net/http2.readFrameHeader({0xc0049084a0, 0x9, 0xc0025c5e10?}, {0x50b1100?, 0xc004d208a0?})
68102026-03-10T19:00:49.280Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x65
68112026-03-10T19:00:49.280Zgolang.org/x/net/http2.(*Framer).ReadFrame(0xc004908460)
68122026-03-10T19:00:49.280Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:501 +0x85
68132026-03-10T19:00:49.280Zgoogle.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc002062d00, 0xc002de4960, 0x431bf68)
68142026-03-10T19:00:49.280Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:611 +0x14e
68152026-03-10T19:00:49.280Zgoogle.golang.org/grpc.(*Server).serveStreams(0xc005fee1c0, {0x5126b60, 0xc002062d00})
68162026-03-10T19:00:49.281Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:906 +0x114
68172026-03-10T19:00:49.281Zgoogle.golang.org/grpc.(*Server).handleRawConn.func1()
68182026-03-10T19:00:49.281Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:848 +0x45
68192026-03-10T19:00:49.281Zcreated by google.golang.org/grpc.(*Server).handleRawConn in goroutine 10149
68202026-03-10T19:00:49.281Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:847 +0x15b
68212026-03-10T19:00:49.281Z
68222026-03-10T19:00:49.281Zgoroutine 9137 [select, 59 minutes]:
68232026-03-10T19:00:49.281Zgithub.com/cockroachdb/cockroach/pkg/util/log.(*fileSink).gcDaemon(0xc003af25b0, {0x50e6118, 0xc004860820})
68242026-03-10T19:00:49.281Z /work/oxidecomputer/cockroach/pkg/util/log/file_log_gc.go:25 +0x79
68252026-03-10T19:00:49.281Zcreated by github.com/cockroachdb/cockroach/pkg/util/log.ApplyConfig in goroutine 9099
68262026-03-10T19:00:49.281Z /work/oxidecomputer/cockroach/pkg/util/log/flags.go:293 +0x10f5
68272026-03-10T19:00:49.281Z
68282026-03-10T19:00:49.281Zgoroutine 9292 [select]:
68292026-03-10T19:00:49.281Zgithub.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1()
68302026-03-10T19:00:49.281Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:71 +0xcb
68312026-03-10T19:00:49.281Zcreated by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 9099
68322026-03-10T19:00:49.281Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:66 +0x58
68332026-03-10T19:00:49.281Z
68342026-03-10T19:00:49.283Zgoroutine 9235 [select, 59 minutes]:
68352026-03-10T19:00:49.283Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc0067f6c08?, 0x0?})
68362026-03-10T19:00:49.283Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
68372026-03-10T19:00:49.283Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
68382026-03-10T19:00:49.283Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
68392026-03-10T19:00:49.287Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
68402026-03-10T19:00:49.288Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
68412026-03-10T19:00:49.288Z
68422026-03-10T19:00:49.288Zgoroutine 9396 [sync.Cond.Wait]:
68432026-03-10T19:00:49.288Zsync.runtime_notifyListWait(0xc0005ac550, 0x3763)
68442026-03-10T19:00:49.288Z /work/go/src/runtime/sema.go:569 +0x159
68452026-03-10T19:00:49.288Zsync.(*Cond).Wait(0x3ae1600?)
68462026-03-10T19:00:49.288Z /work/go/src/sync/cond.go:70 +0x85
68472026-03-10T19:00:49.288Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc0066232f0})
68482026-03-10T19:00:49.288Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
68492026-03-10T19:00:49.288Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
68502026-03-10T19:00:49.288Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
68512026-03-10T19:00:49.288Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
68522026-03-10T19:00:49.288Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
68532026-03-10T19:00:49.288Z
68542026-03-10T19:00:49.288Zgoroutine 9219 [chan receive, 59 minutes]:
68552026-03-10T19:00:49.288Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50e60e0?, 0xc00634c2a0?})
68562026-03-10T19:00:49.288Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
68572026-03-10T19:00:49.288Zruntime/pprof.Do({0x50e6038?, 0x778e7c0?}, {{0xc00030e2c0?, 0xfc5557149f0ea918?, 0xbbee60e7e22501cc?}}, 0xc002883fa8)
68582026-03-10T19:00:49.289Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
68592026-03-10T19:00:49.289Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xa48b3e9fac23155?)
68602026-03-10T19:00:49.289Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
68612026-03-10T19:00:49.289Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9099
68622026-03-10T19:00:49.290Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
68632026-03-10T19:00:49.290Z
68642026-03-10T19:00:49.290Zgoroutine 9415 [sync.Cond.Wait]:
68652026-03-10T19:00:49.290Zsync.runtime_notifyListWait(0xc0005ac550, 0x374c)
68662026-03-10T19:00:49.290Z /work/go/src/runtime/sema.go:569 +0x159
68672026-03-10T19:00:49.290Zsync.(*Cond).Wait(0x3ae1600?)
68682026-03-10T19:00:49.290Z /work/go/src/sync/cond.go:70 +0x85
68692026-03-10T19:00:49.290Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006623710})
68702026-03-10T19:00:49.290Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
68712026-03-10T19:00:49.290Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
68722026-03-10T19:00:49.290Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
68732026-03-10T19:00:49.290Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
68742026-03-10T19:00:49.290Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
68752026-03-10T19:00:49.290Z
68762026-03-10T19:00:49.290Zgoroutine 9870 [select, 59 minutes]:
68772026-03-10T19:00:49.290Zgithub.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run(0xc0013c6000, {0x50e60e0, 0xc0065c93e0})
68782026-03-10T19:00:49.290Z /work/oxidecomputer/cockroach/pkg/internal/client/requestbatcher/batcher.go:475 +0x326
68792026-03-10T19:00:49.291Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
68802026-03-10T19:00:49.291Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
68812026-03-10T19:00:49.291Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
68822026-03-10T19:00:49.291Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
68832026-03-10T19:00:49.291Z
68842026-03-10T19:00:49.291Zgoroutine 10094 [sync.Cond.Wait, 59 minutes]:
68852026-03-10T19:00:49.291Zsync.runtime_notifyListWait(0xc00676e610, 0x3c)
68862026-03-10T19:00:49.291Z /work/go/src/runtime/sema.go:569 +0x159
68872026-03-10T19:00:49.291Zsync.(*Cond).Wait(0x0?)
68882026-03-10T19:00:49.291Z /work/go/src/sync/cond.go:70 +0x85
68892026-03-10T19:00:49.291Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0072ac2a0})
68902026-03-10T19:00:49.291Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
68912026-03-10T19:00:49.291Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
68922026-03-10T19:00:49.291Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
68932026-03-10T19:00:49.291Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
68942026-03-10T19:00:49.291Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
68952026-03-10T19:00:49.291Z
68962026-03-10T19:00:49.291Zgoroutine 10106 [select, 59 minutes]:
68972026-03-10T19:00:49.291Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).startLeaseRenewer.func1({0x50e60e0, 0xc0072ac540})
68982026-03-10T19:00:49.291Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:2256 +0x2ac
68992026-03-10T19:00:49.291Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
69002026-03-10T19:00:49.291Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
69012026-03-10T19:00:49.291Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
69022026-03-10T19:00:49.292Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
69032026-03-10T19:00:49.292Z
69042026-03-10T19:00:49.292Zgoroutine 10066 [sync.Cond.Wait, 59 minutes]:
69052026-03-10T19:00:49.292Zsync.runtime_notifyListWait(0xc00676e610, 0x21)
69062026-03-10T19:00:49.292Z /work/go/src/runtime/sema.go:569 +0x159
69072026-03-10T19:00:49.292Zsync.(*Cond).Wait(0x0?)
69082026-03-10T19:00:49.292Z /work/go/src/sync/cond.go:70 +0x85
69092026-03-10T19:00:49.292Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c9ce0})
69102026-03-10T19:00:49.292Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
69112026-03-10T19:00:49.292Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
69122026-03-10T19:00:49.292Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
69132026-03-10T19:00:49.292Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
69142026-03-10T19:00:49.292Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
69152026-03-10T19:00:49.292Z
69162026-03-10T19:00:49.292Zgoroutine 9284 [chan receive, 59 minutes]:
69172026-03-10T19:00:49.292Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50e60e0?, 0xc005cee6f0?})
69182026-03-10T19:00:49.292Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
69192026-03-10T19:00:49.292Zruntime/pprof.Do({0x50e6038?, 0x778e7c0?}, {{0xc00030e2c0?, 0x0?, 0x778e7c0?}}, 0xc003bb5fa8)
69202026-03-10T19:00:49.292Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
69212026-03-10T19:00:49.292Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
69222026-03-10T19:00:49.292Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
69232026-03-10T19:00:49.293Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9099
69242026-03-10T19:00:49.293Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
69252026-03-10T19:00:49.294Z
69262026-03-10T19:00:49.294Zgoroutine 10067 [sync.Cond.Wait, 59 minutes]:
69272026-03-10T19:00:49.294Zsync.runtime_notifyListWait(0xc00676e610, 0x19)
69282026-03-10T19:00:49.294Z /work/go/src/runtime/sema.go:569 +0x159
69292026-03-10T19:00:49.294Zsync.(*Cond).Wait(0x0?)
69302026-03-10T19:00:49.294Z /work/go/src/sync/cond.go:70 +0x85
69312026-03-10T19:00:49.294Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c9d10})
69322026-03-10T19:00:49.295Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
69332026-03-10T19:00:49.295Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
69342026-03-10T19:00:49.295Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
69352026-03-10T19:00:49.295Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
69362026-03-10T19:00:49.295Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
69372026-03-10T19:00:49.295Z
69382026-03-10T19:00:49.295Zgoroutine 10065 [sync.Cond.Wait, 59 minutes]:
69392026-03-10T19:00:49.299Zsync.runtime_notifyListWait(0xc00676e610, 0x20)
69402026-03-10T19:00:49.299Z /work/go/src/runtime/sema.go:569 +0x159
69412026-03-10T19:00:49.299Zsync.(*Cond).Wait(0x0?)
69422026-03-10T19:00:49.299Z /work/go/src/sync/cond.go:70 +0x85
69432026-03-10T19:00:49.299Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c9cb0})
69442026-03-10T19:00:49.299Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
69452026-03-10T19:00:49.299Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
69462026-03-10T19:00:49.299Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
69472026-03-10T19:00:49.299Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
69482026-03-10T19:00:49.299Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
69492026-03-10T19:00:49.299Z
69502026-03-10T19:00:49.299Zgoroutine 10107 [select]:
69512026-03-10T19:00:49.299Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).startRangefeedUpdater.func1({0x50e60e0, 0xc0072ac570})
69522026-03-10T19:00:49.299Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:2307 +0x28a
69532026-03-10T19:00:49.299Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
69542026-03-10T19:00:49.299Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
69552026-03-10T19:00:49.299Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
69562026-03-10T19:00:49.300Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
69572026-03-10T19:00:49.300Z
69582026-03-10T19:00:49.300Zgoroutine 9335 [select]:
69592026-03-10T19:00:49.300Zgoogle.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc003811680, 0x1)
69602026-03-10T19:00:49.300Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:407 +0x113
69612026-03-10T19:00:49.300Zgoogle.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc006aacae0)
69622026-03-10T19:00:49.300Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:534 +0x7b
69632026-03-10T19:00:49.300Zgoogle.golang.org/grpc/internal/transport.NewServerTransport.func2()
69642026-03-10T19:00:49.300Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:332 +0xc6
69652026-03-10T19:00:49.300Zcreated by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 9331
69662026-03-10T19:00:49.300Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:329 +0x184c
69672026-03-10T19:00:49.300Z
69682026-03-10T19:00:49.300Zgoroutine 9307 [select, 59 minutes]:
69692026-03-10T19:00:49.300Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc004376008?, 0xc002c03ce0?})
69702026-03-10T19:00:49.300Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
69712026-03-10T19:00:49.300Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
69722026-03-10T19:00:49.300Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
69732026-03-10T19:00:49.300Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
69742026-03-10T19:00:49.300Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
69752026-03-10T19:00:49.300Z
69762026-03-10T19:00:49.300Zgoroutine 9375 [sync.Cond.Wait]:
69772026-03-10T19:00:49.300Zsync.runtime_notifyListWait(0xc0005ac550, 0x3753)
69782026-03-10T19:00:49.300Z /work/go/src/runtime/sema.go:569 +0x159
69792026-03-10T19:00:49.300Zsync.(*Cond).Wait(0x3ae1600?)
69802026-03-10T19:00:49.300Z /work/go/src/sync/cond.go:70 +0x85
69812026-03-10T19:00:49.300Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006622de0})
69822026-03-10T19:00:49.300Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
69832026-03-10T19:00:49.300Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
69842026-03-10T19:00:49.300Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
69852026-03-10T19:00:49.300Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
69862026-03-10T19:00:49.300Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
69872026-03-10T19:00:49.300Z
69882026-03-10T19:00:49.300Zgoroutine 9575 [select, 59 minutes]:
69892026-03-10T19:00:49.300Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1({0x50e6118, 0xc003c22aa0})
69902026-03-10T19:00:49.300Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:116 +0x157
69912026-03-10T19:00:49.300Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.Group.GoCtx.func3()
69922026-03-10T19:00:49.300Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
69932026-03-10T19:00:49.300Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
69942026-03-10T19:00:49.301Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
69952026-03-10T19:00:49.301Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9573
69962026-03-10T19:00:49.301Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
69972026-03-10T19:00:49.301Z
69982026-03-10T19:00:49.301Zgoroutine 9167 [select]:
69992026-03-10T19:00:49.301Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaScanner).waitAndProcess(0xc001a628c0, {0x50e60e0, 0xc001eac5d0}, {0x17f8de0?, 0x50c6000?, 0x0?}, 0xc0019ca408)
70002026-03-10T19:00:49.301Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scanner.go:224 +0x1f1
70012026-03-10T19:00:49.301Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaScanner).scanLoop.func1.1(0x50c6000?)
70022026-03-10T19:00:49.301Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scanner.go:286 +0x45
70032026-03-10T19:00:49.301Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*storeReplicaVisitor).Visit(0xc0065261e0, 0xc0210c2c80)
70042026-03-10T19:00:49.301Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:436 +0x202
70052026-03-10T19:00:49.301Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaScanner).scanLoop.func1({0x50e60e0, 0xc001eac5d0})
70062026-03-10T19:00:49.301Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scanner.go:284 +0x1a3
70072026-03-10T19:00:49.301Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
70082026-03-10T19:00:49.301Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
70092026-03-10T19:00:49.301Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9437
70102026-03-10T19:00:49.301Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
70112026-03-10T19:00:49.301Z
70122026-03-10T19:00:49.301Zgoroutine 10643 [select]:
70132026-03-10T19:00:49.301Zgoogle.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc0080b3950, {0xc0072688e0, 0x5, 0x5})
70142026-03-10T19:00:49.301Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:191 +0x95
70152026-03-10T19:00:49.301Zgoogle.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc0080b3950, {0xc0072688e0?, 0xc004c0d3b0?, 0xc005ed9048?})
70162026-03-10T19:00:49.301Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:171 +0x12d
70172026-03-10T19:00:49.301Zgoogle.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0080eaa20, {0xc0072688e0?, 0xc005ed90c0?, 0xa418a5?})
70182026-03-10T19:00:49.301Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:485 +0x2c
70192026-03-10T19:00:49.301Zio.ReadAtLeast({0x50bb160, 0xc0080eaa20}, {0xc0072688e0, 0x5, 0x5}, 0x5)
70202026-03-10T19:00:49.301Z /work/go/src/io/io.go:335 +0x90
70212026-03-10T19:00:49.301Zio.ReadFull(...)
70222026-03-10T19:00:49.301Z /work/go/src/io/io.go:354
70232026-03-10T19:00:49.301Zgoogle.golang.org/grpc/internal/transport.(*Stream).Read(0xc0080ee120, {0xc0072688e0, 0x5, 0x5})
70242026-03-10T19:00:49.301Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:469 +0x96
70252026-03-10T19:00:49.301Zgoogle.golang.org/grpc.(*parser).recvMsg(0xc0072688d0, 0x7fffffff)
70262026-03-10T19:00:49.301Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:559 +0x42
70272026-03-10T19:00:49.301Zgoogle.golang.org/grpc.recvAndDecompress(0x60?, 0xc0080ee120, {0x0, 0x0}, 0x7fffffff, 0xc005ed9378, {0x50ddc20, 0x778e7c0})
70282026-03-10T19:00:49.301Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:690 +0x5a
70292026-03-10T19:00:49.301Zgoogle.golang.org/grpc.recv(0x50c4e80?, {0x7f784b7a48e8, 0xc000567440}, 0xc005ed9300?, {0x0?, 0x0?}, {0x3fa0a40, 0xc01de59a80}, 0x10?, 0xc005ed9378, ...)
70302026-03-10T19:00:49.301Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:756 +0x68
70312026-03-10T19:00:49.301Zgoogle.golang.org/grpc.(*csAttempt).recvMsg(0xc007f0a840, {0x3fa0a40, 0xc01de59a80}, 0x10?)
70322026-03-10T19:00:49.301Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:983 +0x27e
70332026-03-10T19:00:49.302Zgoogle.golang.org/grpc.(*clientStream).RecvMsg.func1(0x41f225?)
70342026-03-10T19:00:49.302Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:834 +0x1f
70352026-03-10T19:00:49.302Zgoogle.golang.org/grpc.(*clientStream).withRetry(0xc00468fc20, 0xc005ed94a8, 0xc005ed9498)
70362026-03-10T19:00:49.302Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:688 +0x2ee
70372026-03-10T19:00:49.302Zgoogle.golang.org/grpc.(*clientStream).RecvMsg(0xc00468fc20, {0x3fa0a40?, 0xc01de59a80?})
70382026-03-10T19:00:49.302Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:833 +0x107
70392026-03-10T19:00:49.306Zgithub.com/cockroachdb/cockroach/pkg/util/tracing.(*tracingClientStream).RecvMsg(0xc0075f2100, {0x3fa0a40?, 0xc01de59a80?})
70402026-03-10T19:00:49.306Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:441 +0x31
70412026-03-10T19:00:49.306Zgithub.com/cockroachdb/cockroach/pkg/roachpb.(*internalRangeFeedClient).Recv(0xc006c193e0)
70422026-03-10T19:00:49.306Z /work/oxidecomputer/cockroach/pkg/roachpb/api.pb.go:9289 +0x46
70432026-03-10T19:00:49.306Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed.func3()
70442026-03-10T19:00:49.307Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:516 +0x25
70452026-03-10T19:00:49.307Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*stuckRangeFeedCanceler).do(0xc0080b3810, 0xc005ed9a50)
70462026-03-10T19:00:49.307Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed_canceler.go:100 +0x1b3
70472026-03-10T19:00:49.307Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed(0xc0021b6f08, {0x50e6118?, 0xc003810050?}, {{0xc0065d6538, 0x1, 0x8}, {0xc0065d6540, 0x1, 0x8}}, {0x189b8cc3dbda5d50, ...}, ...)
70482026-03-10T19:00:49.307Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:515 +0xc98
70492026-03-10T19:00:49.307Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).partialRangeFeed(0xc0021b6f08, {0x50e6118, 0xc003810050}, 0xc003810000, {{0xc0065d6538, 0x1, 0x8}, {0xc0065d6540, 0x1, 0x8}}, ...)
70502026-03-10T19:00:49.307Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:336 +0x745
70512026-03-10T19:00:49.307Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.1({0x50e6118?, 0xc003810050?})
70522026-03-10T19:00:49.308Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:120 +0xa9
70532026-03-10T19:00:49.308Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.Group.GoCtx.2()
70542026-03-10T19:00:49.308Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
70552026-03-10T19:00:49.308Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
70562026-03-10T19:00:49.308Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
70572026-03-10T19:00:49.308Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9149
70582026-03-10T19:00:49.308Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
70592026-03-10T19:00:49.308Z
70602026-03-10T19:00:49.308Zgoroutine 10329 [sync.Cond.Wait, 59 minutes]:
70612026-03-10T19:00:49.308Zsync.runtime_notifyListWait(0xc00518f9d0, 0x3)
70622026-03-10T19:00:49.308Z /work/go/src/runtime/sema.go:569 +0x159
70632026-03-10T19:00:49.308Zsync.(*Cond).Wait(0xc0078e37d8?)
70642026-03-10T19:00:49.308Z /work/go/src/sync/cond.go:70 +0x85
70652026-03-10T19:00:49.308Zgithub.com/cockroachdb/cockroach/pkg/sql.(*StmtBuf).CurCmd(0xc00483d0e8)
70662026-03-10T19:00:49.308Z /work/oxidecomputer/cockroach/pkg/sql/conn_io.go:453 +0xad
70672026-03-10T19:00:49.308Zgithub.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd(0xc00486c008)
70682026-03-10T19:00:49.308Z /work/oxidecomputer/cockroach/pkg/sql/conn_executor.go:1984 +0x66
70692026-03-10T19:00:49.308Zgithub.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0xc00486c008, {0x50e6118, 0xc0069e6f50}, 0x9df5ef4f?, {0x5400, 0x15000, 0xc002c5c0f0, 0x0, 0x0}, 0xc003fcccf0)
70702026-03-10T19:00:49.308Z /work/oxidecomputer/cockroach/pkg/sql/conn_executor.go:1958 +0x265
70712026-03-10T19:00:49.308Zgithub.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn(0x207bbf?, {0x50e6118?, 0xc0069e6f50?}, {0x3?}, {0x5400, 0x15000, 0xc002c5c0f0, 0x0, 0x0}, 0xc003fcccf0)
70722026-03-10T19:00:49.309Z /work/oxidecomputer/cockroach/pkg/sql/conn_executor.go:763 +0xe6
70732026-03-10T19:00:49.309Zgithub.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1()
70742026-03-10T19:00:49.309Z /work/oxidecomputer/cockroach/pkg/sql/pgwire/conn.go:730 +0x3ee
70752026-03-10T19:00:49.309Zcreated by github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync in goroutine 10302
70762026-03-10T19:00:49.309Z /work/oxidecomputer/cockroach/pkg/sql/pgwire/conn.go:641 +0x247
70772026-03-10T19:00:49.309Z
70782026-03-10T19:00:49.309Zgoroutine 9300 [select, 59 minutes]:
70792026-03-10T19:00:49.309Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc002c06308?, 0x1?})
70802026-03-10T19:00:49.309Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
70812026-03-10T19:00:49.309Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
70822026-03-10T19:00:49.309Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
70832026-03-10T19:00:49.309Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
70842026-03-10T19:00:49.309Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
70852026-03-10T19:00:49.309Z
70862026-03-10T19:00:49.310Zgoroutine 9272 [select]:
70872026-03-10T19:00:49.310Zgithub.com/cockroachdb/cockroach/pkg/gossip.newInfoStore.func1({0xc002a80308?, 0xc004f9a2d0?})
70882026-03-10T19:00:49.310Z /work/oxidecomputer/cockroach/pkg/gossip/infostore.go:192 +0x74
70892026-03-10T19:00:49.310Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
70902026-03-10T19:00:49.310Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
70912026-03-10T19:00:49.310Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
70922026-03-10T19:00:49.310Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
70932026-03-10T19:00:49.310Z
70942026-03-10T19:00:49.310Zgoroutine 9369 [sync.Cond.Wait]:
70952026-03-10T19:00:49.310Zsync.runtime_notifyListWait(0xc0005ac550, 0x374a)
70962026-03-10T19:00:49.310Z /work/go/src/runtime/sema.go:569 +0x159
70972026-03-10T19:00:49.310Zsync.(*Cond).Wait(0x3ae1600?)
70982026-03-10T19:00:49.310Z /work/go/src/sync/cond.go:70 +0x85
70992026-03-10T19:00:49.310Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006622cc0})
71002026-03-10T19:00:49.310Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
71012026-03-10T19:00:49.310Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
71022026-03-10T19:00:49.310Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
71032026-03-10T19:00:49.310Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
71042026-03-10T19:00:49.310Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
71052026-03-10T19:00:49.310Z
71062026-03-10T19:00:49.310Zgoroutine 9397 [sync.Cond.Wait]:
71072026-03-10T19:00:49.310Zsync.runtime_notifyListWait(0xc0005ac550, 0x3757)
71082026-03-10T19:00:49.310Z /work/go/src/runtime/sema.go:569 +0x159
71092026-03-10T19:00:49.310Zsync.(*Cond).Wait(0x3ae1600?)
71102026-03-10T19:00:49.310Z /work/go/src/sync/cond.go:70 +0x85
71112026-03-10T19:00:49.310Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006623320})
71122026-03-10T19:00:49.310Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
71132026-03-10T19:00:49.310Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
71142026-03-10T19:00:49.310Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
71152026-03-10T19:00:49.310Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
71162026-03-10T19:00:49.310Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
71172026-03-10T19:00:49.310Z
71182026-03-10T19:00:49.310Zgoroutine 9194 [chan receive, 59 minutes]:
71192026-03-10T19:00:49.310Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50e60e0?, 0xc005cee090?})
71202026-03-10T19:00:49.310Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
71212026-03-10T19:00:49.310Zruntime/pprof.Do({0x50e6038?, 0x778e7c0?}, {{0xc00030e2c0?, 0x10000c00494fd80?, 0x5126b60?}}, 0xc00004ffa8)
71222026-03-10T19:00:49.310Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
71232026-03-10T19:00:49.310Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc00004ffb8?)
71242026-03-10T19:00:49.311Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
71252026-03-10T19:00:49.311Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9099
71262026-03-10T19:00:49.311Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
71272026-03-10T19:00:49.311Z
71282026-03-10T19:00:49.311Zgoroutine 10191 [select, 59 minutes]:
71292026-03-10T19:00:49.311Zgoogle.golang.org/grpc.newClientStreamWithParams.func4()
71302026-03-10T19:00:49.311Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:341 +0x8c
71312026-03-10T19:00:49.311Zcreated by google.golang.org/grpc.newClientStreamWithParams in goroutine 10156
71322026-03-10T19:00:49.311Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:340 +0xb2a
71332026-03-10T19:00:49.311Z
71342026-03-10T19:00:49.311Zgoroutine 9259 [chan receive, 59 minutes]:
71352026-03-10T19:00:49.311Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50e60e0?, 0xc00634ca80?})
71362026-03-10T19:00:49.311Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
71372026-03-10T19:00:49.312Zruntime/pprof.Do({0x50e6038?, 0x778e7c0?}, {{0xc00030e2c0?, 0x0?, 0x0?}}, 0xc0022c57a8)
71382026-03-10T19:00:49.312Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
71392026-03-10T19:00:49.316Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
71402026-03-10T19:00:49.316Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
71412026-03-10T19:00:49.316Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9099
71422026-03-10T19:00:49.317Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
71432026-03-10T19:00:49.317Z
71442026-03-10T19:00:49.317Zgoroutine 9281 [select]:
71452026-03-10T19:00:49.317Zgithub.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue.func2()
71462026-03-10T19:00:49.317Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:366 +0x85
71472026-03-10T19:00:49.317Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue in goroutine 9099
71482026-03-10T19:00:49.317Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:363 +0x3ed
71492026-03-10T19:00:49.317Z
71502026-03-10T19:00:49.317Zgoroutine 10644 [select, 59 minutes]:
71512026-03-10T19:00:49.317Zgoogle.golang.org/grpc.newClientStreamWithParams.func4()
71522026-03-10T19:00:49.317Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:341 +0x8c
71532026-03-10T19:00:49.317Zcreated by google.golang.org/grpc.newClientStreamWithParams in goroutine 10643
71542026-03-10T19:00:49.317Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:340 +0xb2a
71552026-03-10T19:00:49.317Z
71562026-03-10T19:00:49.317Zgoroutine 10115 [select, 2 minutes]:
71572026-03-10T19:00:49.317Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/protectedts/ptcache.(*Cache).periodicallyRefreshProtectedtsCache(0xc007475340, {0x50e60e0, 0xc0072ac960})
71582026-03-10T19:00:49.317Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/protectedts/ptcache/cache.go:175 +0x216
71592026-03-10T19:00:49.317Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
71602026-03-10T19:00:49.317Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
71612026-03-10T19:00:49.317Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
71622026-03-10T19:00:49.317Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
71632026-03-10T19:00:49.317Z
71642026-03-10T19:00:49.317Zgoroutine 3451788 [select]:
71652026-03-10T19:00:49.317Zgithub.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1()
71662026-03-10T19:00:49.318Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:71 +0xcb
71672026-03-10T19:00:49.318Zcreated by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 9409
71682026-03-10T19:00:49.318Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:66 +0x58
71692026-03-10T19:00:49.318Z
71702026-03-10T19:00:49.318Zgoroutine 9433 [select, 59 minutes]:
71712026-03-10T19:00:49.318Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Start.func3({0xc0026f7808?, 0xc001b30930?})
71722026-03-10T19:00:49.318Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:1978 +0xb2
71732026-03-10T19:00:49.318Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
71742026-03-10T19:00:49.318Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
71752026-03-10T19:00:49.318Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
71762026-03-10T19:00:49.318Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
71772026-03-10T19:00:49.318Z
71782026-03-10T19:00:49.318Zgoroutine 10099 [select]:
71792026-03-10T19:00:49.318Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).raftTickLoop(0xc007076008, {0x2773?, 0x0?})
71802026-03-10T19:00:49.318Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store_raft.go:618 +0x145
71812026-03-10T19:00:49.318Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
71822026-03-10T19:00:49.318Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
71832026-03-10T19:00:49.318Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
71842026-03-10T19:00:49.318Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
71852026-03-10T19:00:49.318Z
71862026-03-10T19:00:49.318Zgoroutine 10114 [select]:
71872026-03-10T19:00:49.318Zgithub.com/cockroachdb/cockroach/pkg/server.startSampleEnvironment.func1({0x50e60e0, 0xc0072ac930})
71882026-03-10T19:00:49.318Z /work/oxidecomputer/cockroach/pkg/server/env_sampler.go:138 +0x1ad
71892026-03-10T19:00:49.318Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
71902026-03-10T19:00:49.318Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
71912026-03-10T19:00:49.318Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
71922026-03-10T19:00:49.318Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
71932026-03-10T19:00:49.318Z
71942026-03-10T19:00:49.318Zgoroutine 10485 [select]:
71952026-03-10T19:00:49.318Zgoogle.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc004f9bae0, {0xc004b207c0, 0x5, 0x5})
71962026-03-10T19:00:49.318Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:191 +0x95
71972026-03-10T19:00:49.318Zgoogle.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc004f9bae0, {0xc004b207c0?, 0xc004046d98?, 0xc001f6f048?})
71982026-03-10T19:00:49.318Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:171 +0x12d
71992026-03-10T19:00:49.318Zgoogle.golang.org/grpc/internal/transport.(*transportReader).Read(0xc006fe60c0, {0xc004b207c0?, 0xc001f6f0c0?, 0xa418a5?})
72002026-03-10T19:00:49.318Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:485 +0x2c
72012026-03-10T19:00:49.318Zio.ReadAtLeast({0x50bb160, 0xc006fe60c0}, {0xc004b207c0, 0x5, 0x5}, 0x5)
72022026-03-10T19:00:49.318Z /work/go/src/io/io.go:335 +0x90
72032026-03-10T19:00:49.318Zio.ReadFull(...)
72042026-03-10T19:00:49.319Z /work/go/src/io/io.go:354
72052026-03-10T19:00:49.319Zgoogle.golang.org/grpc/internal/transport.(*Stream).Read(0xc00480be60, {0xc004b207c0, 0x5, 0x5})
72062026-03-10T19:00:49.319Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:469 +0x96
72072026-03-10T19:00:49.319Zgoogle.golang.org/grpc.(*parser).recvMsg(0xc004b207b0, 0x7fffffff)
72082026-03-10T19:00:49.319Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:559 +0x42
72092026-03-10T19:00:49.320Zgoogle.golang.org/grpc.recvAndDecompress(0x60?, 0xc00480be60, {0x0, 0x0}, 0x7fffffff, 0xc001f6f378, {0x50ddc20, 0x778e7c0})
72102026-03-10T19:00:49.320Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:690 +0x5a
72112026-03-10T19:00:49.320Zgoogle.golang.org/grpc.recv(0x50c4e80?, {0x7f784b7a48e8, 0xc000567440}, 0x2b?, {0x0?, 0x0?}, {0x3fa0a40, 0xc02c675d60}, 0x10?, 0xc001f6f378, ...)
72122026-03-10T19:00:49.320Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:756 +0x68
72132026-03-10T19:00:49.320Zgoogle.golang.org/grpc.(*csAttempt).recvMsg(0xc004c188f0, {0x3fa0a40, 0xc02c675d60}, 0x10?)
72142026-03-10T19:00:49.320Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:983 +0x27e
72152026-03-10T19:00:49.320Zgoogle.golang.org/grpc.(*clientStream).RecvMsg.func1(0x41f225?)
72162026-03-10T19:00:49.320Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:834 +0x1f
72172026-03-10T19:00:49.320Zgoogle.golang.org/grpc.(*clientStream).withRetry(0xc00480bc20, 0xc001f6f4a8, 0xc001f6f498)
72182026-03-10T19:00:49.320Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:688 +0x2ee
72192026-03-10T19:00:49.320Zgoogle.golang.org/grpc.(*clientStream).RecvMsg(0xc00480bc20, {0x3fa0a40?, 0xc02c675d60?})
72202026-03-10T19:00:49.320Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:833 +0x107
72212026-03-10T19:00:49.320Zgithub.com/cockroachdb/cockroach/pkg/util/tracing.(*tracingClientStream).RecvMsg(0xc004c14d00, {0x3fa0a40?, 0xc02c675d60?})
72222026-03-10T19:00:49.320Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:441 +0x31
72232026-03-10T19:00:49.320Zgithub.com/cockroachdb/cockroach/pkg/roachpb.(*internalRangeFeedClient).Recv(0xc005efe6e0)
72242026-03-10T19:00:49.320Z /work/oxidecomputer/cockroach/pkg/roachpb/api.pb.go:9289 +0x46
72252026-03-10T19:00:49.320Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed.func3()
72262026-03-10T19:00:49.320Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:516 +0x25
72272026-03-10T19:00:49.320Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*stuckRangeFeedCanceler).do(0xc004f9b9a0, 0xc001f6fa50)
72282026-03-10T19:00:49.320Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed_canceler.go:100 +0x1b3
72292026-03-10T19:00:49.320Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed(0xc0021b6f08, {0x50e6118?, 0xc004f9b8b0?}, {{0xc005b260b0, 0x1, 0x8}, {0xc005b260b8, 0x1, 0x8}}, {0x189b8cc3e6444b58, ...}, ...)
72302026-03-10T19:00:49.320Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:515 +0xc98
72312026-03-10T19:00:49.321Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).partialRangeFeed(0xc0021b6f08, {0x50e6118, 0xc004f9b8b0}, 0xc004f9b860, {{0xc005b260b0, 0x1, 0x8}, {0xc005b260b8, 0x1, 0x8}}, ...)
72322026-03-10T19:00:49.321Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:336 +0x745
72332026-03-10T19:00:49.321Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.1({0x50e6118?, 0xc004f9b8b0?})
72342026-03-10T19:00:49.321Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:120 +0xa9
72352026-03-10T19:00:49.321Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.Group.GoCtx.2()
72362026-03-10T19:00:49.321Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
72372026-03-10T19:00:49.321Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
72382026-03-10T19:00:49.321Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
72392026-03-10T19:00:49.325Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10483
72402026-03-10T19:00:49.325Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
72412026-03-10T19:00:49.325Z
72422026-03-10T19:00:49.325Zgoroutine 9258 [chan receive, 59 minutes]:
72432026-03-10T19:00:49.325Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50e60e0?, 0xc00634cb40?})
72442026-03-10T19:00:49.325Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
72452026-03-10T19:00:49.325Zruntime/pprof.Do({0x50e6038?, 0x778e7c0?}, {{0xc00030e2c0?, 0x0?, 0x0?}}, 0xc0022c4fa8)
72462026-03-10T19:00:49.325Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
72472026-03-10T19:00:49.325Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
72482026-03-10T19:00:49.325Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
72492026-03-10T19:00:49.326Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9099
72502026-03-10T19:00:49.326Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
72512026-03-10T19:00:49.327Z
72522026-03-10T19:00:49.327Zgoroutine 10100 [select]:
72532026-03-10T19:00:49.327Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).coalescedHeartbeatsLoop(0xc007076008, {0x50e60e0, 0xc0072ac3c0})
72542026-03-10T19:00:49.327Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store_raft.go:678 +0xec
72552026-03-10T19:00:49.327Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
72562026-03-10T19:00:49.327Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
72572026-03-10T19:00:49.327Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
72582026-03-10T19:00:49.327Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
72592026-03-10T19:00:49.327Z
72602026-03-10T19:00:49.327Zgoroutine 10204 [select]:
72612026-03-10T19:00:49.327Zgithub.com/cockroachdb/cockroach/pkg/sql/stats.(*Refresher).Start.func1({0x50e60e0, 0xc005c71830})
72622026-03-10T19:00:49.327Z /work/oxidecomputer/cockroach/pkg/sql/stats/automatic_stats.go:374 +0x1b9
72632026-03-10T19:00:49.327Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
72642026-03-10T19:00:49.327Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
72652026-03-10T19:00:49.327Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
72662026-03-10T19:00:49.327Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
72672026-03-10T19:00:49.327Z
72682026-03-10T19:00:49.327Zgoroutine 9195 [chan receive, 59 minutes]:
72692026-03-10T19:00:49.327Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50e60e0?, 0xc00634c1b0?})
72702026-03-10T19:00:49.327Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
72712026-03-10T19:00:49.327Zruntime/pprof.Do({0x50e6038?, 0x778e7c0?}, {{0xc00030e2c0?, 0x10000c0046027c0?, 0x5126b60?}}, 0xc00004cfa8)
72722026-03-10T19:00:49.327Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
72732026-03-10T19:00:49.327Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc00004cfb8?)
72742026-03-10T19:00:49.327Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
72752026-03-10T19:00:49.327Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9099
72762026-03-10T19:00:49.327Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
72772026-03-10T19:00:49.327Z
72782026-03-10T19:00:49.327Zgoroutine 10227 [select]:
72792026-03-10T19:00:49.327Zgithub.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10({0x50e60e0?, 0xc005c71ce0?})
72802026-03-10T19:00:49.327Z /work/oxidecomputer/cockroach/pkg/jobs/registry.go:866 +0x1bd
72812026-03-10T19:00:49.327Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
72822026-03-10T19:00:49.327Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
72832026-03-10T19:00:49.327Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
72842026-03-10T19:00:49.327Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
72852026-03-10T19:00:49.327Z
72862026-03-10T19:00:49.327Zgoroutine 9257 [chan receive, 59 minutes]:
72872026-03-10T19:00:49.327Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50e60e0?, 0xc005cee510?})
72882026-03-10T19:00:49.327Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
72892026-03-10T19:00:49.327Zruntime/pprof.Do({0x50e6038?, 0x778e7c0?}, {{0xc00030e2c0?, 0xc001e80b00?, 0x41f225?}}, 0xc0030c87a8)
72902026-03-10T19:00:49.328Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
72912026-03-10T19:00:49.328Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
72922026-03-10T19:00:49.328Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
72932026-03-10T19:00:49.328Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9099
72942026-03-10T19:00:49.328Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
72952026-03-10T19:00:49.328Z
72962026-03-10T19:00:49.328Zgoroutine 10101 [select, 59 minutes]:
72972026-03-10T19:00:49.328Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Start.func3({0xc00733e008?, 0x0?})
72982026-03-10T19:00:49.328Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:1978 +0xb2
72992026-03-10T19:00:49.328Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
73002026-03-10T19:00:49.328Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
73012026-03-10T19:00:49.328Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
73022026-03-10T19:00:49.329Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
73032026-03-10T19:00:49.329Z
73042026-03-10T19:00:49.329Zgoroutine 10102 [select]:
73052026-03-10T19:00:49.329Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).startGossip.func3({0x50e60e0, 0xc0072ac480})
73062026-03-10T19:00:49.329Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:2152 +0x225
73072026-03-10T19:00:49.329Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
73082026-03-10T19:00:49.329Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
73092026-03-10T19:00:49.329Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
73102026-03-10T19:00:49.329Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
73112026-03-10T19:00:49.329Z
73122026-03-10T19:00:49.329Zgoroutine 9187 [select, 59 minutes]:
73132026-03-10T19:00:49.329Zgithub.com/cockroachdb/cockroach/pkg/util/log.(*fileSink).gcDaemon(0xc003af2750, {0x50e6118, 0xc004860820})
73142026-03-10T19:00:49.329Z /work/oxidecomputer/cockroach/pkg/util/log/file_log_gc.go:25 +0x79
73152026-03-10T19:00:49.329Zcreated by github.com/cockroachdb/cockroach/pkg/util/log.ApplyConfig in goroutine 9099
73162026-03-10T19:00:49.329Z /work/oxidecomputer/cockroach/pkg/util/log/flags.go:293 +0x10f5
73172026-03-10T19:00:49.329Z
73182026-03-10T19:00:49.329Zgoroutine 10103 [select]:
73192026-03-10T19:00:49.329Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).startGossip.func3({0x50e60e0, 0xc0072ac4b0})
73202026-03-10T19:00:49.329Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:2152 +0x225
73212026-03-10T19:00:49.329Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
73222026-03-10T19:00:49.329Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
73232026-03-10T19:00:49.330Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
73242026-03-10T19:00:49.330Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
73252026-03-10T19:00:49.330Z
73262026-03-10T19:00:49.330Zgoroutine 9191 [chan receive, 59 minutes]:
73272026-03-10T19:00:49.330Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50e60e0?, 0xc00634c030?})
73282026-03-10T19:00:49.330Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
73292026-03-10T19:00:49.330Zruntime/pprof.Do({0x50e6038?, 0x778e7c0?}, {{0xc00030e2c0?, 0x100000000000000?, 0xc0000ae7d0?}}, 0xc0000ae7a8)
73302026-03-10T19:00:49.330Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
73312026-03-10T19:00:49.330Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
73322026-03-10T19:00:49.330Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
73332026-03-10T19:00:49.330Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9099
73342026-03-10T19:00:49.330Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
73352026-03-10T19:00:49.330Z
73362026-03-10T19:00:49.330Zgoroutine 10344 [IO wait, 59 minutes]:
73372026-03-10T19:00:49.330Zinternal/poll.runtime_pollWait(0x7f784b7f26c0, 0x72)
73382026-03-10T19:00:49.330Z /work/go/src/runtime/netpoll.go:345 +0x85
73392026-03-10T19:00:49.336Zinternal/poll.(*pollDesc).wait(0xf?, 0xc0022b1c90?, 0x0)
73402026-03-10T19:00:49.336Z /work/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
73412026-03-10T19:00:49.336Zinternal/poll.(*pollDesc).waitRead(...)
73422026-03-10T19:00:49.336Z /work/go/src/internal/poll/fd_poll_runtime.go:89
73432026-03-10T19:00:49.336Zinternal/poll.(*FD).Accept(0xc001287a00)
73442026-03-10T19:00:49.336Z /work/go/src/internal/poll/fd_unix.go:611 +0x2ac
73452026-03-10T19:00:49.336Znet.(*netFD).accept(0xc001287a00)
73462026-03-10T19:00:49.336Z /work/go/src/net/fd_unix.go:172 +0x29
73472026-03-10T19:00:49.336Znet.(*TCPListener).accept(0xc0075f3520)
73482026-03-10T19:00:49.336Z /work/go/src/net/tcpsock_posix.go:159 +0x1e
73492026-03-10T19:00:49.336Znet.(*TCPListener).Accept(0xc0075f3520)
73502026-03-10T19:00:49.336Z /work/go/src/net/tcpsock.go:327 +0x30
73512026-03-10T19:00:49.336Zgithub.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith(0xc0031d5bb0, {0x50e60e0, 0xc0030c2630}, 0xc003659b80, {0x50d9250, 0xc0075f3520}, 0xc004e7bae8)
73522026-03-10T19:00:49.336Z /work/oxidecomputer/cockroach/pkg/util/netutil/net.go:138 +0x77
73532026-03-10T19:00:49.336Zgithub.com/cockroachdb/cockroach/pkg/server.(*SQLServer).startServeSQL.func1({0x50e60e0, 0xc0030c2630})
73542026-03-10T19:00:49.336Z /work/oxidecomputer/cockroach/pkg/server/server_sql.go:1374 +0xcb
73552026-03-10T19:00:49.337Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
73562026-03-10T19:00:49.337Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
73572026-03-10T19:00:49.337Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
73582026-03-10T19:00:49.337Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
73592026-03-10T19:00:49.337Z
73602026-03-10T19:00:49.337Zgoroutine 10233 [select]:
73612026-03-10T19:00:49.337Zgoogle.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000fb10e0, 0x1)
73622026-03-10T19:00:49.337Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:407 +0x113
73632026-03-10T19:00:49.337Zgoogle.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0044c7320)
73642026-03-10T19:00:49.338Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:534 +0x7b
73652026-03-10T19:00:49.338Zgoogle.golang.org/grpc/internal/transport.newHTTP2Client.func3()
73662026-03-10T19:00:49.338Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:415 +0x67
73672026-03-10T19:00:49.338Zcreated by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 10158
73682026-03-10T19:00:49.338Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:413 +0x1dcb
73692026-03-10T19:00:49.338Z
73702026-03-10T19:00:49.338Zgoroutine 10325 [select]:
73712026-03-10T19:00:49.338Zgithub.com/cockroachdb/cockroach/pkg/kv/kvprober.(*Prober).Start.func1.1({0x50e60e0, 0xc002e688d0})
73722026-03-10T19:00:49.338Z /work/oxidecomputer/cockroach/pkg/kv/kvprober/kvprober.go:293 +0x314
73732026-03-10T19:00:49.338Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
73742026-03-10T19:00:49.338Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
73752026-03-10T19:00:49.338Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
73762026-03-10T19:00:49.338Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
73772026-03-10T19:00:49.338Z
73782026-03-10T19:00:49.338Zgoroutine 9186 [select, 59 minutes]:
73792026-03-10T19:00:49.338Zgithub.com/cockroachdb/cockroach/pkg/util/log.(*fileSink).gcDaemon(0xc003af2680, {0x50e6118, 0xc004860820})
73802026-03-10T19:00:49.338Z /work/oxidecomputer/cockroach/pkg/util/log/file_log_gc.go:25 +0x79
73812026-03-10T19:00:49.338Zcreated by github.com/cockroachdb/cockroach/pkg/util/log.ApplyConfig in goroutine 9099
73822026-03-10T19:00:49.338Z /work/oxidecomputer/cockroach/pkg/util/log/flags.go:293 +0x10f5
73832026-03-10T19:00:49.338Z
73842026-03-10T19:00:49.338Zgoroutine 10468 [select]:
73852026-03-10T19:00:49.339Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).outputLoop(0xc003c31420, {0x50e6118, 0xc002d6a5f0})
73862026-03-10T19:00:49.339Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:310 +0x247
73872026-03-10T19:00:49.339Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).runOutputLoop(0xc003c31420, {0x50e60e0, 0xc003267ce0}, 0x1b?)
73882026-03-10T19:00:49.339Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:335 +0xba
73892026-03-10T19:00:49.339Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run.func1({0x50e60e0?, 0xc003267ce0?})
73902026-03-10T19:00:49.339Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:297 +0x46
73912026-03-10T19:00:49.339Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
73922026-03-10T19:00:49.339Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
73932026-03-10T19:00:49.339Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9817
73942026-03-10T19:00:49.339Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
73952026-03-10T19:00:49.339Z
73962026-03-10T19:00:49.339Zgoroutine 10469 [semacquire, 59 minutes]:
73972026-03-10T19:00:49.339Zsync.runtime_Semacquire(0xc0016ddbc0?)
73982026-03-10T19:00:49.339Z /work/go/src/runtime/sema.go:62 +0x25
73992026-03-10T19:00:49.339Zsync.(*WaitGroup).Wait(0x44f7e0?)
74002026-03-10T19:00:49.339Z /work/go/src/sync/waitgroup.go:116 +0x48
74012026-03-10T19:00:49.339Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc000790bc0)
74022026-03-10T19:00:49.339Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
74032026-03-10T19:00:49.339Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc000790bc0?, {0x50e6118?, 0xc002cfb9a0?}})
74042026-03-10T19:00:49.339Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
74052026-03-10T19:00:49.339Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed(0xc0021b6f08, {0x50e6118?, 0xc002d6a780?}, {0xc003288b40, 0x1, 0xc003671418?}, {0x2?, 0x8?, 0x0?}, 0x0, ...)
74062026-03-10T19:00:49.340Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:139 +0x6c5
74072026-03-10T19:00:49.340Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*dbAdapter).RangeFeed(0xc001207e40?, {0x50e6118?, 0xc002d6a780?}, {0xc003288b40?, 0x50e6118?, 0xc002d6a0a0?}, {0xc002d7d3e0?, 0x41fe5a0?, 0xc0?}, 0x0, ...)
74082026-03-10T19:00:49.340Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/db_adapter.go:78 +0x3e
74092026-03-10T19:00:49.340Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func1({0x50e6118?, 0xc002d6a780?})
74102026-03-10T19:00:49.340Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:285 +0x6c
74112026-03-10T19:00:49.340Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
74122026-03-10T19:00:49.340Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
74132026-03-10T19:00:49.340Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
74142026-03-10T19:00:49.340Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
74152026-03-10T19:00:49.340Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10210
74162026-03-10T19:00:49.340Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
74172026-03-10T19:00:49.340Z
74182026-03-10T19:00:49.340Zgoroutine 10467 [chan receive, 59 minutes]:
74192026-03-10T19:00:49.340Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).rangeFeedWithRangeID(0xc0019cd408, 0x3dbc860?, 0xc00523fc20, {0x7f784cb862a8, 0xc0067c1f50})
74202026-03-10T19:00:49.340Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:249 +0x76b
74212026-03-10T19:00:49.340Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed(...)
74222026-03-10T19:00:49.340Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:149
74232026-03-10T19:00:49.340Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).RangeFeed(0xc001668008, 0xc00523fc20, {0x5119e20, 0xc0067c1f50})
74242026-03-10T19:00:49.340Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:3082 +0x16f
74252026-03-10T19:00:49.340Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).RangeFeed(0xc006849e00, 0xc00523fc20, {0x5119e20, 0xc0067c1f50})
74262026-03-10T19:00:49.340Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/stores.go:216 +0x19a
74272026-03-10T19:00:49.340Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed(0x7366f90?, 0x403a5e0?, {0x5119e20, 0xc0067c1f50})
74282026-03-10T19:00:49.341Z /work/oxidecomputer/cockroach/pkg/server/node.go:1232 +0x28
74292026-03-10T19:00:49.341Zgithub.com/cockroachdb/cockroach/pkg/roachpb._Internal_RangeFeed_Handler({0x403a5e0, 0xc001668908}, {0x5114010, 0xc001207ec0})
74302026-03-10T19:00:49.341Z /work/oxidecomputer/cockroach/pkg/roachpb/api.pb.go:9523 +0x107
74312026-03-10T19:00:49.341Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.StreamServerInterceptor.func14({0x403a5e0, 0xc001668908}, {0x51147c0, 0xc0066152b0}, 0xc0000133c8, 0x4319298)
74322026-03-10T19:00:49.341Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:205 +0x35b
74332026-03-10T19:00:49.341Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x403a5e0?, 0xc001668908?}, {0x51147c0?, 0xc0066152b0?})
74342026-03-10T19:00:49.341Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1408 +0x59
74352026-03-10T19:00:49.341Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func4({0x403a5e0, 0xc001668908}, {0x51147c0, 0xc0066152b0}, 0xc004d95918?, 0xc0033490c0)
74362026-03-10T19:00:49.341Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:247 +0x6c
74372026-03-10T19:00:49.341Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x403a5e0?, 0xc001668908?}, {0x51147c0?, 0xc0066152b0?})
74382026-03-10T19:00:49.341Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
74392026-03-10T19:00:49.345Zgithub.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({{{0x2b?}}}, {0x403a5e0, 0xc001668908}, {0x51147c0, 0xc0066152b0}, 0xc0000133c8, 0xc0033490c0)
74402026-03-10T19:00:49.345Z /work/oxidecomputer/cockroach/pkg/rpc/auth.go:98 +0x19b
74412026-03-10T19:00:49.345Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x403a5e0?, 0xc001668908?}, {0x51147c0?, 0xc0066152b0?})
74422026-03-10T19:00:49.345Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
74432026-03-10T19:00:49.345Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func2.1({0x66?, 0x40?})
74442026-03-10T19:00:49.345Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:216 +0x27
74452026-03-10T19:00:49.345Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc006848400, {0x50e60e0, 0xc0032679b0}, {0xc004d95b28?, 0x416dfb?}, 0xc004d95ac0)
74462026-03-10T19:00:49.345Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:344 +0xcb
74472026-03-10T19:00:49.345Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func2({0x403a5e0, 0xc001668908}, {0x51147c0, 0xc0066152b0}, 0xc0000133c8, 0xc0033490c0)
74482026-03-10T19:00:49.345Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:215 +0xdf
74492026-03-10T19:00:49.345Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x403a5e0?, 0xc001668908?}, {0x51147c0?, 0xc0066152b0?})
74502026-03-10T19:00:49.345Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
74512026-03-10T19:00:49.345Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x403a5e0, 0xc001668908}, {0x51147c0, 0xc0066152b0}, 0xc0000133c8, 0x4319298)
74522026-03-10T19:00:49.345Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1413 +0x108
74532026-03-10T19:00:49.345Zgoogle.golang.org/grpc.(*Server).processStreamingRPC(0xc005fee1c0, {0x5126b60, 0xc00130c340}, 0xc00523fb00, 0xc0077eb920, 0x73e2b80, 0x0)
74542026-03-10T19:00:49.345Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1549 +0xe51
74552026-03-10T19:00:49.345Zgoogle.golang.org/grpc.(*Server).handleStream(0xc005fee1c0, {0x5126b60, 0xc00130c340}, 0xc00523fb00, 0x0)
74562026-03-10T19:00:49.345Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1624 +0x999
74572026-03-10T19:00:49.345Zgoogle.golang.org/grpc.(*Server).serveStreams.func1.2()
74582026-03-10T19:00:49.345Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:922 +0x8d
74592026-03-10T19:00:49.345Zcreated by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 10187
74602026-03-10T19:00:49.345Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:920 +0x226
74612026-03-10T19:00:49.345Z
74622026-03-10T19:00:49.346Zgoroutine 10470 [select]:
74632026-03-10T19:00:49.346Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents(0xc001085a20, {0x50e6118, 0xc002d6a780}, 0xc003288a20, 0xc0009b9260)
74642026-03-10T19:00:49.346Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:331 +0xc5
74652026-03-10T19:00:49.346Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func2({0x50e6118?, 0xc002d6a780?})
74662026-03-10T19:00:49.346Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:288 +0x32
74672026-03-10T19:00:49.346Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
74682026-03-10T19:00:49.346Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
74692026-03-10T19:00:49.346Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
74702026-03-10T19:00:49.346Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
74712026-03-10T19:00:49.346Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10210
74722026-03-10T19:00:49.346Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
74732026-03-10T19:00:49.346Z
74742026-03-10T19:00:49.346Zgoroutine 9378 [sync.Cond.Wait]:
74752026-03-10T19:00:49.346Zsync.runtime_notifyListWait(0xc0005ac550, 0x374d)
74762026-03-10T19:00:49.347Z /work/go/src/runtime/sema.go:569 +0x159
74772026-03-10T19:00:49.347Zsync.(*Cond).Wait(0x3ae1600?)
74782026-03-10T19:00:49.347Z /work/go/src/sync/cond.go:70 +0x85
74792026-03-10T19:00:49.347Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006622e70})
74802026-03-10T19:00:49.347Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
74812026-03-10T19:00:49.347Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
74822026-03-10T19:00:49.347Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
74832026-03-10T19:00:49.347Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
74842026-03-10T19:00:49.347Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
74852026-03-10T19:00:49.347Z
74862026-03-10T19:00:49.347Zgoroutine 10073 [sync.Cond.Wait, 59 minutes]:
74872026-03-10T19:00:49.347Zsync.runtime_notifyListWait(0xc00676e610, 0x28)
74882026-03-10T19:00:49.347Z /work/go/src/runtime/sema.go:569 +0x159
74892026-03-10T19:00:49.347Zsync.(*Cond).Wait(0x0?)
74902026-03-10T19:00:49.347Z /work/go/src/sync/cond.go:70 +0x85
74912026-03-10T19:00:49.347Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c9e30})
74922026-03-10T19:00:49.347Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
74932026-03-10T19:00:49.347Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
74942026-03-10T19:00:49.347Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
74952026-03-10T19:00:49.347Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
74962026-03-10T19:00:49.347Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
74972026-03-10T19:00:49.347Z
74982026-03-10T19:00:49.347Zgoroutine 9323 [chan receive, 59 minutes]:
74992026-03-10T19:00:49.347Zgithub.com/cockroachdb/cmux.muxListener.Accept(...)
75002026-03-10T19:00:49.347Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184
75012026-03-10T19:00:49.347Zcrypto/tls.(*listener).Accept(0xc0036a7530)
75022026-03-10T19:00:49.347Z /work/go/src/crypto/tls/tls.go:66 +0x27
75032026-03-10T19:00:49.347Znet/http.(*Server).Serve(0xc00464dd10, {0x50d12d0, 0xc0036a7530})
75042026-03-10T19:00:49.347Z /work/go/src/net/http/server.go:3260 +0x33e
75052026-03-10T19:00:49.347Zgithub.com/cockroachdb/cockroach/pkg/server.(*httpServer).start.func4({0xc0015ea908?, 0xc001b30300?})
75062026-03-10T19:00:49.347Z /work/oxidecomputer/cockroach/pkg/server/server_http.go:277 +0x25
75072026-03-10T19:00:49.347Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
75082026-03-10T19:00:49.347Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
75092026-03-10T19:00:49.347Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
75102026-03-10T19:00:49.348Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
75112026-03-10T19:00:49.348Z
75122026-03-10T19:00:49.348Zgoroutine 9333 [select]:
75132026-03-10T19:00:49.348Znet.(*pipe).read(0xc00318c180, {0xc0028c8700, 0x700, 0x5?})
75142026-03-10T19:00:49.348Z /work/go/src/net/pipe.go:159 +0x155
75152026-03-10T19:00:49.348Znet.(*pipe).Read(0xc0001bfa40?, {0xc0028c8700?, 0x20?, 0x3043900?})
75162026-03-10T19:00:49.348Z /work/go/src/net/pipe.go:142 +0x1c
75172026-03-10T19:00:49.348Zcrypto/tls.(*atLeastReader).Read(0xc029b832d8, {0xc0028c8700?, 0x0?, 0xc029b832d8?})
75182026-03-10T19:00:49.348Z /work/go/src/crypto/tls/conn.go:806 +0x3b
75192026-03-10T19:00:49.348Zbytes.(*Buffer).ReadFrom(0xc0001bfb30, {0x50b6dc0, 0xc029b832d8})
75202026-03-10T19:00:49.348Z /work/go/src/bytes/buffer.go:211 +0x98
75212026-03-10T19:00:49.348Zcrypto/tls.(*Conn).readFromUntil(0xc0001bf888, {0x7f784ca591f8, 0xc00318c180}, 0xc003043958?)
75222026-03-10T19:00:49.348Z /work/go/src/crypto/tls/conn.go:828 +0xde
75232026-03-10T19:00:49.348Zcrypto/tls.(*Conn).readRecordOrCCS(0xc0001bf888, 0x0)
75242026-03-10T19:00:49.348Z /work/go/src/crypto/tls/conn.go:626 +0x3cf
75252026-03-10T19:00:49.348Zcrypto/tls.(*Conn).readRecord(...)
75262026-03-10T19:00:49.348Z /work/go/src/crypto/tls/conn.go:588
75272026-03-10T19:00:49.348Zcrypto/tls.(*Conn).Read(0xc0001bf888, {0xc000fce000, 0x8000, 0xc003043ca8?})
75282026-03-10T19:00:49.348Z /work/go/src/crypto/tls/conn.go:1370 +0x156
75292026-03-10T19:00:49.348Zbufio.(*Reader).Read(0xc005f1c960, {0xc001b1b1c0, 0x9, 0xc003043d30?})
75302026-03-10T19:00:49.349Z /work/go/src/bufio/bufio.go:241 +0x197
75312026-03-10T19:00:49.349Zio.ReadAtLeast({0x50b1100, 0xc005f1c960}, {0xc001b1b1c0, 0x9, 0x9}, 0x9)
75322026-03-10T19:00:49.349Z /work/go/src/io/io.go:335 +0x90
75332026-03-10T19:00:49.349Zio.ReadFull(...)
75342026-03-10T19:00:49.349Z /work/go/src/io/io.go:354
75352026-03-10T19:00:49.349Zgolang.org/x/net/http2.readFrameHeader({0xc001b1b1c0, 0x9, 0x3c5cd00?}, {0x50b1100?, 0xc005f1c960?})
75362026-03-10T19:00:49.349Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x65
75372026-03-10T19:00:49.349Zgolang.org/x/net/http2.(*Framer).ReadFrame(0xc001b1b180)
75382026-03-10T19:00:49.349Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:501 +0x85
75392026-03-10T19:00:49.353Zgoogle.golang.org/grpc/internal/transport.(*http2Client).reader(0xc000e870e0)
75402026-03-10T19:00:49.353Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1498 +0x41d
75412026-03-10T19:00:49.353Zcreated by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 9353
75422026-03-10T19:00:49.353Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:365 +0x184d
75432026-03-10T19:00:49.353Z
75442026-03-10T19:00:49.353Zgoroutine 10057 [sync.Cond.Wait, 59 minutes]:
75452026-03-10T19:00:49.353Zsync.runtime_notifyListWait(0xc00676e610, 0x18)
75462026-03-10T19:00:49.353Z /work/go/src/runtime/sema.go:569 +0x159
75472026-03-10T19:00:49.353Zsync.(*Cond).Wait(0x0?)
75482026-03-10T19:00:49.353Z /work/go/src/sync/cond.go:70 +0x85
75492026-03-10T19:00:49.353Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c9b30})
75502026-03-10T19:00:49.353Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
75512026-03-10T19:00:49.353Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
75522026-03-10T19:00:49.353Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
75532026-03-10T19:00:49.353Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
75542026-03-10T19:00:49.353Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
75552026-03-10T19:00:49.353Z
75562026-03-10T19:00:49.353Zgoroutine 10058 [sync.Cond.Wait, 59 minutes]:
75572026-03-10T19:00:49.353Zsync.runtime_notifyListWait(0xc00676e610, 0xd)
75582026-03-10T19:00:49.353Z /work/go/src/runtime/sema.go:569 +0x159
75592026-03-10T19:00:49.353Zsync.(*Cond).Wait(0x100000003f1c0a0?)
75602026-03-10T19:00:49.353Z /work/go/src/sync/cond.go:70 +0x85
75612026-03-10T19:00:49.353Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c9b60})
75622026-03-10T19:00:49.355Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
75632026-03-10T19:00:49.355Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
75642026-03-10T19:00:49.355Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
75652026-03-10T19:00:49.355Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
75662026-03-10T19:00:49.355Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
75672026-03-10T19:00:49.355Z
75682026-03-10T19:00:49.355Zgoroutine 10059 [sync.Cond.Wait, 59 minutes]:
75692026-03-10T19:00:49.355Zsync.runtime_notifyListWait(0xc00676e610, 0x1a)
75702026-03-10T19:00:49.355Z /work/go/src/runtime/sema.go:569 +0x159
75712026-03-10T19:00:49.355Zsync.(*Cond).Wait(0x3f1b920?)
75722026-03-10T19:00:49.355Z /work/go/src/sync/cond.go:70 +0x85
75732026-03-10T19:00:49.355Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c9b90})
75742026-03-10T19:00:49.355Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
75752026-03-10T19:00:49.355Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
75762026-03-10T19:00:49.355Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
75772026-03-10T19:00:49.355Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
75782026-03-10T19:00:49.355Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
75792026-03-10T19:00:49.355Z
75802026-03-10T19:00:49.355Zgoroutine 9431 [select]:
75812026-03-10T19:00:49.355Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).raftTickLoop(0xc001668008, {0x24d7?, 0xc004bf8f80?})
75822026-03-10T19:00:49.358Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store_raft.go:618 +0x145
75832026-03-10T19:00:49.358Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
75842026-03-10T19:00:49.358Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
75852026-03-10T19:00:49.358Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
75862026-03-10T19:00:49.358Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
75872026-03-10T19:00:49.358Z
75882026-03-10T19:00:49.358Zgoroutine 10056 [sync.Cond.Wait, 59 minutes]:
75892026-03-10T19:00:49.358Zsync.runtime_notifyListWait(0xc00676e610, 0x17)
75902026-03-10T19:00:49.358Z /work/go/src/runtime/sema.go:569 +0x159
75912026-03-10T19:00:49.359Zsync.(*Cond).Wait(0x0?)
75922026-03-10T19:00:49.359Z /work/go/src/sync/cond.go:70 +0x85
75932026-03-10T19:00:49.359Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c9b00})
75942026-03-10T19:00:49.359Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
75952026-03-10T19:00:49.359Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
75962026-03-10T19:00:49.359Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
75972026-03-10T19:00:49.359Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
75982026-03-10T19:00:49.359Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
75992026-03-10T19:00:49.359Z
76002026-03-10T19:00:49.359Zgoroutine 9337 [select]:
76012026-03-10T19:00:49.359Znet.(*pipe).read(0xc00318c280, {0xc000214c00, 0x600, 0x5?})
76022026-03-10T19:00:49.359Z /work/go/src/net/pipe.go:159 +0x155
76032026-03-10T19:00:49.359Znet.(*pipe).Read(0xc0001bf6c0?, {0xc000214c00?, 0x50e6188?, 0x452d90?})
76042026-03-10T19:00:49.359Z /work/go/src/net/pipe.go:142 +0x1c
76052026-03-10T19:00:49.360Zcrypto/tls.(*atLeastReader).Read(0xc029b832f0, {0xc000214c00?, 0x0?, 0xc029b832f0?})
76062026-03-10T19:00:49.360Z /work/go/src/crypto/tls/conn.go:806 +0x3b
76072026-03-10T19:00:49.360Zbytes.(*Buffer).ReadFrom(0xc0001bf7b0, {0x50b6dc0, 0xc029b832f0})
76082026-03-10T19:00:49.360Z /work/go/src/bytes/buffer.go:211 +0x98
76092026-03-10T19:00:49.360Zcrypto/tls.(*Conn).readFromUntil(0xc0001bf508, {0x7f784ca591f8, 0xc00318c280}, 0xc001074990?)
76102026-03-10T19:00:49.360Z /work/go/src/crypto/tls/conn.go:828 +0xde
76112026-03-10T19:00:49.360Zcrypto/tls.(*Conn).readRecordOrCCS(0xc0001bf508, 0x0)
76122026-03-10T19:00:49.360Z /work/go/src/crypto/tls/conn.go:626 +0x3cf
76132026-03-10T19:00:49.360Zcrypto/tls.(*Conn).readRecord(...)
76142026-03-10T19:00:49.360Z /work/go/src/crypto/tls/conn.go:588
76152026-03-10T19:00:49.360Zcrypto/tls.(*Conn).Read(0xc0001bf508, {0xc000d72000, 0x8000, 0x7f784b769398?})
76162026-03-10T19:00:49.360Z /work/go/src/crypto/tls/conn.go:1370 +0x156
76172026-03-10T19:00:49.360Zbufio.(*Reader).Read(0xc005f1c840, {0xc001b1b0e0, 0x9, 0xc000748008?})
76182026-03-10T19:00:49.360Z /work/go/src/bufio/bufio.go:241 +0x197
76192026-03-10T19:00:49.361Zio.ReadAtLeast({0x50b1100, 0xc005f1c840}, {0xc001b1b0e0, 0x9, 0x9}, 0x9)
76202026-03-10T19:00:49.361Z /work/go/src/io/io.go:335 +0x90
76212026-03-10T19:00:49.361Zio.ReadFull(...)
76222026-03-10T19:00:49.361Z /work/go/src/io/io.go:354
76232026-03-10T19:00:49.361Zgolang.org/x/net/http2.readFrameHeader({0xc001b1b0e0, 0x9, 0xc029a0aab0?}, {0x50b1100?, 0xc005f1c840?})
76242026-03-10T19:00:49.361Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x65
76252026-03-10T19:00:49.361Zgolang.org/x/net/http2.(*Framer).ReadFrame(0xc001b1b0a0)
76262026-03-10T19:00:49.361Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:501 +0x85
76272026-03-10T19:00:49.361Zgoogle.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc002134340, 0xc00768e2d0, 0x431bf68)
76282026-03-10T19:00:49.361Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:611 +0x14e
76292026-03-10T19:00:49.361Zgoogle.golang.org/grpc.(*Server).serveStreams(0xc005fee1c0, {0x5126b60, 0xc002134340})
76302026-03-10T19:00:49.361Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:906 +0x114
76312026-03-10T19:00:49.361Zgoogle.golang.org/grpc.(*Server).handleRawConn.func1()
76322026-03-10T19:00:49.361Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:848 +0x45
76332026-03-10T19:00:49.362Zcreated by google.golang.org/grpc.(*Server).handleRawConn in goroutine 9331
76342026-03-10T19:00:49.362Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:847 +0x15b
76352026-03-10T19:00:49.362Z
76362026-03-10T19:00:49.362Zgoroutine 10060 [sync.Cond.Wait, 59 minutes]:
76372026-03-10T19:00:49.362Zsync.runtime_notifyListWait(0xc00676e610, 0x1b)
76382026-03-10T19:00:49.362Z /work/go/src/runtime/sema.go:569 +0x159
76392026-03-10T19:00:49.366Zsync.(*Cond).Wait(0x0?)
76402026-03-10T19:00:49.366Z /work/go/src/sync/cond.go:70 +0x85
76412026-03-10T19:00:49.366Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c9bc0})
76422026-03-10T19:00:49.366Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
76432026-03-10T19:00:49.366Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
76442026-03-10T19:00:49.366Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
76452026-03-10T19:00:49.366Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
76462026-03-10T19:00:49.366Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
76472026-03-10T19:00:49.366Z
76482026-03-10T19:00:49.366Zgoroutine 10063 [sync.Cond.Wait, 59 minutes]:
76492026-03-10T19:00:49.366Zsync.runtime_notifyListWait(0xc00676e610, 0x1e)
76502026-03-10T19:00:49.366Z /work/go/src/runtime/sema.go:569 +0x159
76512026-03-10T19:00:49.366Zsync.(*Cond).Wait(0x0?)
76522026-03-10T19:00:49.367Z /work/go/src/sync/cond.go:70 +0x85
76532026-03-10T19:00:49.367Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c9c50})
76542026-03-10T19:00:49.367Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
76552026-03-10T19:00:49.367Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
76562026-03-10T19:00:49.367Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
76572026-03-10T19:00:49.367Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
76582026-03-10T19:00:49.367Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
76592026-03-10T19:00:49.368Z
76602026-03-10T19:00:49.368Zgoroutine 10062 [sync.Cond.Wait, 59 minutes]:
76612026-03-10T19:00:49.368Zsync.runtime_notifyListWait(0xc00676e610, 0x1d)
76622026-03-10T19:00:49.368Z /work/go/src/runtime/sema.go:569 +0x159
76632026-03-10T19:00:49.368Zsync.(*Cond).Wait(0x0?)
76642026-03-10T19:00:49.368Z /work/go/src/sync/cond.go:70 +0x85
76652026-03-10T19:00:49.368Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c9c20})
76662026-03-10T19:00:49.368Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
76672026-03-10T19:00:49.368Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
76682026-03-10T19:00:49.368Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
76692026-03-10T19:00:49.368Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
76702026-03-10T19:00:49.368Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
76712026-03-10T19:00:49.368Z
76722026-03-10T19:00:49.368Zgoroutine 10061 [sync.Cond.Wait, 59 minutes]:
76732026-03-10T19:00:49.368Zsync.runtime_notifyListWait(0xc00676e610, 0x1c)
76742026-03-10T19:00:49.368Z /work/go/src/runtime/sema.go:569 +0x159
76752026-03-10T19:00:49.368Zsync.(*Cond).Wait(0x0?)
76762026-03-10T19:00:49.368Z /work/go/src/sync/cond.go:70 +0x85
76772026-03-10T19:00:49.368Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c9bf0})
76782026-03-10T19:00:49.368Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
76792026-03-10T19:00:49.368Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
76802026-03-10T19:00:49.368Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
76812026-03-10T19:00:49.368Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
76822026-03-10T19:00:49.368Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
76832026-03-10T19:00:49.368Z
76842026-03-10T19:00:49.368Zgoroutine 10082 [sync.Cond.Wait, 59 minutes]:
76852026-03-10T19:00:49.368Zsync.runtime_notifyListWait(0xc00676e610, 0x31)
76862026-03-10T19:00:49.368Z /work/go/src/runtime/sema.go:569 +0x159
76872026-03-10T19:00:49.368Zsync.(*Cond).Wait(0x0?)
76882026-03-10T19:00:49.368Z /work/go/src/sync/cond.go:70 +0x85
76892026-03-10T19:00:49.368Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0072ac060})
76902026-03-10T19:00:49.368Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
76912026-03-10T19:00:49.368Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
76922026-03-10T19:00:49.368Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
76932026-03-10T19:00:49.368Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
76942026-03-10T19:00:49.368Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
76952026-03-10T19:00:49.368Z
76962026-03-10T19:00:49.369Zgoroutine 9217 [chan receive, 59 minutes]:
76972026-03-10T19:00:49.369Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50e60e0?, 0xc005cee1e0?})
76982026-03-10T19:00:49.369Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
76992026-03-10T19:00:49.369Zruntime/pprof.Do({0x50e6038?, 0x778e7c0?}, {{0xc00030e2c0?, 0xfec8f31b73e7ee5f?, 0xbd5a478e7fff4749?}}, 0xc002882fa8)
77002026-03-10T19:00:49.369Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
77012026-03-10T19:00:49.369Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x892a724bf048c934?)
77022026-03-10T19:00:49.369Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
77032026-03-10T19:00:49.369Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9099
77042026-03-10T19:00:49.369Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
77052026-03-10T19:00:49.369Z
77062026-03-10T19:00:49.369Zgoroutine 9218 [chan receive, 59 minutes]:
77072026-03-10T19:00:49.369Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50e60e0?, 0xc005cee180?})
77082026-03-10T19:00:49.369Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
77092026-03-10T19:00:49.369Zruntime/pprof.Do({0x50e6038?, 0x778e7c0?}, {{0xc00030e2c0?, 0xd024a39e607c331?, 0x5d96d2cb5cef29a4?}}, 0xc0028837a8)
77102026-03-10T19:00:49.369Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
77112026-03-10T19:00:49.369Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x52ad258863ce66b5?)
77122026-03-10T19:00:49.369Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
77132026-03-10T19:00:49.369Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9099
77142026-03-10T19:00:49.369Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
77152026-03-10T19:00:49.369Z
77162026-03-10T19:00:49.369Zgoroutine 9232 [select]:
77172026-03-10T19:00:49.369Zgithub.com/cockroachdb/cockroach/pkg/sql/contention.(*eventStore).startResolver.func1({0x50e60e0, 0xc001ead0e0})
77182026-03-10T19:00:49.369Z /work/oxidecomputer/cockroach/pkg/sql/contention/event_store.go:219 +0x1f4
77192026-03-10T19:00:49.369Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
77202026-03-10T19:00:49.370Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
77212026-03-10T19:00:49.370Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
77222026-03-10T19:00:49.370Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
77232026-03-10T19:00:49.370Z
77242026-03-10T19:00:49.370Zgoroutine 10076 [sync.Cond.Wait, 59 minutes]:
77252026-03-10T19:00:49.370Zsync.runtime_notifyListWait(0xc00676e610, 0x2b)
77262026-03-10T19:00:49.370Z /work/go/src/runtime/sema.go:569 +0x159
77272026-03-10T19:00:49.370Zsync.(*Cond).Wait(0x0?)
77282026-03-10T19:00:49.370Z /work/go/src/sync/cond.go:70 +0x85
77292026-03-10T19:00:49.370Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c9ec0})
77302026-03-10T19:00:49.370Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
77312026-03-10T19:00:49.370Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
77322026-03-10T19:00:49.370Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
77332026-03-10T19:00:49.370Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
77342026-03-10T19:00:49.370Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
77352026-03-10T19:00:49.370Z
77362026-03-10T19:00:49.370Zgoroutine 9423 [sync.Cond.Wait]:
77372026-03-10T19:00:49.370Zsync.runtime_notifyListWait(0xc0005ac550, 0x373c)
77382026-03-10T19:00:49.370Z /work/go/src/runtime/sema.go:569 +0x159
77392026-03-10T19:00:49.374Zsync.(*Cond).Wait(0x3ae1600?)
77402026-03-10T19:00:49.374Z /work/go/src/sync/cond.go:70 +0x85
77412026-03-10T19:00:49.374Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006623920})
77422026-03-10T19:00:49.374Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
77432026-03-10T19:00:49.374Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
77442026-03-10T19:00:49.375Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
77452026-03-10T19:00:49.375Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
77462026-03-10T19:00:49.375Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
77472026-03-10T19:00:49.375Z
77482026-03-10T19:00:49.375Zgoroutine 10108 [select]:
77492026-03-10T19:00:49.375Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*StoreRebalancer).Start.func1({0x50e60e0, 0xc0072ac630})
77502026-03-10T19:00:49.375Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store_rebalancer.go:235 +0x1c5
77512026-03-10T19:00:49.375Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
77522026-03-10T19:00:49.375Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
77532026-03-10T19:00:49.375Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
77542026-03-10T19:00:49.375Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
77552026-03-10T19:00:49.375Z
77562026-03-10T19:00:49.375Zgoroutine 10091 [sync.Cond.Wait, 59 minutes]:
77572026-03-10T19:00:49.375Zsync.runtime_notifyListWait(0xc00676e610, 0x3a)
77582026-03-10T19:00:49.375Z /work/go/src/runtime/sema.go:569 +0x159
77592026-03-10T19:00:49.375Zsync.(*Cond).Wait(0xc000094f08?)
77602026-03-10T19:00:49.375Z /work/go/src/sync/cond.go:70 +0x85
77612026-03-10T19:00:49.375Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0072ac210})
77622026-03-10T19:00:49.375Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
77632026-03-10T19:00:49.375Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
77642026-03-10T19:00:49.376Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
77652026-03-10T19:00:49.376Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
77662026-03-10T19:00:49.376Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
77672026-03-10T19:00:49.378Z
77682026-03-10T19:00:49.378Zgoroutine 9446 [select]:
77692026-03-10T19:00:49.378Zgithub.com/cockroachdb/cockroach/pkg/server.startSampleEnvironment.func1({0x50e60e0, 0xc006810240})
77702026-03-10T19:00:49.378Z /work/oxidecomputer/cockroach/pkg/server/env_sampler.go:138 +0x1ad
77712026-03-10T19:00:49.378Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
77722026-03-10T19:00:49.378Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
77732026-03-10T19:00:49.378Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
77742026-03-10T19:00:49.378Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
77752026-03-10T19:00:49.378Z
77762026-03-10T19:00:49.378Zgoroutine 10618 [chan receive, 59 minutes]:
77772026-03-10T19:00:49.378Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).rangeFeedWithRangeID(0xc00128b008, 0x3dbc860?, 0xc008005440, {0x7f784cb862a8, 0xc007075460})
77782026-03-10T19:00:49.378Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:249 +0x76b
77792026-03-10T19:00:49.378Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed(...)
77802026-03-10T19:00:49.378Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:149
77812026-03-10T19:00:49.378Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).RangeFeed(0xc001668008, 0xc008005440, {0x5119e20, 0xc007075460})
77822026-03-10T19:00:49.378Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:3082 +0x16f
77832026-03-10T19:00:49.378Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).RangeFeed(0xc006849e00, 0xc008005440, {0x5119e20, 0xc007075460})
77842026-03-10T19:00:49.378Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/stores.go:216 +0x19a
77852026-03-10T19:00:49.378Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed(0x7366f90?, 0x403a5e0?, {0x5119e20, 0xc007075460})
77862026-03-10T19:00:49.378Z /work/oxidecomputer/cockroach/pkg/server/node.go:1232 +0x28
77872026-03-10T19:00:49.378Zgithub.com/cockroachdb/cockroach/pkg/roachpb._Internal_RangeFeed_Handler({0x403a5e0, 0xc001668908}, {0x5114010, 0xc005ee6aa0})
77882026-03-10T19:00:49.378Z /work/oxidecomputer/cockroach/pkg/roachpb/api.pb.go:9523 +0x107
77892026-03-10T19:00:49.378Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.StreamServerInterceptor.func14({0x403a5e0, 0xc001668908}, {0x51147c0, 0xc007f824e0}, 0xc0062deba0, 0x4319298)
77902026-03-10T19:00:49.378Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:205 +0x35b
77912026-03-10T19:00:49.378Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x403a5e0?, 0xc001668908?}, {0x51147c0?, 0xc007f824e0?})
77922026-03-10T19:00:49.378Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1408 +0x59
77932026-03-10T19:00:49.378Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func4({0x403a5e0, 0xc001668908}, {0x51147c0, 0xc007f824e0}, 0x30?, 0xc00810c200)
77942026-03-10T19:00:49.378Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:247 +0x6c
77952026-03-10T19:00:49.378Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x403a5e0?, 0xc001668908?}, {0x51147c0?, 0xc007f824e0?})
77962026-03-10T19:00:49.378Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
77972026-03-10T19:00:49.378Zgithub.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({{{0x3afe8c0?}}}, {0x403a5e0, 0xc001668908}, {0x51147c0, 0xc007f824e0}, 0xc0062deba0, 0xc00810c200)
77982026-03-10T19:00:49.378Z /work/oxidecomputer/cockroach/pkg/rpc/auth.go:98 +0x19b
77992026-03-10T19:00:49.379Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x403a5e0?, 0xc001668908?}, {0x51147c0?, 0xc007f824e0?})
78002026-03-10T19:00:49.379Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
78012026-03-10T19:00:49.379Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func2.1({0x66?, 0x40?})
78022026-03-10T19:00:49.379Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:216 +0x27
78032026-03-10T19:00:49.379Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc006848400, {0x50e60e0, 0xc008003e30}, {0xc000bd6b28?, 0x416dfb?}, 0xc000bd6ac0)
78042026-03-10T19:00:49.379Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:344 +0xcb
78052026-03-10T19:00:49.379Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func2({0x403a5e0, 0xc001668908}, {0x51147c0, 0xc007f824e0}, 0xc0062deba0, 0xc00810c200)
78062026-03-10T19:00:49.379Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:215 +0xdf
78072026-03-10T19:00:49.379Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x403a5e0?, 0xc001668908?}, {0x51147c0?, 0xc007f824e0?})
78082026-03-10T19:00:49.379Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
78092026-03-10T19:00:49.380Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x403a5e0, 0xc001668908}, {0x51147c0, 0xc007f824e0}, 0xc0062deba0, 0x4319298)
78102026-03-10T19:00:49.380Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1413 +0x108
78112026-03-10T19:00:49.380Zgoogle.golang.org/grpc.(*Server).processStreamingRPC(0xc005fee1c0, {0x5126b60, 0xc00130c340}, 0xc008005320, 0xc0077eb920, 0x73e2b80, 0x0)
78122026-03-10T19:00:49.380Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1549 +0xe51
78132026-03-10T19:00:49.380Zgoogle.golang.org/grpc.(*Server).handleStream(0xc005fee1c0, {0x5126b60, 0xc00130c340}, 0xc008005320, 0x0)
78142026-03-10T19:00:49.380Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1624 +0x999
78152026-03-10T19:00:49.380Zgoogle.golang.org/grpc.(*Server).serveStreams.func1.2()
78162026-03-10T19:00:49.380Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:922 +0x8d
78172026-03-10T19:00:49.380Zcreated by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 10187
78182026-03-10T19:00:49.380Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:920 +0x226
78192026-03-10T19:00:49.380Z
78202026-03-10T19:00:49.380Zgoroutine 10206 [semacquire, 59 minutes]:
78212026-03-10T19:00:49.380Zsync.runtime_Semacquire(0xc006e92a38?)
78222026-03-10T19:00:49.380Z /work/go/src/runtime/sema.go:62 +0x25
78232026-03-10T19:00:49.380Zsync.(*WaitGroup).Wait(0x44f7e0?)
78242026-03-10T19:00:49.380Z /work/go/src/sync/waitgroup.go:116 +0x48
78252026-03-10T19:00:49.380Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc0043fa700)
78262026-03-10T19:00:49.380Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
78272026-03-10T19:00:49.381Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc0043fa700?, {0x50e6118?, 0xc00717cc80?}})
78282026-03-10T19:00:49.381Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
78292026-03-10T19:00:49.381Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait({0x50e60e0?, 0xc006527590?}, {0xc004427d78, 0x2, 0x4000000000000000?})
78302026-03-10T19:00:49.381Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:193 +0xff
78312026-03-10T19:00:49.381Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0xc0009d6b00, {0x50e60e0, 0xc006527590}, 0xc005c718f0)
78322026-03-10T19:00:49.381Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:291 +0x59c
78332026-03-10T19:00:49.381Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).Start.func2({0x50e60e0, 0xc005c71aa0})
78342026-03-10T19:00:49.381Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:215 +0x2f3
78352026-03-10T19:00:49.381Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
78362026-03-10T19:00:49.381Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
78372026-03-10T19:00:49.381Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
78382026-03-10T19:00:49.381Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
78392026-03-10T19:00:49.385Z
78402026-03-10T19:00:49.385Zgoroutine 9432 [select]:
78412026-03-10T19:00:49.385Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).coalescedHeartbeatsLoop(0xc001668008, {0x50e60e0, 0xc006623ad0})
78422026-03-10T19:00:49.385Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store_raft.go:678 +0xec
78432026-03-10T19:00:49.385Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
78442026-03-10T19:00:49.385Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
78452026-03-10T19:00:49.385Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
78462026-03-10T19:00:49.385Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
78472026-03-10T19:00:49.385Z
78482026-03-10T19:00:49.387Zgoroutine 10159 [select]:
78492026-03-10T19:00:49.387Zgithub.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc0036bf400, {0x50e60e0, 0xc0065f86f0}, 0xc00717ceb0, {0xc00322b550, 0xf}, 0xc006968660)
78502026-03-10T19:00:49.387Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:1462 +0x27b
78512026-03-10T19:00:49.387Zgithub.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1({0x50e60e0, 0xc0065f86f0})
78522026-03-10T19:00:49.387Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:1380 +0x85
78532026-03-10T19:00:49.387Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
78542026-03-10T19:00:49.387Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
78552026-03-10T19:00:49.387Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10156
78562026-03-10T19:00:49.387Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
78572026-03-10T19:00:49.387Z
78582026-03-10T19:00:49.387Zgoroutine 10231 [select]:
78592026-03-10T19:00:49.387Zgoogle.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc000076960)
78602026-03-10T19:00:49.387Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1572 +0x137
78612026-03-10T19:00:49.387Zcreated by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 10158
78622026-03-10T19:00:49.387Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:360 +0x180b
78632026-03-10T19:00:49.387Z
78642026-03-10T19:00:49.387Zgoroutine 10228 [select, 10 minutes]:
78652026-03-10T19:00:49.388Zgithub.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigmanager.(*Manager).run(0xc007961810, {0x50e60e0, 0xc005c71d40})
78662026-03-10T19:00:49.388Z /work/oxidecomputer/cockroach/pkg/spanconfig/spanconfigmanager/manager.go:159 +0x3ba
78672026-03-10T19:00:49.388Zgithub.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigmanager.(*Manager).Start.func1({0x50e60e0?, 0xc005c71d40?})
78682026-03-10T19:00:49.388Z /work/oxidecomputer/cockroach/pkg/spanconfig/spanconfigmanager/manager.go:101 +0x2a
78692026-03-10T19:00:49.388Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
78702026-03-10T19:00:49.388Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
78712026-03-10T19:00:49.388Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
78722026-03-10T19:00:49.388Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
78732026-03-10T19:00:49.388Z
78742026-03-10T19:00:49.388Zgoroutine 9420 [sync.Cond.Wait]:
78752026-03-10T19:00:49.388Zsync.runtime_notifyListWait(0xc0005ac550, 0x376d)
78762026-03-10T19:00:49.388Z /work/go/src/runtime/sema.go:569 +0x159
78772026-03-10T19:00:49.388Zsync.(*Cond).Wait(0x3ae1600?)
78782026-03-10T19:00:49.388Z /work/go/src/sync/cond.go:70 +0x85
78792026-03-10T19:00:49.388Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006623800})
78802026-03-10T19:00:49.388Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
78812026-03-10T19:00:49.388Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
78822026-03-10T19:00:49.388Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
78832026-03-10T19:00:49.388Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
78842026-03-10T19:00:49.388Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
78852026-03-10T19:00:49.388Z
78862026-03-10T19:00:49.388Zgoroutine 9419 [sync.Cond.Wait]:
78872026-03-10T19:00:49.389Zsync.runtime_notifyListWait(0xc0005ac550, 0x376e)
78882026-03-10T19:00:49.389Z /work/go/src/runtime/sema.go:569 +0x159
78892026-03-10T19:00:49.389Zsync.(*Cond).Wait(0x3ae1600?)
78902026-03-10T19:00:49.389Z /work/go/src/sync/cond.go:70 +0x85
78912026-03-10T19:00:49.389Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc0066237d0})
78922026-03-10T19:00:49.389Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
78932026-03-10T19:00:49.389Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
78942026-03-10T19:00:49.389Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
78952026-03-10T19:00:49.389Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
78962026-03-10T19:00:49.389Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
78972026-03-10T19:00:49.389Z
78982026-03-10T19:00:49.389Zgoroutine 9401 [sync.Cond.Wait]:
78992026-03-10T19:00:49.389Zsync.runtime_notifyListWait(0xc0005ac550, 0x375a)
79002026-03-10T19:00:49.389Z /work/go/src/runtime/sema.go:569 +0x159
79012026-03-10T19:00:49.389Zsync.(*Cond).Wait(0x3ae1600?)
79022026-03-10T19:00:49.389Z /work/go/src/sync/cond.go:70 +0x85
79032026-03-10T19:00:49.389Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc0066233e0})
79042026-03-10T19:00:49.389Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
79052026-03-10T19:00:49.389Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
79062026-03-10T19:00:49.389Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
79072026-03-10T19:00:49.389Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
79082026-03-10T19:00:49.389Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
79092026-03-10T19:00:49.389Z
79102026-03-10T19:00:49.389Zgoroutine 9434 [select]:
79112026-03-10T19:00:49.389Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).startGossip.func3({0x50e60e0, 0xc006623c20})
79122026-03-10T19:00:49.389Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:2152 +0x225
79132026-03-10T19:00:49.389Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
79142026-03-10T19:00:49.389Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
79152026-03-10T19:00:49.389Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
79162026-03-10T19:00:49.389Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
79172026-03-10T19:00:49.390Z
79182026-03-10T19:00:49.390Zgoroutine 9436 [select, 1 minutes]:
79192026-03-10T19:00:49.390Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).startGossip.func3({0x50e60e0, 0xc006623c80})
79202026-03-10T19:00:49.390Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:2152 +0x225
79212026-03-10T19:00:49.390Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
79222026-03-10T19:00:49.390Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
79232026-03-10T19:00:49.390Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
79242026-03-10T19:00:49.390Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
79252026-03-10T19:00:49.390Z
79262026-03-10T19:00:49.390Zgoroutine 9371 [sync.Cond.Wait]:
79272026-03-10T19:00:49.390Zsync.runtime_notifyListWait(0xc0005ac550, 0x3746)
79282026-03-10T19:00:49.390Z /work/go/src/runtime/sema.go:569 +0x159
79292026-03-10T19:00:49.390Zsync.(*Cond).Wait(0x3ae1600?)
79302026-03-10T19:00:49.390Z /work/go/src/sync/cond.go:70 +0x85
79312026-03-10T19:00:49.390Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006622d20})
79322026-03-10T19:00:49.390Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
79332026-03-10T19:00:49.390Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
79342026-03-10T19:00:49.390Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
79352026-03-10T19:00:49.391Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
79362026-03-10T19:00:49.391Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
79372026-03-10T19:00:49.391Z
79382026-03-10T19:00:49.391Zgoroutine 9372 [sync.Cond.Wait]:
79392026-03-10T19:00:49.395Zsync.runtime_notifyListWait(0xc0005ac550, 0x3744)
79402026-03-10T19:00:49.395Z /work/go/src/runtime/sema.go:569 +0x159
79412026-03-10T19:00:49.395Zsync.(*Cond).Wait(0x3ae1600?)
79422026-03-10T19:00:49.395Z /work/go/src/sync/cond.go:70 +0x85
79432026-03-10T19:00:49.395Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006622d50})
79442026-03-10T19:00:49.395Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
79452026-03-10T19:00:49.395Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
79462026-03-10T19:00:49.395Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
79472026-03-10T19:00:49.395Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
79482026-03-10T19:00:49.395Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
79492026-03-10T19:00:49.395Z
79502026-03-10T19:00:49.395Zgoroutine 10092 [sync.Cond.Wait, 59 minutes]:
79512026-03-10T19:00:49.395Zsync.runtime_notifyListWait(0xc00676e610, 0x2d)
79522026-03-10T19:00:49.395Z /work/go/src/runtime/sema.go:569 +0x159
79532026-03-10T19:00:49.395Zsync.(*Cond).Wait(0xc000094f08?)
79542026-03-10T19:00:49.395Z /work/go/src/sync/cond.go:70 +0x85
79552026-03-10T19:00:49.395Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0072ac240})
79562026-03-10T19:00:49.396Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
79572026-03-10T19:00:49.396Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
79582026-03-10T19:00:49.396Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
79592026-03-10T19:00:49.396Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
79602026-03-10T19:00:49.396Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
79612026-03-10T19:00:49.396Z
79622026-03-10T19:00:49.396Zgoroutine 10128 [select]:
79632026-03-10T19:00:49.396Zgoogle.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc0013f61e0)
79642026-03-10T19:00:49.398Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1572 +0x137
79652026-03-10T19:00:49.398Zcreated by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 10124
79662026-03-10T19:00:49.398Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:360 +0x180b
79672026-03-10T19:00:49.398Z
79682026-03-10T19:00:49.398Zgoroutine 9424 [sync.Cond.Wait]:
79692026-03-10T19:00:49.398Zsync.runtime_notifyListWait(0xc0005ac550, 0x3739)
79702026-03-10T19:00:49.398Z /work/go/src/runtime/sema.go:569 +0x159
79712026-03-10T19:00:49.398Zsync.(*Cond).Wait(0x3ae1600?)
79722026-03-10T19:00:49.398Z /work/go/src/sync/cond.go:70 +0x85
79732026-03-10T19:00:49.398Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006623950})
79742026-03-10T19:00:49.399Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
79752026-03-10T19:00:49.399Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
79762026-03-10T19:00:49.399Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
79772026-03-10T19:00:49.399Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
79782026-03-10T19:00:49.399Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
79792026-03-10T19:00:49.399Z
79802026-03-10T19:00:49.399Zgoroutine 9442 [select, 10 minutes]:
79812026-03-10T19:00:49.399Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.KVAdmissionControllerImpl.SetTenantWeightProvider.func1()
79822026-03-10T19:00:49.399Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:3809 +0xd5
79832026-03-10T19:00:49.399Zcreated by github.com/cockroachdb/cockroach/pkg/kv/kvserver.KVAdmissionControllerImpl.SetTenantWeightProvider in goroutine 9099
79842026-03-10T19:00:49.399Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:3802 +0xcc
79852026-03-10T19:00:49.399Z
79862026-03-10T19:00:49.399Zgoroutine 10172 [semacquire, 59 minutes]:
79872026-03-10T19:00:49.401Zsync.runtime_Semacquire(0xc000013080?)
79882026-03-10T19:00:49.401Z /work/go/src/runtime/sema.go:62 +0x25
79892026-03-10T19:00:49.401Zsync.(*WaitGroup).Wait(0x44f7e0?)
79902026-03-10T19:00:49.401Z /work/go/src/sync/waitgroup.go:116 +0x48
79912026-03-10T19:00:49.401Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc003348e40)
79922026-03-10T19:00:49.402Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
79932026-03-10T19:00:49.402Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc003348e40?, {0x50e6118?, 0xc002d6a0a0?}})
79942026-03-10T19:00:49.402Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
79952026-03-10T19:00:49.402Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait({0x50e60e0?, 0xc002d7d620?}, {0xc005b37d78, 0x2, 0xc005b37c70?})
79962026-03-10T19:00:49.402Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:193 +0xff
79972026-03-10T19:00:49.402Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0xc0045fcf20, {0x50e60e0, 0xc002d7d620}, 0xc002d7d3e0)
79982026-03-10T19:00:49.402Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:291 +0x59c
79992026-03-10T19:00:49.403Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).Start.func2({0x50e60e0, 0xc002d7d500})
80002026-03-10T19:00:49.403Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:215 +0x2f3
80012026-03-10T19:00:49.403Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
80022026-03-10T19:00:49.403Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
80032026-03-10T19:00:49.403Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9981
80042026-03-10T19:00:49.403Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
80052026-03-10T19:00:49.403Z
80062026-03-10T19:00:49.403Zgoroutine 9352 [select, 59 minutes]:
80072026-03-10T19:00:49.403Zgoogle.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc003348080)
80082026-03-10T19:00:49.403Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:112 +0x6d
80092026-03-10T19:00:49.404Zcreated by google.golang.org/grpc.newCCBalancerWrapper in goroutine 9099
80102026-03-10T19:00:49.405Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:73 +0x22b
80112026-03-10T19:00:49.405Z
80122026-03-10T19:00:49.405Zgoroutine 10077 [sync.Cond.Wait, 59 minutes]:
80132026-03-10T19:00:49.405Zsync.runtime_notifyListWait(0xc00676e610, 0x2a)
80142026-03-10T19:00:49.405Z /work/go/src/runtime/sema.go:569 +0x159
80152026-03-10T19:00:49.405Zsync.(*Cond).Wait(0x0?)
80162026-03-10T19:00:49.405Z /work/go/src/sync/cond.go:70 +0x85
80172026-03-10T19:00:49.405Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c9ef0})
80182026-03-10T19:00:49.405Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
80192026-03-10T19:00:49.405Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
80202026-03-10T19:00:49.405Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
80212026-03-10T19:00:49.405Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
80222026-03-10T19:00:49.405Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
80232026-03-10T19:00:49.405Z
80242026-03-10T19:00:49.405Zgoroutine 3631404 [select]:
80252026-03-10T19:00:49.405Zgithub.com/cockroachdb/cockroach/pkg/sql/stats.(*Refresher).Start.func1.1({0x50e60e0, 0xc00743f0b0})
80262026-03-10T19:00:49.405Z /work/oxidecomputer/cockroach/pkg/sql/stats/automatic_stats.go:410 +0x145
80272026-03-10T19:00:49.406Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
80282026-03-10T19:00:49.406Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
80292026-03-10T19:00:49.406Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9634
80302026-03-10T19:00:49.406Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
80312026-03-10T19:00:49.406Z
80322026-03-10T19:00:49.407Zgoroutine 10011 [select, 59 minutes]:
80332026-03-10T19:00:49.407Zgoogle.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc00670a900)
80342026-03-10T19:00:49.407Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:112 +0x6d
80352026-03-10T19:00:49.407Zcreated by google.golang.org/grpc.newCCBalancerWrapper in goroutine 9099
80362026-03-10T19:00:49.407Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:73 +0x22b
80372026-03-10T19:00:49.407Z
80382026-03-10T19:00:49.407Zgoroutine 10429 [select]:
80392026-03-10T19:00:49.415Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents(0xc0045fcf20, {0x50e6118, 0xc002d6a0a0}, 0xc002d7d3e0, 0xc0041fe5a0)
80402026-03-10T19:00:49.415Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:331 +0xc5
80412026-03-10T19:00:49.415Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func2({0x50e6118?, 0xc002d6a0a0?})
80422026-03-10T19:00:49.415Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:288 +0x32
80432026-03-10T19:00:49.415Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
80442026-03-10T19:00:49.415Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
80452026-03-10T19:00:49.415Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
80462026-03-10T19:00:49.415Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
80472026-03-10T19:00:49.415Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10172
80482026-03-10T19:00:49.415Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
80492026-03-10T19:00:49.415Z
80502026-03-10T19:00:49.416Zgoroutine 9390 [sync.Cond.Wait]:
80512026-03-10T19:00:49.416Zsync.runtime_notifyListWait(0xc0005ac550, 0x373e)
80522026-03-10T19:00:49.416Z /work/go/src/runtime/sema.go:569 +0x159
80532026-03-10T19:00:49.416Zsync.(*Cond).Wait(0x3ae1600?)
80542026-03-10T19:00:49.416Z /work/go/src/sync/cond.go:70 +0x85
80552026-03-10T19:00:49.416Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006623140})
80562026-03-10T19:00:49.416Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
80572026-03-10T19:00:49.416Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
80582026-03-10T19:00:49.416Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
80592026-03-10T19:00:49.416Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
80602026-03-10T19:00:49.416Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
80612026-03-10T19:00:49.416Z
80622026-03-10T19:00:49.416Zgoroutine 9440 [select]:
80632026-03-10T19:00:49.416Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*StoreRebalancer).Start.func1({0x50e60e0, 0xc006623e30})
80642026-03-10T19:00:49.416Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store_rebalancer.go:235 +0x1c5
80652026-03-10T19:00:49.416Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
80662026-03-10T19:00:49.416Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
80672026-03-10T19:00:49.417Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
80682026-03-10T19:00:49.417Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
80692026-03-10T19:00:49.417Z
80702026-03-10T19:00:49.417Zgoroutine 10090 [sync.Cond.Wait, 59 minutes]:
80712026-03-10T19:00:49.417Zsync.runtime_notifyListWait(0xc00676e610, 0x39)
80722026-03-10T19:00:49.417Z /work/go/src/runtime/sema.go:569 +0x159
80732026-03-10T19:00:49.417Zsync.(*Cond).Wait(0x6f7cfab57b700be?)
80742026-03-10T19:00:49.417Z /work/go/src/sync/cond.go:70 +0x85
80752026-03-10T19:00:49.417Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0072ac1e0})
80762026-03-10T19:00:49.417Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
80772026-03-10T19:00:49.417Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
80782026-03-10T19:00:49.417Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
80792026-03-10T19:00:49.417Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
80802026-03-10T19:00:49.417Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
80812026-03-10T19:00:49.417Z
80822026-03-10T19:00:49.417Zgoroutine 10081 [sync.Cond.Wait, 59 minutes]:
80832026-03-10T19:00:49.417Zsync.runtime_notifyListWait(0xc00676e610, 0x30)
80842026-03-10T19:00:49.417Z /work/go/src/runtime/sema.go:569 +0x159
80852026-03-10T19:00:49.417Zsync.(*Cond).Wait(0x2db9160e9a9341b4?)
80862026-03-10T19:00:49.417Z /work/go/src/sync/cond.go:70 +0x85
80872026-03-10T19:00:49.417Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0072ac030})
80882026-03-10T19:00:49.417Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
80892026-03-10T19:00:49.417Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
80902026-03-10T19:00:49.417Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
80912026-03-10T19:00:49.417Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
80922026-03-10T19:00:49.417Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
80932026-03-10T19:00:49.417Z
80942026-03-10T19:00:49.417Zgoroutine 9316 [semacquire, 59 minutes]:
80952026-03-10T19:00:49.417Zsync.runtime_Semacquire(0xc0064f0c90?)
80962026-03-10T19:00:49.417Z /work/go/src/runtime/sema.go:62 +0x25
80972026-03-10T19:00:49.417Zsync.(*WaitGroup).Wait(0x44f7e0?)
80982026-03-10T19:00:49.417Z /work/go/src/sync/waitgroup.go:116 +0x48
80992026-03-10T19:00:49.417Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc0005ac040)
81002026-03-10T19:00:49.418Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
81012026-03-10T19:00:49.418Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc0005ac040?, {0x50e6118?, 0xc000d02050?}})
81022026-03-10T19:00:49.418Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
81032026-03-10T19:00:49.418Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait({0x50e60e0?, 0xc00401b800?}, {0xc004334d78, 0x2, 0x4000000000000000?})
81042026-03-10T19:00:49.418Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:193 +0xff
81052026-03-10T19:00:49.418Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0xc006028840, {0x50e60e0, 0xc00401b800}, 0xc00634c960)
81062026-03-10T19:00:49.418Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:291 +0x59c
81072026-03-10T19:00:49.418Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).Start.func2({0x50e60e0, 0xc00634cb70})
81082026-03-10T19:00:49.418Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:215 +0x2f3
81092026-03-10T19:00:49.418Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
81102026-03-10T19:00:49.418Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
81112026-03-10T19:00:49.418Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
81122026-03-10T19:00:49.418Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
81132026-03-10T19:00:49.418Z
81142026-03-10T19:00:49.419Zgoroutine 9336 [select]:
81152026-03-10T19:00:49.419Zgoogle.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc002134340)
81162026-03-10T19:00:49.419Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1131 +0x205
81172026-03-10T19:00:49.419Zcreated by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 9331
81182026-03-10T19:00:49.419Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:341 +0x188e
81192026-03-10T19:00:49.419Z
81202026-03-10T19:00:49.419Zgoroutine 10186 [select]:
81212026-03-10T19:00:49.419Zgoogle.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc00130c340)
81222026-03-10T19:00:49.419Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1131 +0x205
81232026-03-10T19:00:49.419Zcreated by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 10242
81242026-03-10T19:00:49.419Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:341 +0x188e
81252026-03-10T19:00:49.419Z
81262026-03-10T19:00:49.419Zgoroutine 10185 [select]:
81272026-03-10T19:00:49.420Zgoogle.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc006c41d10, 0x1)
81282026-03-10T19:00:49.420Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:407 +0x113
81292026-03-10T19:00:49.420Zgoogle.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc004c45320)
81302026-03-10T19:00:49.420Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:534 +0x7b
81312026-03-10T19:00:49.420Zgoogle.golang.org/grpc/internal/transport.NewServerTransport.func2()
81322026-03-10T19:00:49.420Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:332 +0xc6
81332026-03-10T19:00:49.420Zcreated by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 10242
81342026-03-10T19:00:49.420Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:329 +0x184c
81352026-03-10T19:00:49.420Z
81362026-03-10T19:00:49.420Zgoroutine 9362 [select]:
81372026-03-10T19:00:49.420Zgithub.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run(0xc006cae840, {0x50e60e0, 0xc006526b40})
81382026-03-10T19:00:49.420Z /work/oxidecomputer/cockroach/pkg/internal/client/requestbatcher/batcher.go:475 +0x326
81392026-03-10T19:00:49.424Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
81402026-03-10T19:00:49.424Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
81412026-03-10T19:00:49.424Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
81422026-03-10T19:00:49.424Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
81432026-03-10T19:00:49.424Z
81442026-03-10T19:00:49.424Zgoroutine 10013 [chan receive, 59 minutes]:
81452026-03-10T19:00:49.424Zgithub.com/cockroachdb/cockroach/pkg/server.configureGRPCGateway.func5({0x50e60e0, 0xc006709530})
81462026-03-10T19:00:49.424Z /work/oxidecomputer/cockroach/pkg/server/grpc_gateway.go:127 +0x45
81472026-03-10T19:00:49.424Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
81482026-03-10T19:00:49.424Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
81492026-03-10T19:00:49.424Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
81502026-03-10T19:00:49.424Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
81512026-03-10T19:00:49.424Z
81522026-03-10T19:00:49.424Zgoroutine 9709 [select]:
81532026-03-10T19:00:49.424Zgoogle.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc006dd6cd0, 0x1)
81542026-03-10T19:00:49.425Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:407 +0x113
81552026-03-10T19:00:49.425Zgoogle.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00435f560)
81562026-03-10T19:00:49.425Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:534 +0x7b
81572026-03-10T19:00:49.425Zgoogle.golang.org/grpc/internal/transport.NewServerTransport.func2()
81582026-03-10T19:00:49.425Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:332 +0xc6
81592026-03-10T19:00:49.425Zcreated by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 9706
81602026-03-10T19:00:49.425Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:329 +0x184c
81612026-03-10T19:00:49.425Z
81622026-03-10T19:00:49.425Zgoroutine 10119 [select]:
81632026-03-10T19:00:49.425Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/liveness.(*NodeLiveness).Start.func1({0xc007378c08?, 0x0?})
81642026-03-10T19:00:49.425Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/liveness/liveness.go:789 +0x37f
81652026-03-10T19:00:49.425Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
81662026-03-10T19:00:49.425Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
81672026-03-10T19:00:49.425Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
81682026-03-10T19:00:49.426Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
81692026-03-10T19:00:49.426Z
81702026-03-10T19:00:49.426Zgoroutine 10178 [select]:
81712026-03-10T19:00:49.426Zgoogle.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc006c40190, 0x1)
81722026-03-10T19:00:49.426Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:407 +0x113
81732026-03-10T19:00:49.426Zgoogle.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc003bb8180)
81742026-03-10T19:00:49.426Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:534 +0x7b
81752026-03-10T19:00:49.426Zgoogle.golang.org/grpc/internal/transport.newHTTP2Client.func3()
81762026-03-10T19:00:49.426Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:415 +0x67
81772026-03-10T19:00:49.426Zcreated by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 10124
81782026-03-10T19:00:49.426Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:413 +0x1dcb
81792026-03-10T19:00:49.426Z
81802026-03-10T19:00:49.426Zgoroutine 10422 [select]:
81812026-03-10T19:00:49.426Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).outputLoop(0xc003c31340, {0x50e6118, 0xc0053af950})
81822026-03-10T19:00:49.426Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:310 +0x247
81832026-03-10T19:00:49.426Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).runOutputLoop(0xc003c31340, {0x50e60e0, 0xc00244bf20}, 0x1b?)
81842026-03-10T19:00:49.426Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:335 +0xba
81852026-03-10T19:00:49.426Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run.func1({0x50e60e0?, 0xc00244bf20?})
81862026-03-10T19:00:49.426Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:297 +0x46
81872026-03-10T19:00:49.428Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
81882026-03-10T19:00:49.428Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
81892026-03-10T19:00:49.428Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10420
81902026-03-10T19:00:49.428Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
81912026-03-10T19:00:49.428Z
81922026-03-10T19:00:49.428Zgoroutine 10074 [sync.Cond.Wait, 59 minutes]:
81932026-03-10T19:00:49.428Zsync.runtime_notifyListWait(0xc00676e610, 0x29)
81942026-03-10T19:00:49.428Z /work/go/src/runtime/sema.go:569 +0x159
81952026-03-10T19:00:49.428Zsync.(*Cond).Wait(0x0?)
81962026-03-10T19:00:49.428Z /work/go/src/sync/cond.go:70 +0x85
81972026-03-10T19:00:49.428Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c9e60})
81982026-03-10T19:00:49.428Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
81992026-03-10T19:00:49.428Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
82002026-03-10T19:00:49.428Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
82012026-03-10T19:00:49.428Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
82022026-03-10T19:00:49.428Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
82032026-03-10T19:00:49.428Z
82042026-03-10T19:00:49.428Zgoroutine 9421 [sync.Cond.Wait]:
82052026-03-10T19:00:49.428Zsync.runtime_notifyListWait(0xc0005ac550, 0x3767)
82062026-03-10T19:00:49.428Z /work/go/src/runtime/sema.go:569 +0x159
82072026-03-10T19:00:49.428Zsync.(*Cond).Wait(0x3ae1600?)
82082026-03-10T19:00:49.428Z /work/go/src/sync/cond.go:70 +0x85
82092026-03-10T19:00:49.428Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006623860})
82102026-03-10T19:00:49.428Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
82112026-03-10T19:00:49.429Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
82122026-03-10T19:00:49.429Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
82132026-03-10T19:00:49.429Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
82142026-03-10T19:00:49.429Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
82152026-03-10T19:00:49.429Z
82162026-03-10T19:00:49.429Zgoroutine 9428 [sync.Cond.Wait]:
82172026-03-10T19:00:49.429Zsync.runtime_notifyListWait(0xc0005ac550, 0x3770)
82182026-03-10T19:00:49.429Z /work/go/src/runtime/sema.go:569 +0x159
82192026-03-10T19:00:49.429Zsync.(*Cond).Wait(0x3ae1600?)
82202026-03-10T19:00:49.429Z /work/go/src/sync/cond.go:70 +0x85
82212026-03-10T19:00:49.429Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006623a10})
82222026-03-10T19:00:49.429Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
82232026-03-10T19:00:49.429Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
82242026-03-10T19:00:49.429Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
82252026-03-10T19:00:49.429Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
82262026-03-10T19:00:49.429Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
82272026-03-10T19:00:49.429Z
82282026-03-10T19:00:49.429Zgoroutine 9391 [sync.Cond.Wait]:
82292026-03-10T19:00:49.429Zsync.runtime_notifyListWait(0xc0005ac550, 0x3749)
82302026-03-10T19:00:49.429Z /work/go/src/runtime/sema.go:569 +0x159
82312026-03-10T19:00:49.429Zsync.(*Cond).Wait(0x3ae1600?)
82322026-03-10T19:00:49.429Z /work/go/src/sync/cond.go:70 +0x85
82332026-03-10T19:00:49.429Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006623170})
82342026-03-10T19:00:49.429Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
82352026-03-10T19:00:49.430Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
82362026-03-10T19:00:49.430Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
82372026-03-10T19:00:49.430Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
82382026-03-10T19:00:49.430Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
82392026-03-10T19:00:49.435Z
82402026-03-10T19:00:49.435Zgoroutine 10187 [IO wait]:
82412026-03-10T19:00:49.435Zinternal/poll.runtime_pollWait(0x7f784b7f1170, 0x72)
82422026-03-10T19:00:49.435Z /work/go/src/runtime/netpoll.go:345 +0x85
82432026-03-10T19:00:49.435Zinternal/poll.(*pollDesc).wait(0xc002a9e000?, 0xc000215800?, 0x0)
82442026-03-10T19:00:49.435Z /work/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
82452026-03-10T19:00:49.435Zinternal/poll.(*pollDesc).waitRead(...)
82462026-03-10T19:00:49.435Z /work/go/src/internal/poll/fd_poll_runtime.go:89
82472026-03-10T19:00:49.435Zinternal/poll.(*FD).Read(0xc002a9e000, {0xc000215800, 0x600, 0x600})
82482026-03-10T19:00:49.435Z /work/go/src/internal/poll/fd_unix.go:164 +0x27a
82492026-03-10T19:00:49.435Znet.(*netFD).Read(0xc002a9e000, {0xc000215800?, 0xc0048f1868?, 0x41f225?})
82502026-03-10T19:00:49.435Z /work/go/src/net/fd_posix.go:55 +0x25
82512026-03-10T19:00:49.435Znet.(*conn).Read(0xc0064de008, {0xc000215800?, 0x7f784aa413a0?, 0xc0216988a0?})
82522026-03-10T19:00:49.435Z /work/go/src/net/net.go:185 +0x45
82532026-03-10T19:00:49.435Zgithub.com/cockroachdb/cmux.(*MuxConn).Read(0xc001f062a0, {0xc000215800, 0x600, 0x600})
82542026-03-10T19:00:49.435Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x14f
82552026-03-10T19:00:49.435Zcrypto/tls.(*atLeastReader).Read(0xc0216988a0, {0xc000215800?, 0x0?, 0xc0216988a0?})
82562026-03-10T19:00:49.435Z /work/go/src/crypto/tls/conn.go:806 +0x3b
82572026-03-10T19:00:49.435Zbytes.(*Buffer).ReadFrom(0xc0049ea2b0, {0x50b6dc0, 0xc0216988a0})
82582026-03-10T19:00:49.435Z /work/go/src/bytes/buffer.go:211 +0x98
82592026-03-10T19:00:49.435Zcrypto/tls.(*Conn).readFromUntil(0xc0049ea008, {0x7f784ca77e58, 0xc001f062a0}, 0xc0048f1990?)
82602026-03-10T19:00:49.436Z /work/go/src/crypto/tls/conn.go:828 +0xde
82612026-03-10T19:00:49.436Zcrypto/tls.(*Conn).readRecordOrCCS(0xc0049ea008, 0x0)
82622026-03-10T19:00:49.436Z /work/go/src/crypto/tls/conn.go:626 +0x3cf
82632026-03-10T19:00:49.436Zcrypto/tls.(*Conn).readRecord(...)
82642026-03-10T19:00:49.436Z /work/go/src/crypto/tls/conn.go:588
82652026-03-10T19:00:49.436Zcrypto/tls.(*Conn).Read(0xc0049ea008, {0xc002f1e000, 0x8000, 0xc002b0e000?})
82662026-03-10T19:00:49.436Z /work/go/src/crypto/tls/conn.go:1370 +0x156
82672026-03-10T19:00:49.436Zbufio.(*Reader).Read(0xc004c452c0, {0xc002f8d1c0, 0x9, 0xc017ead560?})
82682026-03-10T19:00:49.436Z /work/go/src/bufio/bufio.go:241 +0x197
82692026-03-10T19:00:49.437Zio.ReadAtLeast({0x50b1100, 0xc004c452c0}, {0xc002f8d1c0, 0x9, 0x9}, 0x9)
82702026-03-10T19:00:49.437Z /work/go/src/io/io.go:335 +0x90
82712026-03-10T19:00:49.437Zio.ReadFull(...)
82722026-03-10T19:00:49.437Z /work/go/src/io/io.go:354
82732026-03-10T19:00:49.437Zgolang.org/x/net/http2.readFrameHeader({0xc002f8d1c0, 0x9, 0xa37773?}, {0x50b1100?, 0xc004c452c0?})
82742026-03-10T19:00:49.437Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x65
82752026-03-10T19:00:49.437Zgolang.org/x/net/http2.(*Framer).ReadFrame(0xc002f8d180)
82762026-03-10T19:00:49.437Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:501 +0x85
82772026-03-10T19:00:49.437Zgoogle.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc00130c340, 0xc00401af90, 0x431bf68)
82782026-03-10T19:00:49.437Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:611 +0x14e
82792026-03-10T19:00:49.437Zgoogle.golang.org/grpc.(*Server).serveStreams(0xc005fee1c0, {0x5126b60, 0xc00130c340})
82802026-03-10T19:00:49.437Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:906 +0x114
82812026-03-10T19:00:49.438Zgoogle.golang.org/grpc.(*Server).handleRawConn.func1()
82822026-03-10T19:00:49.438Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:848 +0x45
82832026-03-10T19:00:49.438Zcreated by google.golang.org/grpc.(*Server).handleRawConn in goroutine 10242
82842026-03-10T19:00:49.438Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:847 +0x15b
82852026-03-10T19:00:49.439Z
82862026-03-10T19:00:49.439Zgoroutine 9318 [select, 59 minutes]:
82872026-03-10T19:00:49.439Zgithub.com/cockroachdb/cockroach/pkg/util/hlc.(*Clock).StartMonitoringForwardClockJumps.func1()
82882026-03-10T19:00:49.439Z /work/oxidecomputer/cockroach/pkg/util/hlc/hlc.go:263 +0x148
82892026-03-10T19:00:49.439Zcreated by github.com/cockroachdb/cockroach/pkg/util/hlc.(*Clock).StartMonitoringForwardClockJumps in goroutine 9099
82902026-03-10T19:00:49.439Z /work/oxidecomputer/cockroach/pkg/util/hlc/hlc.go:255 +0x106
82912026-03-10T19:00:49.439Z
82922026-03-10T19:00:49.439Zgoroutine 10109 [select]:
82932026-03-10T19:00:49.439Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).startComputePeriodicMetrics.func1({0x50e60e0, 0xc0072ac7e0})
82942026-03-10T19:00:49.439Z /work/oxidecomputer/cockroach/pkg/server/node.go:748 +0xff
82952026-03-10T19:00:49.439Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
82962026-03-10T19:00:49.439Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
82972026-03-10T19:00:49.439Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
82982026-03-10T19:00:49.439Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
82992026-03-10T19:00:49.439Z
83002026-03-10T19:00:49.439Zgoroutine 9960 [select, 59 minutes]:
83012026-03-10T19:00:49.439Zgithub.com/cockroachdb/cockroach/pkg/util/tracing.newTracingClientStream.func2()
83022026-03-10T19:00:49.439Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:386 +0x67
83032026-03-10T19:00:49.439Zcreated by github.com/cockroachdb/cockroach/pkg/util/tracing.newTracingClientStream in goroutine 10023
83042026-03-10T19:00:49.440Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:385 +0x125
83052026-03-10T19:00:49.440Z
83062026-03-10T19:00:49.440Zgoroutine 9654 [select]:
83072026-03-10T19:00:49.440Zgithub.com/cockroachdb/cockroach/pkg/ts.(*poller).start.func1({0x50e60e0, 0xc0067083c0})
83082026-03-10T19:00:49.440Z /work/oxidecomputer/cockroach/pkg/ts/db.go:176 +0xea
83092026-03-10T19:00:49.440Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
83102026-03-10T19:00:49.440Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
83112026-03-10T19:00:49.440Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
83122026-03-10T19:00:49.440Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
83132026-03-10T19:00:49.440Z
83142026-03-10T19:00:49.440Zgoroutine 9847 [select]:
83152026-03-10T19:00:49.440Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).outputLoop(0xc001389880, {0x50e6118, 0xc003b0a280})
83162026-03-10T19:00:49.440Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:310 +0x247
83172026-03-10T19:00:49.440Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).runOutputLoop(0xc001389880, {0x50e60e0, 0xc00399d560}, 0x1?)
83182026-03-10T19:00:49.441Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:335 +0xba
83192026-03-10T19:00:49.441Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run.func1({0x50e60e0?, 0xc00399d560?})
83202026-03-10T19:00:49.441Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:297 +0x46
83212026-03-10T19:00:49.441Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
83222026-03-10T19:00:49.441Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
83232026-03-10T19:00:49.441Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9579
83242026-03-10T19:00:49.441Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
83252026-03-10T19:00:49.441Z
83262026-03-10T19:00:49.442Zgoroutine 9385 [sync.Cond.Wait]:
83272026-03-10T19:00:49.442Zsync.runtime_notifyListWait(0xc0005ac550, 0x3751)
83282026-03-10T19:00:49.442Z /work/go/src/runtime/sema.go:569 +0x159
83292026-03-10T19:00:49.442Zsync.(*Cond).Wait(0x3ae1600?)
83302026-03-10T19:00:49.442Z /work/go/src/sync/cond.go:70 +0x85
83312026-03-10T19:00:49.442Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006623050})
83322026-03-10T19:00:49.442Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
83332026-03-10T19:00:49.442Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
83342026-03-10T19:00:49.442Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
83352026-03-10T19:00:49.442Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
83362026-03-10T19:00:49.442Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
83372026-03-10T19:00:49.442Z
83382026-03-10T19:00:49.442Zgoroutine 10129 [IO wait]:
83392026-03-10T19:00:49.447Zinternal/poll.runtime_pollWait(0x7f784abc3320, 0x72)
83402026-03-10T19:00:49.447Z /work/go/src/runtime/netpoll.go:345 +0x85
83412026-03-10T19:00:49.447Zinternal/poll.(*pollDesc).wait(0xc0064f3c80?, 0xc00120a000?, 0x0)
83422026-03-10T19:00:49.447Z /work/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
83432026-03-10T19:00:49.447Zinternal/poll.(*pollDesc).waitRead(...)
83442026-03-10T19:00:49.447Z /work/go/src/internal/poll/fd_poll_runtime.go:89
83452026-03-10T19:00:49.447Zinternal/poll.(*FD).Read(0xc0064f3c80, {0xc00120a000, 0x700, 0x700})
83462026-03-10T19:00:49.447Z /work/go/src/internal/poll/fd_unix.go:164 +0x27a
83472026-03-10T19:00:49.447Znet.(*netFD).Read(0xc0064f3c80, {0xc00120a000?, 0x7f7846ff9ca8?, 0xc02f86df08?})
83482026-03-10T19:00:49.447Z /work/go/src/net/fd_posix.go:55 +0x25
83492026-03-10T19:00:49.447Znet.(*conn).Read(0xc001f20fd0, {0xc00120a000?, 0xc0073738e0?, 0x416dfb?})
83502026-03-10T19:00:49.448Z /work/go/src/net/net.go:185 +0x45
83512026-03-10T19:00:49.448Zcrypto/tls.(*atLeastReader).Read(0xc02f86df08, {0xc00120a000?, 0x0?, 0xc02f86df08?})
83522026-03-10T19:00:49.448Z /work/go/src/crypto/tls/conn.go:806 +0x3b
83532026-03-10T19:00:49.448Zbytes.(*Buffer).ReadFrom(0xc0065f17b0, {0x50b6dc0, 0xc02f86df08})
83542026-03-10T19:00:49.448Z /work/go/src/bytes/buffer.go:211 +0x98
83552026-03-10T19:00:49.448Zcrypto/tls.(*Conn).readFromUntil(0xc0065f1508, {0x50b6fa0, 0xc001f20fd0}, 0xc007373928?)
83562026-03-10T19:00:49.448Z /work/go/src/crypto/tls/conn.go:828 +0xde
83572026-03-10T19:00:49.448Zcrypto/tls.(*Conn).readRecordOrCCS(0xc0065f1508, 0x0)
83582026-03-10T19:00:49.448Z /work/go/src/crypto/tls/conn.go:626 +0x3cf
83592026-03-10T19:00:49.448Zcrypto/tls.(*Conn).readRecord(...)
83602026-03-10T19:00:49.448Z /work/go/src/crypto/tls/conn.go:588
83612026-03-10T19:00:49.448Zcrypto/tls.(*Conn).Read(0xc0065f1508, {0xc0077f6000, 0x8000, 0xc03154f7a0?})
83622026-03-10T19:00:49.448Z /work/go/src/crypto/tls/conn.go:1370 +0x156
83632026-03-10T19:00:49.448Zbufio.(*Reader).Read(0xc003bb8120, {0xc002f8c200, 0x9, 0x0?})
83642026-03-10T19:00:49.449Z /work/go/src/bufio/bufio.go:241 +0x197
83652026-03-10T19:00:49.449Zio.ReadAtLeast({0x50b1100, 0xc003bb8120}, {0xc002f8c200, 0x9, 0x9}, 0x9)
83662026-03-10T19:00:49.449Z /work/go/src/io/io.go:335 +0x90
83672026-03-10T19:00:49.449Zio.ReadFull(...)
83682026-03-10T19:00:49.449Z /work/go/src/io/io.go:354
83692026-03-10T19:00:49.449Zgolang.org/x/net/http2.readFrameHeader({0xc002f8c200, 0x9, 0x0?}, {0x50b1100?, 0xc003bb8120?})
83702026-03-10T19:00:49.449Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x65
83712026-03-10T19:00:49.450Zgolang.org/x/net/http2.(*Framer).ReadFrame(0xc002f8c1c0)
83722026-03-10T19:00:49.450Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:501 +0x85
83732026-03-10T19:00:49.450Zgoogle.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0013f61e0)
83742026-03-10T19:00:49.450Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1498 +0x41d
83752026-03-10T19:00:49.450Zcreated by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 10124
83762026-03-10T19:00:49.450Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:365 +0x184d
83772026-03-10T19:00:49.450Z
83782026-03-10T19:00:49.450Zgoroutine 9149 [select, 59 minutes]:
83792026-03-10T19:00:49.451Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1({0x50e6118, 0xc003810050})
83802026-03-10T19:00:49.451Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:116 +0x157
83812026-03-10T19:00:49.451Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.Group.GoCtx.func3()
83822026-03-10T19:00:49.451Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
83832026-03-10T19:00:49.451Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
83842026-03-10T19:00:49.451Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
83852026-03-10T19:00:49.451Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9034
83862026-03-10T19:00:49.451Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
83872026-03-10T19:00:49.451Z
83882026-03-10T19:00:49.451Zgoroutine 10355 [select]:
83892026-03-10T19:00:49.451Zgithub.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc00168ca00, {0x50e60e0, 0xc00242fe90}, 0xc0073ccaf0, {0xc007bc39c0, 0xf}, 0xc001f854a0)
83902026-03-10T19:00:49.451Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:1462 +0x27b
83912026-03-10T19:00:49.451Zgithub.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1({0x50e60e0, 0xc00242fe90})
83922026-03-10T19:00:49.451Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:1380 +0x85
83932026-03-10T19:00:49.451Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
83942026-03-10T19:00:49.451Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
83952026-03-10T19:00:49.452Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
83962026-03-10T19:00:49.452Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
83972026-03-10T19:00:49.452Z
83982026-03-10T19:00:49.452Zgoroutine 9377 [sync.Cond.Wait]:
83992026-03-10T19:00:49.453Zsync.runtime_notifyListWait(0xc0005ac550, 0x3755)
84002026-03-10T19:00:49.453Z /work/go/src/runtime/sema.go:569 +0x159
84012026-03-10T19:00:49.453Zsync.(*Cond).Wait(0x3ae1600?)
84022026-03-10T19:00:49.453Z /work/go/src/sync/cond.go:70 +0x85
84032026-03-10T19:00:49.453Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006622e40})
84042026-03-10T19:00:49.453Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
84052026-03-10T19:00:49.453Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
84062026-03-10T19:00:49.453Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
84072026-03-10T19:00:49.453Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
84082026-03-10T19:00:49.453Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
84092026-03-10T19:00:49.453Z
84102026-03-10T19:00:49.453Zgoroutine 10071 [sync.Cond.Wait, 59 minutes]:
84112026-03-10T19:00:49.453Zsync.runtime_notifyListWait(0xc00676e610, 0x26)
84122026-03-10T19:00:49.453Z /work/go/src/runtime/sema.go:569 +0x159
84132026-03-10T19:00:49.453Zsync.(*Cond).Wait(0x0?)
84142026-03-10T19:00:49.453Z /work/go/src/sync/cond.go:70 +0x85
84152026-03-10T19:00:49.453Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c9dd0})
84162026-03-10T19:00:49.453Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
84172026-03-10T19:00:49.453Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
84182026-03-10T19:00:49.453Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
84192026-03-10T19:00:49.455Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
84202026-03-10T19:00:49.455Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
84212026-03-10T19:00:49.455Z
84222026-03-10T19:00:49.455Zgoroutine 10072 [sync.Cond.Wait, 59 minutes]:
84232026-03-10T19:00:49.455Zsync.runtime_notifyListWait(0xc00676e610, 0x22)
84242026-03-10T19:00:49.455Z /work/go/src/runtime/sema.go:569 +0x159
84252026-03-10T19:00:49.455Zsync.(*Cond).Wait(0x0?)
84262026-03-10T19:00:49.455Z /work/go/src/sync/cond.go:70 +0x85
84272026-03-10T19:00:49.455Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c9e00})
84282026-03-10T19:00:49.455Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
84292026-03-10T19:00:49.455Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
84302026-03-10T19:00:49.455Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
84312026-03-10T19:00:49.455Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
84322026-03-10T19:00:49.455Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
84332026-03-10T19:00:49.455Z
84342026-03-10T19:00:49.455Zgoroutine 9166 [select, 59 minutes]:
84352026-03-10T19:00:49.455Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2({0x50e60e0, 0xc001eac450})
84362026-03-10T19:00:49.455Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/queue.go:823 +0xff
84372026-03-10T19:00:49.455Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
84382026-03-10T19:00:49.455Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
84392026-03-10T19:00:49.461Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9437
84402026-03-10T19:00:49.463Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
84412026-03-10T19:00:49.463Z
84422026-03-10T19:00:49.463Zgoroutine 9370 [sync.Cond.Wait]:
84432026-03-10T19:00:49.463Zsync.runtime_notifyListWait(0xc0005ac550, 0x373f)
84442026-03-10T19:00:49.463Z /work/go/src/runtime/sema.go:569 +0x159
84452026-03-10T19:00:49.463Zsync.(*Cond).Wait(0x3ae1600?)
84462026-03-10T19:00:49.463Z /work/go/src/sync/cond.go:70 +0x85
84472026-03-10T19:00:49.463Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006622cf0})
84482026-03-10T19:00:49.463Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
84492026-03-10T19:00:49.463Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
84502026-03-10T19:00:49.463Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
84512026-03-10T19:00:49.463Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
84522026-03-10T19:00:49.463Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
84532026-03-10T19:00:49.463Z
84542026-03-10T19:00:49.463Zgoroutine 9430 [semacquire, 59 minutes]:
84552026-03-10T19:00:49.463Zsync.runtime_Semacquire(0xc004bfdf08?)
84562026-03-10T19:00:49.464Z /work/go/src/runtime/sema.go:62 +0x25
84572026-03-10T19:00:49.464Zsync.(*WaitGroup).Wait(0x48a03c?)
84582026-03-10T19:00:49.464Z /work/go/src/sync/waitgroup.go:116 +0x48
84592026-03-10T19:00:49.464Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Wait(...)
84602026-03-10T19:00:49.464Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:236
84612026-03-10T19:00:49.464Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
84622026-03-10T19:00:49.464Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
84632026-03-10T19:00:49.464Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
84642026-03-10T19:00:49.465Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
84652026-03-10T19:00:49.465Z
84662026-03-10T19:00:49.465Zgoroutine 10064 [sync.Cond.Wait, 59 minutes]:
84672026-03-10T19:00:49.465Zsync.runtime_notifyListWait(0xc00676e610, 0x1f)
84682026-03-10T19:00:49.465Z /work/go/src/runtime/sema.go:569 +0x159
84692026-03-10T19:00:49.465Zsync.(*Cond).Wait(0x0?)
84702026-03-10T19:00:49.465Z /work/go/src/sync/cond.go:70 +0x85
84712026-03-10T19:00:49.466Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c9c80})
84722026-03-10T19:00:49.466Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
84732026-03-10T19:00:49.466Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
84742026-03-10T19:00:49.466Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
84752026-03-10T19:00:49.466Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
84762026-03-10T19:00:49.466Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
84772026-03-10T19:00:49.466Z
84782026-03-10T19:00:49.467Zgoroutine 10413 [select, 59 minutes]:
84792026-03-10T19:00:49.467Zgithub.com/cockroachdb/cockroach/pkg/util/tracing.newTracingClientStream.func2()
84802026-03-10T19:00:49.467Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:386 +0x67
84812026-03-10T19:00:49.467Zcreated by github.com/cockroachdb/cockroach/pkg/util/tracing.newTracingClientStream in goroutine 10411
84822026-03-10T19:00:49.467Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:385 +0x125
84832026-03-10T19:00:49.467Z
84842026-03-10T19:00:49.467Zgoroutine 10125 [select]:
84852026-03-10T19:00:49.467Zgithub.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc0036bf400, {0x50e60e0, 0xc0072ad7d0}, 0xc0064fd180, {0xc00322b550, 0xf}, 0xc000da0a20)
84862026-03-10T19:00:49.467Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:1462 +0x27b
84872026-03-10T19:00:49.467Zgithub.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1({0x50e60e0, 0xc0072ad7d0})
84882026-03-10T19:00:49.467Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:1380 +0x85
84892026-03-10T19:00:49.467Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
84902026-03-10T19:00:49.467Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
84912026-03-10T19:00:49.467Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10121
84922026-03-10T19:00:49.467Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
84932026-03-10T19:00:49.467Z
84942026-03-10T19:00:49.467Zgoroutine 9036 [chan receive, 59 minutes]:
84952026-03-10T19:00:49.467Zgithub.com/cockroachdb/cmux.muxListener.Accept(...)
84962026-03-10T19:00:49.467Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184
84972026-03-10T19:00:49.467Zgoogle.golang.org/grpc.(*Server).Serve(0xc005fee1c0, {0x50d12a0, 0xc0036a76b0})
84982026-03-10T19:00:49.467Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:780 +0x469
84992026-03-10T19:00:49.467Zgithub.com/cockroachdb/cockroach/pkg/server.startListenRPCAndSQL.func5.1({0xc0016ee008?, 0x0?})
85002026-03-10T19:00:49.467Z /work/oxidecomputer/cockroach/pkg/server/start_listen.go:134 +0x25
85012026-03-10T19:00:49.467Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
85022026-03-10T19:00:49.467Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
85032026-03-10T19:00:49.467Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
85042026-03-10T19:00:49.467Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
85052026-03-10T19:00:49.468Z
85062026-03-10T19:00:49.468Zgoroutine 10140 [select]:
85072026-03-10T19:00:49.468Zgithub.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc0036bf400, {0x50e60e0, 0xc006a18c00}, 0xc006dd7b80, {0xc00322b550, 0xf}, 0xc000da1ce0)
85082026-03-10T19:00:49.468Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:1462 +0x27b
85092026-03-10T19:00:49.468Zgithub.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1({0x50e60e0, 0xc006a18c00})
85102026-03-10T19:00:49.468Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:1380 +0x85
85112026-03-10T19:00:49.468Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
85122026-03-10T19:00:49.468Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
85132026-03-10T19:00:49.468Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
85142026-03-10T19:00:49.468Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
85152026-03-10T19:00:49.468Z
85162026-03-10T19:00:49.468Zgoroutine 10123 [select, 59 minutes]:
85172026-03-10T19:00:49.468Zgoogle.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0065efe00)
85182026-03-10T19:00:49.468Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:112 +0x6d
85192026-03-10T19:00:49.468Zcreated by google.golang.org/grpc.newCCBalancerWrapper in goroutine 10121
85202026-03-10T19:00:49.468Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:73 +0x22b
85212026-03-10T19:00:49.468Z
85222026-03-10T19:00:49.468Zgoroutine 10180 [IO wait]:
85232026-03-10T19:00:49.468Zinternal/poll.runtime_pollWait(0x7f784abc38f0, 0x72)
85242026-03-10T19:00:49.468Z /work/go/src/runtime/netpoll.go:345 +0x85
85252026-03-10T19:00:49.468Zinternal/poll.(*pollDesc).wait(0xc006660b80?, 0xc006ad8000?, 0x0)
85262026-03-10T19:00:49.468Z /work/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
85272026-03-10T19:00:49.468Zinternal/poll.(*pollDesc).waitRead(...)
85282026-03-10T19:00:49.468Z /work/go/src/internal/poll/fd_poll_runtime.go:89
85292026-03-10T19:00:49.468Zinternal/poll.(*FD).Read(0xc006660b80, {0xc006ad8000, 0x5500, 0x5500})
85302026-03-10T19:00:49.468Z /work/go/src/internal/poll/fd_unix.go:164 +0x27a
85312026-03-10T19:00:49.468Znet.(*netFD).Read(0xc006660b80, {0xc006ad8000?, 0x7f7846f4d928?, 0xc0159e8a98?})
85322026-03-10T19:00:49.469Z /work/go/src/net/fd_posix.go:55 +0x25
85332026-03-10T19:00:49.469Znet.(*conn).Read(0xc001f20278, {0xc006ad8000?, 0xc0073bf8e0?, 0x416dfb?})
85342026-03-10T19:00:49.469Z /work/go/src/net/net.go:185 +0x45
85352026-03-10T19:00:49.469Zcrypto/tls.(*atLeastReader).Read(0xc0159e8a98, {0xc006ad8000?, 0x0?, 0xc0159e8a98?})
85362026-03-10T19:00:49.469Z /work/go/src/crypto/tls/conn.go:806 +0x3b
85372026-03-10T19:00:49.469Zbytes.(*Buffer).ReadFrom(0xc0066a4630, {0x50b6dc0, 0xc0159e8a98})
85382026-03-10T19:00:49.469Z /work/go/src/bytes/buffer.go:211 +0x98
85392026-03-10T19:00:49.473Zcrypto/tls.(*Conn).readFromUntil(0xc0066a4388, {0x50b6fa0, 0xc001f20278}, 0xc0073bf928?)
85402026-03-10T19:00:49.473Z /work/go/src/crypto/tls/conn.go:828 +0xde
85412026-03-10T19:00:49.473Zcrypto/tls.(*Conn).readRecordOrCCS(0xc0066a4388, 0x0)
85422026-03-10T19:00:49.473Z /work/go/src/crypto/tls/conn.go:626 +0x3cf
85432026-03-10T19:00:49.473Zcrypto/tls.(*Conn).readRecord(...)
85442026-03-10T19:00:49.474Z /work/go/src/crypto/tls/conn.go:588
85452026-03-10T19:00:49.474Zcrypto/tls.(*Conn).Read(0xc0066a4388, {0xc00198a000, 0x8000, 0xc0117fefc0?})
85462026-03-10T19:00:49.474Z /work/go/src/crypto/tls/conn.go:1370 +0x156
85472026-03-10T19:00:49.474Zbufio.(*Reader).Read(0xc001bd0300, {0xc002f8cac0, 0x9, 0x0?})
85482026-03-10T19:00:49.474Z /work/go/src/bufio/bufio.go:241 +0x197
85492026-03-10T19:00:49.474Zio.ReadAtLeast({0x50b1100, 0xc001bd0300}, {0xc002f8cac0, 0x9, 0x9}, 0x9)
85502026-03-10T19:00:49.474Z /work/go/src/io/io.go:335 +0x90
85512026-03-10T19:00:49.474Zio.ReadFull(...)
85522026-03-10T19:00:49.474Z /work/go/src/io/io.go:354
85532026-03-10T19:00:49.474Zgolang.org/x/net/http2.readFrameHeader({0xc002f8cac0, 0x9, 0x0?}, {0x50b1100?, 0xc001bd0300?})
85542026-03-10T19:00:49.474Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x65
85552026-03-10T19:00:49.474Zgolang.org/x/net/http2.(*Framer).ReadFrame(0xc002f8ca80)
85562026-03-10T19:00:49.475Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:501 +0x85
85572026-03-10T19:00:49.475Zgoogle.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0013f6780)
85582026-03-10T19:00:49.475Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1498 +0x41d
85592026-03-10T19:00:49.475Zcreated by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 10139
85602026-03-10T19:00:49.475Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:365 +0x184d
85612026-03-10T19:00:49.475Z
85622026-03-10T19:00:49.475Zgoroutine 10118 [select]:
85632026-03-10T19:00:49.475Zgithub.com/cockroachdb/cockroach/pkg/util/admission.(*StoreGrantCoordinators).SetPebbleMetricsProvider.func1()
85642026-03-10T19:00:49.475Z /work/oxidecomputer/cockroach/pkg/util/admission/granter.go:1261 +0xd5
85652026-03-10T19:00:49.475Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.(*StoreGrantCoordinators).SetPebbleMetricsProvider in goroutine 9099
85662026-03-10T19:00:49.475Z /work/oxidecomputer/cockroach/pkg/util/admission/granter.go:1256 +0x28d
85672026-03-10T19:00:49.475Z
85682026-03-10T19:00:49.475Zgoroutine 10117 [select]:
85692026-03-10T19:00:49.475Zgithub.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
85702026-03-10T19:00:49.475Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:439 +0x1c9
85712026-03-10T19:00:49.475Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 9099
85722026-03-10T19:00:49.476Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:412 +0x4f
85732026-03-10T19:00:49.476Z
85742026-03-10T19:00:49.476Zgoroutine 10290 [select]:
85752026-03-10T19:00:49.476Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).outputLoop(0xc003c0e000, {0x50e6118, 0xc000d264b0})
85762026-03-10T19:00:49.476Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:310 +0x247
85772026-03-10T19:00:49.476Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).runOutputLoop(0xc003c0e000, {0x50e60e0, 0xc001b3d320}, 0x1b?)
85782026-03-10T19:00:49.477Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:335 +0xba
85792026-03-10T19:00:49.477Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run.func1({0x50e60e0?, 0xc001b3d320?})
85802026-03-10T19:00:49.477Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:297 +0x46
85812026-03-10T19:00:49.477Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
85822026-03-10T19:00:49.477Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
85832026-03-10T19:00:49.477Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9579
85842026-03-10T19:00:49.477Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
85852026-03-10T19:00:49.477Z
85862026-03-10T19:00:49.477Zgoroutine 10116 [select]:
85872026-03-10T19:00:49.477Zgithub.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue.func2()
85882026-03-10T19:00:49.477Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:366 +0x85
85892026-03-10T19:00:49.478Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue in goroutine 9099
85902026-03-10T19:00:49.478Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:363 +0x3ed
85912026-03-10T19:00:49.478Z
85922026-03-10T19:00:49.478Zgoroutine 9815 [select]:
85932026-03-10T19:00:49.478Zgithub.com/cockroachdb/cockroach/pkg/rpc.respStreamClientAdapter.recvInternal(...)
85942026-03-10T19:00:49.478Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:694
85952026-03-10T19:00:49.478Zgithub.com/cockroachdb/cockroach/pkg/rpc.rangeFeedClientAdapter.Recv(...)
85962026-03-10T19:00:49.478Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:723
85972026-03-10T19:00:49.478Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed.func3()
85982026-03-10T19:00:49.478Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:516 +0x25
85992026-03-10T19:00:49.478Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*stuckRangeFeedCanceler).do(0xc001749bd0, 0xc006123a50)
86002026-03-10T19:00:49.478Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed_canceler.go:100 +0x1b3
86012026-03-10T19:00:49.478Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed(0xc00432cf08, {0x50e6118?, 0xc001749a90?}, {{0xc00652cac0, 0x2, 0x8}, {0xc00652cac8, 0x2, 0x8}}, {0x189b8cc3df84b2bb, ...}, ...)
86022026-03-10T19:00:49.478Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:515 +0xc98
86032026-03-10T19:00:49.478Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).partialRangeFeed(0xc00432cf08, {0x50e6118, 0xc001749a90}, 0xc001749a40, {{0xc00652cac0, 0x2, 0x8}, {0xc00652cac8, 0x2, 0x8}}, ...)
86042026-03-10T19:00:49.478Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:336 +0x745
86052026-03-10T19:00:49.478Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.1({0x50e6118?, 0xc001749a90?})
86062026-03-10T19:00:49.478Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:120 +0xa9
86072026-03-10T19:00:49.478Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.Group.GoCtx.2()
86082026-03-10T19:00:49.478Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
86092026-03-10T19:00:49.478Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
86102026-03-10T19:00:49.478Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
86112026-03-10T19:00:49.478Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9813
86122026-03-10T19:00:49.478Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
86132026-03-10T19:00:49.478Z
86142026-03-10T19:00:49.478Zgoroutine 10156 [select]:
86152026-03-10T19:00:49.478Zgoogle.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc000d26280, {0xc00054b5a0, 0x5, 0x5})
86162026-03-10T19:00:49.478Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:191 +0x95
86172026-03-10T19:00:49.478Zgoogle.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc000d26280, {0xc00054b5a0?, 0xc007272420?, 0xc00378f048?})
86182026-03-10T19:00:49.478Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:171 +0x12d
86192026-03-10T19:00:49.478Zgoogle.golang.org/grpc/internal/transport.(*transportReader).Read(0xc000c59e00, {0xc00054b5a0?, 0xc00378f0c0?, 0xa418a5?})
86202026-03-10T19:00:49.478Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:485 +0x2c
86212026-03-10T19:00:49.478Zio.ReadAtLeast({0x50bb160, 0xc000c59e00}, {0xc00054b5a0, 0x5, 0x5}, 0x5)
86222026-03-10T19:00:49.478Z /work/go/src/io/io.go:335 +0x90
86232026-03-10T19:00:49.478Zio.ReadFull(...)
86242026-03-10T19:00:49.478Z /work/go/src/io/io.go:354
86252026-03-10T19:00:49.478Zgoogle.golang.org/grpc/internal/transport.(*Stream).Read(0xc0016e9b00, {0xc00054b5a0, 0x5, 0x5})
86262026-03-10T19:00:49.479Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:469 +0x96
86272026-03-10T19:00:49.479Zgoogle.golang.org/grpc.(*parser).recvMsg(0xc00054b590, 0x7fffffff)
86282026-03-10T19:00:49.479Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:559 +0x42
86292026-03-10T19:00:49.479Zgoogle.golang.org/grpc.recvAndDecompress(0x60?, 0xc0016e9b00, {0x0, 0x0}, 0x7fffffff, 0xc00378f378, {0x50ddc20, 0x778e7c0})
86302026-03-10T19:00:49.479Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:690 +0x5a
86312026-03-10T19:00:49.479Zgoogle.golang.org/grpc.recv(0x50c4e80?, {0x7f784b7a48e8, 0xc000567440}, 0xc00378f300?, {0x0?, 0x0?}, {0x3fa0a40, 0xc015082760}, 0x10?, 0xc00378f378, ...)
86322026-03-10T19:00:49.479Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:756 +0x68
86332026-03-10T19:00:49.479Zgoogle.golang.org/grpc.(*csAttempt).recvMsg(0xc000821130, {0x3fa0a40, 0xc015082760}, 0x10?)
86342026-03-10T19:00:49.479Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:983 +0x27e
86352026-03-10T19:00:49.479Zgoogle.golang.org/grpc.(*clientStream).RecvMsg.func1(0x41f225?)
86362026-03-10T19:00:49.479Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:834 +0x1f
86372026-03-10T19:00:49.479Zgoogle.golang.org/grpc.(*clientStream).withRetry(0xc0016e98c0, 0xc00378f4a8, 0xc00378f498)
86382026-03-10T19:00:49.479Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:688 +0x2ee
86392026-03-10T19:00:49.484Zgoogle.golang.org/grpc.(*clientStream).RecvMsg(0xc0016e98c0, {0x3fa0a40?, 0xc015082760?})
86402026-03-10T19:00:49.484Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:833 +0x107
86412026-03-10T19:00:49.484Zgithub.com/cockroachdb/cockroach/pkg/util/tracing.(*tracingClientStream).RecvMsg(0xc007429400, {0x3fa0a40?, 0xc015082760?})
86422026-03-10T19:00:49.484Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:441 +0x31
86432026-03-10T19:00:49.484Zgithub.com/cockroachdb/cockroach/pkg/roachpb.(*internalRangeFeedClient).Recv(0xc006877a20)
86442026-03-10T19:00:49.485Z /work/oxidecomputer/cockroach/pkg/roachpb/api.pb.go:9289 +0x46
86452026-03-10T19:00:49.485Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed.func3()
86462026-03-10T19:00:49.485Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:516 +0x25
86472026-03-10T19:00:49.485Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*stuckRangeFeedCanceler).do(0xc00717ce60, 0xc00378fa50)
86482026-03-10T19:00:49.485Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed_canceler.go:100 +0x1b3
86492026-03-10T19:00:49.485Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed(0xc0021b6f08, {0x50e6118?, 0xc00717cd20?}, {{0xc005de1378, 0x1, 0x8}, {0xc005de1380, 0x1, 0x8}}, {0x0, ...}, ...)
86502026-03-10T19:00:49.485Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:515 +0xc98
86512026-03-10T19:00:49.485Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).partialRangeFeed(0xc0021b6f08, {0x50e6118, 0xc00717cd20}, 0xc00717ccd0, {{0xc005de1378, 0x1, 0x8}, {0xc005de1380, 0x1, 0x8}}, ...)
86522026-03-10T19:00:49.485Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:336 +0x745
86532026-03-10T19:00:49.485Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.1({0x50e6118?, 0xc00717cd20?})
86542026-03-10T19:00:49.485Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:120 +0xa9
86552026-03-10T19:00:49.485Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.Group.GoCtx.2()
86562026-03-10T19:00:49.485Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
86572026-03-10T19:00:49.485Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
86582026-03-10T19:00:49.485Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
86592026-03-10T19:00:49.485Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10154
86602026-03-10T19:00:49.486Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
86612026-03-10T19:00:49.486Z
86622026-03-10T19:00:49.486Zgoroutine 9409 [sync.Cond.Wait]:
86632026-03-10T19:00:49.486Zsync.runtime_notifyListWait(0xc0005ac550, 0x3759)
86642026-03-10T19:00:49.486Z /work/go/src/runtime/sema.go:569 +0x159
86652026-03-10T19:00:49.486Zsync.(*Cond).Wait(0x3ae1600?)
86662026-03-10T19:00:49.486Z /work/go/src/sync/cond.go:70 +0x85
86672026-03-10T19:00:49.486Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc0066235f0})
86682026-03-10T19:00:49.486Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
86692026-03-10T19:00:49.486Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
86702026-03-10T19:00:49.486Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
86712026-03-10T19:00:49.486Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
86722026-03-10T19:00:49.486Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
86732026-03-10T19:00:49.487Z
86742026-03-10T19:00:49.487Zgoroutine 9405 [sync.Cond.Wait]:
86752026-03-10T19:00:49.487Zsync.runtime_notifyListWait(0xc0005ac550, 0x3738)
86762026-03-10T19:00:49.487Z /work/go/src/runtime/sema.go:569 +0x159
86772026-03-10T19:00:49.487Zsync.(*Cond).Wait(0x3ae1600?)
86782026-03-10T19:00:49.487Z /work/go/src/sync/cond.go:70 +0x85
86792026-03-10T19:00:49.487Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc0066234a0})
86802026-03-10T19:00:49.487Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
86812026-03-10T19:00:49.487Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
86822026-03-10T19:00:49.487Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
86832026-03-10T19:00:49.487Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
86842026-03-10T19:00:49.487Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
86852026-03-10T19:00:49.487Z
86862026-03-10T19:00:49.487Zgoroutine 9723 [semacquire, 59 minutes]:
86872026-03-10T19:00:49.487Zsync.runtime_Semacquire(0xc006a554b8?)
86882026-03-10T19:00:49.487Z /work/go/src/runtime/sema.go:62 +0x25
86892026-03-10T19:00:49.487Zsync.(*WaitGroup).Wait(0x44f7e0?)
86902026-03-10T19:00:49.487Z /work/go/src/sync/waitgroup.go:116 +0x48
86912026-03-10T19:00:49.487Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc0021f5c00)
86922026-03-10T19:00:49.487Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
86932026-03-10T19:00:49.487Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc0021f5c00?, {0x50e6118?, 0xc0012db180?}})
86942026-03-10T19:00:49.487Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
86952026-03-10T19:00:49.487Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed(0xc00432cf08, {0x50e6118?, 0xc0012db0e0?}, {0xc000c58d20, 0x1, 0x1d2ada0?}, {0xc001ea3e48?, 0x610d2b8?, 0xc0?}, 0x0, ...)
86962026-03-10T19:00:49.487Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:139 +0x6c5
86972026-03-10T19:00:49.487Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*dbAdapter).RangeFeed(0x50e60e0?, {0x50e6118?, 0xc0012db0e0?}, {0xc000c58d20?, 0x50e60e0?, 0xc006295dd0?}, {0x707000000000001?, 0x50e60e0?, 0x0?}, 0x0, ...)
86982026-03-10T19:00:49.487Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/db_adapter.go:78 +0x3e
86992026-03-10T19:00:49.487Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func1({0x50e6118?, 0xc0012db0e0?})
87002026-03-10T19:00:49.487Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:285 +0x6c
87012026-03-10T19:00:49.487Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
87022026-03-10T19:00:49.488Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
87032026-03-10T19:00:49.488Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
87042026-03-10T19:00:49.488Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
87052026-03-10T19:00:49.488Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9485
87062026-03-10T19:00:49.488Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
87072026-03-10T19:00:49.488Z
87082026-03-10T19:00:49.488Zgoroutine 9334 [select]:
87092026-03-10T19:00:49.488Zgoogle.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc003811720, 0x1)
87102026-03-10T19:00:49.488Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:407 +0x113
87112026-03-10T19:00:49.488Zgoogle.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc005f1c9c0)
87122026-03-10T19:00:49.488Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:534 +0x7b
87132026-03-10T19:00:49.488Zgoogle.golang.org/grpc/internal/transport.newHTTP2Client.func3()
87142026-03-10T19:00:49.488Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:415 +0x67
87152026-03-10T19:00:49.488Zcreated by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 9353
87162026-03-10T19:00:49.488Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:413 +0x1dcb
87172026-03-10T19:00:49.488Z
87182026-03-10T19:00:49.488Zgoroutine 9350 [chan receive, 59 minutes]:
87192026-03-10T19:00:49.488Zgithub.com/cockroachdb/cockroach/pkg/server.configureGRPCGateway.func1({0xc0015ebb08?, 0xc0025de930?})
87202026-03-10T19:00:49.488Z /work/oxidecomputer/cockroach/pkg/server/grpc_gateway.go:86 +0x31
87212026-03-10T19:00:49.488Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
87222026-03-10T19:00:49.489Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
87232026-03-10T19:00:49.489Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
87242026-03-10T19:00:49.489Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
87252026-03-10T19:00:49.489Z
87262026-03-10T19:00:49.489Zgoroutine 10047 [sync.Cond.Wait, 59 minutes]:
87272026-03-10T19:00:49.489Zsync.runtime_notifyListWait(0xc00676e610, 0xe)
87282026-03-10T19:00:49.489Z /work/go/src/runtime/sema.go:569 +0x159
87292026-03-10T19:00:49.489Zsync.(*Cond).Wait(0x100000003f1b920?)
87302026-03-10T19:00:49.489Z /work/go/src/sync/cond.go:70 +0x85
87312026-03-10T19:00:49.489Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c9950})
87322026-03-10T19:00:49.489Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
87332026-03-10T19:00:49.490Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
87342026-03-10T19:00:49.490Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
87352026-03-10T19:00:49.490Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
87362026-03-10T19:00:49.490Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
87372026-03-10T19:00:49.490Z
87382026-03-10T19:00:49.490Zgoroutine 9290 [chan receive, 59 minutes]:
87392026-03-10T19:00:49.493Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50e60e0?, 0xc002ff0150?})
87402026-03-10T19:00:49.493Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
87412026-03-10T19:00:49.493Zruntime/pprof.Do({0x50e6038?, 0x778e7c0?}, {{0xc00030e2c0?, 0x10?, 0xc26434f4b3011213?}}, 0xc003bb0fa8)
87422026-03-10T19:00:49.493Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
87432026-03-10T19:00:49.493Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
87442026-03-10T19:00:49.493Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
87452026-03-10T19:00:49.493Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9099
87462026-03-10T19:00:49.494Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
87472026-03-10T19:00:49.494Z
87482026-03-10T19:00:49.494Zgoroutine 10200 [select]:
87492026-03-10T19:00:49.494Zgithub.com/cockroachdb/cockroach/pkg/sql/sqlstats/persistedsqlstats.(*PersistedSQLStats).startSQLStatsFlushLoop.func1({0x50e60e0, 0xc005c71770})
87502026-03-10T19:00:49.494Z /work/oxidecomputer/cockroach/pkg/sql/sqlstats/persistedsqlstats/provider.go:137 +0x236
87512026-03-10T19:00:49.494Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
87522026-03-10T19:00:49.494Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
87532026-03-10T19:00:49.494Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
87542026-03-10T19:00:49.494Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
87552026-03-10T19:00:49.494Z
87562026-03-10T19:00:49.494Zgoroutine 9675 [select]:
87572026-03-10T19:00:49.494Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents(0xc00133c840, {0x50e6118, 0xc0046c7540}, 0xc005f94450, 0xc005f04660)
87582026-03-10T19:00:49.494Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:331 +0xc5
87592026-03-10T19:00:49.494Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func2({0x50e6118?, 0xc0046c7540?})
87602026-03-10T19:00:49.494Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:288 +0x32
87612026-03-10T19:00:49.494Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
87622026-03-10T19:00:49.494Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
87632026-03-10T19:00:49.494Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
87642026-03-10T19:00:49.494Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
87652026-03-10T19:00:49.494Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9673
87662026-03-10T19:00:49.494Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
87672026-03-10T19:00:49.494Z
87682026-03-10T19:00:49.494Zgoroutine 9410 [sync.Cond.Wait]:
87692026-03-10T19:00:49.494Zsync.runtime_notifyListWait(0xc0005ac550, 0x3769)
87702026-03-10T19:00:49.494Z /work/go/src/runtime/sema.go:569 +0x159
87712026-03-10T19:00:49.494Zsync.(*Cond).Wait(0x3ae1600?)
87722026-03-10T19:00:49.494Z /work/go/src/sync/cond.go:70 +0x85
87732026-03-10T19:00:49.494Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006623620})
87742026-03-10T19:00:49.494Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
87752026-03-10T19:00:49.494Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
87762026-03-10T19:00:49.494Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
87772026-03-10T19:00:49.494Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
87782026-03-10T19:00:49.494Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
87792026-03-10T19:00:49.494Z
87802026-03-10T19:00:49.494Zgoroutine 9528 [semacquire, 59 minutes]:
87812026-03-10T19:00:49.494Zsync.runtime_Semacquire(0xc0074c14b8?)
87822026-03-10T19:00:49.494Z /work/go/src/runtime/sema.go:62 +0x25
87832026-03-10T19:00:49.495Zsync.(*WaitGroup).Wait(0x44f7e0?)
87842026-03-10T19:00:49.495Z /work/go/src/sync/waitgroup.go:116 +0x48
87852026-03-10T19:00:49.495Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc00607e900)
87862026-03-10T19:00:49.495Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
87872026-03-10T19:00:49.495Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc00607e900?, {0x50e6118?, 0xc003b0a2d0?}})
87882026-03-10T19:00:49.495Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
87892026-03-10T19:00:49.495Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait({0x50e60e0?, 0xc00398d800?}, {0xc005b83d78, 0x2, 0xc005b83c70?})
87902026-03-10T19:00:49.495Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:193 +0xff
87912026-03-10T19:00:49.495Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0xc0045fc000, {0x50e60e0, 0xc00398d800}, 0xc00398d650)
87922026-03-10T19:00:49.495Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:291 +0x59c
87932026-03-10T19:00:49.495Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).Start.func2({0x50e60e0, 0xc00398d770})
87942026-03-10T19:00:49.495Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:215 +0x2f3
87952026-03-10T19:00:49.495Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
87962026-03-10T19:00:49.495Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
87972026-03-10T19:00:49.495Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9527
87982026-03-10T19:00:49.495Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
87992026-03-10T19:00:49.495Z
88002026-03-10T19:00:49.495Zgoroutine 9813 [select, 59 minutes]:
88012026-03-10T19:00:49.495Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1({0x50e6118, 0xc001749a90})
88022026-03-10T19:00:49.495Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:116 +0x157
88032026-03-10T19:00:49.495Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.Group.GoCtx.func3()
88042026-03-10T19:00:49.495Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
88052026-03-10T19:00:49.495Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
88062026-03-10T19:00:49.495Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
88072026-03-10T19:00:49.495Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9811
88082026-03-10T19:00:49.495Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
88092026-03-10T19:00:49.495Z
88102026-03-10T19:00:49.495Zgoroutine 10367 [select, 59 minutes]:
88112026-03-10T19:00:49.495Zgoogle.golang.org/grpc.newClientStreamWithParams.func4()
88122026-03-10T19:00:49.495Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:341 +0x8c
88132026-03-10T19:00:49.495Zcreated by google.golang.org/grpc.newClientStreamWithParams in goroutine 10366
88142026-03-10T19:00:49.495Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:340 +0xb2a
88152026-03-10T19:00:49.495Z
88162026-03-10T19:00:49.495Zgoroutine 9453 [select]:
88172026-03-10T19:00:49.495Zgithub.com/cockroachdb/cockroach/pkg/rpc.respStreamClientAdapter.recvInternal(...)
88182026-03-10T19:00:49.495Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:694
88192026-03-10T19:00:49.496Zgithub.com/cockroachdb/cockroach/pkg/rpc.rangeFeedClientAdapter.Recv(...)
88202026-03-10T19:00:49.496Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:723
88212026-03-10T19:00:49.496Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed.func3()
88222026-03-10T19:00:49.496Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:516 +0x25
88232026-03-10T19:00:49.496Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*stuckRangeFeedCanceler).do(0xc0039249b0, 0xc0028c7a50)
88242026-03-10T19:00:49.496Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed_canceler.go:100 +0x1b3
88252026-03-10T19:00:49.496Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed(0xc00432cf08, {0x50e6118?, 0xc000ff8320?}, {{0xc006771ab8, 0x1, 0x8}, {0xc006771ac0, 0x1, 0x8}}, {0x189b8cc3dad60f54, ...}, ...)
88262026-03-10T19:00:49.496Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:515 +0xc98
88272026-03-10T19:00:49.496Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).partialRangeFeed(0xc00432cf08, {0x50e6118, 0xc000ff8320}, 0xc000ff82d0, {{0xc006771ab8, 0x1, 0x8}, {0xc006771ac0, 0x1, 0x8}}, ...)
88282026-03-10T19:00:49.496Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:336 +0x745
88292026-03-10T19:00:49.496Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.1({0x50e6118?, 0xc000ff8320?})
88302026-03-10T19:00:49.496Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:120 +0xa9
88312026-03-10T19:00:49.496Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.Group.GoCtx.2()
88322026-03-10T19:00:49.496Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
88332026-03-10T19:00:49.496Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
88342026-03-10T19:00:49.496Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
88352026-03-10T19:00:49.496Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9059
88362026-03-10T19:00:49.496Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
88372026-03-10T19:00:49.496Z
88382026-03-10T19:00:49.496Zgoroutine 10201 [select]:
88392026-03-10T19:00:49.500Zgithub.com/cockroachdb/cockroach/pkg/sql/sqlstats/persistedsqlstats.(*jobMonitor).start.func1({0x50e60e0, 0xc005c717a0})
88402026-03-10T19:00:49.500Z /work/oxidecomputer/cockroach/pkg/sql/sqlstats/persistedsqlstats/scheduled_job_monitor.go:96 +0x296
88412026-03-10T19:00:49.500Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
88422026-03-10T19:00:49.500Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
88432026-03-10T19:00:49.500Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
88442026-03-10T19:00:49.500Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
88452026-03-10T19:00:49.500Z
88462026-03-10T19:00:49.500Zgoroutine 9845 [select]:
88472026-03-10T19:00:49.500Zgithub.com/cockroachdb/cockroach/pkg/rpc.respStreamClientAdapter.recvInternal(...)
88482026-03-10T19:00:49.501Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:694
88492026-03-10T19:00:49.501Zgithub.com/cockroachdb/cockroach/pkg/rpc.rangeFeedClientAdapter.Recv(...)
88502026-03-10T19:00:49.501Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:723
88512026-03-10T19:00:49.501Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed.func3()
88522026-03-10T19:00:49.501Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:516 +0x25
88532026-03-10T19:00:49.501Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*stuckRangeFeedCanceler).do(0xc003b0a0f0, 0xc00376ba50)
88542026-03-10T19:00:49.501Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed_canceler.go:100 +0x1b3
88552026-03-10T19:00:49.501Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed(0xc00432cf08, {0x50e6118?, 0xc000ed9ef0?}, {{0xc000784490, 0x1, 0x8}, {0xc000784498, 0x1, 0x8}}, {0x189b8cc3df96684f, ...}, ...)
88562026-03-10T19:00:49.501Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:515 +0xc98
88572026-03-10T19:00:49.501Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).partialRangeFeed(0xc00432cf08, {0x50e6118, 0xc000ed9ef0}, 0xc000ed9ea0, {{0xc000784490, 0x1, 0x8}, {0xc000784498, 0x1, 0x8}}, ...)
88582026-03-10T19:00:49.501Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:336 +0x745
88592026-03-10T19:00:49.501Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.1({0x50e6118?, 0xc000ed9ef0?})
88602026-03-10T19:00:49.501Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:120 +0xa9
88612026-03-10T19:00:49.501Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.Group.GoCtx.2()
88622026-03-10T19:00:49.501Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
88632026-03-10T19:00:49.501Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
88642026-03-10T19:00:49.501Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
88652026-03-10T19:00:49.501Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9843
88662026-03-10T19:00:49.501Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
88672026-03-10T19:00:49.501Z
88682026-03-10T19:00:49.501Zgoroutine 10019 [select]:
88692026-03-10T19:00:49.501Znet.(*pipe).read(0xc006dd1000, {0xc003bf6700, 0x700, 0x5?})
88702026-03-10T19:00:49.501Z /work/go/src/net/pipe.go:159 +0x155
88712026-03-10T19:00:49.501Znet.(*pipe).Read(0xc006ca08c0?, {0xc003bf6700?, 0x0?, 0x0?})
88722026-03-10T19:00:49.501Z /work/go/src/net/pipe.go:142 +0x1c
88732026-03-10T19:00:49.501Zcrypto/tls.(*atLeastReader).Read(0xc027ff0d08, {0xc003bf6700?, 0x0?, 0xc027ff0d08?})
88742026-03-10T19:00:49.501Z /work/go/src/crypto/tls/conn.go:806 +0x3b
88752026-03-10T19:00:49.502Zbytes.(*Buffer).ReadFrom(0xc006ca09b0, {0x50b6dc0, 0xc027ff0d08})
88762026-03-10T19:00:49.502Z /work/go/src/bytes/buffer.go:211 +0x98
88772026-03-10T19:00:49.503Zcrypto/tls.(*Conn).readFromUntil(0xc006ca0708, {0x7f784ca591f8, 0xc006dd1000}, 0xc004c68958?)
88782026-03-10T19:00:49.503Z /work/go/src/crypto/tls/conn.go:828 +0xde
88792026-03-10T19:00:49.503Zcrypto/tls.(*Conn).readRecordOrCCS(0xc006ca0708, 0x0)
88802026-03-10T19:00:49.503Z /work/go/src/crypto/tls/conn.go:626 +0x3cf
88812026-03-10T19:00:49.503Zcrypto/tls.(*Conn).readRecord(...)
88822026-03-10T19:00:49.503Z /work/go/src/crypto/tls/conn.go:588
88832026-03-10T19:00:49.503Zcrypto/tls.(*Conn).Read(0xc006ca0708, {0xc006f5e000, 0x8000, 0xc004c68ca8?})
88842026-03-10T19:00:49.503Z /work/go/src/crypto/tls/conn.go:1370 +0x156
88852026-03-10T19:00:49.503Zbufio.(*Reader).Read(0xc004dc85a0, {0xc00326a9e0, 0x9, 0xc004c68d30?})
88862026-03-10T19:00:49.503Z /work/go/src/bufio/bufio.go:241 +0x197
88872026-03-10T19:00:49.503Zio.ReadAtLeast({0x50b1100, 0xc004dc85a0}, {0xc00326a9e0, 0x9, 0x9}, 0x9)
88882026-03-10T19:00:49.503Z /work/go/src/io/io.go:335 +0x90
88892026-03-10T19:00:49.503Zio.ReadFull(...)
88902026-03-10T19:00:49.503Z /work/go/src/io/io.go:354
88912026-03-10T19:00:49.503Zgolang.org/x/net/http2.readFrameHeader({0xc00326a9e0, 0x9, 0x3c5cd00?}, {0x50b1100?, 0xc004dc85a0?})
88922026-03-10T19:00:49.503Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x65
88932026-03-10T19:00:49.503Zgolang.org/x/net/http2.(*Framer).ReadFrame(0xc00326a9a0)
88942026-03-10T19:00:49.503Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:501 +0x85
88952026-03-10T19:00:49.503Zgoogle.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0013f7860)
88962026-03-10T19:00:49.503Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1498 +0x41d
88972026-03-10T19:00:49.503Zcreated by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 10012
88982026-03-10T19:00:49.503Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:365 +0x184d
88992026-03-10T19:00:49.503Z
89002026-03-10T19:00:49.506Zgoroutine 9659 [chan receive, 59 minutes]:
89012026-03-10T19:00:49.506Zgithub.com/cockroachdb/cockroach/pkg/server.(*httpServer).start.func1({0x50e60e0, 0xc0067089c0})
89022026-03-10T19:00:49.506Z /work/oxidecomputer/cockroach/pkg/server/server_http.go:226 +0x45
89032026-03-10T19:00:49.506Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
89042026-03-10T19:00:49.506Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
89052026-03-10T19:00:49.506Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
89062026-03-10T19:00:49.506Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
89072026-03-10T19:00:49.506Z
89082026-03-10T19:00:49.506Zgoroutine 10046 [sync.Cond.Wait, 59 minutes]:
89092026-03-10T19:00:49.506Zsync.runtime_notifyListWait(0xc00676e610, 0x0)
89102026-03-10T19:00:49.506Z /work/go/src/runtime/sema.go:569 +0x159
89112026-03-10T19:00:49.506Zsync.(*Cond).Wait(0x0?)
89122026-03-10T19:00:49.506Z /work/go/src/sync/cond.go:70 +0x85
89132026-03-10T19:00:49.506Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c9920})
89142026-03-10T19:00:49.506Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
89152026-03-10T19:00:49.506Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
89162026-03-10T19:00:49.506Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
89172026-03-10T19:00:49.506Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
89182026-03-10T19:00:49.506Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
89192026-03-10T19:00:49.506Z
89202026-03-10T19:00:49.506Zgoroutine 10145 [IO wait]:
89212026-03-10T19:00:49.506Zinternal/poll.runtime_pollWait(0x7f784b7f09b0, 0x72)
89222026-03-10T19:00:49.506Z /work/go/src/runtime/netpoll.go:345 +0x85
89232026-03-10T19:00:49.506Zinternal/poll.(*pollDesc).wait(0xc00673b300?, 0xc0078f6000?, 0x0)
89242026-03-10T19:00:49.507Z /work/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
89252026-03-10T19:00:49.507Zinternal/poll.(*pollDesc).waitRead(...)
89262026-03-10T19:00:49.507Z /work/go/src/internal/poll/fd_poll_runtime.go:89
89272026-03-10T19:00:49.507Zinternal/poll.(*FD).Read(0xc00673b300, {0xc0078f6000, 0xa000, 0xa000})
89282026-03-10T19:00:49.507Z /work/go/src/internal/poll/fd_unix.go:164 +0x27a
89292026-03-10T19:00:49.507Znet.(*netFD).Read(0xc00673b300, {0xc0078f6000?, 0xc002e57868?, 0x41f225?})
89302026-03-10T19:00:49.507Z /work/go/src/net/fd_posix.go:55 +0x25
89312026-03-10T19:00:49.507Znet.(*conn).Read(0xc006b4e2a8, {0xc0078f6000?, 0x7f7846f4d928?, 0xc0159e8930?})
89322026-03-10T19:00:49.507Z /work/go/src/net/net.go:185 +0x45
89332026-03-10T19:00:49.507Zgithub.com/cockroachdb/cmux.(*MuxConn).Read(0xc004b2ea80, {0xc0078f6000, 0xa000, 0xa000})
89342026-03-10T19:00:49.507Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x14f
89352026-03-10T19:00:49.507Zcrypto/tls.(*atLeastReader).Read(0xc0159e8930, {0xc0078f6000?, 0x0?, 0xc0159e8930?})
89362026-03-10T19:00:49.507Z /work/go/src/crypto/tls/conn.go:806 +0x3b
89372026-03-10T19:00:49.507Zbytes.(*Buffer).ReadFrom(0xc00659f0b0, {0x50b6dc0, 0xc0159e8930})
89382026-03-10T19:00:49.507Z /work/go/src/bytes/buffer.go:211 +0x98
89392026-03-10T19:00:49.512Zcrypto/tls.(*Conn).readFromUntil(0xc00659ee08, {0x7f784ca77e58, 0xc004b2ea80}, 0xc002e57990?)
89402026-03-10T19:00:49.512Z /work/go/src/crypto/tls/conn.go:828 +0xde
89412026-03-10T19:00:49.512Zcrypto/tls.(*Conn).readRecordOrCCS(0xc00659ee08, 0x0)
89422026-03-10T19:00:49.512Z /work/go/src/crypto/tls/conn.go:626 +0x3cf
89432026-03-10T19:00:49.512Zcrypto/tls.(*Conn).readRecord(...)
89442026-03-10T19:00:49.512Z /work/go/src/crypto/tls/conn.go:588
89452026-03-10T19:00:49.512Zcrypto/tls.(*Conn).Read(0xc00659ee08, {0xc001aa8000, 0x8000, 0xc002e57ce8?})
89462026-03-10T19:00:49.512Z /work/go/src/crypto/tls/conn.go:1370 +0x156
89472026-03-10T19:00:49.512Zbufio.(*Reader).Read(0xc004c46ea0, {0xc004908ba0, 0x9, 0xc0159e8918?})
89482026-03-10T19:00:49.513Z /work/go/src/bufio/bufio.go:241 +0x197
89492026-03-10T19:00:49.514Zio.ReadAtLeast({0x50b1100, 0xc004c46ea0}, {0xc004908ba0, 0x9, 0x9}, 0x9)
89502026-03-10T19:00:49.514Z /work/go/src/io/io.go:335 +0x90
89512026-03-10T19:00:49.514Zio.ReadFull(...)
89522026-03-10T19:00:49.514Z /work/go/src/io/io.go:354
89532026-03-10T19:00:49.514Zgolang.org/x/net/http2.readFrameHeader({0xc004908ba0, 0x9, 0xc002e57e10?}, {0x50b1100?, 0xc004c46ea0?})
89542026-03-10T19:00:49.514Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x65
89552026-03-10T19:00:49.514Zgolang.org/x/net/http2.(*Framer).ReadFrame(0xc004908b60)
89562026-03-10T19:00:49.514Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:501 +0x85
89572026-03-10T19:00:49.514Zgoogle.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc002063d40, 0xc006a196b0, 0x431bf68)
89582026-03-10T19:00:49.514Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:611 +0x14e
89592026-03-10T19:00:49.514Zgoogle.golang.org/grpc.(*Server).serveStreams(0xc005fee1c0, {0x5126b60, 0xc002063d40})
89602026-03-10T19:00:49.514Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:906 +0x114
89612026-03-10T19:00:49.514Zgoogle.golang.org/grpc.(*Server).handleRawConn.func1()
89622026-03-10T19:00:49.514Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:848 +0x45
89632026-03-10T19:00:49.514Zcreated by google.golang.org/grpc.(*Server).handleRawConn in goroutine 10169
89642026-03-10T19:00:49.514Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:847 +0x15b
89652026-03-10T19:00:49.514Z
89662026-03-10T19:00:49.514Zgoroutine 9710 [select]:
89672026-03-10T19:00:49.514Zgoogle.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc001619ba0)
89682026-03-10T19:00:49.514Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1131 +0x205
89692026-03-10T19:00:49.514Zcreated by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 9706
89702026-03-10T19:00:49.514Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:341 +0x188e
89712026-03-10T19:00:49.514Z
89722026-03-10T19:00:49.514Zgoroutine 10162 [select, 59 minutes]:
89732026-03-10T19:00:49.514Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2({0x50e60e0, 0xc00672be30})
89742026-03-10T19:00:49.514Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/queue.go:823 +0xff
89752026-03-10T19:00:49.514Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
89762026-03-10T19:00:49.514Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
89772026-03-10T19:00:49.514Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10105
89782026-03-10T19:00:49.514Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
89792026-03-10T19:00:49.514Z
89802026-03-10T19:00:49.514Zgoroutine 9812 [select]:
89812026-03-10T19:00:49.514Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents(0xc0019f0000, {0x50e6118, 0xc0017499f0}, 0xc0067bbb30, 0xc0006cd320)
89822026-03-10T19:00:49.514Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:331 +0xc5
89832026-03-10T19:00:49.514Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func2({0x50e6118?, 0xc0017499f0?})
89842026-03-10T19:00:49.514Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:288 +0x32
89852026-03-10T19:00:49.515Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
89862026-03-10T19:00:49.515Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
89872026-03-10T19:00:49.515Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
89882026-03-10T19:00:49.515Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
89892026-03-10T19:00:49.515Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9470
89902026-03-10T19:00:49.515Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
89912026-03-10T19:00:49.515Z
89922026-03-10T19:00:49.515Zgoroutine 9206 [select]:
89932026-03-10T19:00:49.515Zgithub.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue.func2()
89942026-03-10T19:00:49.515Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:366 +0x85
89952026-03-10T19:00:49.515Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue in goroutine 9099
89962026-03-10T19:00:49.515Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:363 +0x3ed
89972026-03-10T19:00:49.515Z
89982026-03-10T19:00:49.515Zgoroutine 9599 [select]:
89992026-03-10T19:00:49.516Zgithub.com/cockroachdb/cockroach/pkg/sql/sqlstats/persistedsqlstats.(*jobMonitor).start.func1({0x50e60e0, 0xc006037c50})
90002026-03-10T19:00:49.516Z /work/oxidecomputer/cockroach/pkg/sql/sqlstats/persistedsqlstats/scheduled_job_monitor.go:96 +0x296
90012026-03-10T19:00:49.516Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
90022026-03-10T19:00:49.516Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
90032026-03-10T19:00:49.516Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
90042026-03-10T19:00:49.516Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
90052026-03-10T19:00:49.516Z
90062026-03-10T19:00:49.516Zgoroutine 10164 [select, 59 minutes]:
90072026-03-10T19:00:49.516Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2({0x50e60e0, 0xc00672be90})
90082026-03-10T19:00:49.516Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/queue.go:823 +0xff
90092026-03-10T19:00:49.516Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
90102026-03-10T19:00:49.516Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
90112026-03-10T19:00:49.516Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10105
90122026-03-10T19:00:49.516Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
90132026-03-10T19:00:49.516Z
90142026-03-10T19:00:49.516Zgoroutine 9968 [select, 59 minutes]:
90152026-03-10T19:00:49.516Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2({0x50e60e0, 0xc00672bdd0})
90162026-03-10T19:00:49.516Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/queue.go:823 +0xff
90172026-03-10T19:00:49.516Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
90182026-03-10T19:00:49.516Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
90192026-03-10T19:00:49.516Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10105
90202026-03-10T19:00:49.516Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
90212026-03-10T19:00:49.516Z
90222026-03-10T19:00:49.516Zgoroutine 9439 [select]:
90232026-03-10T19:00:49.516Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).startRangefeedUpdater.func1({0x50e60e0, 0xc006623d40})
90242026-03-10T19:00:49.516Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:2307 +0x28a
90252026-03-10T19:00:49.516Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
90262026-03-10T19:00:49.516Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
90272026-03-10T19:00:49.516Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
90282026-03-10T19:00:49.516Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
90292026-03-10T19:00:49.516Z
90302026-03-10T19:00:49.516Zgoroutine 10202 [select, 59 minutes]:
90312026-03-10T19:00:49.516Zgithub.com/cockroachdb/cockroach/pkg/sql/sqlstats/sslocal.(*SQLStats).Start.func1({0x50e60e0, 0xc005c717d0})
90322026-03-10T19:00:49.516Z /work/oxidecomputer/cockroach/pkg/sql/sqlstats/sslocal/sslocal_provider.go:80 +0x306
90332026-03-10T19:00:49.516Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
90342026-03-10T19:00:49.516Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
90352026-03-10T19:00:49.516Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
90362026-03-10T19:00:49.516Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
90372026-03-10T19:00:49.517Z
90382026-03-10T19:00:49.517Zgoroutine 9399 [sync.Cond.Wait]:
90392026-03-10T19:00:49.521Zsync.runtime_notifyListWait(0xc0005ac550, 0x376f)
90402026-03-10T19:00:49.521Z /work/go/src/runtime/sema.go:569 +0x159
90412026-03-10T19:00:49.521Zsync.(*Cond).Wait(0x3ae1600?)
90422026-03-10T19:00:49.521Z /work/go/src/sync/cond.go:70 +0x85
90432026-03-10T19:00:49.521Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006623380})
90442026-03-10T19:00:49.521Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
90452026-03-10T19:00:49.521Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
90462026-03-10T19:00:49.521Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
90472026-03-10T19:00:49.521Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
90482026-03-10T19:00:49.521Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
90492026-03-10T19:00:49.521Z
90502026-03-10T19:00:49.521Zgoroutine 9388 [sync.Cond.Wait]:
90512026-03-10T19:00:49.521Zsync.runtime_notifyListWait(0xc0005ac550, 0x3766)
90522026-03-10T19:00:49.521Z /work/go/src/runtime/sema.go:569 +0x159
90532026-03-10T19:00:49.521Zsync.(*Cond).Wait(0x3ae1600?)
90542026-03-10T19:00:49.521Z /work/go/src/sync/cond.go:70 +0x85
90552026-03-10T19:00:49.521Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc0066230e0})
90562026-03-10T19:00:49.521Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
90572026-03-10T19:00:49.521Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
90582026-03-10T19:00:49.521Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
90592026-03-10T19:00:49.521Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
90602026-03-10T19:00:49.521Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
90612026-03-10T19:00:49.522Z
90622026-03-10T19:00:49.522Zgoroutine 9952 [select]:
90632026-03-10T19:00:49.522Zgithub.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).manage.func1({0x50e60e0, 0xc00677a090})
90642026-03-10T19:00:49.522Z /work/oxidecomputer/cockroach/pkg/gossip/gossip.go:1420 +0x2d2
90652026-03-10T19:00:49.522Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
90662026-03-10T19:00:49.522Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
90672026-03-10T19:00:49.522Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
90682026-03-10T19:00:49.522Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
90692026-03-10T19:00:49.522Z
90702026-03-10T19:00:49.522Zgoroutine 9387 [sync.Cond.Wait]:
90712026-03-10T19:00:49.522Zsync.runtime_notifyListWait(0xc0005ac550, 0x3768)
90722026-03-10T19:00:49.522Z /work/go/src/runtime/sema.go:569 +0x159
90732026-03-10T19:00:49.522Zsync.(*Cond).Wait(0x3ae1600?)
90742026-03-10T19:00:49.522Z /work/go/src/sync/cond.go:70 +0x85
90752026-03-10T19:00:49.522Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc0066230b0})
90762026-03-10T19:00:49.522Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
90772026-03-10T19:00:49.523Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
90782026-03-10T19:00:49.523Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
90792026-03-10T19:00:49.523Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
90802026-03-10T19:00:49.523Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
90812026-03-10T19:00:49.523Z
90822026-03-10T19:00:49.523Zgoroutine 9389 [sync.Cond.Wait]:
90832026-03-10T19:00:49.523Zsync.runtime_notifyListWait(0xc0005ac550, 0x375b)
90842026-03-10T19:00:49.523Z /work/go/src/runtime/sema.go:569 +0x159
90852026-03-10T19:00:49.523Zsync.(*Cond).Wait(0x3ae1600?)
90862026-03-10T19:00:49.523Z /work/go/src/sync/cond.go:70 +0x85
90872026-03-10T19:00:49.523Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006623110})
90882026-03-10T19:00:49.523Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
90892026-03-10T19:00:49.523Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
90902026-03-10T19:00:49.523Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
90912026-03-10T19:00:49.523Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
90922026-03-10T19:00:49.523Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
90932026-03-10T19:00:49.523Z
90942026-03-10T19:00:49.523Zgoroutine 9386 [sync.Cond.Wait]:
90952026-03-10T19:00:49.523Zsync.runtime_notifyListWait(0xc0005ac550, 0x375e)
90962026-03-10T19:00:49.523Z /work/go/src/runtime/sema.go:569 +0x159
90972026-03-10T19:00:49.523Zsync.(*Cond).Wait(0x3ae1600?)
90982026-03-10T19:00:49.523Z /work/go/src/sync/cond.go:70 +0x85
90992026-03-10T19:00:49.523Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006623080})
91002026-03-10T19:00:49.523Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
91012026-03-10T19:00:49.523Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
91022026-03-10T19:00:49.523Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
91032026-03-10T19:00:49.523Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
91042026-03-10T19:00:49.523Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
91052026-03-10T19:00:49.523Z
91062026-03-10T19:00:49.523Zgoroutine 10035 [sync.Cond.Wait, 59 minutes]:
91072026-03-10T19:00:49.523Zsync.runtime_notifyListWait(0xc00676e610, 0x2)
91082026-03-10T19:00:49.523Z /work/go/src/runtime/sema.go:569 +0x159
91092026-03-10T19:00:49.525Zsync.(*Cond).Wait(0xc007238bb0?)
91102026-03-10T19:00:49.525Z /work/go/src/sync/cond.go:70 +0x85
91112026-03-10T19:00:49.525Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c9710})
91122026-03-10T19:00:49.525Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
91132026-03-10T19:00:49.525Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
91142026-03-10T19:00:49.525Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
91152026-03-10T19:00:49.525Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
91162026-03-10T19:00:49.525Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
91172026-03-10T19:00:49.525Z
91182026-03-10T19:00:49.525Zgoroutine 9205 [select]:
91192026-03-10T19:00:49.525Zgithub.com/cockroachdb/cockroach/pkg/gossip.newInfoStore.func1({0xc0034f4608?, 0x0?})
91202026-03-10T19:00:49.525Z /work/oxidecomputer/cockroach/pkg/gossip/infostore.go:192 +0x74
91212026-03-10T19:00:49.525Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
91222026-03-10T19:00:49.525Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
91232026-03-10T19:00:49.525Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
91242026-03-10T19:00:49.525Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
91252026-03-10T19:00:49.525Z
91262026-03-10T19:00:49.525Zgoroutine 9438 [select]:
91272026-03-10T19:00:49.525Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).startLeaseRenewer.func1({0x50e60e0, 0xc006623d10})
91282026-03-10T19:00:49.525Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:2256 +0x2ac
91292026-03-10T19:00:49.525Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
91302026-03-10T19:00:49.525Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
91312026-03-10T19:00:49.525Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
91322026-03-10T19:00:49.528Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
91332026-03-10T19:00:49.528Z
91342026-03-10T19:00:49.528Zgoroutine 10039 [sync.Cond.Wait, 59 minutes]:
91352026-03-10T19:00:49.528Zsync.runtime_notifyListWait(0xc00676e610, 0x6)
91362026-03-10T19:00:49.528Z /work/go/src/runtime/sema.go:569 +0x159
91372026-03-10T19:00:49.528Zsync.(*Cond).Wait(0xc001668008?)
91382026-03-10T19:00:49.528Z /work/go/src/sync/cond.go:70 +0x85
91392026-03-10T19:00:49.532Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c97d0})
91402026-03-10T19:00:49.532Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
91412026-03-10T19:00:49.532Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
91422026-03-10T19:00:49.532Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
91432026-03-10T19:00:49.532Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
91442026-03-10T19:00:49.532Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
91452026-03-10T19:00:49.532Z
91462026-03-10T19:00:49.532Zgoroutine 9967 [select, 59 minutes]:
91472026-03-10T19:00:49.532Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2({0x50e60e0, 0xc00672bda0})
91482026-03-10T19:00:49.532Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/queue.go:823 +0xff
91492026-03-10T19:00:49.532Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
91502026-03-10T19:00:49.532Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
91512026-03-10T19:00:49.532Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10105
91522026-03-10T19:00:49.532Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
91532026-03-10T19:00:49.532Z
91542026-03-10T19:00:49.532Zgoroutine 10412 [select, 59 minutes]:
91552026-03-10T19:00:49.532Zgoogle.golang.org/grpc.newClientStreamWithParams.func4()
91562026-03-10T19:00:49.532Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:341 +0x8c
91572026-03-10T19:00:49.533Zcreated by google.golang.org/grpc.newClientStreamWithParams in goroutine 10411
91582026-03-10T19:00:49.533Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:340 +0xb2a
91592026-03-10T19:00:49.533Z
91602026-03-10T19:00:49.533Zgoroutine 9872 [select, 59 minutes]:
91612026-03-10T19:00:49.533Zgithub.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run(0xc0013c6580, {0x50e60e0, 0xc0065c94a0})
91622026-03-10T19:00:49.533Z /work/oxidecomputer/cockroach/pkg/internal/client/requestbatcher/batcher.go:475 +0x326
91632026-03-10T19:00:49.533Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
91642026-03-10T19:00:49.533Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
91652026-03-10T19:00:49.533Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
91662026-03-10T19:00:49.533Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
91672026-03-10T19:00:49.533Z
91682026-03-10T19:00:49.533Zgoroutine 9964 [select, 59 minutes]:
91692026-03-10T19:00:49.533Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2({0x50e60e0, 0xc00672bd10})
91702026-03-10T19:00:49.533Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/queue.go:823 +0xff
91712026-03-10T19:00:49.533Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
91722026-03-10T19:00:49.533Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
91732026-03-10T19:00:49.533Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10105
91742026-03-10T19:00:49.533Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
91752026-03-10T19:00:49.533Z
91762026-03-10T19:00:49.533Zgoroutine 9681 [semacquire, 59 minutes]:
91772026-03-10T19:00:49.533Zsync.runtime_Semacquire(0xc0074c1350?)
91782026-03-10T19:00:49.534Z /work/go/src/runtime/sema.go:62 +0x25
91792026-03-10T19:00:49.534Zsync.(*WaitGroup).Wait(0x44f7e0?)
91802026-03-10T19:00:49.534Z /work/go/src/sync/waitgroup.go:116 +0x48
91812026-03-10T19:00:49.534Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc00607e700)
91822026-03-10T19:00:49.534Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
91832026-03-10T19:00:49.535Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc00607e700?, {0x50e6118?, 0xc000ed9ef0?}})
91842026-03-10T19:00:49.535Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
91852026-03-10T19:00:49.535Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed(0xc00432cf08, {0x50e6118?, 0xc000ed9db0?}, {0xc000819da0, 0x1, 0x1d2ada0?}, {0xb92f80?, 0x2d3e000?, 0xc0?}, 0x0, ...)
91862026-03-10T19:00:49.535Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:139 +0x6c5
91872026-03-10T19:00:49.535Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*dbAdapter).RangeFeed(0x1106d5c5361b99?, {0x50e6118?, 0xc000ed9db0?}, {0xc000819da0?, 0x0?, 0xc006068008?}, {0x50e60e0?, 0x2a50a50?, 0xc0?}, 0x0, ...)
91882026-03-10T19:00:49.535Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/db_adapter.go:78 +0x3e
91892026-03-10T19:00:49.535Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func1({0x50e6118?, 0xc000ed9db0?})
91902026-03-10T19:00:49.535Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:285 +0x6c
91912026-03-10T19:00:49.535Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
91922026-03-10T19:00:49.535Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
91932026-03-10T19:00:49.535Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
91942026-03-10T19:00:49.535Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
91952026-03-10T19:00:49.535Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9532
91962026-03-10T19:00:49.535Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
91972026-03-10T19:00:49.535Z
91982026-03-10T19:00:49.535Zgoroutine 9799 [select]:
91992026-03-10T19:00:49.535Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run(0xc005f52300, {0x50e60e0?, 0xc004fe6db0?}, 0xb991a6?, 0xc007270480, 0xc006848400)
92002026-03-10T19:00:49.535Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:269 +0x4dc
92012026-03-10T19:00:49.535Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).Start.func1({0x50e60e0?, 0xc004fe6db0?})
92022026-03-10T19:00:49.535Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:222 +0x36
92032026-03-10T19:00:49.535Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
92042026-03-10T19:00:49.535Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
92052026-03-10T19:00:49.535Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9454
92062026-03-10T19:00:49.535Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
92072026-03-10T19:00:49.535Z
92082026-03-10T19:00:49.535Zgoroutine 9693 [select, 59 minutes]:
92092026-03-10T19:00:49.535Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1({0x50e6118, 0xc0046a69b0})
92102026-03-10T19:00:49.535Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:116 +0x157
92112026-03-10T19:00:49.535Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.Group.GoCtx.func3()
92122026-03-10T19:00:49.535Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
92132026-03-10T19:00:49.535Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
92142026-03-10T19:00:49.535Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
92152026-03-10T19:00:49.535Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9691
92162026-03-10T19:00:49.535Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
92172026-03-10T19:00:49.535Z
92182026-03-10T19:00:49.535Zgoroutine 9414 [sync.Cond.Wait]:
92192026-03-10T19:00:49.535Zsync.runtime_notifyListWait(0xc0005ac550, 0x376b)
92202026-03-10T19:00:49.535Z /work/go/src/runtime/sema.go:569 +0x159
92212026-03-10T19:00:49.537Zsync.(*Cond).Wait(0x3ae1600?)
92222026-03-10T19:00:49.537Z /work/go/src/sync/cond.go:70 +0x85
92232026-03-10T19:00:49.537Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc0066236e0})
92242026-03-10T19:00:49.537Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
92252026-03-10T19:00:49.537Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
92262026-03-10T19:00:49.537Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
92272026-03-10T19:00:49.537Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
92282026-03-10T19:00:49.537Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
92292026-03-10T19:00:49.537Z
92302026-03-10T19:00:49.537Zgoroutine 9641 [select]:
92312026-03-10T19:00:49.537Zgithub.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10({0x50e60e0?, 0xc0035b4180?})
92322026-03-10T19:00:49.537Z /work/oxidecomputer/cockroach/pkg/jobs/registry.go:866 +0x1bd
92332026-03-10T19:00:49.537Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
92342026-03-10T19:00:49.537Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
92352026-03-10T19:00:49.537Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
92362026-03-10T19:00:49.537Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
92372026-03-10T19:00:49.537Z
92382026-03-10T19:00:49.537Zgoroutine 10044 [sync.Cond.Wait, 59 minutes]:
92392026-03-10T19:00:49.542Zsync.runtime_notifyListWait(0xc00676e610, 0xb)
92402026-03-10T19:00:49.542Z /work/go/src/runtime/sema.go:569 +0x159
92412026-03-10T19:00:49.542Zsync.(*Cond).Wait(0x0?)
92422026-03-10T19:00:49.542Z /work/go/src/sync/cond.go:70 +0x85
92432026-03-10T19:00:49.542Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c98c0})
92442026-03-10T19:00:49.543Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
92452026-03-10T19:00:49.543Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
92462026-03-10T19:00:49.543Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
92472026-03-10T19:00:49.543Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
92482026-03-10T19:00:49.543Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
92492026-03-10T19:00:49.543Z
92502026-03-10T19:00:49.543Zgoroutine 10037 [sync.Cond.Wait, 59 minutes]:
92512026-03-10T19:00:49.543Zsync.runtime_notifyListWait(0xc00676e610, 0x4)
92522026-03-10T19:00:49.543Z /work/go/src/runtime/sema.go:569 +0x159
92532026-03-10T19:00:49.543Zsync.(*Cond).Wait(0xc006237508?)
92542026-03-10T19:00:49.543Z /work/go/src/sync/cond.go:70 +0x85
92552026-03-10T19:00:49.543Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c9770})
92562026-03-10T19:00:49.543Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
92572026-03-10T19:00:49.543Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
92582026-03-10T19:00:49.543Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
92592026-03-10T19:00:49.543Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
92602026-03-10T19:00:49.543Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
92612026-03-10T19:00:49.543Z
92622026-03-10T19:00:49.543Zgoroutine 9846 [chan receive, 59 minutes]:
92632026-03-10T19:00:49.543Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).rangeFeedWithRangeID(0xc004e66008, 0x3f72e40?, 0xc006040120, {0x7f784b79a928, 0xc006042180})
92642026-03-10T19:00:49.543Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:249 +0x76b
92652026-03-10T19:00:49.543Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed(...)
92662026-03-10T19:00:49.543Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:149
92672026-03-10T19:00:49.543Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).RangeFeed(0xc001668008, 0xc006040120, {0x511a910, 0xc006042180})
92682026-03-10T19:00:49.543Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:3082 +0x16f
92692026-03-10T19:00:49.543Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).RangeFeed(0xc006849e00, 0xc006040120, {0x511a910, 0xc006042180})
92702026-03-10T19:00:49.543Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/stores.go:216 +0x19a
92712026-03-10T19:00:49.543Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed(0x3f72e40?, 0xc003b0a1c0?, {0x511a910, 0xc006042180})
92722026-03-10T19:00:49.543Z /work/oxidecomputer/cockroach/pkg/server/node.go:1232 +0x28
92732026-03-10T19:00:49.543Zgithub.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed.func1()
92742026-03-10T19:00:49.543Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:753 +0xa7
92752026-03-10T19:00:49.543Zcreated by github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed in goroutine 9845
92762026-03-10T19:00:49.543Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:750 +0x17c
92772026-03-10T19:00:49.543Z
92782026-03-10T19:00:49.544Zgoroutine 9407 [sync.Cond.Wait]:
92792026-03-10T19:00:49.545Zsync.runtime_notifyListWait(0xc0005ac550, 0x374f)
92802026-03-10T19:00:49.545Z /work/go/src/runtime/sema.go:569 +0x159
92812026-03-10T19:00:49.545Zsync.(*Cond).Wait(0x3ae1600?)
92822026-03-10T19:00:49.545Z /work/go/src/sync/cond.go:70 +0x85
92832026-03-10T19:00:49.545Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006623500})
92842026-03-10T19:00:49.545Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
92852026-03-10T19:00:49.545Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
92862026-03-10T19:00:49.545Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
92872026-03-10T19:00:49.545Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
92882026-03-10T19:00:49.546Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
92892026-03-10T19:00:49.546Z
92902026-03-10T19:00:49.546Zgoroutine 9692 [select]:
92912026-03-10T19:00:49.546Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents(0xc00133c580, {0x50e6118, 0xc0046a68c0}, 0xc005f941b0, 0xc0043c2960)
92922026-03-10T19:00:49.546Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:331 +0xc5
92932026-03-10T19:00:49.546Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func2({0x50e6118?, 0xc0046a68c0?})
92942026-03-10T19:00:49.546Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:288 +0x32
92952026-03-10T19:00:49.546Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
92962026-03-10T19:00:49.546Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
92972026-03-10T19:00:49.546Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
92982026-03-10T19:00:49.546Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
92992026-03-10T19:00:49.546Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9671
93002026-03-10T19:00:49.546Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
93012026-03-10T19:00:49.546Z
93022026-03-10T19:00:49.546Zgoroutine 9772 [chan receive, 59 minutes]:
93032026-03-10T19:00:49.546Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).rangeFeedWithRangeID(0xc001694008, 0x3f72e40?, 0xc005d41560, {0x7f784b79a928, 0xc002369380})
93042026-03-10T19:00:49.546Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:249 +0x76b
93052026-03-10T19:00:49.546Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed(...)
93062026-03-10T19:00:49.546Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:149
93072026-03-10T19:00:49.546Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).RangeFeed(0xc001668008, 0xc005d41560, {0x511a910, 0xc002369380})
93082026-03-10T19:00:49.546Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:3082 +0x16f
93092026-03-10T19:00:49.547Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).RangeFeed(0xc006849e00, 0xc005d41560, {0x511a910, 0xc002369380})
93102026-03-10T19:00:49.547Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/stores.go:216 +0x19a
93112026-03-10T19:00:49.547Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed(0x3f72e40?, 0xc000e29b10?, {0x511a910, 0xc002369380})
93122026-03-10T19:00:49.547Z /work/oxidecomputer/cockroach/pkg/server/node.go:1232 +0x28
93132026-03-10T19:00:49.547Zgithub.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed.func1()
93142026-03-10T19:00:49.547Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:753 +0xa7
93152026-03-10T19:00:49.547Zcreated by github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed in goroutine 9771
93162026-03-10T19:00:49.548Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:750 +0x17c
93172026-03-10T19:00:49.548Z
93182026-03-10T19:00:49.548Zgoroutine 9963 [select]:
93192026-03-10T19:00:49.548Zgoogle.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc0065fb5e0, {0xc00654fd50, 0x5, 0x5})
93202026-03-10T19:00:49.548Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:179 +0x8a
93212026-03-10T19:00:49.548Zgoogle.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc0065fb5e0, {0xc00654fd50?, 0xc006ababd0?, 0xc00e4997d8?})
93222026-03-10T19:00:49.548Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:173 +0x15b
93232026-03-10T19:00:49.548Zgoogle.golang.org/grpc/internal/transport.(*transportReader).Read(0xc00672b500, {0xc00654fd50?, 0xc00e499850?, 0xa504c5?})
93242026-03-10T19:00:49.548Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:485 +0x2c
93252026-03-10T19:00:49.548Zio.ReadAtLeast({0x50bb160, 0xc00672b500}, {0xc00654fd50, 0x5, 0x5}, 0x5)
93262026-03-10T19:00:49.548Z /work/go/src/io/io.go:335 +0x90
93272026-03-10T19:00:49.548Zio.ReadFull(...)
93282026-03-10T19:00:49.548Z /work/go/src/io/io.go:354
93292026-03-10T19:00:49.548Zgoogle.golang.org/grpc/internal/transport.(*Stream).Read(0xc0067286c0, {0xc00654fd50, 0x5, 0x5})
93302026-03-10T19:00:49.548Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:469 +0x96
93312026-03-10T19:00:49.549Zgoogle.golang.org/grpc.(*parser).recvMsg(0xc00654fd40, 0x7fffffff)
93322026-03-10T19:00:49.549Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:559 +0x42
93332026-03-10T19:00:49.549Zgoogle.golang.org/grpc.recvAndDecompress(0x36?, 0xc0067286c0, {0x0, 0x0}, 0x7fffffff, 0xc00e499b00, {0x50ddc20, 0x778e7c0})
93342026-03-10T19:00:49.549Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:690 +0x5a
93352026-03-10T19:00:49.549Zgoogle.golang.org/grpc.recv(0xc001fa1d40?, {0x7f784b7a48e8, 0xc000567440}, 0x38?, {0x0?, 0x0?}, {0x3f300a0, 0xc00e3d70e0}, 0xc00e499a48?, 0xc00e499b00, ...)
93362026-03-10T19:00:49.549Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:756 +0x68
93372026-03-10T19:00:49.549Zgoogle.golang.org/grpc.(*serverStream).RecvMsg(0xc001fa1d40, {0x3f300a0, 0xc00e3d70e0})
93382026-03-10T19:00:49.549Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:1583 +0x16a
93392026-03-10T19:00:49.553Zgithub.com/cockroachdb/cockroach/pkg/gossip.(*gossipGossipServer).Recv(0xc006d31790)
93402026-03-10T19:00:49.553Z /work/oxidecomputer/cockroach/pkg/gossip/gossip.pb.go:679 +0x46
93412026-03-10T19:00:49.553Zgithub.com/cockroachdb/cockroach/pkg/gossip.(*server).gossipReceiver(0xc006332300, {0x50e60e0, 0xc00672b7a0}, 0xc006b4fc40, 0xc0065bf140, 0xc00e499f00)
93422026-03-10T19:00:49.553Z /work/oxidecomputer/cockroach/pkg/gossip/server.go:331 +0xa7f
93432026-03-10T19:00:49.553Zgithub.com/cockroachdb/cockroach/pkg/gossip.(*server).Gossip.func3({0x50e60e0?, 0xc00672b7a0?})
93442026-03-10T19:00:49.553Z /work/oxidecomputer/cockroach/pkg/gossip/server.go:139 +0x74
93452026-03-10T19:00:49.553Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
93462026-03-10T19:00:49.553Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
93472026-03-10T19:00:49.553Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9962
93482026-03-10T19:00:49.553Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
93492026-03-10T19:00:49.553Z
93502026-03-10T19:00:49.553Zgoroutine 9204 [select, 59 minutes]:
93512026-03-10T19:00:49.553Zgithub.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler.func1()
93522026-03-10T19:00:49.553Z /work/oxidecomputer/cockroach/pkg/security/certificate_manager.go:241 +0x113
93532026-03-10T19:00:49.553Zcreated by github.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler in goroutine 9099
93542026-03-10T19:00:49.553Z /work/oxidecomputer/cockroach/pkg/security/certificate_manager.go:238 +0x7d
93552026-03-10T19:00:49.553Z
93562026-03-10T19:00:49.553Zgoroutine 9711 [select]:
93572026-03-10T19:00:49.553Znet.(*pipe).read(0xc006dd1080, {0xc003616c00, 0x600, 0x5?})
93582026-03-10T19:00:49.554Z /work/go/src/net/pipe.go:159 +0x155
93592026-03-10T19:00:49.554Znet.(*pipe).Read(0xc006ca0540?, {0xc003616c00?, 0x0?, 0x0?})
93602026-03-10T19:00:49.554Z /work/go/src/net/pipe.go:142 +0x1c
93612026-03-10T19:00:49.554Zcrypto/tls.(*atLeastReader).Read(0xc027ff0d20, {0xc003616c00?, 0x0?, 0xc027ff0d20?})
93622026-03-10T19:00:49.554Z /work/go/src/crypto/tls/conn.go:806 +0x3b
93632026-03-10T19:00:49.554Zbytes.(*Buffer).ReadFrom(0xc006ca0630, {0x50b6dc0, 0xc027ff0d20})
93642026-03-10T19:00:49.554Z /work/go/src/bytes/buffer.go:211 +0x98
93652026-03-10T19:00:49.554Zcrypto/tls.(*Conn).readFromUntil(0xc006ca0388, {0x7f784ca591f8, 0xc006dd1080}, 0xc0028d1990?)
93662026-03-10T19:00:49.554Z /work/go/src/crypto/tls/conn.go:828 +0xde
93672026-03-10T19:00:49.554Zcrypto/tls.(*Conn).readRecordOrCCS(0xc006ca0388, 0x0)
93682026-03-10T19:00:49.554Z /work/go/src/crypto/tls/conn.go:626 +0x3cf
93692026-03-10T19:00:49.554Zcrypto/tls.(*Conn).readRecord(...)
93702026-03-10T19:00:49.554Z /work/go/src/crypto/tls/conn.go:588
93712026-03-10T19:00:49.554Zcrypto/tls.(*Conn).Read(0xc006ca0388, {0xc006fac000, 0x8000, 0x7f784ab88e48?})
93722026-03-10T19:00:49.554Z /work/go/src/crypto/tls/conn.go:1370 +0x156
93732026-03-10T19:00:49.554Zbufio.(*Reader).Read(0xc004c468a0, {0xc001ee9b60, 0x9, 0xc000f25808?})
93742026-03-10T19:00:49.554Z /work/go/src/bufio/bufio.go:241 +0x197
93752026-03-10T19:00:49.554Zio.ReadAtLeast({0x50b1100, 0xc004c468a0}, {0xc001ee9b60, 0x9, 0x9}, 0x9)
93762026-03-10T19:00:49.554Z /work/go/src/io/io.go:335 +0x90
93772026-03-10T19:00:49.554Zio.ReadFull(...)
93782026-03-10T19:00:49.554Z /work/go/src/io/io.go:354
93792026-03-10T19:00:49.554Zgolang.org/x/net/http2.readFrameHeader({0xc001ee9b60, 0x9, 0xc02787bc38?}, {0x50b1100?, 0xc004c468a0?})
93802026-03-10T19:00:49.554Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x65
93812026-03-10T19:00:49.554Zgolang.org/x/net/http2.(*Framer).ReadFrame(0xc001ee9b20)
93822026-03-10T19:00:49.554Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:501 +0x85
93832026-03-10T19:00:49.554Zgoogle.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc001619ba0, 0xc006e971d0, 0x431bf68)
93842026-03-10T19:00:49.554Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:611 +0x14e
93852026-03-10T19:00:49.554Zgoogle.golang.org/grpc.(*Server).serveStreams(0xc003999340, {0x5126b60, 0xc001619ba0})
93862026-03-10T19:00:49.554Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:906 +0x114
93872026-03-10T19:00:49.554Zgoogle.golang.org/grpc.(*Server).handleRawConn.func1()
93882026-03-10T19:00:49.554Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:848 +0x45
93892026-03-10T19:00:49.554Zcreated by google.golang.org/grpc.(*Server).handleRawConn in goroutine 9706
93902026-03-10T19:00:49.554Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:847 +0x15b
93912026-03-10T19:00:49.554Z
93922026-03-10T19:00:49.554Zgoroutine 9951 [select, 59 minutes]:
93932026-03-10T19:00:49.554Zgithub.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).bootstrap.func1({0x50e60e0?, 0xc00677a060?})
93942026-03-10T19:00:49.555Z /work/oxidecomputer/cockroach/pkg/gossip/gossip.go:1385 +0x253
93952026-03-10T19:00:49.555Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
93962026-03-10T19:00:49.555Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
93972026-03-10T19:00:49.555Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
93982026-03-10T19:00:49.555Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
93992026-03-10T19:00:49.555Z
94002026-03-10T19:00:49.555Zgoroutine 9849 [select]:
94012026-03-10T19:00:49.555Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents(0xc0045fc000, {0x50e6118, 0xc003b0a2d0}, 0xc00398d650, 0xc00032b6e0)
94022026-03-10T19:00:49.555Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:331 +0xc5
94032026-03-10T19:00:49.555Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func2({0x50e6118?, 0xc003b0a2d0?})
94042026-03-10T19:00:49.555Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:288 +0x32
94052026-03-10T19:00:49.555Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
94062026-03-10T19:00:49.555Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
94072026-03-10T19:00:49.555Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
94082026-03-10T19:00:49.555Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
94092026-03-10T19:00:49.555Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9528
94102026-03-10T19:00:49.555Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
94112026-03-10T19:00:49.555Z
94122026-03-10T19:00:49.555Zgoroutine 10041 [sync.Cond.Wait, 59 minutes]:
94132026-03-10T19:00:49.555Zsync.runtime_notifyListWait(0xc00676e610, 0x8)
94142026-03-10T19:00:49.555Z /work/go/src/runtime/sema.go:569 +0x159
94152026-03-10T19:00:49.555Zsync.(*Cond).Wait(0x0?)
94162026-03-10T19:00:49.555Z /work/go/src/sync/cond.go:70 +0x85
94172026-03-10T19:00:49.555Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c9830})
94182026-03-10T19:00:49.555Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
94192026-03-10T19:00:49.556Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
94202026-03-10T19:00:49.556Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
94212026-03-10T19:00:49.556Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
94222026-03-10T19:00:49.556Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
94232026-03-10T19:00:49.556Z
94242026-03-10T19:00:49.556Zgoroutine 9850 [select, 59 minutes]:
94252026-03-10T19:00:49.556Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1({0x50e6118, 0xc003b0a370})
94262026-03-10T19:00:49.556Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:116 +0x157
94272026-03-10T19:00:49.556Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.Group.GoCtx.func3()
94282026-03-10T19:00:49.556Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
94292026-03-10T19:00:49.556Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
94302026-03-10T19:00:49.556Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
94312026-03-10T19:00:49.556Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9848
94322026-03-10T19:00:49.556Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
94332026-03-10T19:00:49.556Z
94342026-03-10T19:00:49.556Zgoroutine 9595 [select, 59 minutes]:
94352026-03-10T19:00:49.556Zgithub.com/cockroachdb/cockroach/pkg/sql/gcjob/gcjobnotifier.(*Notifier).run(0xc0043a6c60, {0x257b?, 0xc0060372f0?})
94362026-03-10T19:00:49.556Z /work/oxidecomputer/cockroach/pkg/sql/gcjob/gcjobnotifier/notifier.go:145 +0xc8
94372026-03-10T19:00:49.556Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
94382026-03-10T19:00:49.556Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
94392026-03-10T19:00:49.560Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
94402026-03-10T19:00:49.560Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
94412026-03-10T19:00:49.560Z
94422026-03-10T19:00:49.560Zgoroutine 10049 [sync.Cond.Wait, 59 minutes]:
94432026-03-10T19:00:49.560Zsync.runtime_notifyListWait(0xc00676e610, 0x10)
94442026-03-10T19:00:49.560Z /work/go/src/runtime/sema.go:569 +0x159
94452026-03-10T19:00:49.560Zsync.(*Cond).Wait(0xc001f23340?)
94462026-03-10T19:00:49.561Z /work/go/src/sync/cond.go:70 +0x85
94472026-03-10T19:00:49.561Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c99b0})
94482026-03-10T19:00:49.561Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
94492026-03-10T19:00:49.561Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
94502026-03-10T19:00:49.561Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
94512026-03-10T19:00:49.561Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
94522026-03-10T19:00:49.561Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
94532026-03-10T19:00:49.561Z
94542026-03-10T19:00:49.561Zgoroutine 9724 [select]:
94552026-03-10T19:00:49.561Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents(0xc006caf080, {0x50e6118, 0xc0012db0e0}, 0xc000c58c60, 0xc005cf6900)
94562026-03-10T19:00:49.561Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:331 +0xc5
94572026-03-10T19:00:49.561Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func2({0x50e6118?, 0xc0012db0e0?})
94582026-03-10T19:00:49.561Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:288 +0x32
94592026-03-10T19:00:49.561Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
94602026-03-10T19:00:49.561Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
94612026-03-10T19:00:49.561Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
94622026-03-10T19:00:49.561Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
94632026-03-10T19:00:49.561Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9485
94642026-03-10T19:00:49.562Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
94652026-03-10T19:00:49.562Z
94662026-03-10T19:00:49.562Zgoroutine 9783 [select]:
94672026-03-10T19:00:49.562Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).outputLoop(0xc004724540, {0x50e6118, 0xc004129d10})
94682026-03-10T19:00:49.562Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:310 +0x247
94692026-03-10T19:00:49.562Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).runOutputLoop(0xc004724540, {0x50e60e0, 0xc00457af60}, 0xc003cb45a0?)
94702026-03-10T19:00:49.562Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:335 +0xba
94712026-03-10T19:00:49.562Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run.func1({0x50e60e0?, 0xc00457af60?})
94722026-03-10T19:00:49.562Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:297 +0x46
94732026-03-10T19:00:49.562Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
94742026-03-10T19:00:49.562Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
94752026-03-10T19:00:49.562Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9579
94762026-03-10T19:00:49.562Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
94772026-03-10T19:00:49.562Z
94782026-03-10T19:00:49.562Zgoroutine 9843 [select, 59 minutes]:
94792026-03-10T19:00:49.562Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1({0x50e6118, 0xc000ed9ef0})
94802026-03-10T19:00:49.562Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:116 +0x157
94812026-03-10T19:00:49.563Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.Group.GoCtx.func3()
94822026-03-10T19:00:49.564Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
94832026-03-10T19:00:49.564Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
94842026-03-10T19:00:49.564Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
94852026-03-10T19:00:49.564Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9681
94862026-03-10T19:00:49.564Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
94872026-03-10T19:00:49.564Z
94882026-03-10T19:00:49.564Zgoroutine 10043 [sync.Cond.Wait, 59 minutes]:
94892026-03-10T19:00:49.564Zsync.runtime_notifyListWait(0xc00676e610, 0xa)
94902026-03-10T19:00:49.564Z /work/go/src/runtime/sema.go:569 +0x159
94912026-03-10T19:00:49.564Zsync.(*Cond).Wait(0x100000003f1b920?)
94922026-03-10T19:00:49.564Z /work/go/src/sync/cond.go:70 +0x85
94932026-03-10T19:00:49.564Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c9890})
94942026-03-10T19:00:49.564Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
94952026-03-10T19:00:49.564Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
94962026-03-10T19:00:49.564Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
94972026-03-10T19:00:49.564Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
94982026-03-10T19:00:49.564Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
94992026-03-10T19:00:49.564Z
95002026-03-10T19:00:49.564Zgoroutine 10157 [select, 59 minutes]:
95012026-03-10T19:00:49.564Zgoogle.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0043fa840)
95022026-03-10T19:00:49.564Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:112 +0x6d
95032026-03-10T19:00:49.564Zcreated by google.golang.org/grpc.newCCBalancerWrapper in goroutine 10156
95042026-03-10T19:00:49.564Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:73 +0x22b
95052026-03-10T19:00:49.564Z
95062026-03-10T19:00:49.564Zgoroutine 10619 [select]:
95072026-03-10T19:00:49.564Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).outputLoop(0xc004fa28c0, {0x50e6118, 0xc008001400})
95082026-03-10T19:00:49.565Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:310 +0x247
95092026-03-10T19:00:49.565Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).runOutputLoop(0xc004fa28c0, {0x50e60e0, 0xc008122150}, 0x1b?)
95102026-03-10T19:00:49.565Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:335 +0xba
95112026-03-10T19:00:49.565Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run.func1({0x50e60e0?, 0xc008122150?})
95122026-03-10T19:00:49.565Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:297 +0x46
95132026-03-10T19:00:49.565Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
95142026-03-10T19:00:49.565Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
95152026-03-10T19:00:49.565Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9799
95162026-03-10T19:00:49.565Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
95172026-03-10T19:00:49.565Z
95182026-03-10T19:00:49.565Zgoroutine 9354 [chan receive, 59 minutes]:
95192026-03-10T19:00:49.565Zgithub.com/cockroachdb/cockroach/pkg/server.configureGRPCGateway.func5({0x50e60e0, 0xc0064d4cc0})
95202026-03-10T19:00:49.565Z /work/oxidecomputer/cockroach/pkg/server/grpc_gateway.go:127 +0x45
95212026-03-10T19:00:49.565Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
95222026-03-10T19:00:49.565Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
95232026-03-10T19:00:49.565Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
95242026-03-10T19:00:49.565Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
95252026-03-10T19:00:49.566Z
95262026-03-10T19:00:49.566Zgoroutine 9769 [select, 59 minutes]:
95272026-03-10T19:00:49.566Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1({0x50e6118, 0xc000e29900})
95282026-03-10T19:00:49.566Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:116 +0x157
95292026-03-10T19:00:49.566Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.Group.GoCtx.func3()
95302026-03-10T19:00:49.566Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
95312026-03-10T19:00:49.566Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
95322026-03-10T19:00:49.566Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
95332026-03-10T19:00:49.566Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9674
95342026-03-10T19:00:49.566Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
95352026-03-10T19:00:49.567Z
95362026-03-10T19:00:49.567Zgoroutine 9658 [chan receive, 59 minutes]:
95372026-03-10T19:00:49.567Zgithub.com/cockroachdb/cockroach/pkg/util/netutil.MakeServer.func2({0xc00670c308?, 0xc0004de960?})
95382026-03-10T19:00:49.567Z /work/oxidecomputer/cockroach/pkg/util/netutil/net.go:113 +0x49
95392026-03-10T19:00:49.571Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
95402026-03-10T19:00:49.571Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
95412026-03-10T19:00:49.571Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
95422026-03-10T19:00:49.571Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
95432026-03-10T19:00:49.571Z
95442026-03-10T19:00:49.571Zgoroutine 9725 [select, 59 minutes]:
95452026-03-10T19:00:49.571Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1({0x50e6118, 0xc0012db180})
95462026-03-10T19:00:49.571Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:116 +0x157
95472026-03-10T19:00:49.571Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.Group.GoCtx.func3()
95482026-03-10T19:00:49.571Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
95492026-03-10T19:00:49.571Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
95502026-03-10T19:00:49.571Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
95512026-03-10T19:00:49.571Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9723
95522026-03-10T19:00:49.571Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
95532026-03-10T19:00:49.571Z
95542026-03-10T19:00:49.571Zgoroutine 9775 [select]:
95552026-03-10T19:00:49.571Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).outputLoop(0xc00463a620, {0x50e6118, 0xc000e29c20})
95562026-03-10T19:00:49.571Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:310 +0x247
95572026-03-10T19:00:49.572Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).runOutputLoop(0xc00463a620, {0x50e60e0, 0xc0025d7b00}, 0x3?)
95582026-03-10T19:00:49.572Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:335 +0xba
95592026-03-10T19:00:49.572Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run.func1({0x50e60e0?, 0xc0025d7b00?})
95602026-03-10T19:00:49.572Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:297 +0x46
95612026-03-10T19:00:49.572Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
95622026-03-10T19:00:49.572Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
95632026-03-10T19:00:49.572Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9773
95642026-03-10T19:00:49.572Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
95652026-03-10T19:00:49.572Z
95662026-03-10T19:00:49.572Zgoroutine 9413 [sync.Cond.Wait]:
95672026-03-10T19:00:49.572Zsync.runtime_notifyListWait(0xc0005ac550, 0x3737)
95682026-03-10T19:00:49.572Z /work/go/src/runtime/sema.go:569 +0x159
95692026-03-10T19:00:49.572Zsync.(*Cond).Wait(0x3ae1600?)
95702026-03-10T19:00:49.572Z /work/go/src/sync/cond.go:70 +0x85
95712026-03-10T19:00:49.572Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc0066236b0})
95722026-03-10T19:00:49.572Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
95732026-03-10T19:00:49.573Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
95742026-03-10T19:00:49.573Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
95752026-03-10T19:00:49.573Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
95762026-03-10T19:00:49.573Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
95772026-03-10T19:00:49.573Z
95782026-03-10T19:00:49.573Zgoroutine 9848 [semacquire, 59 minutes]:
95792026-03-10T19:00:49.573Zsync.runtime_Semacquire(0xc0074c1518?)
95802026-03-10T19:00:49.573Z /work/go/src/runtime/sema.go:62 +0x25
95812026-03-10T19:00:49.573Zsync.(*WaitGroup).Wait(0x44f7e0?)
95822026-03-10T19:00:49.573Z /work/go/src/sync/waitgroup.go:116 +0x48
95832026-03-10T19:00:49.573Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc00607e940)
95842026-03-10T19:00:49.573Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
95852026-03-10T19:00:49.573Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc00607e940?, {0x50e6118?, 0xc003b0a370?}})
95862026-03-10T19:00:49.573Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
95872026-03-10T19:00:49.573Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed(0xc00432cf08, {0x50e6118?, 0xc003b0a2d0?}, {0xc00398d6e0, 0x1, 0x50e60e0?}, {0xb92f80?, 0x2d3e000?, 0xc0?}, 0x0, ...)
95882026-03-10T19:00:49.573Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:139 +0x6c5
95892026-03-10T19:00:49.573Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*dbAdapter).RangeFeed(0x2677?, {0x50e6118?, 0xc003b0a2d0?}, {0xc00398d6e0?, 0x50e60e0?, 0xc00399d560?}, {0x707000000000001?, 0x50e60e0?, 0x0?}, 0x0, ...)
95902026-03-10T19:00:49.573Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/db_adapter.go:78 +0x3e
95912026-03-10T19:00:49.573Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func1({0x50e6118?, 0xc003b0a2d0?})
95922026-03-10T19:00:49.573Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:285 +0x6c
95932026-03-10T19:00:49.573Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
95942026-03-10T19:00:49.574Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
95952026-03-10T19:00:49.575Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
95962026-03-10T19:00:49.575Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
95972026-03-10T19:00:49.575Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9528
95982026-03-10T19:00:49.575Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
95992026-03-10T19:00:49.575Z
96002026-03-10T19:00:49.575Zgoroutine 9674 [semacquire, 59 minutes]:
96012026-03-10T19:00:49.575Zsync.runtime_Semacquire(0xc006fcad80?)
96022026-03-10T19:00:49.575Z /work/go/src/runtime/sema.go:62 +0x25
96032026-03-10T19:00:49.575Zsync.(*WaitGroup).Wait(0x44f7e0?)
96042026-03-10T19:00:49.575Z /work/go/src/sync/waitgroup.go:116 +0x48
96052026-03-10T19:00:49.575Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc005d2d9c0)
96062026-03-10T19:00:49.575Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
96072026-03-10T19:00:49.575Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc005d2d9c0?, {0x50e6118?, 0xc000e29900?}})
96082026-03-10T19:00:49.575Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
96092026-03-10T19:00:49.575Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed(0xc00432cf08, {0x50e6118?, 0xc0046c7540?}, {0xc005f944e0, 0x1, 0xc004367cd8?}, {0x1eca800?, 0x4367d30?, 0xc0?}, 0x1, ...)
96102026-03-10T19:00:49.575Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:139 +0x6c5
96112026-03-10T19:00:49.575Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*dbAdapter).RangeFeed(0x50e60e0?, {0x50e6118?, 0xc0046c7540?}, {0xc005f944e0?, 0x50e60e0?, 0xc006623140?}, {0x70700c00236e920?, 0x50e60e0?, 0x0?}, 0x1, ...)
96122026-03-10T19:00:49.575Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/db_adapter.go:78 +0x3e
96132026-03-10T19:00:49.575Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func1({0x50e6118?, 0xc0046c7540?})
96142026-03-10T19:00:49.575Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:285 +0x6c
96152026-03-10T19:00:49.575Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
96162026-03-10T19:00:49.575Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
96172026-03-10T19:00:49.575Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
96182026-03-10T19:00:49.575Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
96192026-03-10T19:00:49.575Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9673
96202026-03-10T19:00:49.575Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
96212026-03-10T19:00:49.575Z
96222026-03-10T19:00:49.575Zgoroutine 9708 [select]:
96232026-03-10T19:00:49.575Zgoogle.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc006f466e0, 0x1)
96242026-03-10T19:00:49.575Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:407 +0x113
96252026-03-10T19:00:49.575Zgoogle.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc004c46900)
96262026-03-10T19:00:49.575Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:534 +0x7b
96272026-03-10T19:00:49.575Zgoogle.golang.org/grpc/internal/transport.newHTTP2Client.func3()
96282026-03-10T19:00:49.575Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:415 +0x67
96292026-03-10T19:00:49.575Zcreated by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 10012
96302026-03-10T19:00:49.575Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:413 +0x1dcb
96312026-03-10T19:00:49.575Z
96322026-03-10T19:00:49.576Zgoroutine 10042 [sync.Cond.Wait, 59 minutes]:
96332026-03-10T19:00:49.576Zsync.runtime_notifyListWait(0xc00676e610, 0x9)
96342026-03-10T19:00:49.576Z /work/go/src/runtime/sema.go:569 +0x159
96352026-03-10T19:00:49.576Zsync.(*Cond).Wait(0xc001768060?)
96362026-03-10T19:00:49.576Z /work/go/src/sync/cond.go:70 +0x85
96372026-03-10T19:00:49.576Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c9860})
96382026-03-10T19:00:49.576Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
96392026-03-10T19:00:49.580Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
96402026-03-10T19:00:49.580Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
96412026-03-10T19:00:49.580Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
96422026-03-10T19:00:49.580Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
96432026-03-10T19:00:49.581Z
96442026-03-10T19:00:49.581Zgoroutine 9771 [select]:
96452026-03-10T19:00:49.581Zgithub.com/cockroachdb/cockroach/pkg/rpc.respStreamClientAdapter.recvInternal(...)
96462026-03-10T19:00:49.581Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:694
96472026-03-10T19:00:49.581Zgithub.com/cockroachdb/cockroach/pkg/rpc.rangeFeedClientAdapter.Recv(...)
96482026-03-10T19:00:49.581Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:723
96492026-03-10T19:00:49.581Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed.func3()
96502026-03-10T19:00:49.581Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:516 +0x25
96512026-03-10T19:00:49.581Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*stuckRangeFeedCanceler).do(0xc000e299f0, 0xc003765a50)
96522026-03-10T19:00:49.581Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed_canceler.go:100 +0x1b3
96532026-03-10T19:00:49.581Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed(0xc00432cf08, {0x50e6118?, 0xc000e29900?}, {{0xc00357fc08, 0x1, 0x8}, {0xc00357fc10, 0x1, 0x8}}, {0x189b8cc3e00145ab, ...}, ...)
96542026-03-10T19:00:49.581Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:515 +0xc98
96552026-03-10T19:00:49.581Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).partialRangeFeed(0xc00432cf08, {0x50e6118, 0xc000e29900}, 0xc000e298b0, {{0xc00357fc08, 0x1, 0x8}, {0xc00357fc10, 0x1, 0x8}}, ...)
96562026-03-10T19:00:49.581Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:336 +0x745
96572026-03-10T19:00:49.581Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.1({0x50e6118?, 0xc000e29900?})
96582026-03-10T19:00:49.581Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:120 +0xa9
96592026-03-10T19:00:49.581Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.Group.GoCtx.2()
96602026-03-10T19:00:49.581Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
96612026-03-10T19:00:49.581Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
96622026-03-10T19:00:49.581Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
96632026-03-10T19:00:49.581Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9769
96642026-03-10T19:00:49.581Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
96652026-03-10T19:00:49.581Z
96662026-03-10T19:00:49.581Zgoroutine 9034 [semacquire, 59 minutes]:
96672026-03-10T19:00:49.581Zsync.runtime_Semacquire(0xc003f70078?)
96682026-03-10T19:00:49.581Z /work/go/src/runtime/sema.go:62 +0x25
96692026-03-10T19:00:49.581Zsync.(*WaitGroup).Wait(0x44f7e0?)
96702026-03-10T19:00:49.581Z /work/go/src/sync/waitgroup.go:116 +0x48
96712026-03-10T19:00:49.581Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc00242c180)
96722026-03-10T19:00:49.581Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
96732026-03-10T19:00:49.581Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc00242c180?, {0x50e6118?, 0xc003810050?}})
96742026-03-10T19:00:49.582Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
96752026-03-10T19:00:49.582Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed(0xc0021b6f08, {0x50e6118?, 0xc000d02050?}, {0xc00634ca80, 0x1, 0x0?}, {0xc0000bff00?, 0xb985f9?, 0x0?}, 0x0, ...)
96762026-03-10T19:00:49.582Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:139 +0x6c5
96772026-03-10T19:00:49.582Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*dbAdapter).RangeFeed(0x50e60e0?, {0x50e6118?, 0xc000d02050?}, {0xc00634ca80?, 0xc004bef208?, 0x0?}, {0x100c0000bffa8?, 0x50e60e0?, 0x0?}, 0x0, ...)
96782026-03-10T19:00:49.582Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/db_adapter.go:78 +0x3e
96792026-03-10T19:00:49.582Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func1({0x50e6118?, 0xc000d02050?})
96802026-03-10T19:00:49.582Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:285 +0x6c
96812026-03-10T19:00:49.582Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
96822026-03-10T19:00:49.582Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
96832026-03-10T19:00:49.584Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
96842026-03-10T19:00:49.584Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
96852026-03-10T19:00:49.584Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9316
96862026-03-10T19:00:49.584Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
96872026-03-10T19:00:49.584Z
96882026-03-10T19:00:49.585Zgoroutine 9412 [sync.Cond.Wait]:
96892026-03-10T19:00:49.585Zsync.runtime_notifyListWait(0xc0005ac550, 0x3772)
96902026-03-10T19:00:49.585Z /work/go/src/runtime/sema.go:569 +0x159
96912026-03-10T19:00:49.585Zsync.(*Cond).Wait(0x3ae1600?)
96922026-03-10T19:00:49.585Z /work/go/src/sync/cond.go:70 +0x85
96932026-03-10T19:00:49.585Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006623680})
96942026-03-10T19:00:49.585Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
96952026-03-10T19:00:49.585Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
96962026-03-10T19:00:49.585Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
96972026-03-10T19:00:49.585Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
96982026-03-10T19:00:49.585Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
96992026-03-10T19:00:49.585Z
97002026-03-10T19:00:49.585Zgoroutine 9852 [select]:
97012026-03-10T19:00:49.585Zgithub.com/cockroachdb/cockroach/pkg/rpc.respStreamClientAdapter.recvInternal(...)
97022026-03-10T19:00:49.586Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:694
97032026-03-10T19:00:49.586Zgithub.com/cockroachdb/cockroach/pkg/rpc.rangeFeedClientAdapter.Recv(...)
97042026-03-10T19:00:49.586Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:723
97052026-03-10T19:00:49.586Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed.func3()
97062026-03-10T19:00:49.586Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:516 +0x25
97072026-03-10T19:00:49.586Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*stuckRangeFeedCanceler).do(0xc003b0a4b0, 0xc00618da50)
97082026-03-10T19:00:49.586Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed_canceler.go:100 +0x1b3
97092026-03-10T19:00:49.586Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed(0xc00432cf08, {0x50e6118?, 0xc003b0a370?}, {{0xc000784238, 0x1, 0x8}, {0xc000784240, 0x1, 0x8}}, {0x189b8cc3df91d3ff, ...}, ...)
97102026-03-10T19:00:49.586Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:515 +0xc98
97112026-03-10T19:00:49.586Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).partialRangeFeed(0xc00432cf08, {0x50e6118, 0xc003b0a370}, 0xc003b0a320, {{0xc000784238, 0x1, 0x8}, {0xc000784240, 0x1, 0x8}}, ...)
97122026-03-10T19:00:49.586Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:336 +0x745
97132026-03-10T19:00:49.586Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.1({0x50e6118?, 0xc003b0a370?})
97142026-03-10T19:00:49.586Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:120 +0xa9
97152026-03-10T19:00:49.586Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.Group.GoCtx.2()
97162026-03-10T19:00:49.586Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
97172026-03-10T19:00:49.586Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
97182026-03-10T19:00:49.586Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
97192026-03-10T19:00:49.586Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9850
97202026-03-10T19:00:49.586Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
97212026-03-10T19:00:49.587Z
97222026-03-10T19:00:49.587Zgoroutine 9782 [chan receive, 59 minutes]:
97232026-03-10T19:00:49.587Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).rangeFeedWithRangeID(0xc004e66008, 0x3f72e40?, 0xc005edefc0, {0x7f784b79a928, 0xc005dcdf40})
97242026-03-10T19:00:49.587Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:249 +0x76b
97252026-03-10T19:00:49.587Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed(...)
97262026-03-10T19:00:49.587Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:149
97272026-03-10T19:00:49.587Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).RangeFeed(0xc001668008, 0xc005edefc0, {0x511a910, 0xc005dcdf40})
97282026-03-10T19:00:49.587Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:3082 +0x16f
97292026-03-10T19:00:49.587Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).RangeFeed(0xc006849e00, 0xc005edefc0, {0x511a910, 0xc005dcdf40})
97302026-03-10T19:00:49.587Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/stores.go:216 +0x19a
97312026-03-10T19:00:49.587Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed(0x3f72e40?, 0xc004129cf0?, {0x511a910, 0xc005dcdf40})
97322026-03-10T19:00:49.587Z /work/oxidecomputer/cockroach/pkg/server/node.go:1232 +0x28
97332026-03-10T19:00:49.587Zgithub.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed.func1()
97342026-03-10T19:00:49.587Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:753 +0xa7
97352026-03-10T19:00:49.587Zcreated by github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed in goroutine 9781
97362026-03-10T19:00:49.587Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:750 +0x17c
97372026-03-10T19:00:49.587Z
97382026-03-10T19:00:49.587Zgoroutine 9598 [select, 9 minutes]:
97392026-03-10T19:00:49.591Zgithub.com/cockroachdb/cockroach/pkg/sql/sqlstats/persistedsqlstats.(*PersistedSQLStats).startSQLStatsFlushLoop.func1({0x50e60e0, 0xc006037c20})
97402026-03-10T19:00:49.591Z /work/oxidecomputer/cockroach/pkg/sql/sqlstats/persistedsqlstats/provider.go:137 +0x236
97412026-03-10T19:00:49.591Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
97422026-03-10T19:00:49.592Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
97432026-03-10T19:00:49.592Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
97442026-03-10T19:00:49.592Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
97452026-03-10T19:00:49.592Z
97462026-03-10T19:00:49.592Zgoroutine 10036 [sync.Cond.Wait, 59 minutes]:
97472026-03-10T19:00:49.592Zsync.runtime_notifyListWait(0xc00676e610, 0x3)
97482026-03-10T19:00:49.592Z /work/go/src/runtime/sema.go:569 +0x159
97492026-03-10T19:00:49.592Zsync.(*Cond).Wait(0x0?)
97502026-03-10T19:00:49.592Z /work/go/src/sync/cond.go:70 +0x85
97512026-03-10T19:00:49.592Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c9740})
97522026-03-10T19:00:49.592Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
97532026-03-10T19:00:49.592Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
97542026-03-10T19:00:49.593Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
97552026-03-10T19:00:49.593Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
97562026-03-10T19:00:49.593Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
97572026-03-10T19:00:49.593Z
97582026-03-10T19:00:49.593Zgoroutine 9676 [select]:
97592026-03-10T19:00:49.593Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).outputLoop(0xc001388a80, {0x50e6118, 0xc0046c75e0})
97602026-03-10T19:00:49.593Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:310 +0x247
97612026-03-10T19:00:49.593Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).runOutputLoop(0xc001388a80, {0x50e60e0, 0xc005f94ae0}, 0xc0064fe970?)
97622026-03-10T19:00:49.593Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:335 +0xba
97632026-03-10T19:00:49.593Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run.func1({0x50e60e0?, 0xc005f94ae0?})
97642026-03-10T19:00:49.593Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:297 +0x46
97652026-03-10T19:00:49.593Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
97662026-03-10T19:00:49.593Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
97672026-03-10T19:00:49.593Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9579
97682026-03-10T19:00:49.593Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
97692026-03-10T19:00:49.594Z
97702026-03-10T19:00:49.594Zgoroutine 9989 [semacquire, 59 minutes]:
97712026-03-10T19:00:49.594Zsync.runtime_Semacquire(0xc0049be4c8?)
97722026-03-10T19:00:49.594Z /work/go/src/runtime/sema.go:62 +0x25
97732026-03-10T19:00:49.594Zsync.(*WaitGroup).Wait(0x44f7e0?)
97742026-03-10T19:00:49.594Z /work/go/src/sync/waitgroup.go:116 +0x48
97752026-03-10T19:00:49.594Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc003422b80)
97762026-03-10T19:00:49.594Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
97772026-03-10T19:00:49.594Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc003422b80?, {0x50e6118?, 0xc007a220a0?}})
97782026-03-10T19:00:49.594Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
97792026-03-10T19:00:49.594Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait({0x50e60e0?, 0xc00655e090?}, {0xc0066c7d78, 0x2, 0xc0066c7c70?})
97802026-03-10T19:00:49.594Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:193 +0xff
97812026-03-10T19:00:49.594Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0xc00146c580, {0x50e60e0, 0xc00655e090}, 0xc007a03800)
97822026-03-10T19:00:49.594Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:291 +0x59c
97832026-03-10T19:00:49.594Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).Start.func2({0x50e60e0, 0xc007a03ce0})
97842026-03-10T19:00:49.594Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:215 +0x2f3
97852026-03-10T19:00:49.594Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
97862026-03-10T19:00:49.595Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
97872026-03-10T19:00:49.595Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9988
97882026-03-10T19:00:49.595Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
97892026-03-10T19:00:49.595Z
97902026-03-10T19:00:49.595Zgoroutine 9842 [select]:
97912026-03-10T19:00:49.595Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents(0xc0045fc2c0, {0x50e6118, 0xc000ed9db0}, 0xc000819d10, 0xc00032bc80)
97922026-03-10T19:00:49.595Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:331 +0xc5
97932026-03-10T19:00:49.595Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func2({0x50e6118?, 0xc000ed9db0?})
97942026-03-10T19:00:49.595Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:288 +0x32
97952026-03-10T19:00:49.595Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
97962026-03-10T19:00:49.595Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
97972026-03-10T19:00:49.595Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
97982026-03-10T19:00:49.595Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
97992026-03-10T19:00:49.595Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9532
98002026-03-10T19:00:49.595Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
98012026-03-10T19:00:49.595Z
98022026-03-10T19:00:49.595Zgoroutine 10486 [select, 59 minutes]:
98032026-03-10T19:00:49.595Zgoogle.golang.org/grpc.newClientStreamWithParams.func4()
98042026-03-10T19:00:49.595Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:341 +0x8c
98052026-03-10T19:00:49.595Zcreated by google.golang.org/grpc.newClientStreamWithParams in goroutine 10485
98062026-03-10T19:00:49.595Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:340 +0xb2a
98072026-03-10T19:00:49.595Z
98082026-03-10T19:00:49.596Zgoroutine 9363 [select]:
98092026-03-10T19:00:49.596Zgithub.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run(0xc006cae9a0, {0x50e60e0, 0xc006526ba0})
98102026-03-10T19:00:49.596Z /work/oxidecomputer/cockroach/pkg/internal/client/requestbatcher/batcher.go:475 +0x326
98112026-03-10T19:00:49.596Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
98122026-03-10T19:00:49.596Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
98132026-03-10T19:00:49.596Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
98142026-03-10T19:00:49.596Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
98152026-03-10T19:00:49.596Z
98162026-03-10T19:00:49.596Zgoroutine 9364 [select, 9 minutes]:
98172026-03-10T19:00:49.596Zgithub.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run(0xc006caeb00, {0x50e60e0, 0xc006526c00})
98182026-03-10T19:00:49.596Z /work/oxidecomputer/cockroach/pkg/internal/client/requestbatcher/batcher.go:475 +0x326
98192026-03-10T19:00:49.596Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
98202026-03-10T19:00:49.596Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
98212026-03-10T19:00:49.596Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
98222026-03-10T19:00:49.596Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
98232026-03-10T19:00:49.596Z
98242026-03-10T19:00:49.596Zgoroutine 9532 [semacquire, 59 minutes]:
98252026-03-10T19:00:49.596Zsync.runtime_Semacquire(0xc0074c12f0?)
98262026-03-10T19:00:49.596Z /work/go/src/runtime/sema.go:62 +0x25
98272026-03-10T19:00:49.596Zsync.(*WaitGroup).Wait(0x44f7e0?)
98282026-03-10T19:00:49.596Z /work/go/src/sync/waitgroup.go:116 +0x48
98292026-03-10T19:00:49.596Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc00607e6c0)
98302026-03-10T19:00:49.596Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
98312026-03-10T19:00:49.597Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc00607e6c0?, {0x50e6118?, 0xc000ed9db0?}})
98322026-03-10T19:00:49.597Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
98332026-03-10T19:00:49.597Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait({0x50e60e0?, 0xc000819ef0?}, {0xc005d13d78, 0x2, 0xc005d13c70?})
98342026-03-10T19:00:49.597Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:193 +0xff
98352026-03-10T19:00:49.597Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0xc0045fc2c0, {0x50e60e0, 0xc000819ef0}, 0xc000819d10)
98362026-03-10T19:00:49.597Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:291 +0x59c
98372026-03-10T19:00:49.597Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).Start.func2({0x50e60e0, 0xc000819e30})
98382026-03-10T19:00:49.597Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:215 +0x2f3
98392026-03-10T19:00:49.602Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
98402026-03-10T19:00:49.602Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
98412026-03-10T19:00:49.602Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9530
98422026-03-10T19:00:49.602Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
98432026-03-10T19:00:49.602Z
98442026-03-10T19:00:49.602Zgoroutine 10038 [sync.Cond.Wait, 59 minutes]:
98452026-03-10T19:00:49.602Zsync.runtime_notifyListWait(0xc00676e610, 0x5)
98462026-03-10T19:00:49.602Z /work/go/src/runtime/sema.go:569 +0x159
98472026-03-10T19:00:49.602Zsync.(*Cond).Wait(0xc001668008?)
98482026-03-10T19:00:49.602Z /work/go/src/sync/cond.go:70 +0x85
98492026-03-10T19:00:49.602Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c97a0})
98502026-03-10T19:00:49.602Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
98512026-03-10T19:00:49.602Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
98522026-03-10T19:00:49.602Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
98532026-03-10T19:00:49.602Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
98542026-03-10T19:00:49.603Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
98552026-03-10T19:00:49.603Z
98562026-03-10T19:00:49.603Zgoroutine 9406 [sync.Cond.Wait]:
98572026-03-10T19:00:49.603Zsync.runtime_notifyListWait(0xc0005ac550, 0x3748)
98582026-03-10T19:00:49.603Z /work/go/src/runtime/sema.go:569 +0x159
98592026-03-10T19:00:49.603Zsync.(*Cond).Wait(0x3ae1600?)
98602026-03-10T19:00:49.603Z /work/go/src/sync/cond.go:70 +0x85
98612026-03-10T19:00:49.603Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc0066234d0})
98622026-03-10T19:00:49.603Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
98632026-03-10T19:00:49.603Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
98642026-03-10T19:00:49.603Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
98652026-03-10T19:00:49.603Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
98662026-03-10T19:00:49.603Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
98672026-03-10T19:00:49.603Z
98682026-03-10T19:00:49.603Zgoroutine 9203 [chan receive, 59 minutes]:
98692026-03-10T19:00:49.603Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewContext.func2({0xc0034f4308?, 0x0?})
98702026-03-10T19:00:49.603Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:541 +0x3f
98712026-03-10T19:00:49.603Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
98722026-03-10T19:00:49.603Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
98732026-03-10T19:00:49.603Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
98742026-03-10T19:00:49.603Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
98752026-03-10T19:00:49.603Z
98762026-03-10T19:00:49.603Zgoroutine 9657 [select, 59 minutes]:
98772026-03-10T19:00:49.603Zgithub.com/cockroachdb/cockroach/pkg/util/hlc.(*Clock).StartMonitoringForwardClockJumps.func1()
98782026-03-10T19:00:49.603Z /work/oxidecomputer/cockroach/pkg/util/hlc/hlc.go:263 +0x148
98792026-03-10T19:00:49.603Zcreated by github.com/cockroachdb/cockroach/pkg/util/hlc.(*Clock).StartMonitoringForwardClockJumps in goroutine 9099
98802026-03-10T19:00:49.603Z /work/oxidecomputer/cockroach/pkg/util/hlc/hlc.go:255 +0x106
98812026-03-10T19:00:49.603Z
98822026-03-10T19:00:49.603Zgoroutine 9404 [sync.Cond.Wait]:
98832026-03-10T19:00:49.603Zsync.runtime_notifyListWait(0xc0005ac550, 0x375f)
98842026-03-10T19:00:49.603Z /work/go/src/runtime/sema.go:569 +0x159
98852026-03-10T19:00:49.603Zsync.(*Cond).Wait(0x3ae1600?)
98862026-03-10T19:00:49.603Z /work/go/src/sync/cond.go:70 +0x85
98872026-03-10T19:00:49.603Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006623470})
98882026-03-10T19:00:49.603Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
98892026-03-10T19:00:49.603Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
98902026-03-10T19:00:49.603Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
98912026-03-10T19:00:49.604Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
98922026-03-10T19:00:49.604Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
98932026-03-10T19:00:49.604Z
98942026-03-10T19:00:49.604Zgoroutine 9959 [select, 59 minutes]:
98952026-03-10T19:00:49.604Zgoogle.golang.org/grpc.newClientStreamWithParams.func4()
98962026-03-10T19:00:49.604Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:341 +0x8c
98972026-03-10T19:00:49.604Zcreated by google.golang.org/grpc.newClientStreamWithParams in goroutine 10023
98982026-03-10T19:00:49.604Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:340 +0xb2a
98992026-03-10T19:00:49.604Z
99002026-03-10T19:00:49.604Zgoroutine 9655 [IO wait, 59 minutes]:
99012026-03-10T19:00:49.604Zinternal/poll.runtime_pollWait(0x7f784abc3bd8, 0x72)
99022026-03-10T19:00:49.604Z /work/go/src/runtime/netpoll.go:345 +0x85
99032026-03-10T19:00:49.604Zinternal/poll.(*pollDesc).wait(0xc?, 0x0?, 0x0)
99042026-03-10T19:00:49.604Z /work/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
99052026-03-10T19:00:49.605Zinternal/poll.(*pollDesc).waitRead(...)
99062026-03-10T19:00:49.605Z /work/go/src/internal/poll/fd_poll_runtime.go:89
99072026-03-10T19:00:49.605Zinternal/poll.(*FD).Accept(0xc003009480)
99082026-03-10T19:00:49.605Z /work/go/src/internal/poll/fd_unix.go:611 +0x2ac
99092026-03-10T19:00:49.605Znet.(*netFD).accept(0xc003009480)
99102026-03-10T19:00:49.605Z /work/go/src/net/fd_unix.go:172 +0x29
99112026-03-10T19:00:49.605Znet.(*TCPListener).accept(0xc006f5dbe0)
99122026-03-10T19:00:49.605Z /work/go/src/net/tcpsock_posix.go:159 +0x1e
99132026-03-10T19:00:49.605Znet.(*TCPListener).Accept(0xc006f5dbe0)
99142026-03-10T19:00:49.605Z /work/go/src/net/tcpsock.go:327 +0x30
99152026-03-10T19:00:49.605Zgithub.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith(0xc00077a5e8, {0x50e60e0, 0xc006708450}, 0xc006848400, {0x50d9250, 0xc006f5dbe0}, 0xc00669dcc8)
99162026-03-10T19:00:49.605Z /work/oxidecomputer/cockroach/pkg/util/netutil/net.go:138 +0x77
99172026-03-10T19:00:49.605Zgithub.com/cockroachdb/cockroach/pkg/server.(*SQLServer).startServeSQL.func1({0x50e60e0, 0xc006708450})
99182026-03-10T19:00:49.605Z /work/oxidecomputer/cockroach/pkg/server/server_sql.go:1374 +0xcb
99192026-03-10T19:00:49.605Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
99202026-03-10T19:00:49.605Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
99212026-03-10T19:00:49.605Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
99222026-03-10T19:00:49.605Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
99232026-03-10T19:00:49.605Z
99242026-03-10T19:00:49.605Zgoroutine 9662 [chan receive, 59 minutes]:
99252026-03-10T19:00:49.605Zgithub.com/cockroachdb/cmux.muxListener.Accept(...)
99262026-03-10T19:00:49.605Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184
99272026-03-10T19:00:49.605Zcrypto/tls.(*listener).Accept(0xc0034385a0)
99282026-03-10T19:00:49.605Z /work/go/src/crypto/tls/tls.go:66 +0x27
99292026-03-10T19:00:49.605Znet/http.(*Server).Serve(0xc00211aff0, {0x50d12d0, 0xc0034385a0})
99302026-03-10T19:00:49.605Z /work/go/src/net/http/server.go:3260 +0x33e
99312026-03-10T19:00:49.605Zgithub.com/cockroachdb/cockroach/pkg/server.(*httpServer).start.func4({0xc00670cf08?, 0xc0066230b0?})
99322026-03-10T19:00:49.605Z /work/oxidecomputer/cockroach/pkg/server/server_http.go:277 +0x25
99332026-03-10T19:00:49.605Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
99342026-03-10T19:00:49.605Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
99352026-03-10T19:00:49.605Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
99362026-03-10T19:00:49.605Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
99372026-03-10T19:00:49.605Z
99382026-03-10T19:00:49.605Zgoroutine 9969 [select, 59 minutes]:
99392026-03-10T19:00:49.610Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2({0x50e60e0, 0xc00672be00})
99402026-03-10T19:00:49.610Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/queue.go:823 +0xff
99412026-03-10T19:00:49.610Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
99422026-03-10T19:00:49.610Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
99432026-03-10T19:00:49.610Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10105
99442026-03-10T19:00:49.610Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
99452026-03-10T19:00:49.610Z
99462026-03-10T19:00:49.610Zgoroutine 9773 [select]:
99472026-03-10T19:00:49.610Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run(0xc005d21800, {0x50e60e0?, 0xc0025d79e0?}, 0xb991a6?, 0xc006fcae40, 0xc006848400)
99482026-03-10T19:00:49.610Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:269 +0x4dc
99492026-03-10T19:00:49.610Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).Start.func1({0x50e60e0?, 0xc0025d79e0?})
99502026-03-10T19:00:49.610Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:222 +0x36
99512026-03-10T19:00:49.610Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
99522026-03-10T19:00:49.610Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
99532026-03-10T19:00:49.610Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9772
99542026-03-10T19:00:49.610Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
99552026-03-10T19:00:49.610Z
99562026-03-10T19:00:49.610Zgoroutine 10154 [select, 59 minutes]:
99572026-03-10T19:00:49.610Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1({0x50e6118, 0xc00717cd20})
99582026-03-10T19:00:49.610Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:116 +0x157
99592026-03-10T19:00:49.610Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.Group.GoCtx.func3()
99602026-03-10T19:00:49.610Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
99612026-03-10T19:00:49.610Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
99622026-03-10T19:00:49.610Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
99632026-03-10T19:00:49.610Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10152
99642026-03-10T19:00:49.610Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
99652026-03-10T19:00:49.610Z
99662026-03-10T19:00:49.610Zgoroutine 9853 [chan receive, 59 minutes]:
99672026-03-10T19:00:49.610Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).rangeFeedWithRangeID(0xc004e66008, 0x3f72e40?, 0xc006040480, {0x7f784b79a928, 0xc006042320})
99682026-03-10T19:00:49.611Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:249 +0x76b
99692026-03-10T19:00:49.611Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed(...)
99702026-03-10T19:00:49.611Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:149
99712026-03-10T19:00:49.611Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).RangeFeed(0xc001668008, 0xc006040480, {0x511a910, 0xc006042320})
99722026-03-10T19:00:49.611Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:3082 +0x16f
99732026-03-10T19:00:49.611Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).RangeFeed(0xc006849e00, 0xc006040480, {0x511a910, 0xc006042320})
99742026-03-10T19:00:49.611Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/stores.go:216 +0x19a
99752026-03-10T19:00:49.611Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed(0x3f72e40?, 0xc003b0a580?, {0x511a910, 0xc006042320})
99762026-03-10T19:00:49.611Z /work/oxidecomputer/cockroach/pkg/server/node.go:1232 +0x28
99772026-03-10T19:00:49.611Zgithub.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed.func1()
99782026-03-10T19:00:49.611Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:753 +0xa7
99792026-03-10T19:00:49.611Zcreated by github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed in goroutine 9852
99802026-03-10T19:00:49.611Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:750 +0x17c
99812026-03-10T19:00:49.612Z
99822026-03-10T19:00:49.612Zgoroutine 9035 [select]:
99832026-03-10T19:00:49.612Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents(0xc006028840, {0x50e6118, 0xc000d02050}, 0xc00634c960, 0xc0021ba600)
99842026-03-10T19:00:49.612Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:331 +0xc5
99852026-03-10T19:00:49.612Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func2({0x50e6118?, 0xc000d02050?})
99862026-03-10T19:00:49.612Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:288 +0x32
99872026-03-10T19:00:49.612Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
99882026-03-10T19:00:49.613Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
99892026-03-10T19:00:49.613Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
99902026-03-10T19:00:49.613Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
99912026-03-10T19:00:49.613Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9316
99922026-03-10T19:00:49.613Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
99932026-03-10T19:00:49.613Z
99942026-03-10T19:00:49.613Zgoroutine 9741 [select]:
99952026-03-10T19:00:49.613Zgithub.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigsqlwatcher.(*SQLWatcher).watch(0xc00472f950, {0x50e60e0, 0xc002e438c0}, {0x18?, 0x3ceaec0?, 0x0?}, 0xc0021b4780)
99962026-03-10T19:00:49.613Z /work/oxidecomputer/cockroach/pkg/spanconfig/spanconfigsqlwatcher/sqlwatcher.go:159 +0x5ff
99972026-03-10T19:00:49.613Zgithub.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigsqlwatcher.(*SQLWatcher).WatchForSQLUpdates(0xc00276ef00?, {0x50e60e0?, 0xc002e438c0?}, {0xc0049c2cd0?, 0x148aa50?, 0xc0?}, 0x40ed6e8?)
99982026-03-10T19:00:49.613Z /work/oxidecomputer/cockroach/pkg/spanconfig/spanconfigsqlwatcher/sqlwatcher.go:89 +0x1d
99992026-03-10T19:00:49.613Zgithub.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigreconciler.(*incrementalReconciler).reconcile(...)
100002026-03-10T19:00:49.613Z /work/oxidecomputer/cockroach/pkg/spanconfig/spanconfigreconciler/reconciler.go:453
100012026-03-10T19:00:49.613Zgithub.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigreconciler.(*Reconciler).Reconcile(0xc00276eea0, {0x50e60e0, 0xc002e438c0}, {0x6fc23ac00?, 0xcccccccd?, 0xcc?}, {0x50ef178, 0xc0036b86c0}, 0xc00035f500)
100022026-03-10T19:00:49.613Z /work/oxidecomputer/cockroach/pkg/spanconfig/spanconfigreconciler/reconciler.go:186 +0x419
100032026-03-10T19:00:49.613Zgithub.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigjob.(*resumer).Resume(0xc000b1e630, {0x50e60e0, 0xc002e438c0}, {0x3e94a00?, 0xc000f82968})
100042026-03-10T19:00:49.613Z /work/oxidecomputer/cockroach/pkg/spanconfig/spanconfigjob/job.go:162 +0xb8a
100052026-03-10T19:00:49.613Zgithub.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func2(0x50e60e0?, 0x50e60e0?, 0xc03782f9d0, {0x50c6550?, 0xc000b1e630?}, {0x50e60e0?, 0xc002e438c0?}, {0x3e94a00?, 0xc000f82968?})
100062026-03-10T19:00:49.613Z /work/oxidecomputer/cockroach/pkg/jobs/registry.go:1243 +0xc2
100072026-03-10T19:00:49.614Zgithub.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine(0xc002c20f08, {0x50e60e0, 0xc001fa6a50}, {0x3e94a00, 0xc000f82968}, {0x50c6550, 0xc000b1e630}, 0xc0033196c0, {0xc00127af98, 0x7}, ...)
100082026-03-10T19:00:49.614Z /work/oxidecomputer/cockroach/pkg/jobs/registry.go:1244 +0x67f
100092026-03-10T19:00:49.614Zgithub.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob(0xc002c20f08, {0x50e60e0, 0xc004c0fcb0}, {0x50c6550, 0xc000b1e630}, 0xc0033196c0, {0xc00127af98, 0x7}, {0xc0072b9860, 0x17})
100102026-03-10T19:00:49.614Z /work/oxidecomputer/cockroach/pkg/jobs/adopt.go:415 +0x570
100112026-03-10T19:00:49.614Zgithub.com/cockroachdb/cockroach/pkg/jobs.(*Registry).resumeJob.func1({0xc002afa608?, 0x17ef6b5?})
100122026-03-10T19:00:49.614Z /work/oxidecomputer/cockroach/pkg/jobs/adopt.go:336 +0xee
100132026-03-10T19:00:49.614Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
100142026-03-10T19:00:49.614Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
100152026-03-10T19:00:49.614Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9738
100162026-03-10T19:00:49.614Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
100172026-03-10T19:00:49.614Z
100182026-03-10T19:00:49.614Zgoroutine 9642 [select, 10 minutes]:
100192026-03-10T19:00:49.614Zgithub.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigmanager.(*Manager).run(0xc00472f9a0, {0x50e60e0, 0xc0035b41b0})
100202026-03-10T19:00:49.614Z /work/oxidecomputer/cockroach/pkg/spanconfig/spanconfigmanager/manager.go:159 +0x3ba
100212026-03-10T19:00:49.614Zgithub.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigmanager.(*Manager).Start.func1({0x50e60e0?, 0xc0035b41b0?})
100222026-03-10T19:00:49.614Z /work/oxidecomputer/cockroach/pkg/spanconfig/spanconfigmanager/manager.go:101 +0x2a
100232026-03-10T19:00:49.614Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
100242026-03-10T19:00:49.614Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
100252026-03-10T19:00:49.614Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
100262026-03-10T19:00:49.614Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
100272026-03-10T19:00:49.615Z
100282026-03-10T19:00:49.615Zgoroutine 9761 [semacquire, 59 minutes]:
100292026-03-10T19:00:49.615Zsync.runtime_Semacquire(0xc007269b90?)
100302026-03-10T19:00:49.615Z /work/go/src/runtime/sema.go:62 +0x25
100312026-03-10T19:00:49.615Zsync.(*WaitGroup).Wait(0x44f7e0?)
100322026-03-10T19:00:49.615Z /work/go/src/sync/waitgroup.go:116 +0x48
100332026-03-10T19:00:49.615Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc005e9f900)
100342026-03-10T19:00:49.615Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
100352026-03-10T19:00:49.615Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc005e9f900?, {0x50e6118?, 0xc004129950?}})
100362026-03-10T19:00:49.615Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
100372026-03-10T19:00:49.615Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed(0xc00432cf08, {0x50e6118?, 0xc004129770?}, {0xc005f94390, 0x1, 0xc00432cf08?}, {0xc001a06740?, 0x5ee0230?, 0xc0?}, 0x0, ...)
100382026-03-10T19:00:49.615Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:139 +0x6c5
100392026-03-10T19:00:49.619Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*dbAdapter).RangeFeed(0x2620?, {0x50e6118?, 0xc004129770?}, {0xc005f94390?, 0x50e60e0?, 0xc00457a1b0?}, {0x7070000050e60e0?, 0x50e60e0?, 0x0?}, 0x0, ...)
100402026-03-10T19:00:49.619Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/db_adapter.go:78 +0x3e
100412026-03-10T19:00:49.619Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func1({0x50e6118?, 0xc004129770?})
100422026-03-10T19:00:49.619Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:285 +0x6c
100432026-03-10T19:00:49.619Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
100442026-03-10T19:00:49.619Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
100452026-03-10T19:00:49.619Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
100462026-03-10T19:00:49.620Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
100472026-03-10T19:00:49.620Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9672
100482026-03-10T19:00:49.620Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
100492026-03-10T19:00:49.620Z
100502026-03-10T19:00:49.620Zgoroutine 9988 [select]:
100512026-03-10T19:00:49.620Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.(*Watcher).Run(0xc003c80870, {0x50e6118, 0xc0014520f0})
100522026-03-10T19:00:49.620Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:319 +0x955
100532026-03-10T19:00:49.620Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.Start.func1({0x50e60e0?, 0xc007a036e0?})
100542026-03-10T19:00:49.620Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:189 +0x2d1
100552026-03-10T19:00:49.620Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
100562026-03-10T19:00:49.620Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
100572026-03-10T19:00:49.620Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
100582026-03-10T19:00:49.620Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
100592026-03-10T19:00:49.620Z
100602026-03-10T19:00:49.620Zgoroutine 10034 [sync.Cond.Wait, 59 minutes]:
100612026-03-10T19:00:49.620Zsync.runtime_notifyListWait(0xc00676e610, 0x1)
100622026-03-10T19:00:49.620Z /work/go/src/runtime/sema.go:569 +0x159
100632026-03-10T19:00:49.620Zsync.(*Cond).Wait(0x0?)
100642026-03-10T19:00:49.620Z /work/go/src/sync/cond.go:70 +0x85
100652026-03-10T19:00:49.620Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c96e0})
100662026-03-10T19:00:49.620Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
100672026-03-10T19:00:49.620Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
100682026-03-10T19:00:49.620Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
100692026-03-10T19:00:49.620Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
100702026-03-10T19:00:49.620Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
100712026-03-10T19:00:49.620Z
100722026-03-10T19:00:49.620Zgoroutine 9348 [chan receive, 59 minutes]:
100732026-03-10T19:00:49.620Zgithub.com/cockroachdb/cockroach/pkg/server.startListenRPCAndSQL.func1({0x50e60e0, 0xc0064d4870})
100742026-03-10T19:00:49.620Z /work/oxidecomputer/cockroach/pkg/server/start_listen.go:64 +0x45
100752026-03-10T19:00:49.620Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
100762026-03-10T19:00:49.620Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
100772026-03-10T19:00:49.620Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
100782026-03-10T19:00:49.620Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
100792026-03-10T19:00:49.620Z
100802026-03-10T19:00:49.620Zgoroutine 9629 [IO wait]:
100812026-03-10T19:00:49.620Zinternal/poll.runtime_pollWait(0x7f784b7f1740, 0x72)
100822026-03-10T19:00:49.620Z /work/go/src/runtime/netpoll.go:345 +0x85
100832026-03-10T19:00:49.621Zinternal/poll.(*pollDesc).wait(0xc0064f3580?, 0xc00362e700?, 0x0)
100842026-03-10T19:00:49.621Z /work/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
100852026-03-10T19:00:49.621Zinternal/poll.(*pollDesc).waitRead(...)
100862026-03-10T19:00:49.621Z /work/go/src/internal/poll/fd_poll_runtime.go:89
100872026-03-10T19:00:49.621Zinternal/poll.(*FD).Read(0xc0064f3580, {0xc00362e700, 0x700, 0x700})
100882026-03-10T19:00:49.621Z /work/go/src/internal/poll/fd_unix.go:164 +0x27a
100892026-03-10T19:00:49.621Znet.(*netFD).Read(0xc0064f3580, {0xc00362e700?, 0x7f784ab5aaa0?, 0xc007e49668?})
100902026-03-10T19:00:49.621Z /work/go/src/net/fd_posix.go:55 +0x25
100912026-03-10T19:00:49.622Znet.(*conn).Read(0xc001f20e28, {0xc00362e700?, 0xc0048ee8e0?, 0x416dfb?})
100922026-03-10T19:00:49.622Z /work/go/src/net/net.go:185 +0x45
100932026-03-10T19:00:49.622Zcrypto/tls.(*atLeastReader).Read(0xc007e49668, {0xc00362e700?, 0x0?, 0xc007e49668?})
100942026-03-10T19:00:49.623Z /work/go/src/crypto/tls/conn.go:806 +0x3b
100952026-03-10T19:00:49.623Zbytes.(*Buffer).ReadFrom(0xc0065f0630, {0x50b6dc0, 0xc007e49668})
100962026-03-10T19:00:49.623Z /work/go/src/bytes/buffer.go:211 +0x98
100972026-03-10T19:00:49.623Zcrypto/tls.(*Conn).readFromUntil(0xc0065f0388, {0x50b6fa0, 0xc001f20e28}, 0xc0048ee928?)
100982026-03-10T19:00:49.623Z /work/go/src/crypto/tls/conn.go:828 +0xde
100992026-03-10T19:00:49.623Zcrypto/tls.(*Conn).readRecordOrCCS(0xc0065f0388, 0x0)
101002026-03-10T19:00:49.623Z /work/go/src/crypto/tls/conn.go:626 +0x3cf
101012026-03-10T19:00:49.623Zcrypto/tls.(*Conn).readRecord(...)
101022026-03-10T19:00:49.623Z /work/go/src/crypto/tls/conn.go:588
101032026-03-10T19:00:49.623Zcrypto/tls.(*Conn).Read(0xc0065f0388, {0xc007400000, 0x8000, 0xc0048eec88?})
101042026-03-10T19:00:49.623Z /work/go/src/crypto/tls/conn.go:1370 +0x156
101052026-03-10T19:00:49.623Zbufio.(*Reader).Read(0xc005e388a0, {0xc001adbc40, 0x9, 0x7?})
101062026-03-10T19:00:49.623Z /work/go/src/bufio/bufio.go:241 +0x197
101072026-03-10T19:00:49.623Zio.ReadAtLeast({0x50b1100, 0xc005e388a0}, {0xc001adbc40, 0x9, 0x9}, 0x9)
101082026-03-10T19:00:49.623Z /work/go/src/io/io.go:335 +0x90
101092026-03-10T19:00:49.623Zio.ReadFull(...)
101102026-03-10T19:00:49.623Z /work/go/src/io/io.go:354
101112026-03-10T19:00:49.623Zgolang.org/x/net/http2.readFrameHeader({0xc001adbc40, 0x9, 0xc004c4ac00?}, {0x50b1100?, 0xc005e388a0?})
101122026-03-10T19:00:49.623Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x65
101132026-03-10T19:00:49.623Zgolang.org/x/net/http2.(*Framer).ReadFrame(0xc001adbc00)
101142026-03-10T19:00:49.623Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:501 +0x85
101152026-03-10T19:00:49.623Zgoogle.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0016874a0)
101162026-03-10T19:00:49.623Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1498 +0x41d
101172026-03-10T19:00:49.624Zcreated by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 10025
101182026-03-10T19:00:49.624Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:365 +0x184d
101192026-03-10T19:00:49.624Z
101202026-03-10T19:00:49.624Zgoroutine 9819 [select]:
101212026-03-10T19:00:49.624Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).outputLoop(0xc0012167e0, {0x50e6118, 0xc001749e50})
101222026-03-10T19:00:49.624Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:310 +0x247
101232026-03-10T19:00:49.624Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).runOutputLoop(0xc0012167e0, {0x50e60e0, 0xc006295e60}, 0x0?)
101242026-03-10T19:00:49.624Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:335 +0xba
101252026-03-10T19:00:49.624Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run.func1({0x50e60e0?, 0xc006295e60?})
101262026-03-10T19:00:49.624Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:297 +0x46
101272026-03-10T19:00:49.624Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
101282026-03-10T19:00:49.624Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
101292026-03-10T19:00:49.624Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9817
101302026-03-10T19:00:49.624Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
101312026-03-10T19:00:49.624Z
101322026-03-10T19:00:49.624Zgoroutine 10045 [sync.Cond.Wait, 59 minutes]:
101332026-03-10T19:00:49.624Zsync.runtime_notifyListWait(0xc00676e610, 0xc)
101342026-03-10T19:00:49.624Z /work/go/src/runtime/sema.go:569 +0x159
101352026-03-10T19:00:49.624Zsync.(*Cond).Wait(0x1?)
101362026-03-10T19:00:49.624Z /work/go/src/sync/cond.go:70 +0x85
101372026-03-10T19:00:49.624Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c98f0})
101382026-03-10T19:00:49.624Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
101392026-03-10T19:00:49.628Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
101402026-03-10T19:00:49.628Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
101412026-03-10T19:00:49.628Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
101422026-03-10T19:00:49.628Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
101432026-03-10T19:00:49.628Z
101442026-03-10T19:00:49.628Zgoroutine 9673 [semacquire, 59 minutes]:
101452026-03-10T19:00:49.628Zsync.runtime_Semacquire(0xc0021b4990?)
101462026-03-10T19:00:49.628Z /work/go/src/runtime/sema.go:62 +0x25
101472026-03-10T19:00:49.629Zsync.(*WaitGroup).Wait(0x44f7e0?)
101482026-03-10T19:00:49.629Z /work/go/src/sync/waitgroup.go:116 +0x48
101492026-03-10T19:00:49.629Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc000e15080)
101502026-03-10T19:00:49.629Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
101512026-03-10T19:00:49.629Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc000e15080?, {0x50e6118?, 0xc0046c7540?}})
101522026-03-10T19:00:49.629Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
101532026-03-10T19:00:49.629Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait({0x50e60e0?, 0xc005f94600?}, {0xc0028d0d78, 0x2, 0x4000000000000000?})
101542026-03-10T19:00:49.629Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:193 +0xff
101552026-03-10T19:00:49.629Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0xc00133c840, {0x50e60e0, 0xc005f94600}, 0xc005f94450)
101562026-03-10T19:00:49.629Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:291 +0x59c
101572026-03-10T19:00:49.629Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).Start.func2({0x50e60e0, 0xc005f94540})
101582026-03-10T19:00:49.629Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:215 +0x2f3
101592026-03-10T19:00:49.629Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
101602026-03-10T19:00:49.629Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
101612026-03-10T19:00:49.630Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9741
101622026-03-10T19:00:49.630Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
101632026-03-10T19:00:49.630Z
101642026-03-10T19:00:49.630Zgoroutine 10009 [chan receive, 59 minutes]:
101652026-03-10T19:00:49.630Zgithub.com/cockroachdb/cockroach/pkg/server.configureGRPCGateway.func1({0xc00670d808?, 0xc004bb3710?})
101662026-03-10T19:00:49.630Z /work/oxidecomputer/cockroach/pkg/server/grpc_gateway.go:86 +0x31
101672026-03-10T19:00:49.630Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
101682026-03-10T19:00:49.630Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
101692026-03-10T19:00:49.630Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
101702026-03-10T19:00:49.630Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
101712026-03-10T19:00:49.630Z
101722026-03-10T19:00:49.630Zgoroutine 10008 [chan receive, 59 minutes]:
101732026-03-10T19:00:49.630Zgithub.com/cockroachdb/cockroach/pkg/server.startListenRPCAndSQL.func3({0xc00670d508?, 0xc001e7ad80?})
101742026-03-10T19:00:49.630Z /work/oxidecomputer/cockroach/pkg/server/start_listen.go:108 +0x3a
101752026-03-10T19:00:49.630Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
101762026-03-10T19:00:49.630Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
101772026-03-10T19:00:49.630Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
101782026-03-10T19:00:49.630Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
101792026-03-10T19:00:49.630Z
101802026-03-10T19:00:49.630Zgoroutine 9811 [semacquire, 59 minutes]:
101812026-03-10T19:00:49.630Zsync.runtime_Semacquire(0xc001ea3d40?)
101822026-03-10T19:00:49.630Z /work/go/src/runtime/sema.go:62 +0x25
101832026-03-10T19:00:49.630Zsync.(*WaitGroup).Wait(0x44f7e0?)
101842026-03-10T19:00:49.630Z /work/go/src/sync/waitgroup.go:116 +0x48
101852026-03-10T19:00:49.630Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc006102840)
101862026-03-10T19:00:49.630Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
101872026-03-10T19:00:49.630Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc006102840?, {0x50e6118?, 0xc001749a90?}})
101882026-03-10T19:00:49.632Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
101892026-03-10T19:00:49.632Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed(0xc00432cf08, {0x50e6118?, 0xc0017499f0?}, {0xc0067bbec0, 0x1, 0xc004367cd8?}, {0xc0048eaf00?, 0xb985f9?, 0x0?}, 0x1, ...)
101902026-03-10T19:00:49.632Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:139 +0x6c5
101912026-03-10T19:00:49.632Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*dbAdapter).RangeFeed(0x12667db7f636dd?, {0x50e6118?, 0xc0017499f0?}, {0xc0067bbec0?, 0x0?, 0xc006015508?}, {0x50e60e0?, 0x622d320?, 0xc0?}, 0x1, ...)
101922026-03-10T19:00:49.632Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/db_adapter.go:78 +0x3e
101932026-03-10T19:00:49.632Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func1({0x50e6118?, 0xc0017499f0?})
101942026-03-10T19:00:49.632Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:285 +0x6c
101952026-03-10T19:00:49.632Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
101962026-03-10T19:00:49.632Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
101972026-03-10T19:00:49.632Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
101982026-03-10T19:00:49.632Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
101992026-03-10T19:00:49.632Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9470
102002026-03-10T19:00:49.632Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
102012026-03-10T19:00:49.632Z
102022026-03-10T19:00:49.632Zgoroutine 9801 [select]:
102032026-03-10T19:00:49.632Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).outputLoop(0xc00463ac40, {0x50e6118, 0xc00383b180})
102042026-03-10T19:00:49.632Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:310 +0x247
102052026-03-10T19:00:49.632Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).runOutputLoop(0xc00463ac40, {0x50e60e0, 0xc004fe6e70}, 0xc000920000?)
102062026-03-10T19:00:49.632Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:335 +0xba
102072026-03-10T19:00:49.632Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run.func1({0x50e60e0?, 0xc004fe6e70?})
102082026-03-10T19:00:49.632Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:297 +0x46
102092026-03-10T19:00:49.632Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
102102026-03-10T19:00:49.632Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
102112026-03-10T19:00:49.632Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9799
102122026-03-10T19:00:49.632Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
102132026-03-10T19:00:49.632Z
102142026-03-10T19:00:49.632Zgoroutine 9939 [select]:
102152026-03-10T19:00:49.632Zgithub.com/cockroachdb/cockroach/pkg/jobs.(*jobScheduler).runDaemon.func1({0x50e60e0?, 0xc00674c030?})
102162026-03-10T19:00:49.632Z /work/oxidecomputer/cockroach/pkg/jobs/job_scheduler.go:443 +0x293
102172026-03-10T19:00:49.632Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
102182026-03-10T19:00:49.632Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
102192026-03-10T19:00:49.632Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
102202026-03-10T19:00:49.632Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
102212026-03-10T19:00:49.633Z
102222026-03-10T19:00:49.633Zgoroutine 9854 [select]:
102232026-03-10T19:00:49.633Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).outputLoop(0xc001389960, {0x50e6118, 0xc003b0a5a0})
102242026-03-10T19:00:49.633Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:310 +0x247
102252026-03-10T19:00:49.633Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).runOutputLoop(0xc001389960, {0x50e60e0, 0xc00399dad0}, 0x1?)
102262026-03-10T19:00:49.633Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:335 +0xba
102272026-03-10T19:00:49.633Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run.func1({0x50e60e0?, 0xc00399dad0?})
102282026-03-10T19:00:49.633Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:297 +0x46
102292026-03-10T19:00:49.633Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
102302026-03-10T19:00:49.633Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
102312026-03-10T19:00:49.633Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9579
102322026-03-10T19:00:49.633Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
102332026-03-10T19:00:49.634Z
102342026-03-10T19:00:49.634Zgoroutine 10010 [select, 59 minutes]:
102352026-03-10T19:00:49.634Zgithub.com/cockroachdb/cockroach/pkg/server.(*loopbackListener).Accept(0xc006709230)
102362026-03-10T19:00:49.634Z /work/oxidecomputer/cockroach/pkg/server/loopback.go:49 +0x90
102372026-03-10T19:00:49.634Zgoogle.golang.org/grpc.(*Server).Serve(0xc003999340, {0x50d1030, 0xc006709230})
102382026-03-10T19:00:49.634Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:780 +0x469
102392026-03-10T19:00:49.638Zgithub.com/cockroachdb/cockroach/pkg/server.configureGRPCGateway.func2({0xc00670db08?, 0xc005f6e5a0?})
102402026-03-10T19:00:49.638Z /work/oxidecomputer/cockroach/pkg/server/grpc_gateway.go:94 +0x25
102412026-03-10T19:00:49.638Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
102422026-03-10T19:00:49.638Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
102432026-03-10T19:00:49.638Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
102442026-03-10T19:00:49.638Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
102452026-03-10T19:00:49.638Z
102462026-03-10T19:00:49.638Zgoroutine 9727 [select]:
102472026-03-10T19:00:49.638Zgithub.com/cockroachdb/cockroach/pkg/rpc.respStreamClientAdapter.recvInternal(...)
102482026-03-10T19:00:49.638Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:694
102492026-03-10T19:00:49.638Zgithub.com/cockroachdb/cockroach/pkg/rpc.rangeFeedClientAdapter.Recv(...)
102502026-03-10T19:00:49.638Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:723
102512026-03-10T19:00:49.638Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed.func3()
102522026-03-10T19:00:49.638Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:516 +0x25
102532026-03-10T19:00:49.638Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*stuckRangeFeedCanceler).do(0xc0012db270, 0xc001ea7a50)
102542026-03-10T19:00:49.638Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed_canceler.go:100 +0x1b3
102552026-03-10T19:00:49.638Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed(0xc00432cf08, {0x50e6118?, 0xc0012db180?}, {{0xc0067c93a0, 0x1, 0x8}, {0xc0067c93a8, 0x1, 0x8}}, {0x189b8cc3df89d10d, ...}, ...)
102562026-03-10T19:00:49.638Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:515 +0xc98
102572026-03-10T19:00:49.638Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).partialRangeFeed(0xc00432cf08, {0x50e6118, 0xc0012db180}, 0xc0012db130, {{0xc0067c93a0, 0x1, 0x8}, {0xc0067c93a8, 0x1, 0x8}}, ...)
102582026-03-10T19:00:49.638Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:336 +0x745
102592026-03-10T19:00:49.638Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.1({0x50e6118?, 0xc0012db180?})
102602026-03-10T19:00:49.638Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:120 +0xa9
102612026-03-10T19:00:49.638Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.Group.GoCtx.2()
102622026-03-10T19:00:49.638Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
102632026-03-10T19:00:49.638Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
102642026-03-10T19:00:49.638Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
102652026-03-10T19:00:49.638Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9725
102662026-03-10T19:00:49.639Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
102672026-03-10T19:00:49.639Z
102682026-03-10T19:00:49.639Zgoroutine 10007 [chan receive, 59 minutes]:
102692026-03-10T19:00:49.639Zgithub.com/cockroachdb/cockroach/pkg/server.startListenRPCAndSQL.func1({0x50e60e0, 0xc0067090e0})
102702026-03-10T19:00:49.639Z /work/oxidecomputer/cockroach/pkg/server/start_listen.go:64 +0x45
102712026-03-10T19:00:49.639Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
102722026-03-10T19:00:49.639Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
102732026-03-10T19:00:49.639Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
102742026-03-10T19:00:49.639Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
102752026-03-10T19:00:49.639Z
102762026-03-10T19:00:49.639Zgoroutine 9403 [sync.Cond.Wait]:
102772026-03-10T19:00:49.639Zsync.runtime_notifyListWait(0xc0005ac550, 0x3771)
102782026-03-10T19:00:49.639Z /work/go/src/runtime/sema.go:569 +0x159
102792026-03-10T19:00:49.639Zsync.(*Cond).Wait(0x3ae1600?)
102802026-03-10T19:00:49.639Z /work/go/src/sync/cond.go:70 +0x85
102812026-03-10T19:00:49.640Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006623440})
102822026-03-10T19:00:49.640Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
102832026-03-10T19:00:49.640Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
102842026-03-10T19:00:49.640Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
102852026-03-10T19:00:49.640Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
102862026-03-10T19:00:49.640Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
102872026-03-10T19:00:49.640Z
102882026-03-10T19:00:49.640Zgoroutine 9408 [sync.Cond.Wait]:
102892026-03-10T19:00:49.640Zsync.runtime_notifyListWait(0xc0005ac550, 0x375c)
102902026-03-10T19:00:49.640Z /work/go/src/runtime/sema.go:569 +0x159
102912026-03-10T19:00:49.640Zsync.(*Cond).Wait(0x3ae1600?)
102922026-03-10T19:00:49.640Z /work/go/src/sync/cond.go:70 +0x85
102932026-03-10T19:00:49.640Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc004367c70, {0x50e60e0, 0xc006623560})
102942026-03-10T19:00:49.640Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
102952026-03-10T19:00:49.640Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
102962026-03-10T19:00:49.640Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
102972026-03-10T19:00:49.640Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
102982026-03-10T19:00:49.640Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
102992026-03-10T19:00:49.640Z
103002026-03-10T19:00:49.640Zgoroutine 9778 [select]:
103012026-03-10T19:00:49.640Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents(0xc00133c6e0, {0x50e6118, 0xc004129770}, 0xc005f94300, 0xc001bbe060)
103022026-03-10T19:00:49.640Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:331 +0xc5
103032026-03-10T19:00:49.640Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func2({0x50e6118?, 0xc004129770?})
103042026-03-10T19:00:49.640Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:288 +0x32
103052026-03-10T19:00:49.640Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
103062026-03-10T19:00:49.640Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
103072026-03-10T19:00:49.640Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
103082026-03-10T19:00:49.640Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
103092026-03-10T19:00:49.640Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9672
103102026-03-10T19:00:49.640Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
103112026-03-10T19:00:49.640Z
103122026-03-10T19:00:49.641Zgoroutine 9530 [select]:
103132026-03-10T19:00:49.641Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.(*Watcher).Run(0xc0045cf9e0, {0x50e6118, 0xc003154d70})
103142026-03-10T19:00:49.641Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:319 +0x955
103152026-03-10T19:00:49.641Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.Start.func1({0x50e60e0?, 0xc00398da40?})
103162026-03-10T19:00:49.641Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:189 +0x2d1
103172026-03-10T19:00:49.641Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
103182026-03-10T19:00:49.641Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
103192026-03-10T19:00:49.641Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
103202026-03-10T19:00:49.641Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
103212026-03-10T19:00:49.641Z
103222026-03-10T19:00:49.641Zgoroutine 9993 [semacquire, 59 minutes]:
103232026-03-10T19:00:49.641Zsync.runtime_Semacquire(0xc004b20960?)
103242026-03-10T19:00:49.641Z /work/go/src/runtime/sema.go:62 +0x25
103252026-03-10T19:00:49.641Zsync.(*WaitGroup).Wait(0x44f7e0?)
103262026-03-10T19:00:49.641Z /work/go/src/sync/waitgroup.go:116 +0x48
103272026-03-10T19:00:49.641Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc000f9a600)
103282026-03-10T19:00:49.641Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
103292026-03-10T19:00:49.641Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc000f9a600?, {0x50e6118?, 0xc004f9bc70?}})
103302026-03-10T19:00:49.641Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
103312026-03-10T19:00:49.641Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait({0x50e60e0?, 0xc00674ef60?}, {0xc0069e3d78, 0x2, 0xc0069e3c70?})
103322026-03-10T19:00:49.642Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:193 +0xff
103332026-03-10T19:00:49.642Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0xc00146cdc0, {0x50e60e0, 0xc00674ef60}, 0xc00674ed20)
103342026-03-10T19:00:49.642Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:291 +0x59c
103352026-03-10T19:00:49.642Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).Start.func2({0x50e60e0, 0xc00674eed0})
103362026-03-10T19:00:49.642Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:215 +0x2f3
103372026-03-10T19:00:49.642Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
103382026-03-10T19:00:49.642Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
103392026-03-10T19:00:49.646Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9992
103402026-03-10T19:00:49.646Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
103412026-03-10T19:00:49.647Z
103422026-03-10T19:00:49.647Zgoroutine 9596 [select, 29 minutes]:
103432026-03-10T19:00:49.647Zgithub.com/cockroachdb/cockroach/pkg/sql.(*TemporaryObjectCleaner).Start.func1({0x50e60e0, 0xc006037aa0})
103442026-03-10T19:00:49.647Z /work/oxidecomputer/cockroach/pkg/sql/temporary_schema.go:632 +0x22e
103452026-03-10T19:00:49.647Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
103462026-03-10T19:00:49.647Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
103472026-03-10T19:00:49.647Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
103482026-03-10T19:00:49.647Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
103492026-03-10T19:00:49.647Z
103502026-03-10T19:00:49.647Zgoroutine 10398 [select]:
103512026-03-10T19:00:49.647Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents(0xc00146c580, {0x50e6118, 0xc007a220a0}, 0xc007a03800, 0xc0041fea80)
103522026-03-10T19:00:49.647Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:331 +0xc5
103532026-03-10T19:00:49.647Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func2({0x50e6118?, 0xc007a220a0?})
103542026-03-10T19:00:49.647Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:288 +0x32
103552026-03-10T19:00:49.647Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
103562026-03-10T19:00:49.647Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
103572026-03-10T19:00:49.648Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
103582026-03-10T19:00:49.648Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
103592026-03-10T19:00:49.648Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9989
103602026-03-10T19:00:49.648Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
103612026-03-10T19:00:49.648Z
103622026-03-10T19:00:49.648Zgoroutine 9747 [select, 4 minutes]:
103632026-03-10T19:00:49.648Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/protectedts/ptreconcile.(*Reconciler).run(0xc0039ff4f0, {0x50e60e0, 0xc00495cc60}, 0xc006848400)
103642026-03-10T19:00:49.648Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/protectedts/ptreconcile/reconciler.go:104 +0x265
103652026-03-10T19:00:49.648Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/protectedts/ptreconcile.(*Reconciler).StartReconciler.func1({0x50e60e0?, 0xc00495cc60?})
103662026-03-10T19:00:49.648Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/protectedts/ptreconcile/reconciler.go:77 +0x2e
103672026-03-10T19:00:49.648Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
103682026-03-10T19:00:49.648Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
103692026-03-10T19:00:49.648Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9741
103702026-03-10T19:00:49.648Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
103712026-03-10T19:00:49.648Z
103722026-03-10T19:00:49.648Zgoroutine 10397 [semacquire, 59 minutes]:
103732026-03-10T19:00:49.648Zsync.runtime_Semacquire(0xc004b20690?)
103742026-03-10T19:00:49.648Z /work/go/src/runtime/sema.go:62 +0x25
103752026-03-10T19:00:49.648Zsync.(*WaitGroup).Wait(0x44f7e0?)
103762026-03-10T19:00:49.648Z /work/go/src/sync/waitgroup.go:116 +0x48
103772026-03-10T19:00:49.650Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc000f9a380)
103782026-03-10T19:00:49.650Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
103792026-03-10T19:00:49.650Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc000f9a380?, {0x50e6118?, 0xc004f9b8b0?}})
103802026-03-10T19:00:49.650Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
103812026-03-10T19:00:49.650Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed(0xc0021b6f08, {0x50e6118?, 0xc007a220a0?}, {0xc007a03b60, 0x1, 0xc003bb3f08?}, {0xc0077eb920?, 0x357c421?, 0xc0?}, 0x0, ...)
103822026-03-10T19:00:49.650Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:139 +0x6c5
103832026-03-10T19:00:49.650Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*dbAdapter).RangeFeed(0x27a3385f8b8480?, {0x50e6118?, 0xc007a220a0?}, {0xc007a03b60?, 0x0?, 0xc001e9ea08?}, {0x50e60e0?, 0x399c630?, 0xc0?}, 0x0, ...)
103842026-03-10T19:00:49.651Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/db_adapter.go:78 +0x3e
103852026-03-10T19:00:49.651Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func1({0x50e6118?, 0xc007a220a0?})
103862026-03-10T19:00:49.651Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:285 +0x6c
103872026-03-10T19:00:49.651Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
103882026-03-10T19:00:49.651Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
103892026-03-10T19:00:49.651Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
103902026-03-10T19:00:49.651Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
103912026-03-10T19:00:49.651Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9989
103922026-03-10T19:00:49.651Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
103932026-03-10T19:00:49.651Z
103942026-03-10T19:00:49.651Zgoroutine 10368 [select, 59 minutes]:
103952026-03-10T19:00:49.651Zgithub.com/cockroachdb/cockroach/pkg/util/tracing.newTracingClientStream.func2()
103962026-03-10T19:00:49.651Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:386 +0x67
103972026-03-10T19:00:49.652Zcreated by github.com/cockroachdb/cockroach/pkg/util/tracing.newTracingClientStream in goroutine 10366
103982026-03-10T19:00:49.652Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:385 +0x125
103992026-03-10T19:00:49.652Z
104002026-03-10T19:00:49.652Zgoroutine 10334 [semacquire, 59 minutes]:
104012026-03-10T19:00:49.652Zsync.runtime_Semacquire(0xc0000125d0?)
104022026-03-10T19:00:49.652Z /work/go/src/runtime/sema.go:62 +0x25
104032026-03-10T19:00:49.652Zsync.(*WaitGroup).Wait(0x44f7e0?)
104042026-03-10T19:00:49.652Z /work/go/src/sync/waitgroup.go:116 +0x48
104052026-03-10T19:00:49.652Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc003348540)
104062026-03-10T19:00:49.652Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
104072026-03-10T19:00:49.652Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc003348540?, {0x50e6118?, 0xc0053af680?}})
104082026-03-10T19:00:49.652Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
104092026-03-10T19:00:49.652Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed(0xc00432cf08, {0x50e6118?, 0xc0053af5e0?}, {0xc00674c4e0, 0x1, 0xc003bb2720?}, {0xc006ab8930?, 0x1f3bb91?, 0xc0?}, 0x0, ...)
104102026-03-10T19:00:49.652Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:139 +0x6c5
104112026-03-10T19:00:49.653Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*dbAdapter).RangeFeed(0x0?, {0x50e6118?, 0xc0053af5e0?}, {0xc00674c4e0?, 0xa38128?, 0xc00461ac00?}, {0x3ab34c0?, 0x77cd968?, 0xc0?}, 0x0, ...)
104122026-03-10T19:00:49.653Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/db_adapter.go:78 +0x3e
104132026-03-10T19:00:49.653Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func1({0x50e6118?, 0xc0053af5e0?})
104142026-03-10T19:00:49.653Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:285 +0x6c
104152026-03-10T19:00:49.653Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
104162026-03-10T19:00:49.653Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
104172026-03-10T19:00:49.653Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
104182026-03-10T19:00:49.653Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
104192026-03-10T19:00:49.653Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9944
104202026-03-10T19:00:49.653Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
104212026-03-10T19:00:49.653Z
104222026-03-10T19:00:49.653Zgoroutine 10272 [select]:
104232026-03-10T19:00:49.653Zgoogle.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc00274a960)
104242026-03-10T19:00:49.653Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1572 +0x137
104252026-03-10T19:00:49.654Zcreated by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 10354
104262026-03-10T19:00:49.654Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:360 +0x180b
104272026-03-10T19:00:49.654Z
104282026-03-10T19:00:49.654Zgoroutine 10362 [semacquire, 59 minutes]:
104292026-03-10T19:00:49.654Zsync.runtime_Semacquire(0xc004150d80?)
104302026-03-10T19:00:49.654Z /work/go/src/runtime/sema.go:62 +0x25
104312026-03-10T19:00:49.654Zsync.(*WaitGroup).Wait(0x44f7e0?)
104322026-03-10T19:00:49.654Z /work/go/src/sync/waitgroup.go:116 +0x48
104332026-03-10T19:00:49.654Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc001494dc0)
104342026-03-10T19:00:49.654Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
104352026-03-10T19:00:49.654Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc001494dc0?, {0x50e6118?, 0xc001296910?}})
104362026-03-10T19:00:49.654Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
104372026-03-10T19:00:49.654Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed(0xc0021b6f08, {0x50e6118?, 0xc001296870?}, {0xc002cb1e90, 0x1, 0xc0021d4f20?}, {0xc0077eb920?, 0x469cb11?, 0xc0?}, 0x0, ...)
104382026-03-10T19:00:49.654Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:139 +0x6c5
104392026-03-10T19:00:49.658Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*dbAdapter).RangeFeed(0x1b1e12bd020c2f?, {0x50e6118?, 0xc001296870?}, {0xc002cb1e90?, 0x0?, 0xc0015e5508?}, {0x50e60e0?, 0x3b00510?, 0xc0?}, 0x0, ...)
104402026-03-10T19:00:49.658Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/db_adapter.go:78 +0x3e
104412026-03-10T19:00:49.658Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func1({0x50e6118?, 0xc001296870?})
104422026-03-10T19:00:49.658Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:285 +0x6c
104432026-03-10T19:00:49.658Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
104442026-03-10T19:00:49.658Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
104452026-03-10T19:00:49.658Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
104462026-03-10T19:00:49.658Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
104472026-03-10T19:00:49.658Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10276
104482026-03-10T19:00:49.658Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
104492026-03-10T19:00:49.658Z
104502026-03-10T19:00:49.658Zgoroutine 10040 [sync.Cond.Wait, 59 minutes]:
104512026-03-10T19:00:49.658Zsync.runtime_notifyListWait(0xc00676e610, 0x7)
104522026-03-10T19:00:49.658Z /work/go/src/runtime/sema.go:569 +0x159
104532026-03-10T19:00:49.658Zsync.(*Cond).Wait(0x3f1b920?)
104542026-03-10T19:00:49.658Z /work/go/src/sync/cond.go:70 +0x85
104552026-03-10T19:00:49.658Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c9800})
104562026-03-10T19:00:49.658Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
104572026-03-10T19:00:49.658Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
104582026-03-10T19:00:49.658Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
104592026-03-10T19:00:49.658Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
104602026-03-10T19:00:49.658Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
104612026-03-10T19:00:49.658Z
104622026-03-10T19:00:49.659Zgoroutine 10163 [select, 59 minutes]:
104632026-03-10T19:00:49.659Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2({0x50e60e0, 0xc00672be60})
104642026-03-10T19:00:49.659Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/queue.go:823 +0xff
104652026-03-10T19:00:49.659Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
104662026-03-10T19:00:49.659Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
104672026-03-10T19:00:49.659Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10105
104682026-03-10T19:00:49.659Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
104692026-03-10T19:00:49.659Z
104702026-03-10T19:00:49.659Zgoroutine 10645 [select, 59 minutes]:
104712026-03-10T19:00:49.659Zgithub.com/cockroachdb/cockroach/pkg/util/tracing.newTracingClientStream.func2()
104722026-03-10T19:00:49.659Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:386 +0x67
104732026-03-10T19:00:49.659Zcreated by github.com/cockroachdb/cockroach/pkg/util/tracing.newTracingClientStream in goroutine 10643
104742026-03-10T19:00:49.659Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:385 +0x125
104752026-03-10T19:00:49.659Z
104762026-03-10T19:00:49.659Zgoroutine 9672 [semacquire, 59 minutes]:
104772026-03-10T19:00:49.660Zsync.runtime_Semacquire(0xc007269b30?)
104782026-03-10T19:00:49.660Z /work/go/src/runtime/sema.go:62 +0x25
104792026-03-10T19:00:49.660Zsync.(*WaitGroup).Wait(0x44f7e0?)
104802026-03-10T19:00:49.660Z /work/go/src/sync/waitgroup.go:116 +0x48
104812026-03-10T19:00:49.660Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc005e9f8c0)
104822026-03-10T19:00:49.660Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
104832026-03-10T19:00:49.660Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc005e9f8c0?, {0x50e6118?, 0xc004129770?}})
104842026-03-10T19:00:49.660Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
104852026-03-10T19:00:49.660Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait({0x50e60e0?, 0xc00457a5d0?}, {0xc00303ed78, 0x2, 0x4000000000000000?})
104862026-03-10T19:00:49.660Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:193 +0xff
104872026-03-10T19:00:49.660Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0xc00133c6e0, {0x50e60e0, 0xc00457a5d0}, 0xc005f94300)
104882026-03-10T19:00:49.660Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:291 +0x59c
104892026-03-10T19:00:49.660Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).Start.func2({0x50e60e0, 0xc005f943f0})
104902026-03-10T19:00:49.660Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:215 +0x2f3
104912026-03-10T19:00:49.661Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
104922026-03-10T19:00:49.661Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
104932026-03-10T19:00:49.661Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9741
104942026-03-10T19:00:49.661Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
104952026-03-10T19:00:49.661Z
104962026-03-10T19:00:49.661Zgoroutine 9779 [select, 59 minutes]:
104972026-03-10T19:00:49.661Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1({0x50e6118, 0xc004129950})
104982026-03-10T19:00:49.661Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:116 +0x157
104992026-03-10T19:00:49.661Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.Group.GoCtx.func3()
105002026-03-10T19:00:49.661Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
105012026-03-10T19:00:49.661Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
105022026-03-10T19:00:49.661Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
105032026-03-10T19:00:49.661Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9761
105042026-03-10T19:00:49.661Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
105052026-03-10T19:00:49.661Z
105062026-03-10T19:00:49.662Zgoroutine 10419 [chan receive, 59 minutes]:
105072026-03-10T19:00:49.662Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).rangeFeedWithRangeID(0xc001b0a008, 0x3f72e40?, 0xc00523ea20, {0x7f784b79a928, 0xc0012079c0})
105082026-03-10T19:00:49.662Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:249 +0x76b
105092026-03-10T19:00:49.662Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed(...)
105102026-03-10T19:00:49.662Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:149
105112026-03-10T19:00:49.662Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).RangeFeed(0xc001668008, 0xc00523ea20, {0x511a910, 0xc0012079c0})
105122026-03-10T19:00:49.662Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:3082 +0x16f
105132026-03-10T19:00:49.662Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).RangeFeed(0xc006849e00, 0xc00523ea20, {0x511a910, 0xc0012079c0})
105142026-03-10T19:00:49.662Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/stores.go:216 +0x19a
105152026-03-10T19:00:49.662Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed(0x3f72e40?, 0xc0053af840?, {0x511a910, 0xc0012079c0})
105162026-03-10T19:00:49.662Z /work/oxidecomputer/cockroach/pkg/server/node.go:1232 +0x28
105172026-03-10T19:00:49.662Zgithub.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed.func1()
105182026-03-10T19:00:49.662Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:753 +0xa7
105192026-03-10T19:00:49.662Zcreated by github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed in goroutine 10418
105202026-03-10T19:00:49.662Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:750 +0x17c
105212026-03-10T19:00:49.663Z
105222026-03-10T19:00:49.663Zgoroutine 9728 [chan receive, 59 minutes]:
105232026-03-10T19:00:49.663Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).rangeFeedWithRangeID(0xc0019cc808, 0x3f72e40?, 0xc0038665a0, {0x7f784b79a928, 0xc00148c920})
105242026-03-10T19:00:49.663Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:249 +0x76b
105252026-03-10T19:00:49.663Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed(...)
105262026-03-10T19:00:49.663Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:149
105272026-03-10T19:00:49.663Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).RangeFeed(0xc001668008, 0xc0038665a0, {0x511a910, 0xc00148c920})
105282026-03-10T19:00:49.663Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:3082 +0x16f
105292026-03-10T19:00:49.663Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).RangeFeed(0xc006849e00, 0xc0038665a0, {0x511a910, 0xc00148c920})
105302026-03-10T19:00:49.663Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/stores.go:216 +0x19a
105312026-03-10T19:00:49.663Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed(0x3f72e40?, 0xc0012db390?, {0x511a910, 0xc00148c920})
105322026-03-10T19:00:49.663Z /work/oxidecomputer/cockroach/pkg/server/node.go:1232 +0x28
105332026-03-10T19:00:49.663Zgithub.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed.func1()
105342026-03-10T19:00:49.663Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:753 +0xa7
105352026-03-10T19:00:49.663Zcreated by github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed in goroutine 9727
105362026-03-10T19:00:49.663Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:750 +0x17c
105372026-03-10T19:00:49.663Z
105382026-03-10T19:00:49.663Zgoroutine 10399 [chan receive, 59 minutes]:
105392026-03-10T19:00:49.667Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).rangeFeedWithRangeID(0xc004e66008, 0x3dbc860?, 0xc004d24a20, {0x7f784cb862a8, 0xc005cbb1f0})
105402026-03-10T19:00:49.667Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:249 +0x76b
105412026-03-10T19:00:49.667Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed(...)
105422026-03-10T19:00:49.667Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:149
105432026-03-10T19:00:49.667Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).RangeFeed(0xc001668008, 0xc004d24a20, {0x5119e20, 0xc005cbb1f0})
105442026-03-10T19:00:49.667Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:3082 +0x16f
105452026-03-10T19:00:49.667Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).RangeFeed(0xc006849e00, 0xc004d24a20, {0x5119e20, 0xc005cbb1f0})
105462026-03-10T19:00:49.667Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/stores.go:216 +0x19a
105472026-03-10T19:00:49.667Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed(0x7366f90?, 0x403a5e0?, {0x5119e20, 0xc005cbb1f0})
105482026-03-10T19:00:49.668Z /work/oxidecomputer/cockroach/pkg/server/node.go:1232 +0x28
105492026-03-10T19:00:49.668Zgithub.com/cockroachdb/cockroach/pkg/roachpb._Internal_RangeFeed_Handler({0x403a5e0, 0xc001668908}, {0x5114010, 0xc0007868c0})
105502026-03-10T19:00:49.668Z /work/oxidecomputer/cockroach/pkg/roachpb/api.pb.go:9523 +0x107
105512026-03-10T19:00:49.668Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.StreamServerInterceptor.func14({0x403a5e0, 0xc001668908}, {0x51147c0, 0xc00355da00}, 0xc0049be600, 0x4319298)
105522026-03-10T19:00:49.668Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:205 +0x35b
105532026-03-10T19:00:49.668Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x403a5e0?, 0xc001668908?}, {0x51147c0?, 0xc00355da00?})
105542026-03-10T19:00:49.668Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1408 +0x59
105552026-03-10T19:00:49.668Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func4({0x403a5e0, 0xc001668908}, {0x51147c0, 0xc00355da00}, 0x30?, 0xc003422cc0)
105562026-03-10T19:00:49.668Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:247 +0x6c
105572026-03-10T19:00:49.668Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x403a5e0?, 0xc001668908?}, {0x51147c0?, 0xc00355da00?})
105582026-03-10T19:00:49.668Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
105592026-03-10T19:00:49.668Zgithub.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({{{0x3afe8c0?}}}, {0x403a5e0, 0xc001668908}, {0x51147c0, 0xc00355da00}, 0xc0049be600, 0xc003422cc0)
105602026-03-10T19:00:49.668Z /work/oxidecomputer/cockroach/pkg/rpc/auth.go:98 +0x19b
105612026-03-10T19:00:49.668Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x403a5e0?, 0xc001668908?}, {0x51147c0?, 0xc00355da00?})
105622026-03-10T19:00:49.668Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
105632026-03-10T19:00:49.668Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func2.1({0x66?, 0x40?})
105642026-03-10T19:00:49.668Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:216 +0x27
105652026-03-10T19:00:49.668Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc006848400, {0x50e60e0, 0xc0062134d0}, {0x4165af?, 0x7f784b7b2a00?}, 0xc004c69ac0)
105662026-03-10T19:00:49.669Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:344 +0xcb
105672026-03-10T19:00:49.669Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func2({0x403a5e0, 0xc001668908}, {0x51147c0, 0xc00355da00}, 0xc0049be600, 0xc003422cc0)
105682026-03-10T19:00:49.669Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:215 +0xdf
105692026-03-10T19:00:49.669Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x403a5e0?, 0xc001668908?}, {0x51147c0?, 0xc00355da00?})
105702026-03-10T19:00:49.669Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
105712026-03-10T19:00:49.669Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x403a5e0, 0xc001668908}, {0x51147c0, 0xc00355da00}, 0xc0049be600, 0x4319298)
105722026-03-10T19:00:49.669Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1413 +0x108
105732026-03-10T19:00:49.669Zgoogle.golang.org/grpc.(*Server).processStreamingRPC(0xc005fee1c0, {0x5126b60, 0xc00130c340}, 0xc004d24900, 0xc0077eb920, 0x73e2b80, 0x0)
105742026-03-10T19:00:49.669Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1549 +0xe51
105752026-03-10T19:00:49.669Zgoogle.golang.org/grpc.(*Server).handleStream(0xc005fee1c0, {0x5126b60, 0xc00130c340}, 0xc004d24900, 0x0)
105762026-03-10T19:00:49.669Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1624 +0x999
105772026-03-10T19:00:49.669Zgoogle.golang.org/grpc.(*Server).serveStreams.func1.2()
105782026-03-10T19:00:49.669Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:922 +0x8d
105792026-03-10T19:00:49.669Zcreated by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 10187
105802026-03-10T19:00:49.669Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:920 +0x226
105812026-03-10T19:00:49.669Z
105822026-03-10T19:00:49.669Zgoroutine 9214 [select]:
105832026-03-10T19:00:49.669Zgithub.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue.func2()
105842026-03-10T19:00:49.669Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:366 +0x85
105852026-03-10T19:00:49.669Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue in goroutine 9099
105862026-03-10T19:00:49.670Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:363 +0x3ed
105872026-03-10T19:00:49.670Z
105882026-03-10T19:00:49.670Zgoroutine 9213 [select]:
105892026-03-10T19:00:49.670Zgithub.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
105902026-03-10T19:00:49.670Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:439 +0x1c9
105912026-03-10T19:00:49.670Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 9099
105922026-03-10T19:00:49.671Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:412 +0x4f
105932026-03-10T19:00:49.671Z
105942026-03-10T19:00:49.671Zgoroutine 9212 [select]:
105952026-03-10T19:00:49.671Zgithub.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue.func2()
105962026-03-10T19:00:49.671Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:366 +0x85
105972026-03-10T19:00:49.671Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue in goroutine 9099
105982026-03-10T19:00:49.671Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:363 +0x3ed
105992026-03-10T19:00:49.671Z
106002026-03-10T19:00:49.671Zgoroutine 9211 [select]:
106012026-03-10T19:00:49.671Zgithub.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
106022026-03-10T19:00:49.671Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:439 +0x1c9
106032026-03-10T19:00:49.671Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 9099
106042026-03-10T19:00:49.671Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:412 +0x4f
106052026-03-10T19:00:49.671Z
106062026-03-10T19:00:49.671Zgoroutine 9210 [select]:
106072026-03-10T19:00:49.671Zgithub.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue.func2()
106082026-03-10T19:00:49.671Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:366 +0x85
106092026-03-10T19:00:49.671Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue in goroutine 9099
106102026-03-10T19:00:49.671Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:363 +0x3ed
106112026-03-10T19:00:49.671Z
106122026-03-10T19:00:49.671Zgoroutine 9992 [select]:
106132026-03-10T19:00:49.671Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.(*Watcher).Run(0xc003c80c60, {0x50e6118, 0xc0014530e0})
106142026-03-10T19:00:49.671Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:319 +0x955
106152026-03-10T19:00:49.671Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.Start.func1({0x50e60e0?, 0xc00674ecc0?})
106162026-03-10T19:00:49.671Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:189 +0x2d1
106172026-03-10T19:00:49.671Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
106182026-03-10T19:00:49.671Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
106192026-03-10T19:00:49.671Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
106202026-03-10T19:00:49.671Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
106212026-03-10T19:00:49.671Z
106222026-03-10T19:00:49.671Zgoroutine 9208 [select]:
106232026-03-10T19:00:49.671Zgithub.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue.func2()
106242026-03-10T19:00:49.671Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:366 +0x85
106252026-03-10T19:00:49.672Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue in goroutine 9099
106262026-03-10T19:00:49.672Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:363 +0x3ed
106272026-03-10T19:00:49.672Z
106282026-03-10T19:00:49.672Zgoroutine 9966 [select, 59 minutes]:
106292026-03-10T19:00:49.672Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2({0x50e60e0, 0xc00672bd70})
106302026-03-10T19:00:49.672Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/queue.go:823 +0xff
106312026-03-10T19:00:49.672Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
106322026-03-10T19:00:49.672Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
106332026-03-10T19:00:49.672Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10105
106342026-03-10T19:00:49.672Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
106352026-03-10T19:00:49.672Z
106362026-03-10T19:00:49.672Zgoroutine 9209 [select]:
106372026-03-10T19:00:49.672Zgithub.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
106382026-03-10T19:00:49.672Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:439 +0x1c9
106392026-03-10T19:00:49.676Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 9099
106402026-03-10T19:00:49.676Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:412 +0x4f
106412026-03-10T19:00:49.676Z
106422026-03-10T19:00:49.676Zgoroutine 9696 [chan receive, 59 minutes]:
106432026-03-10T19:00:49.676Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).rangeFeedWithRangeID(0xc004e66008, 0x3f72e40?, 0xc00142f440, {0x7f784b79a928, 0xc001754120})
106442026-03-10T19:00:49.676Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:249 +0x76b
106452026-03-10T19:00:49.676Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed(...)
106462026-03-10T19:00:49.676Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:149
106472026-03-10T19:00:49.676Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).RangeFeed(0xc001668008, 0xc00142f440, {0x511a910, 0xc001754120})
106482026-03-10T19:00:49.676Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:3082 +0x16f
106492026-03-10T19:00:49.677Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).RangeFeed(0xc006849e00, 0xc00142f440, {0x511a910, 0xc001754120})
106502026-03-10T19:00:49.677Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/stores.go:216 +0x19a
106512026-03-10T19:00:49.677Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed(0x3f72e40?, 0xc0046a6bc0?, {0x511a910, 0xc001754120})
106522026-03-10T19:00:49.679Z /work/oxidecomputer/cockroach/pkg/server/node.go:1232 +0x28
106532026-03-10T19:00:49.679Zgithub.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed.func1()
106542026-03-10T19:00:49.679Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:753 +0xa7
106552026-03-10T19:00:49.679Zcreated by github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed in goroutine 9695
106562026-03-10T19:00:49.679Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:750 +0x17c
106572026-03-10T19:00:49.679Z
106582026-03-10T19:00:49.679Zgoroutine 9789 [select]:
106592026-03-10T19:00:49.679Zgithub.com/cockroachdb/cockroach/pkg/kv/kvprober.(*Prober).Start.func1.1({0x50e60e0, 0xc00457b5f0})
106602026-03-10T19:00:49.679Z /work/oxidecomputer/cockroach/pkg/kv/kvprober/kvprober.go:293 +0x314
106612026-03-10T19:00:49.679Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
106622026-03-10T19:00:49.679Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
106632026-03-10T19:00:49.679Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
106642026-03-10T19:00:49.679Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
106652026-03-10T19:00:49.679Z
106662026-03-10T19:00:49.679Zgoroutine 9965 [select, 59 minutes]:
106672026-03-10T19:00:49.679Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2({0x50e60e0, 0xc00672bd40})
106682026-03-10T19:00:49.679Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/queue.go:823 +0xff
106692026-03-10T19:00:49.679Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
106702026-03-10T19:00:49.679Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
106712026-03-10T19:00:49.679Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10105
106722026-03-10T19:00:49.679Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
106732026-03-10T19:00:49.679Z
106742026-03-10T19:00:49.680Zgoroutine 9349 [chan receive, 59 minutes]:
106752026-03-10T19:00:49.680Zgithub.com/cockroachdb/cockroach/pkg/server.startListenRPCAndSQL.func3({0xc0015eb808?, 0x1?})
106762026-03-10T19:00:49.680Z /work/oxidecomputer/cockroach/pkg/server/start_listen.go:108 +0x3a
106772026-03-10T19:00:49.680Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
106782026-03-10T19:00:49.680Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
106792026-03-10T19:00:49.680Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
106802026-03-10T19:00:49.680Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
106812026-03-10T19:00:49.680Z
106822026-03-10T19:00:49.680Zgoroutine 10411 [select]:
106832026-03-10T19:00:49.680Zgoogle.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc002cfbbd0, {0xc0016ddcf0, 0x5, 0x5})
106842026-03-10T19:00:49.680Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:191 +0x95
106852026-03-10T19:00:49.680Zgoogle.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc002cfbbd0, {0xc0016ddcf0?, 0xc0040464c8?, 0xc00385d048?})
106862026-03-10T19:00:49.680Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:171 +0x12d
106872026-03-10T19:00:49.680Zgoogle.golang.org/grpc/internal/transport.(*transportReader).Read(0xc002a500f0, {0xc0016ddcf0?, 0xc00385d0c0?, 0xa418a5?})
106882026-03-10T19:00:49.680Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:485 +0x2c
106892026-03-10T19:00:49.680Zio.ReadAtLeast({0x50bb160, 0xc002a500f0}, {0xc0016ddcf0, 0x5, 0x5}, 0x5)
106902026-03-10T19:00:49.680Z /work/go/src/io/io.go:335 +0x90
106912026-03-10T19:00:49.680Zio.ReadFull(...)
106922026-03-10T19:00:49.680Z /work/go/src/io/io.go:354
106932026-03-10T19:00:49.682Zgoogle.golang.org/grpc/internal/transport.(*Stream).Read(0xc002de1680, {0xc0016ddcf0, 0x5, 0x5})
106942026-03-10T19:00:49.682Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:469 +0x96
106952026-03-10T19:00:49.682Zgoogle.golang.org/grpc.(*parser).recvMsg(0xc0016ddce0, 0x7fffffff)
106962026-03-10T19:00:49.682Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:559 +0x42
106972026-03-10T19:00:49.682Zgoogle.golang.org/grpc.recvAndDecompress(0x60?, 0xc002de1680, {0x0, 0x0}, 0x7fffffff, 0xc00385d378, {0x50ddc20, 0x778e7c0})
106982026-03-10T19:00:49.682Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:690 +0x5a
106992026-03-10T19:00:49.682Zgoogle.golang.org/grpc.recv(0x50c4e80?, {0x7f784b7a48e8, 0xc000567440}, 0xc00385d300?, {0x0?, 0x0?}, {0x3fa0a40, 0xc028b6c320}, 0x10?, 0xc00385d378, ...)
107002026-03-10T19:00:49.682Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:756 +0x68
107012026-03-10T19:00:49.682Zgoogle.golang.org/grpc.(*csAttempt).recvMsg(0xc007474160, {0x3fa0a40, 0xc028b6c320}, 0x10?)
107022026-03-10T19:00:49.682Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:983 +0x27e
107032026-03-10T19:00:49.682Zgoogle.golang.org/grpc.(*clientStream).RecvMsg.func1(0x41f225?)
107042026-03-10T19:00:49.682Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:834 +0x1f
107052026-03-10T19:00:49.682Zgoogle.golang.org/grpc.(*clientStream).withRetry(0xc002de1440, 0xc00385d4a8, 0xc00385d498)
107062026-03-10T19:00:49.682Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:688 +0x2ee
107072026-03-10T19:00:49.682Zgoogle.golang.org/grpc.(*clientStream).RecvMsg(0xc002de1440, {0x3fa0a40?, 0xc028b6c320?})
107082026-03-10T19:00:49.682Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:833 +0x107
107092026-03-10T19:00:49.682Zgithub.com/cockroachdb/cockroach/pkg/util/tracing.(*tracingClientStream).RecvMsg(0xc000f106a0, {0x3fa0a40?, 0xc028b6c320?})
107102026-03-10T19:00:49.682Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:441 +0x31
107112026-03-10T19:00:49.682Zgithub.com/cockroachdb/cockroach/pkg/roachpb.(*internalRangeFeedClient).Recv(0xc002773a30)
107122026-03-10T19:00:49.682Z /work/oxidecomputer/cockroach/pkg/roachpb/api.pb.go:9289 +0x46
107132026-03-10T19:00:49.682Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed.func3()
107142026-03-10T19:00:49.682Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:516 +0x25
107152026-03-10T19:00:49.682Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*stuckRangeFeedCanceler).do(0xc002cfba90, 0xc00385da50)
107162026-03-10T19:00:49.683Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed_canceler.go:100 +0x1b3
107172026-03-10T19:00:49.683Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed(0xc0021b6f08, {0x50e6118?, 0xc002cfb9a0?}, {{0xc00342be60, 0x1, 0x8}, {0xc00342be68, 0x1, 0x8}}, {0x189b8cc3e638277a, ...}, ...)
107182026-03-10T19:00:49.683Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:515 +0xc98
107192026-03-10T19:00:49.683Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).partialRangeFeed(0xc0021b6f08, {0x50e6118, 0xc002cfb9a0}, 0xc002cfb950, {{0xc00342be60, 0x1, 0x8}, {0xc00342be68, 0x1, 0x8}}, ...)
107202026-03-10T19:00:49.683Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:336 +0x745
107212026-03-10T19:00:49.683Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.1({0x50e6118?, 0xc002cfb9a0?})
107222026-03-10T19:00:49.683Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:120 +0xa9
107232026-03-10T19:00:49.683Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.Group.GoCtx.2()
107242026-03-10T19:00:49.683Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
107252026-03-10T19:00:49.683Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
107262026-03-10T19:00:49.683Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
107272026-03-10T19:00:49.683Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10409
107282026-03-10T19:00:49.683Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
107292026-03-10T19:00:49.683Z
107302026-03-10T19:00:49.683Zgoroutine 10165 [select]:
107312026-03-10T19:00:49.683Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaScanner).waitAndProcess(0xc0030548c0, {0x50e60e0, 0xc00672bec0}, {0x10000c00148aa50?, 0xc002c96908?, 0x0?}, 0x0)
107322026-03-10T19:00:49.683Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scanner.go:224 +0x1f1
107332026-03-10T19:00:49.683Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaScanner).scanLoop.func1({0x50e60e0, 0xc00672bec0})
107342026-03-10T19:00:49.683Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scanner.go:291 +0x1e5
107352026-03-10T19:00:49.683Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
107362026-03-10T19:00:49.683Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
107372026-03-10T19:00:49.684Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10105
107382026-03-10T19:00:49.684Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
107392026-03-10T19:00:49.688Z
107402026-03-10T19:00:49.688Zgoroutine 9215 [select]:
107412026-03-10T19:00:49.688Zgithub.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
107422026-03-10T19:00:49.688Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:439 +0x1c9
107432026-03-10T19:00:49.688Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 9099
107442026-03-10T19:00:49.688Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:412 +0x4f
107452026-03-10T19:00:49.688Z
107462026-03-10T19:00:49.688Zgoroutine 9961 [select]:
107472026-03-10T19:00:49.688Zgoogle.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc0065fb310, {0xc00654fbb8, 0x5, 0x5})
107482026-03-10T19:00:49.688Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:191 +0x95
107492026-03-10T19:00:49.688Zgoogle.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc0065fb310, {0xc00654fbb8?, 0xc006ababb8?, 0xc003041958?})
107502026-03-10T19:00:49.688Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:171 +0x12d
107512026-03-10T19:00:49.688Zgoogle.golang.org/grpc/internal/transport.(*transportReader).Read(0xc00672b230, {0xc00654fbb8?, 0xc0030419d0?, 0xa418a5?})
107522026-03-10T19:00:49.688Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:485 +0x2c
107532026-03-10T19:00:49.688Zio.ReadAtLeast({0x50bb160, 0xc00672b230}, {0xc00654fbb8, 0x5, 0x5}, 0x5)
107542026-03-10T19:00:49.688Z /work/go/src/io/io.go:335 +0x90
107552026-03-10T19:00:49.688Zio.ReadFull(...)
107562026-03-10T19:00:49.688Z /work/go/src/io/io.go:354
107572026-03-10T19:00:49.688Zgoogle.golang.org/grpc/internal/transport.(*Stream).Read(0xc0067285a0, {0xc00654fbb8, 0x5, 0x5})
107582026-03-10T19:00:49.689Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:469 +0x96
107592026-03-10T19:00:49.689Zgoogle.golang.org/grpc.(*parser).recvMsg(0xc00654fba8, 0x7fffffff)
107602026-03-10T19:00:49.689Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:559 +0x42
107612026-03-10T19:00:49.689Zgoogle.golang.org/grpc.recvAndDecompress(0x3ac3d40?, 0xc0067285a0, {0x0, 0x0}, 0x7fffffff, 0xc003041c88, {0x50ddc20, 0x778e7c0})
107622026-03-10T19:00:49.689Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:690 +0x5a
107632026-03-10T19:00:49.689Zgoogle.golang.org/grpc.recv(0xffffffffffffffff?, {0x7f784b7a48e8, 0xc000567440}, 0x60000c00e0de1e8?, {0x0?, 0x0?}, {0x3f2ff60, 0xc00dd61ae0}, 0x3ac3d40?, 0xc003041c88, ...)
107642026-03-10T19:00:49.689Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:756 +0x68
107652026-03-10T19:00:49.689Zgoogle.golang.org/grpc.(*csAttempt).recvMsg(0xc0067260b0, {0x3f2ff60, 0xc00dd61ae0}, 0x10?)
107662026-03-10T19:00:49.689Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:983 +0x27e
107672026-03-10T19:00:49.689Zgoogle.golang.org/grpc.(*clientStream).RecvMsg.func1(0x41f225?)
107682026-03-10T19:00:49.689Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:834 +0x1f
107692026-03-10T19:00:49.689Zgoogle.golang.org/grpc.(*clientStream).withRetry(0xc006728360, 0xc003041db8, 0xc003041da8)
107702026-03-10T19:00:49.689Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:688 +0x2ee
107712026-03-10T19:00:49.689Zgoogle.golang.org/grpc.(*clientStream).RecvMsg(0xc006728360, {0x3f2ff60?, 0xc00dd61ae0?})
107722026-03-10T19:00:49.689Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:833 +0x107
107732026-03-10T19:00:49.689Zgithub.com/cockroachdb/cockroach/pkg/util/tracing.(*tracingClientStream).RecvMsg(0xc0064c7320, {0x3f2ff60?, 0xc00dd61ae0?})
107742026-03-10T19:00:49.689Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:441 +0x31
107752026-03-10T19:00:49.689Zgithub.com/cockroachdb/cockroach/pkg/gossip.(*gossipGossipClient).Recv(0xc006d31570)
107762026-03-10T19:00:49.689Z /work/oxidecomputer/cockroach/pkg/gossip/gossip.pb.go:636 +0x46
107772026-03-10T19:00:49.689Zgithub.com/cockroachdb/cockroach/pkg/gossip.(*client).gossip.func2.1(0xc002e6c9c0, {0x51208e0, 0xc006d31570}, 0xc006a95790, {0x50e60e0, 0xc00672b380}, 0xc0037169c0)
107782026-03-10T19:00:49.689Z /work/oxidecomputer/cockroach/pkg/gossip/client.go:317 +0x62
107792026-03-10T19:00:49.689Zgithub.com/cockroachdb/cockroach/pkg/gossip.(*client).gossip.func2({0x50e60e0?, 0xc00672b380?})
107802026-03-10T19:00:49.689Z /work/oxidecomputer/cockroach/pkg/gossip/client.go:332 +0x86
107812026-03-10T19:00:49.689Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
107822026-03-10T19:00:49.689Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
107832026-03-10T19:00:49.689Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10023
107842026-03-10T19:00:49.689Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
107852026-03-10T19:00:49.689Z
107862026-03-10T19:00:49.689Zgoroutine 9962 [select]:
107872026-03-10T19:00:49.689Zgithub.com/cockroachdb/cockroach/pkg/gossip.(*server).Gossip(0xc006332300, {0x5120938, 0xc006d31790})
107882026-03-10T19:00:49.689Z /work/oxidecomputer/cockroach/pkg/gossip/server.go:190 +0x87b
107892026-03-10T19:00:49.689Zgithub.com/cockroachdb/cockroach/pkg/gossip._Gossip_Gossip_Handler({0x3f572a0?, 0xc006332300}, {0x5114010, 0xc0064c73a0})
107902026-03-10T19:00:49.689Z /work/oxidecomputer/cockroach/pkg/gossip/gossip.pb.go:660 +0xd8
107912026-03-10T19:00:49.689Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.StreamServerInterceptor.func14({0x3f572a0, 0xc006332300}, {0x51147c0, 0xc001fa1d40}, 0xc00654fd58, 0x431c058)
107922026-03-10T19:00:49.689Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:205 +0x35b
107932026-03-10T19:00:49.689Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x3f572a0?, 0xc006332300?}, {0x51147c0?, 0xc001fa1d40?})
107942026-03-10T19:00:49.690Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1408 +0x59
107952026-03-10T19:00:49.690Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func4({0x3f572a0, 0xc006332300}, {0x51147c0, 0xc001fa1d40}, 0x0?, 0xc0065bf0c0)
107962026-03-10T19:00:49.690Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:247 +0x6c
107972026-03-10T19:00:49.690Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x3f572a0?, 0xc006332300?}, {0x51147c0?, 0xc001fa1d40?})
107982026-03-10T19:00:49.690Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
107992026-03-10T19:00:49.690Zgithub.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({{{0x50e60e0?}}}, {0x3f572a0, 0xc006332300}, {0x51147c0, 0xc001fa1d40}, 0xc00654fd58, 0xc0065bf0c0)
108002026-03-10T19:00:49.690Z /work/oxidecomputer/cockroach/pkg/rpc/auth.go:98 +0x19b
108012026-03-10T19:00:49.690Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x3f572a0?, 0xc006332300?}, {0x51147c0?, 0xc001fa1d40?})
108022026-03-10T19:00:49.690Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
108032026-03-10T19:00:49.690Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func2.1({0x66?, 0x40?})
108042026-03-10T19:00:49.690Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:216 +0x27
108052026-03-10T19:00:49.690Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc006848400, {0x50e60e0, 0xc00672b5f0}, {0xc001706b28?, 0x416e25?}, 0xc001706ac0)
108062026-03-10T19:00:49.690Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:344 +0xcb
108072026-03-10T19:00:49.690Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func2({0x3f572a0, 0xc006332300}, {0x51147c0, 0xc001fa1d40}, 0xc00654fd58, 0xc0065bf0c0)
108082026-03-10T19:00:49.690Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:215 +0xdf
108092026-03-10T19:00:49.690Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x3f572a0?, 0xc006332300?}, {0x51147c0?, 0xc001fa1d40?})
108102026-03-10T19:00:49.690Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
108112026-03-10T19:00:49.690Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x3f572a0, 0xc006332300}, {0x51147c0, 0xc001fa1d40}, 0xc00654fd58, 0x431c058)
108122026-03-10T19:00:49.690Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1413 +0x108
108132026-03-10T19:00:49.690Zgoogle.golang.org/grpc.(*Server).processStreamingRPC(0xc005fee1c0, {0x5126b60, 0xc000a449c0}, 0xc0067286c0, 0xc0077ea300, 0x73be9a0, 0x0)
108142026-03-10T19:00:49.690Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1549 +0xe51
108152026-03-10T19:00:49.691Zgoogle.golang.org/grpc.(*Server).handleStream(0xc005fee1c0, {0x5126b60, 0xc000a449c0}, 0xc0067286c0, 0x0)
108162026-03-10T19:00:49.691Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1624 +0x999
108172026-03-10T19:00:49.691Zgoogle.golang.org/grpc.(*Server).serveStreams.func1.2()
108182026-03-10T19:00:49.691Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:922 +0x8d
108192026-03-10T19:00:49.691Zcreated by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 10131
108202026-03-10T19:00:49.691Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:920 +0x226
108212026-03-10T19:00:49.691Z
108222026-03-10T19:00:49.692Zgoroutine 9207 [select]:
108232026-03-10T19:00:49.692Zgithub.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
108242026-03-10T19:00:49.692Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:439 +0x1c9
108252026-03-10T19:00:49.692Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 9099
108262026-03-10T19:00:49.692Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:412 +0x4f
108272026-03-10T19:00:49.692Z
108282026-03-10T19:00:49.692Zgoroutine 9873 [chan receive, 59 minutes]:
108292026-03-10T19:00:49.692Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func1({0xc00728c608?, 0xc0065c8b10?})
108302026-03-10T19:00:49.692Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:203 +0x37
108312026-03-10T19:00:49.692Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
108322026-03-10T19:00:49.692Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
108332026-03-10T19:00:49.692Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
108342026-03-10T19:00:49.692Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
108352026-03-10T19:00:49.692Z
108362026-03-10T19:00:49.692Zgoroutine 9695 [select]:
108372026-03-10T19:00:49.692Zgithub.com/cockroachdb/cockroach/pkg/rpc.respStreamClientAdapter.recvInternal(...)
108382026-03-10T19:00:49.692Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:694
108392026-03-10T19:00:49.696Zgithub.com/cockroachdb/cockroach/pkg/rpc.rangeFeedClientAdapter.Recv(...)
108402026-03-10T19:00:49.696Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:723
108412026-03-10T19:00:49.696Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed.func3()
108422026-03-10T19:00:49.696Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:516 +0x25
108432026-03-10T19:00:49.696Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*stuckRangeFeedCanceler).do(0xc0046a6aa0, 0xc0012aba50)
108442026-03-10T19:00:49.696Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed_canceler.go:100 +0x1b3
108452026-03-10T19:00:49.696Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed(0xc00432cf08, {0x50e6118?, 0xc0046a69b0?}, {{0xc00357faf8, 0x1, 0x8}, {0xc00357fb00, 0x1, 0x8}}, {0x189b8cc3e00145ab, ...}, ...)
108462026-03-10T19:00:49.696Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:515 +0xc98
108472026-03-10T19:00:49.696Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).partialRangeFeed(0xc00432cf08, {0x50e6118, 0xc0046a69b0}, 0xc0046a6960, {{0xc00357faf8, 0x1, 0x8}, {0xc00357fb00, 0x1, 0x8}}, ...)
108482026-03-10T19:00:49.696Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:336 +0x745
108492026-03-10T19:00:49.696Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.1({0x50e6118?, 0xc0046a69b0?})
108502026-03-10T19:00:49.696Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:120 +0xa9
108512026-03-10T19:00:49.696Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.Group.GoCtx.2()
108522026-03-10T19:00:49.697Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
108532026-03-10T19:00:49.697Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
108542026-03-10T19:00:49.697Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
108552026-03-10T19:00:49.697Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9693
108562026-03-10T19:00:49.697Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
108572026-03-10T19:00:49.697Z
108582026-03-10T19:00:49.697Zgoroutine 9816 [chan receive, 59 minutes]:
108592026-03-10T19:00:49.697Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).rangeFeedWithRangeID(0xc0019cd408, 0x3f72e40?, 0xc006100900, {0x7f784b79a928, 0xc0017551a0})
108602026-03-10T19:00:49.697Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:249 +0x76b
108612026-03-10T19:00:49.697Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed(...)
108622026-03-10T19:00:49.697Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:149
108632026-03-10T19:00:49.697Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).RangeFeed(0xc001668008, 0xc006100900, {0x511a910, 0xc0017551a0})
108642026-03-10T19:00:49.697Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:3082 +0x16f
108652026-03-10T19:00:49.697Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).RangeFeed(0xc006849e00, 0xc006100900, {0x511a910, 0xc0017551a0})
108662026-03-10T19:00:49.698Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/stores.go:216 +0x19a
108672026-03-10T19:00:49.698Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed(0x3f72e40?, 0xc001749ca0?, {0x511a910, 0xc0017551a0})
108682026-03-10T19:00:49.698Z /work/oxidecomputer/cockroach/pkg/server/node.go:1232 +0x28
108692026-03-10T19:00:49.698Zgithub.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed.func1()
108702026-03-10T19:00:49.698Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:753 +0xa7
108712026-03-10T19:00:49.698Zcreated by github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed in goroutine 9815
108722026-03-10T19:00:49.698Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:750 +0x17c
108732026-03-10T19:00:49.698Z
108742026-03-10T19:00:49.698Zgoroutine 9454 [chan receive, 59 minutes]:
108752026-03-10T19:00:49.698Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).rangeFeedWithRangeID(0xc00128b008, 0x3f72e40?, 0xc001ad4fc0, {0x7f784b79a928, 0xc005f461a0})
108762026-03-10T19:00:49.698Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:249 +0x76b
108772026-03-10T19:00:49.698Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed(...)
108782026-03-10T19:00:49.698Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:149
108792026-03-10T19:00:49.699Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).RangeFeed(0xc001668008, 0xc001ad4fc0, {0x511a910, 0xc005f461a0})
108802026-03-10T19:00:49.699Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:3082 +0x16f
108812026-03-10T19:00:49.699Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).RangeFeed(0xc006849e00, 0xc001ad4fc0, {0x511a910, 0xc005f461a0})
108822026-03-10T19:00:49.699Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/stores.go:216 +0x19a
108832026-03-10T19:00:49.699Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed(0x3f72e40?, 0xc003924a80?, {0x511a910, 0xc005f461a0})
108842026-03-10T19:00:49.699Z /work/oxidecomputer/cockroach/pkg/server/node.go:1232 +0x28
108852026-03-10T19:00:49.699Zgithub.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed.func1()
108862026-03-10T19:00:49.699Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:753 +0xa7
108872026-03-10T19:00:49.699Zcreated by github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed in goroutine 9453
108882026-03-10T19:00:49.699Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:750 +0x17c
108892026-03-10T19:00:49.699Z
108902026-03-10T19:00:49.699Zgoroutine 10024 [select, 59 minutes]:
108912026-03-10T19:00:49.699Zgoogle.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc00670b180)
108922026-03-10T19:00:49.699Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:112 +0x6d
108932026-03-10T19:00:49.699Zcreated by google.golang.org/grpc.newCCBalancerWrapper in goroutine 10023
108942026-03-10T19:00:49.699Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:73 +0x22b
108952026-03-10T19:00:49.700Z
108962026-03-10T19:00:49.700Zgoroutine 10199 [select, 59 minutes]:
108972026-03-10T19:00:49.700Zgithub.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowScheduler).Start.func1({0xc001952608?, 0x0?})
108982026-03-10T19:00:49.700Z /work/oxidecomputer/cockroach/pkg/sql/flowinfra/flow_scheduler.go:331 +0x25c
108992026-03-10T19:00:49.700Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
109002026-03-10T19:00:49.700Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
109012026-03-10T19:00:49.700Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
109022026-03-10T19:00:49.700Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
109032026-03-10T19:00:49.700Z
109042026-03-10T19:00:49.700Zgoroutine 10048 [sync.Cond.Wait, 59 minutes]:
109052026-03-10T19:00:49.700Zsync.runtime_notifyListWait(0xc00676e610, 0xf)
109062026-03-10T19:00:49.700Z /work/go/src/runtime/sema.go:569 +0x159
109072026-03-10T19:00:49.700Zsync.(*Cond).Wait(0x3f1b920?)
109082026-03-10T19:00:49.700Z /work/go/src/sync/cond.go:70 +0x85
109092026-03-10T19:00:49.700Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c9980})
109102026-03-10T19:00:49.700Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
109112026-03-10T19:00:49.700Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
109122026-03-10T19:00:49.700Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
109132026-03-10T19:00:49.700Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
109142026-03-10T19:00:49.700Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
109152026-03-10T19:00:49.700Z
109162026-03-10T19:00:49.700Zgoroutine 10050 [sync.Cond.Wait, 59 minutes]:
109172026-03-10T19:00:49.700Zsync.runtime_notifyListWait(0xc00676e610, 0x11)
109182026-03-10T19:00:49.700Z /work/go/src/runtime/sema.go:569 +0x159
109192026-03-10T19:00:49.700Zsync.(*Cond).Wait(0xc0059faeb0?)
109202026-03-10T19:00:49.700Z /work/go/src/sync/cond.go:70 +0x85
109212026-03-10T19:00:49.700Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006732b60, {0x50e60e0, 0xc0065c99e0})
109222026-03-10T19:00:49.700Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
109232026-03-10T19:00:49.700Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
109242026-03-10T19:00:49.701Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
109252026-03-10T19:00:49.701Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9099
109262026-03-10T19:00:49.701Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
109272026-03-10T19:00:49.701ZFAIL github.com/cockroachdb/cockroach/pkg/migration/migrationmanager 3600.161s
109282026-03-10T19:00:49.701Zok github.com/cockroachdb/cockroach/pkg/migration/migrations 46.379s
109292026-03-10T19:00:49.701Zok github.com/cockroachdb/cockroach/pkg/roachpb 0.335s
109302026-03-10T19:00:49.701Zok github.com/cockroachdb/cockroach/pkg/roachprod/cloud 0.006s
109312026-03-10T19:00:49.701Zok github.com/cockroachdb/cockroach/pkg/roachprod/config 0.004s
109322026-03-10T19:00:49.701Zok github.com/cockroachdb/cockroach/pkg/roachprod/install 0.116s
109332026-03-10T19:00:49.701Zok github.com/cockroachdb/cockroach/pkg/roachprod/ssh 0.001s
109342026-03-10T19:00:49.701Zok github.com/cockroachdb/cockroach/pkg/roachprod/vm 0.006s
109352026-03-10T19:00:49.701Zok github.com/cockroachdb/cockroach/pkg/rpc 3.108s
109362026-03-10T19:00:49.701Zok github.com/cockroachdb/cockroach/pkg/rpc/nodedialer 1.551s
109372026-03-10T19:00:49.701Zok github.com/cockroachdb/cockroach/pkg/security 2.975s
109382026-03-10T19:00:49.701Zok github.com/cockroachdb/cockroach/pkg/security/certmgr 0.080s
109392026-03-10T19:00:49.705Zok github.com/cockroachdb/cockroach/pkg/security/sessionrevival 0.026s
109402026-03-10T19:00:49.705Zok github.com/cockroachdb/cockroach/pkg/server 213.248s
109412026-03-10T19:00:49.705Zok github.com/cockroachdb/cockroach/pkg/server/debug 0.053s
109422026-03-10T19:00:49.705Zok github.com/cockroachdb/cockroach/pkg/server/debug/goroutineui 0.019s
109432026-03-10T19:00:49.705Zok github.com/cockroachdb/cockroach/pkg/server/debug/pprofui 0.100s
109442026-03-10T19:00:49.705Zok github.com/cockroachdb/cockroach/pkg/server/diagnostics 0.865s
109452026-03-10T19:00:49.705Zok github.com/cockroachdb/cockroach/pkg/server/dumpstore 0.010s
109462026-03-10T19:00:49.706Zok github.com/cockroachdb/cockroach/pkg/server/goroutinedumper 0.022s
109472026-03-10T19:00:49.706Zok github.com/cockroachdb/cockroach/pkg/server/heapprofiler 0.146s
109482026-03-10T19:00:49.706Zok github.com/cockroachdb/cockroach/pkg/server/pgurl 0.051s
109492026-03-10T19:00:49.706Zok github.com/cockroachdb/cockroach/pkg/server/serverpb 0.015s
109502026-03-10T19:00:49.706Zok github.com/cockroachdb/cockroach/pkg/server/settingswatcher 1.412s
109512026-03-10T19:00:49.706Zok github.com/cockroachdb/cockroach/pkg/server/status 10.308s
109522026-03-10T19:00:49.706Zok github.com/cockroachdb/cockroach/pkg/server/systemconfigwatcher 0.581s
109532026-03-10T19:00:49.706Zok github.com/cockroachdb/cockroach/pkg/server/telemetry 0.014s
109542026-03-10T19:00:49.706Zok github.com/cockroachdb/cockroach/pkg/server/tenantsettingswatcher 1.145s
109552026-03-10T19:00:49.706Zok github.com/cockroachdb/cockroach/pkg/server/tracedumper 0.249s
109562026-03-10T19:00:49.706Zok github.com/cockroachdb/cockroach/pkg/settings 0.032s
109572026-03-10T19:00:49.706Zok github.com/cockroachdb/cockroach/pkg/spanconfig 0.056s
109582026-03-10T19:00:49.706Zok github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigkvaccessor 2.410s
109592026-03-10T19:00:49.706Zok github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigkvsubscriber 5.429s
109602026-03-10T19:00:49.706Zok github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigmanager 5.594s
109612026-03-10T19:00:49.706Zok github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigptsreader 0.041s
109622026-03-10T19:00:49.707Zok github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigreconciler 0.049s [no tests to run]
109632026-03-10T19:00:49.707Zok github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigsqltranslator 0.029s [no tests to run]
109642026-03-10T19:00:49.707Zok github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigsqlwatcher 0.481s
109652026-03-10T19:00:49.707Zok github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigstore 6.601s
109662026-03-10T19:00:49.707Zok github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigtestutils 0.020s
109672026-03-10T19:00:49.708Zok github.com/cockroachdb/cockroach/pkg/sql 367.285s
109682026-03-10T19:00:49.708Zok github.com/cockroachdb/cockroach/pkg/sql/backfill 0.020s
109692026-03-10T19:00:49.708Zok github.com/cockroachdb/cockroach/pkg/sql/catalog 0.192s
109702026-03-10T19:00:49.708Zok github.com/cockroachdb/cockroach/pkg/sql/catalog/bootstrap 0.021s
109712026-03-10T19:00:49.708Zok github.com/cockroachdb/cockroach/pkg/sql/catalog/catalogkeys 0.034s
109722026-03-10T19:00:49.708Zok github.com/cockroachdb/cockroach/pkg/sql/catalog/catformat 0.038s
109732026-03-10T19:00:49.708Zok github.com/cockroachdb/cockroach/pkg/sql/catalog/catpb 0.030s
109742026-03-10T19:00:49.708Zok github.com/cockroachdb/cockroach/pkg/sql/catalog/catprivilege 0.051s
109752026-03-10T19:00:49.708Zok github.com/cockroachdb/cockroach/pkg/sql/catalog/colinfo 0.030s
109762026-03-10T19:00:49.708Zok github.com/cockroachdb/cockroach/pkg/sql/catalog/dbdesc 0.037s
109772026-03-10T19:00:49.708Zok github.com/cockroachdb/cockroach/pkg/sql/catalog/descpb 0.025s
109782026-03-10T19:00:49.708Zok github.com/cockroachdb/cockroach/pkg/sql/catalog/descs 2.379s
109792026-03-10T19:00:49.708Zok github.com/cockroachdb/cockroach/pkg/sql/catalog/hydratedtables 0.016s
109802026-03-10T19:00:49.708Zok github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate 0.058s
109812026-03-10T19:00:49.708Zok github.com/cockroachdb/cockroach/pkg/sql/catalog/lease 20.053s
109822026-03-10T19:00:49.708Zok github.com/cockroachdb/cockroach/pkg/sql/catalog/multiregion 0.030s
109832026-03-10T19:00:49.708Zok github.com/cockroachdb/cockroach/pkg/sql/catalog/nstree 0.011s
109842026-03-10T19:00:49.708Zok github.com/cockroachdb/cockroach/pkg/sql/catalog/resolver 0.034s
109852026-03-10T19:00:49.708Zok github.com/cockroachdb/cockroach/pkg/sql/catalog/schemadesc 0.026s
109862026-03-10T19:00:49.708Zok github.com/cockroachdb/cockroach/pkg/sql/catalog/schemaexpr 0.028s
109872026-03-10T19:00:49.708Zok github.com/cockroachdb/cockroach/pkg/sql/catalog/seqexpr 0.018s
109882026-03-10T19:00:49.708Zok github.com/cockroachdb/cockroach/pkg/sql/catalog/systemschema 0.013s
109892026-03-10T19:00:49.710Zok github.com/cockroachdb/cockroach/pkg/sql/catalog/tabledesc 2.254s
109902026-03-10T19:00:49.710Zok github.com/cockroachdb/cockroach/pkg/sql/catalog/typedesc 0.058s
109912026-03-10T19:00:49.710Zok github.com/cockroachdb/cockroach/pkg/sql/colcontainer 0.500s
109922026-03-10T19:00:49.710Zok github.com/cockroachdb/cockroach/pkg/sql/colconv 0.083s
109932026-03-10T19:00:49.710Zok github.com/cockroachdb/cockroach/pkg/sql/colencoding 0.108s
109942026-03-10T19:00:49.710Zok github.com/cockroachdb/cockroach/pkg/sql/colexec 7.527s
109952026-03-10T19:00:49.710Zok github.com/cockroachdb/cockroach/pkg/sql/colexec/colbuilder 0.453s
109962026-03-10T19:00:49.710Zok github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecagg 0.379s
109972026-03-10T19:00:49.710Zok github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecargs 0.319s
109982026-03-10T19:00:49.710Zok github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase 0.557s
109992026-03-10T19:00:49.710Zok github.com/cockroachdb/cockroach/pkg/sql/colexec/colexeccmp 0.363s
110002026-03-10T19:00:49.710Zok github.com/cockroachdb/cockroach/pkg/sql/colexec/colexechash 0.643s
110012026-03-10T19:00:49.710Zok github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecjoin 0.632s
110022026-03-10T19:00:49.710Zok github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecproj 0.862s
110032026-03-10T19:00:49.710Zok github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecsel 0.409s
110042026-03-10T19:00:49.710Zok github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecspan 0.381s
110052026-03-10T19:00:49.710Zok github.com/cockroachdb/cockroach/pkg/sql/colexec/colexectestutils 0.137s
110062026-03-10T19:00:49.710Zok github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils 1.430s
110072026-03-10T19:00:49.710Zok github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecwindow 0.667s
110082026-03-10T19:00:49.710Zok github.com/cockroachdb/cockroach/pkg/sql/colexec/execgen 0.050s
110092026-03-10T19:00:49.710Zok github.com/cockroachdb/cockroach/pkg/sql/colexec/execgen/cmd/execgen 0.181s
110102026-03-10T19:00:49.710Zok github.com/cockroachdb/cockroach/pkg/sql/colexecerror 0.063s
110112026-03-10T19:00:49.711Zok github.com/cockroachdb/cockroach/pkg/sql/colexecop 0.279s
110122026-03-10T19:00:49.711Zok github.com/cockroachdb/cockroach/pkg/sql/colfetcher 0.325s
110132026-03-10T19:00:49.711Zok github.com/cockroachdb/cockroach/pkg/sql/colflow 1.760s
110142026-03-10T19:00:49.711Zok github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc 0.234s
110152026-03-10T19:00:49.711Zok github.com/cockroachdb/cockroach/pkg/sql/colmem 0.268s
110162026-03-10T19:00:49.711Zok github.com/cockroachdb/cockroach/pkg/sql/contention 0.083s
110172026-03-10T19:00:49.711Zok github.com/cockroachdb/cockroach/pkg/sql/contention/contentionutils 0.914s
110182026-03-10T19:00:49.711Zok github.com/cockroachdb/cockroach/pkg/sql/contention/txnidcache 0.263s
110192026-03-10T19:00:49.711Zok github.com/cockroachdb/cockroach/pkg/sql/contentionpb 0.009s
110202026-03-10T19:00:49.711Zok github.com/cockroachdb/cockroach/pkg/sql/covering 0.005s
110212026-03-10T19:00:49.711Zok github.com/cockroachdb/cockroach/pkg/sql/delegate 0.014s
110222026-03-10T19:00:49.711Zok github.com/cockroachdb/cockroach/pkg/sql/distsql 7.531s
110232026-03-10T19:00:49.711Zok github.com/cockroachdb/cockroach/pkg/sql/doctor 0.039s
110242026-03-10T19:00:49.711Zok github.com/cockroachdb/cockroach/pkg/sql/enum 0.545s
110252026-03-10T19:00:49.711Zok github.com/cockroachdb/cockroach/pkg/sql/execinfra 0.408s
110262026-03-10T19:00:49.711Zok github.com/cockroachdb/cockroach/pkg/sql/execinfrapb 0.124s
110272026-03-10T19:00:49.711Zok github.com/cockroachdb/cockroach/pkg/sql/execstats 0.661s
110282026-03-10T19:00:49.711Zok github.com/cockroachdb/cockroach/pkg/sql/flowinfra 14.391s
110292026-03-10T19:00:49.711Zok github.com/cockroachdb/cockroach/pkg/sql/gcjob 0.188s
110302026-03-10T19:00:49.711Zok github.com/cockroachdb/cockroach/pkg/sql/gcjob/gcjobnotifier 0.078s
110312026-03-10T19:00:49.711Zok github.com/cockroachdb/cockroach/pkg/sql/gcjob_test 13.611s
110322026-03-10T19:00:49.711Zok github.com/cockroachdb/cockroach/pkg/sql/idxusage 0.035s
110332026-03-10T19:00:49.711Zok github.com/cockroachdb/cockroach/pkg/sql/importer 158.188s
110342026-03-10T19:00:49.713Zok github.com/cockroachdb/cockroach/pkg/sql/inverted 0.181s
110352026-03-10T19:00:49.713Zok github.com/cockroachdb/cockroach/pkg/sql/kvstreamer 7.720s
110362026-03-10T19:00:49.713Zok github.com/cockroachdb/cockroach/pkg/sql/lex 0.050s
110372026-03-10T19:00:49.713Zok github.com/cockroachdb/cockroach/pkg/sql/lexbase 0.236s
110382026-03-10T19:00:49.713Zok github.com/cockroachdb/cockroach/pkg/sql/logictest 922.917s
110392026-03-10T19:00:49.717Zok github.com/cockroachdb/cockroach/pkg/sql/oidext 0.017s
110402026-03-10T19:00:49.717Zok github.com/cockroachdb/cockroach/pkg/sql/opt 0.088s
110412026-03-10T19:00:49.717Zok github.com/cockroachdb/cockroach/pkg/sql/opt/bench 0.088s [no tests to run]
110422026-03-10T19:00:49.717Zok github.com/cockroachdb/cockroach/pkg/sql/opt/cat 0.071s
110432026-03-10T19:00:49.717Zok github.com/cockroachdb/cockroach/pkg/sql/opt/constraint 0.146s
110442026-03-10T19:00:49.717Zok github.com/cockroachdb/cockroach/pkg/sql/opt/distribution 0.064s
110452026-03-10T19:00:49.717Zok github.com/cockroachdb/cockroach/pkg/sql/opt/exec/execbuilder 45.746s
110462026-03-10T19:00:49.717Zok github.com/cockroachdb/cockroach/pkg/sql/opt/exec/explain 0.465s
110472026-03-10T19:00:49.717Zok github.com/cockroachdb/cockroach/pkg/sql/opt/idxconstraint 0.182s
110482026-03-10T19:00:49.717Zok github.com/cockroachdb/cockroach/pkg/sql/opt/indexrec 0.137s
110492026-03-10T19:00:49.717Zok github.com/cockroachdb/cockroach/pkg/sql/opt/invertedexpr 0.192s
110502026-03-10T19:00:49.717Zok github.com/cockroachdb/cockroach/pkg/sql/opt/invertedidx 0.220s
110512026-03-10T19:00:49.717Zok github.com/cockroachdb/cockroach/pkg/sql/opt/memo 2.218s
110522026-03-10T19:00:49.717Zok github.com/cockroachdb/cockroach/pkg/sql/opt/norm 1.264s
110532026-03-10T19:00:49.717Zok github.com/cockroachdb/cockroach/pkg/sql/opt/opbench 1.334s
110542026-03-10T19:00:49.717Zok github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder 0.923s
110552026-03-10T19:00:49.717Zok github.com/cockroachdb/cockroach/pkg/sql/opt/optgen/cmd/optfmt 0.086s
110562026-03-10T19:00:49.718Zok github.com/cockroachdb/cockroach/pkg/sql/opt/optgen/cmd/optgen 0.057s
110572026-03-10T19:00:49.718Zok github.com/cockroachdb/cockroach/pkg/sql/opt/optgen/exprgen 0.160s
110582026-03-10T19:00:49.718Zok github.com/cockroachdb/cockroach/pkg/sql/opt/optgen/lang 0.037s
110592026-03-10T19:00:49.718Zok github.com/cockroachdb/cockroach/pkg/sql/opt/ordering 0.107s
110602026-03-10T19:00:49.718Zok github.com/cockroachdb/cockroach/pkg/sql/opt/partialidx 0.141s
110612026-03-10T19:00:49.718Zok github.com/cockroachdb/cockroach/pkg/sql/opt/partition 0.097s
110622026-03-10T19:00:49.718Zok github.com/cockroachdb/cockroach/pkg/sql/opt/props 0.497s
110632026-03-10T19:00:49.718Zok github.com/cockroachdb/cockroach/pkg/sql/opt/props/physical 0.098s
110642026-03-10T19:00:49.718Zok github.com/cockroachdb/cockroach/pkg/sql/opt/testutils 0.137s
110652026-03-10T19:00:49.718Zok github.com/cockroachdb/cockroach/pkg/sql/opt/testutils/opttester 0.118s
110662026-03-10T19:00:49.718Zok github.com/cockroachdb/cockroach/pkg/sql/opt/testutils/testcat 0.114s
110672026-03-10T19:00:49.718Zok github.com/cockroachdb/cockroach/pkg/sql/opt/xform 2.034s
110682026-03-10T19:00:49.718Zok github.com/cockroachdb/cockroach/pkg/sql/parser 0.388s
110692026-03-10T19:00:49.718Zok github.com/cockroachdb/cockroach/pkg/sql/pgwire 41.618s
110702026-03-10T19:00:49.718Zok github.com/cockroachdb/cockroach/pkg/sql/pgwire/hba 0.059s
110712026-03-10T19:00:49.719Zok github.com/cockroachdb/cockroach/pkg/sql/pgwire/identmap 0.048s
110722026-03-10T19:00:49.719Zok github.com/cockroachdb/cockroach/pkg/sql/pgwire/pgerror 0.051s
110732026-03-10T19:00:49.719Zok github.com/cockroachdb/cockroach/pkg/sql/pgwire/pgwirecancel 1.956s
110742026-03-10T19:00:49.719Zok github.com/cockroachdb/cockroach/pkg/sql/physicalplan 12.549s
110752026-03-10T19:00:49.719Zok github.com/cockroachdb/cockroach/pkg/sql/physicalplan/replicaoracle 0.060s
110762026-03-10T19:00:49.719Zok github.com/cockroachdb/cockroach/pkg/sql/privilege 0.032s
110772026-03-10T19:00:49.719Zok github.com/cockroachdb/cockroach/pkg/sql/protoreflect 0.092s
110782026-03-10T19:00:49.719Zok github.com/cockroachdb/cockroach/pkg/sql/querycache 0.383s
110792026-03-10T19:00:49.719Zok github.com/cockroachdb/cockroach/pkg/sql/randgen 0.759s
110802026-03-10T19:00:49.719Zok github.com/cockroachdb/cockroach/pkg/sql/row 2.912s
110812026-03-10T19:00:49.719Zok github.com/cockroachdb/cockroach/pkg/sql/rowcontainer 0.557s
110822026-03-10T19:00:49.719Zok github.com/cockroachdb/cockroach/pkg/sql/rowenc 0.549s
110832026-03-10T19:00:49.719Zok github.com/cockroachdb/cockroach/pkg/sql/rowenc/keyside 0.204s
110842026-03-10T19:00:49.720Zok github.com/cockroachdb/cockroach/pkg/sql/rowenc/valueside 0.238s
110852026-03-10T19:00:49.720Zok github.com/cockroachdb/cockroach/pkg/sql/rowexec 9.221s
110862026-03-10T19:00:49.720Zok github.com/cockroachdb/cockroach/pkg/sql/rowflow 0.532s
110872026-03-10T19:00:49.720Zok github.com/cockroachdb/cockroach/pkg/sql/scanner 0.002s
110882026-03-10T19:00:49.720Zok github.com/cockroachdb/cockroach/pkg/sql/scheduledlogging 3.444s
110892026-03-10T19:00:49.720Zok github.com/cockroachdb/cockroach/pkg/sql/schemachange 5.294s
110902026-03-10T19:00:49.720Zok github.com/cockroachdb/cockroach/pkg/sql/schemachanger 50.083s
110912026-03-10T19:00:49.720Zok github.com/cockroachdb/cockroach/pkg/sql/schemachanger/corpus 0.046s
110922026-03-10T19:00:49.720Zok github.com/cockroachdb/cockroach/pkg/sql/schemachanger/rel 0.053s
110932026-03-10T19:00:49.720Zok github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scbuild 6.251s
110942026-03-10T19:00:49.720Zok github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scdecomp 1.688s
110952026-03-10T19:00:49.720Zok github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scdeps 0.077s
110962026-03-10T19:00:49.720Zok github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scexec 1.707s
110972026-03-10T19:00:49.720Zok github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scplan 3.098s
110982026-03-10T19:00:49.720Zok github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scplan/internal/opgen 0.053s
110992026-03-10T19:00:49.720Zok github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scplan/internal/rules 0.082s
111002026-03-10T19:00:49.721Zok github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scplan/internal/scgraph 0.096s
111012026-03-10T19:00:49.721Zok github.com/cockroachdb/cockroach/pkg/sql/schemachanger/screl 0.065s
111022026-03-10T19:00:49.721Zok github.com/cockroachdb/cockroach/pkg/sql/sem/builtins 7.608s
111032026-03-10T19:00:49.721Zok github.com/cockroachdb/cockroach/pkg/sql/sem/tree 1.139s
111042026-03-10T19:00:49.721Zok github.com/cockroachdb/cockroach/pkg/sql/sem/tree/eval_test 0.642s
111052026-03-10T19:00:49.721Zok github.com/cockroachdb/cockroach/pkg/sql/sessiondata 0.014s
111062026-03-10T19:00:49.721Zok github.com/cockroachdb/cockroach/pkg/sql/sessioninit 0.584s
111072026-03-10T19:00:49.721Zok github.com/cockroachdb/cockroach/pkg/sql/span 0.675s
111082026-03-10T19:00:49.722Zok github.com/cockroachdb/cockroach/pkg/sql/sqlinstance/instanceprovider 0.029s
111092026-03-10T19:00:49.722Zok github.com/cockroachdb/cockroach/pkg/sql/sqlinstance/instancestorage 1.059s
111102026-03-10T19:00:49.722Zok github.com/cockroachdb/cockroach/pkg/sql/sqlliveness/slinstance 0.092s
111112026-03-10T19:00:49.722Zok github.com/cockroachdb/cockroach/pkg/sql/sqlliveness/slstorage 0.965s
111122026-03-10T19:00:49.722Zok github.com/cockroachdb/cockroach/pkg/sql/sqlstats/persistedsqlstats 16.853s
111132026-03-10T19:00:49.722Zok github.com/cockroachdb/cockroach/pkg/sql/sqlstats/persistedsqlstats/sqlstatsutil 0.016s
111142026-03-10T19:00:49.722Zok github.com/cockroachdb/cockroach/pkg/sql/sqlstats/sslocal 2.766s
111152026-03-10T19:00:49.722Zok github.com/cockroachdb/cockroach/pkg/sql/stats 12.596s
111162026-03-10T19:00:49.722Zok github.com/cockroachdb/cockroach/pkg/sql/stmtdiagnostics 3.385s
111172026-03-10T19:00:49.722Zok github.com/cockroachdb/cockroach/pkg/sql/tests 32.919s
111182026-03-10T19:00:49.722Zok github.com/cockroachdb/cockroach/pkg/sql/ttl/ttljob 13.336s
111192026-03-10T19:00:49.722Zok github.com/cockroachdb/cockroach/pkg/sql/types 0.095s
111202026-03-10T19:00:49.722Zok github.com/cockroachdb/cockroach/pkg/startupmigrations 1.126s
111212026-03-10T19:00:49.722Zok github.com/cockroachdb/cockroach/pkg/startupmigrations/leasemanager 0.808s
111222026-03-10T19:00:49.722Zok github.com/cockroachdb/cockroach/pkg/storage 4.749s
111232026-03-10T19:00:49.722Zok github.com/cockroachdb/cockroach/pkg/storage/enginepb 0.009s
111242026-03-10T19:00:49.722Zok github.com/cockroachdb/cockroach/pkg/storage/fs 0.015s
111252026-03-10T19:00:49.722Zok github.com/cockroachdb/cockroach/pkg/storage/metamorphic 38.710s
111262026-03-10T19:00:49.722Zok github.com/cockroachdb/cockroach/pkg/testutils 0.157s
111272026-03-10T19:00:49.722Zok github.com/cockroachdb/cockroach/pkg/testutils/floatcmp 0.002s
111282026-03-10T19:00:49.722Zok github.com/cockroachdb/cockroach/pkg/testutils/keysutils 0.027s
111292026-03-10T19:00:49.722Zok github.com/cockroachdb/cockroach/pkg/testutils/lint/passes/errwrap 0.711s
111302026-03-10T19:00:49.722Zok github.com/cockroachdb/cockroach/pkg/testutils/lint/passes/fmtsafe 0.586s
111312026-03-10T19:00:49.722Zok github.com/cockroachdb/cockroach/pkg/testutils/lint/passes/forbiddenmethod 0.115s
111322026-03-10T19:00:49.722Zok github.com/cockroachdb/cockroach/pkg/testutils/lint/passes/hash 0.495s
111332026-03-10T19:00:49.722Zok github.com/cockroachdb/cockroach/pkg/testutils/lint/passes/leaktestcall 0.530s
111342026-03-10T19:00:49.722Zok github.com/cockroachdb/cockroach/pkg/testutils/lint/passes/nilness 0.142s
111352026-03-10T19:00:49.722Zok github.com/cockroachdb/cockroach/pkg/testutils/lint/passes/nocopy 0.106s
111362026-03-10T19:00:49.722Zok github.com/cockroachdb/cockroach/pkg/testutils/lint/passes/passesutil 0.096s
111372026-03-10T19:00:49.722Zok github.com/cockroachdb/cockroach/pkg/testutils/lint/passes/returnerrcheck 0.393s
111382026-03-10T19:00:49.722Zok github.com/cockroachdb/cockroach/pkg/testutils/lint/passes/timer 0.471s
111392026-03-10T19:00:49.728Zok github.com/cockroachdb/cockroach/pkg/testutils/lint/passes/unconvert 0.090s
111402026-03-10T19:00:49.728Zok github.com/cockroachdb/cockroach/pkg/testutils/sqlutils 1.025s
111412026-03-10T19:00:49.728Zok github.com/cockroachdb/cockroach/pkg/testutils/testcluster 16.634s
111422026-03-10T19:00:49.728Zok github.com/cockroachdb/cockroach/pkg/testutils/zerofields 0.017s
111432026-03-10T19:00:49.730Zok github.com/cockroachdb/cockroach/pkg/ts 9.229s
111442026-03-10T19:00:49.730Zok github.com/cockroachdb/cockroach/pkg/ts/testmodel 0.047s
111452026-03-10T19:00:49.730Zok github.com/cockroachdb/cockroach/pkg/util 0.670s
111462026-03-10T19:00:49.730Zok github.com/cockroachdb/cockroach/pkg/util/admission 4.555s
111472026-03-10T19:00:49.730Zok github.com/cockroachdb/cockroach/pkg/util/binfetcher 0.014s
111482026-03-10T19:00:49.730Zok github.com/cockroachdb/cockroach/pkg/util/bitarray 0.024s
111492026-03-10T19:00:49.731Zok github.com/cockroachdb/cockroach/pkg/util/buildutil 0.037s
111502026-03-10T19:00:49.731Zok github.com/cockroachdb/cockroach/pkg/util/cache 0.034s
111512026-03-10T19:00:49.731Zok github.com/cockroachdb/cockroach/pkg/util/caller 0.065s
111522026-03-10T19:00:49.731Zok github.com/cockroachdb/cockroach/pkg/util/cgroups 0.064s
111532026-03-10T19:00:49.731Zok github.com/cockroachdb/cockroach/pkg/util/circuit 0.140s
111542026-03-10T19:00:49.731Zok github.com/cockroachdb/cockroach/pkg/util/cloudinfo 0.079s
111552026-03-10T19:00:49.731Zok github.com/cockroachdb/cockroach/pkg/util/contextutil 0.063s
111562026-03-10T19:00:49.731Zok github.com/cockroachdb/cockroach/pkg/util/ctxgroup 0.079s
111572026-03-10T19:00:49.732Zok github.com/cockroachdb/cockroach/pkg/util/duration 0.056s
111582026-03-10T19:00:49.732Zok github.com/cockroachdb/cockroach/pkg/util/encoding 0.665s
111592026-03-10T19:00:49.732Zok github.com/cockroachdb/cockroach/pkg/util/encoding/csv 0.062s
111602026-03-10T19:00:49.732Zok github.com/cockroachdb/cockroach/pkg/util/envutil 0.026s
111612026-03-10T19:00:49.732Zok github.com/cockroachdb/cockroach/pkg/util/errorutil 0.031s
111622026-03-10T19:00:49.732Zok github.com/cockroachdb/cockroach/pkg/util/errorutil/unimplemented 0.031s
111632026-03-10T19:00:49.732Zok github.com/cockroachdb/cockroach/pkg/util/flagutil 0.026s
111642026-03-10T19:00:49.733Zok github.com/cockroachdb/cockroach/pkg/util/fsm 0.020s
111652026-03-10T19:00:49.733Zok github.com/cockroachdb/cockroach/pkg/util/fuzzystrmatch 0.044s
111662026-03-10T19:00:49.733Zok github.com/cockroachdb/cockroach/pkg/util/goschedstats 0.055s
111672026-03-10T19:00:49.734Zok github.com/cockroachdb/cockroach/pkg/util/grpcutil 0.114s
111682026-03-10T19:00:49.734Zok github.com/cockroachdb/cockroach/pkg/util/hlc 0.080s
111692026-03-10T19:00:49.734Zok github.com/cockroachdb/cockroach/pkg/util/httputil 0.039s
111702026-03-10T19:00:49.734Zok github.com/cockroachdb/cockroach/pkg/util/humanizeutil 0.039s
111712026-03-10T19:00:49.734Zok github.com/cockroachdb/cockroach/pkg/util/interval 1.827s
111722026-03-10T19:00:49.734Zok github.com/cockroachdb/cockroach/pkg/util/interval/generic 9.897s
111732026-03-10T19:00:49.734Zok github.com/cockroachdb/cockroach/pkg/util/ipaddr 0.046s
111742026-03-10T19:00:49.734Zok github.com/cockroachdb/cockroach/pkg/util/json 0.558s
111752026-03-10T19:00:49.734Zok github.com/cockroachdb/cockroach/pkg/util/limit 0.706s
111762026-03-10T19:00:49.734Zok github.com/cockroachdb/cockroach/pkg/util/log 3.513s
111772026-03-10T19:00:49.735Zok github.com/cockroachdb/cockroach/pkg/util/log/eventpb 0.052s
111782026-03-10T19:00:49.735Zok github.com/cockroachdb/cockroach/pkg/util/log/logconfig 0.086s
111792026-03-10T19:00:49.735Zok github.com/cockroachdb/cockroach/pkg/util/log/logcrash 0.586s
111802026-03-10T19:00:49.735Zok github.com/cockroachdb/cockroach/pkg/util/log/testshout 0.093s
111812026-03-10T19:00:49.735Zok github.com/cockroachdb/cockroach/pkg/util/metric 0.051s
111822026-03-10T19:00:49.735Zok github.com/cockroachdb/cockroach/pkg/util/metric/aggmetric 0.137s
111832026-03-10T19:00:49.735Zok github.com/cockroachdb/cockroach/pkg/util/mon 3.899s
111842026-03-10T19:00:49.735Zok github.com/cockroachdb/cockroach/pkg/util/netutil 0.063s
111852026-03-10T19:00:49.736Zok github.com/cockroachdb/cockroach/pkg/util/netutil/addr 0.094s
111862026-03-10T19:00:49.737Zok github.com/cockroachdb/cockroach/pkg/util/optional 0.031s
111872026-03-10T19:00:49.737Zok github.com/cockroachdb/cockroach/pkg/util/pretty 0.018s
111882026-03-10T19:00:49.737Zok github.com/cockroachdb/cockroach/pkg/util/protoutil 0.064s
111892026-03-10T19:00:49.738Zok github.com/cockroachdb/cockroach/pkg/util/quotapool 0.297s
111902026-03-10T19:00:49.738Zok github.com/cockroachdb/cockroach/pkg/util/randutil 0.009s
111912026-03-10T19:00:49.738Zok github.com/cockroachdb/cockroach/pkg/util/retry 0.018s
111922026-03-10T19:00:49.738Zok github.com/cockroachdb/cockroach/pkg/util/ring 0.042s
111932026-03-10T19:00:49.738Zok github.com/cockroachdb/cockroach/pkg/util/sdnotify 0.008s
111942026-03-10T19:00:49.739Zok github.com/cockroachdb/cockroach/pkg/util/search 0.049s
111952026-03-10T19:00:49.739Zok github.com/cockroachdb/cockroach/pkg/util/shuffle 0.032s
111962026-03-10T19:00:49.739Zok github.com/cockroachdb/cockroach/pkg/util/span 0.079s
111972026-03-10T19:00:49.739Zok github.com/cockroachdb/cockroach/pkg/util/startup 0.065s
111982026-03-10T19:00:49.740Zok github.com/cockroachdb/cockroach/pkg/util/stop 0.644s
111992026-03-10T19:00:49.740Zok github.com/cockroachdb/cockroach/pkg/util/stringarena 0.015s [no tests to run]
112002026-03-10T19:00:49.740Zok github.com/cockroachdb/cockroach/pkg/util/syncutil 0.457s
112012026-03-10T19:00:49.740Zok github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight 0.040s
112022026-03-10T19:00:49.740Zok github.com/cockroachdb/cockroach/pkg/util/sysutil 0.033s
112032026-03-10T19:00:49.740Zok github.com/cockroachdb/cockroach/pkg/util/targz 0.049s
112042026-03-10T19:00:49.740Zok github.com/cockroachdb/cockroach/pkg/util/timeofday 0.042s
112052026-03-10T19:00:49.740Zok github.com/cockroachdb/cockroach/pkg/util/timetz 0.026s
112062026-03-10T19:00:49.740Zok github.com/cockroachdb/cockroach/pkg/util/timeutil 1.657s
112072026-03-10T19:00:49.740Zok github.com/cockroachdb/cockroach/pkg/util/timeutil/pgdate 0.226s
112082026-03-10T19:00:49.741Zok github.com/cockroachdb/cockroach/pkg/util/tracing 0.412s
112092026-03-10T19:00:49.741Zok github.com/cockroachdb/cockroach/pkg/util/tracing/collector 0.418s
112102026-03-10T19:00:49.741Zok github.com/cockroachdb/cockroach/pkg/util/tracing/service 0.016s
112112026-03-10T19:00:49.741Zok github.com/cockroachdb/cockroach/pkg/util/treeprinter 0.001s
112122026-03-10T19:00:49.741Zok github.com/cockroachdb/cockroach/pkg/util/uint128 0.002s
112132026-03-10T19:00:49.741Zok github.com/cockroachdb/cockroach/pkg/util/ulid 1.584s
112142026-03-10T19:00:49.741Zok github.com/cockroachdb/cockroach/pkg/util/unique 0.015s
112152026-03-10T19:00:49.741Zok github.com/cockroachdb/cockroach/pkg/util/uuid 0.006s
112162026-03-10T19:00:49.741Zok github.com/cockroachdb/cockroach/pkg/util/version 0.034s
112172026-03-10T19:00:49.741Zok github.com/cockroachdb/cockroach/pkg/workload 0.032s
112182026-03-10T19:00:49.741Zok github.com/cockroachdb/cockroach/pkg/workload/bank 0.595s
112192026-03-10T19:00:49.741Zok github.com/cockroachdb/cockroach/pkg/workload/cli 0.010s
112202026-03-10T19:00:49.742Zok github.com/cockroachdb/cockroach/pkg/workload/faker 0.002s
112212026-03-10T19:00:49.742Zok github.com/cockroachdb/cockroach/pkg/workload/histogram 0.009s
112222026-03-10T19:00:49.742Zok github.com/cockroachdb/cockroach/pkg/workload/movr 0.019s
112232026-03-10T19:00:49.742Zok github.com/cockroachdb/cockroach/pkg/workload/tpcc 0.017s
112242026-03-10T19:00:49.742Zok github.com/cockroachdb/cockroach/pkg/workload/tpch 0.018s
112252026-03-10T19:00:49.742Zok github.com/cockroachdb/cockroach/pkg/workload/workloadimpl 0.002s
112262026-03-10T19:00:49.742Zok github.com/cockroachdb/cockroach/pkg/workload/workloadsql 2.565s
112272026-03-10T19:00:49.742Zok github.com/cockroachdb/cockroach/pkg/workload/ycsb 7.380s
112282026-03-10T19:00:49.742ZFAIL
112292026-03-10T19:00:51.398Zgmake: *** [Makefile:949: test] Error 1
112302026-03-10T19:00:51.404Zprocess exited: duration 4009732 ms, exit code 2
 
112312026-03-10T19:00:51.411Zfound 0 output files