01KKF6KQKMAT648G4B7GJ54QK1: test-linux

BasicConfig {
    output_rules: [],
    rust_toolchain: None,
    target: Some(
        "ubuntu-22.04-large",
    ),
    access_repos: [],
    publish: [],
    skip_clone: false,
}

Buildomat Job: 01KKF6M3N8CZN7X4WQE8BWDB4B

Tags:

Output:

SEQ GLOBAL TIME DETAILS
12026-03-11T19:40:45.813Zjob assigned to worker 01KKF6M8ZCCA2NS3HXSK2R4KD7 [factory aws, i-0a3079a9162fa192b] (queued for 27 s)
 
22026-03-11T19:40:46.249Zstarting task 0: "setup"
32026-03-11T19:40:46.259Z++ uname -s
42026-03-11T19:40:46.262Z+ kern=Linux
52026-03-11T19:40:46.262Z+ build_user=build
62026-03-11T19:40:46.262Z+ build_uid=12345
72026-03-11T19:40:46.262Z+ work_dir=/work
82026-03-11T19:40:46.262Z+ input_dir=/input
92026-03-11T19:40:46.262Z+ [[ 0 == 12345 ]]
102026-03-11T19:40:46.262Z+ case "$kern" in
112026-03-11T19:40:46.262Z+ apt-get -y update
122026-03-11T19:40:46.338ZHit:1 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy InRelease
132026-03-11T19:40:46.338ZGet:2 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB]
142026-03-11T19:40:46.346ZGet:3 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB]
152026-03-11T19:40:46.394ZGet:4 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/universe amd64 Packages [14.1 MB]
162026-03-11T19:40:46.604ZGet:5 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB]
172026-03-11T19:40:46.622ZGet:6 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/universe Translation-en [5,652 kB]
182026-03-11T19:40:46.715ZGet:7 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/universe amd64 c-n-f Metadata [286 kB]
192026-03-11T19:40:46.721ZGet:8 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/multiverse amd64 Packages [217 kB]
202026-03-11T19:40:46.728ZGet:9 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/multiverse Translation-en [112 kB]
212026-03-11T19:40:46.731ZGet:10 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/multiverse amd64 c-n-f Metadata [8,372 B]
222026-03-11T19:40:46.734ZGet:11 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [3,285 kB]
232026-03-11T19:40:46.772ZGet:12 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [499 kB]
242026-03-11T19:40:46.781ZGet:13 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [19.2 kB]
252026-03-11T19:40:46.784ZGet:14 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [5,414 kB]
262026-03-11T19:40:46.818ZGet:15 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [1,028 kB]
272026-03-11T19:40:46.829ZGet:16 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [676 B]
282026-03-11T19:40:46.832ZGet:17 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,256 kB]
292026-03-11T19:40:46.846ZGet:18 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [314 kB]
302026-03-11T19:40:46.851ZGet:19 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [30.4 kB]
312026-03-11T19:40:46.854ZGet:20 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [59.0 kB]
322026-03-11T19:40:46.857ZGet:21 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [13.5 kB]
332026-03-11T19:40:46.860ZGet:22 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [612 B]
342026-03-11T19:40:46.860ZGet:23 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [69.4 kB]
352026-03-11T19:40:46.862ZGet:24 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.5 kB]
362026-03-11T19:40:46.862ZGet:25 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [412 B]
372026-03-11T19:40:46.866ZGet:26 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-backports/restricted amd64 c-n-f Metadata [116 B]
382026-03-11T19:40:46.866ZGet:27 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [31.7 kB]
392026-03-11T19:40:46.868ZGet:28 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.9 kB]
402026-03-11T19:40:46.868ZGet:29 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B]
412026-03-11T19:40:46.868ZGet:30 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-backports/multiverse amd64 c-n-f Metadata [116 B]
422026-03-11T19:40:47.264ZGet:31 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [3,023 kB]
432026-03-11T19:40:48.311ZGet:32 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [431 kB]
442026-03-11T19:40:48.325ZGet:33 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [14.1 kB]
452026-03-11T19:40:48.328ZGet:34 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [5,251 kB]
462026-03-11T19:40:48.745ZGet:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [1,000 kB]
472026-03-11T19:40:48.860ZGet:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [680 B]
482026-03-11T19:40:48.860ZGet:37 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [1,019 kB]
492026-03-11T19:40:48.891ZGet:38 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [225 kB]
502026-03-11T19:40:48.899ZGet:39 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [22.7 kB]
512026-03-11T19:40:48.902ZGet:40 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [51.9 kB]
522026-03-11T19:40:48.902ZGet:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [10.6 kB]
532026-03-11T19:40:48.902ZGet:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [388 B]
542026-03-11T19:40:53.452ZFetched 43.8 MB in 3s (13.6 MB/s)
552026-03-11T19:40:53.944ZReading package lists...
562026-03-11T19:40:53.954Z+ apt-get -y install sysvbanner build-essential
572026-03-11T19:40:53.980ZReading package lists...
582026-03-11T19:40:54.217ZBuilding dependency tree...
592026-03-11T19:40:54.232ZReading state information...
602026-03-11T19:40:54.232ZThe following additional packages will be installed:
612026-03-11T19:40:54.232Z bzip2 cpp cpp-11 dpkg-dev fakeroot fontconfig-config fonts-dejavu-core g++
622026-03-11T19:40:54.232Z g++-11 gcc gcc-11 gcc-11-base gcc-12-base libalgorithm-diff-perl
632026-03-11T19:40:54.232Z libalgorithm-diff-xs-perl libalgorithm-merge-perl libasan6 libatomic1
642026-03-11T19:40:54.232Z libc-dev-bin libc-devtools libc6 libc6-dev libcc1-0 libcrypt-dev libdeflate0
652026-03-11T19:40:54.232Z libdpkg-perl libfakeroot libfile-fcntllock-perl libfontconfig1 libgcc-11-dev
662026-03-11T19:40:54.232Z libgcc-s1 libgd3 libgomp1 libisl23 libitm1 libjbig0 libjpeg-turbo8 libjpeg8
672026-03-11T19:40:54.232Z liblsan0 libmpc3 libnsl-dev libquadmath0 libstdc++-11-dev libstdc++6
682026-03-11T19:40:54.232Z libtiff5 libtirpc-common libtirpc-dev libtirpc3 libtsan0 libubsan1 libwebp7
692026-03-11T19:40:54.233Z libxpm4 linux-libc-dev lto-disabled-list make manpages-dev rpcsvc-proto
702026-03-11T19:40:54.233ZSuggested packages:
712026-03-11T19:40:54.233Z bzip2-doc cpp-doc gcc-11-locales debian-keyring g++-multilib g++-11-multilib
722026-03-11T19:40:54.233Z gcc-11-doc gcc-multilib autoconf automake libtool flex bison gdb gcc-doc
732026-03-11T19:40:54.233Z gcc-11-multilib glibc-doc bzr libgd-tools libstdc++-11-doc make-doc
742026-03-11T19:40:54.233ZRecommended packages:
752026-03-11T19:40:54.233Z libnss-nis libnss-nisplus
762026-03-11T19:40:54.257ZThe following NEW packages will be installed:
772026-03-11T19:40:54.262Z build-essential bzip2 cpp cpp-11 dpkg-dev fakeroot fontconfig-config
782026-03-11T19:40:54.262Z fonts-dejavu-core g++ g++-11 gcc gcc-11 gcc-11-base libalgorithm-diff-perl
792026-03-11T19:40:54.262Z libalgorithm-diff-xs-perl libalgorithm-merge-perl libasan6 libatomic1
802026-03-11T19:40:54.262Z libc-dev-bin libc-devtools libc6-dev libcc1-0 libcrypt-dev libdeflate0
812026-03-11T19:40:54.262Z libdpkg-perl libfakeroot libfile-fcntllock-perl libfontconfig1 libgcc-11-dev
822026-03-11T19:40:54.262Z libgd3 libgomp1 libisl23 libitm1 libjbig0 libjpeg-turbo8 libjpeg8 liblsan0
832026-03-11T19:40:54.262Z libmpc3 libnsl-dev libquadmath0 libstdc++-11-dev libtiff5 libtirpc-dev
842026-03-11T19:40:54.262Z libtsan0 libubsan1 libwebp7 libxpm4 linux-libc-dev lto-disabled-list make
852026-03-11T19:40:54.262Z manpages-dev rpcsvc-proto sysvbanner
862026-03-11T19:40:54.262ZThe following packages will be upgraded:
872026-03-11T19:40:54.262Z gcc-12-base libc6 libgcc-s1 libstdc++6 libtirpc-common libtirpc3
882026-03-11T19:40:54.273Z6 upgraded, 53 newly installed, 0 to remove and 303 not upgraded.
892026-03-11T19:40:54.299ZNeed to get 68.3 MB of archives.
902026-03-11T19:40:54.299ZAfter this operation, 208 MB of additional disk space will be used.
912026-03-11T19:40:54.299ZGet:1 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 gcc-12-base amd64 12.3.0-1ubuntu1~22.04.3 [216 kB]
922026-03-11T19:40:54.313ZGet:2 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libstdc++6 amd64 12.3.0-1ubuntu1~22.04.3 [699 kB]
932026-03-11T19:40:54.333ZGet:3 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libgcc-s1 amd64 12.3.0-1ubuntu1~22.04.3 [53.9 kB]
942026-03-11T19:40:54.339ZGet:4 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libc6 amd64 2.35-0ubuntu3.13 [3,235 kB]
952026-03-11T19:40:54.370ZGet:5 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libtirpc-common all 1.3.2-2ubuntu0.1 [7,766 B]
962026-03-11T19:40:54.373ZGet:6 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libtirpc3 amd64 1.3.2-2ubuntu0.1 [82.3 kB]
972026-03-11T19:40:54.373ZGet:7 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libc-dev-bin amd64 2.35-0ubuntu3.13 [20.3 kB]
982026-03-11T19:40:54.373ZGet:8 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-libc-dev amd64 5.15.0-171.181 [1,316 kB]
992026-03-11T19:40:54.379ZGet:9 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 libcrypt-dev amd64 1:4.4.27-1 [112 kB]
1002026-03-11T19:40:54.382ZGet:10 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 rpcsvc-proto amd64 1.4.2-0ubuntu6 [68.5 kB]
1012026-03-11T19:40:54.384ZGet:11 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libtirpc-dev amd64 1.3.2-2ubuntu0.1 [192 kB]
1022026-03-11T19:40:54.387ZGet:12 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 libnsl-dev amd64 1.3.0-2build2 [71.3 kB]
1032026-03-11T19:40:54.390ZGet:13 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libc6-dev amd64 2.35-0ubuntu3.13 [2,101 kB]
1042026-03-11T19:40:54.410ZGet:14 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 gcc-11-base amd64 11.4.0-1ubuntu1~22.04.3 [216 kB]
1052026-03-11T19:40:54.414ZGet:15 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 libisl23 amd64 0.24-2build1 [727 kB]
1062026-03-11T19:40:54.418ZGet:16 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 libmpc3 amd64 1.2.1-2build1 [46.9 kB]
1072026-03-11T19:40:54.421ZGet:17 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 cpp-11 amd64 11.4.0-1ubuntu1~22.04.3 [10.0 MB]
1082026-03-11T19:40:54.505ZGet:18 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 cpp amd64 4:11.2.0-1ubuntu1 [27.7 kB]
1092026-03-11T19:40:54.508ZGet:19 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libcc1-0 amd64 12.3.0-1ubuntu1~22.04.3 [48.3 kB]
1102026-03-11T19:40:54.508ZGet:20 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libgomp1 amd64 12.3.0-1ubuntu1~22.04.3 [127 kB]
1112026-03-11T19:40:54.512ZGet:21 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libitm1 amd64 12.3.0-1ubuntu1~22.04.3 [30.2 kB]
1122026-03-11T19:40:54.515ZGet:22 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libatomic1 amd64 12.3.0-1ubuntu1~22.04.3 [10.5 kB]
1132026-03-11T19:40:54.515ZGet:23 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libasan6 amd64 11.4.0-1ubuntu1~22.04.3 [2,283 kB]
1142026-03-11T19:40:54.544ZGet:24 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 liblsan0 amd64 12.3.0-1ubuntu1~22.04.3 [1,069 kB]
1152026-03-11T19:40:54.553ZGet:25 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libtsan0 amd64 11.4.0-1ubuntu1~22.04.3 [2,260 kB]
1162026-03-11T19:40:54.574ZGet:26 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libubsan1 amd64 12.3.0-1ubuntu1~22.04.3 [976 kB]
1172026-03-11T19:40:54.582ZGet:27 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libquadmath0 amd64 12.3.0-1ubuntu1~22.04.3 [154 kB]
1182026-03-11T19:40:54.585ZGet:28 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libgcc-11-dev amd64 11.4.0-1ubuntu1~22.04.3 [2,517 kB]
1192026-03-11T19:40:54.608ZGet:29 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 gcc-11 amd64 11.4.0-1ubuntu1~22.04.3 [20.1 MB]
1202026-03-11T19:40:54.792ZGet:30 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 gcc amd64 4:11.2.0-1ubuntu1 [5,112 B]
1212026-03-11T19:40:54.792ZGet:31 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libstdc++-11-dev amd64 11.4.0-1ubuntu1~22.04.3 [2,101 kB]
1222026-03-11T19:40:54.805ZGet:32 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 g++-11 amd64 11.4.0-1ubuntu1~22.04.3 [11.4 MB]
1232026-03-11T19:40:54.911ZGet:33 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 g++ amd64 4:11.2.0-1ubuntu1 [1,412 B]
1242026-03-11T19:40:54.913ZGet:34 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 make amd64 4.3-4.1build1 [180 kB]
1252026-03-11T19:40:54.913ZGet:35 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libdpkg-perl all 1.21.1ubuntu2.6 [237 kB]
1262026-03-11T19:40:54.916ZGet:36 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 bzip2 amd64 1.0.8-5build1 [34.8 kB]
1272026-03-11T19:40:54.920ZGet:37 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 lto-disabled-list all 24 [12.5 kB]
1282026-03-11T19:40:54.923ZGet:38 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 dpkg-dev all 1.21.1ubuntu2.6 [922 kB]
1292026-03-11T19:40:54.935ZGet:39 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 build-essential amd64 12.9ubuntu3 [4,744 B]
1302026-03-11T19:40:54.938ZGet:40 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 libfakeroot amd64 1.28-1ubuntu1 [31.5 kB]
1312026-03-11T19:40:54.941ZGet:41 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 fakeroot amd64 1.28-1ubuntu1 [60.4 kB]
1322026-03-11T19:40:54.944ZGet:42 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 fonts-dejavu-core all 2.37-2build1 [1,041 kB]
1332026-03-11T19:40:54.954ZGet:43 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 fontconfig-config all 2.13.1-4.2ubuntu5 [29.1 kB]
1342026-03-11T19:40:54.957ZGet:44 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 libalgorithm-diff-perl all 1.201-1 [41.8 kB]
1352026-03-11T19:40:54.960ZGet:45 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 libalgorithm-diff-xs-perl amd64 0.04-6build3 [11.9 kB]
1362026-03-11T19:40:54.963ZGet:46 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 libalgorithm-merge-perl all 0.08-3 [12.0 kB]
1372026-03-11T19:40:54.963ZGet:47 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 libfontconfig1 amd64 2.13.1-4.2ubuntu5 [131 kB]
1382026-03-11T19:40:54.965ZGet:48 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 libjpeg-turbo8 amd64 2.1.2-0ubuntu1 [134 kB]
1392026-03-11T19:40:54.968ZGet:49 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 libjpeg8 amd64 8c-2ubuntu10 [2,264 B]
1402026-03-11T19:40:54.971ZGet:50 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 libdeflate0 amd64 1.10-2 [70.9 kB]
1412026-03-11T19:40:54.974ZGet:51 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libjbig0 amd64 2.1-3.1ubuntu0.22.04.1 [29.2 kB]
1422026-03-11T19:40:54.977ZGet:52 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libwebp7 amd64 1.2.2-2ubuntu0.22.04.2 [206 kB]
1432026-03-11T19:40:54.981ZGet:53 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libtiff5 amd64 4.3.0-6ubuntu0.12 [185 kB]
1442026-03-11T19:40:54.985ZGet:54 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libxpm4 amd64 1:3.5.12-1ubuntu0.22.04.2 [36.7 kB]
1452026-03-11T19:40:55.006ZGet:55 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libgd3 amd64 2.3.0-2ubuntu2.3 [129 kB]
1462026-03-11T19:40:55.006ZGet:56 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libc-devtools amd64 2.35-0ubuntu3.13 [29.0 kB]
1472026-03-11T19:40:55.006ZGet:57 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 libfile-fcntllock-perl amd64 0.22-3build7 [33.9 kB]
1482026-03-11T19:40:55.006ZGet:58 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 manpages-dev all 5.10-1ubuntu1 [2,309 kB]
1492026-03-11T19:40:55.016ZGet:59 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/universe amd64 sysvbanner amd64 1:1.0-17fakesync1 [7,112 B]
1502026-03-11T19:40:55.145Zdebconf: unable to initialize frontend: Dialog
1512026-03-11T19:40:55.145Zdebconf: (TERM is not set, so the dialog frontend is not usable.)
1522026-03-11T19:40:55.145Zdebconf: falling back to frontend: Readline
1532026-03-11T19:40:55.149Zdebconf: unable to initialize frontend: Readline
1542026-03-11T19:40:55.149Zdebconf: (This frontend requires a controlling tty.)
1552026-03-11T19:40:55.149Zdebconf: falling back to frontend: Teletype
1562026-03-11T19:40:55.152Zdpkg-preconfigure: unable to re-open stdin:
1572026-03-11T19:40:55.166ZFetched 68.3 MB in 1s (93.6 MB/s)
1582026-03-11T19:40:55.435Z(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 63625 files and directories currently installed.)
1592026-03-11T19:40:55.438ZPreparing to unpack .../gcc-12-base_12.3.0-1ubuntu1~22.04.3_amd64.deb ...
1602026-03-11T19:40:55.460ZUnpacking gcc-12-base:amd64 (12.3.0-1ubuntu1~22.04.3) over (12-20220319-1ubuntu1) ...
1612026-03-11T19:40:55.521ZSetting up gcc-12-base:amd64 (12.3.0-1ubuntu1~22.04.3) ...
1622026-03-11T19:40:55.598Z(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 63625 files and directories currently installed.)
1632026-03-11T19:40:55.601ZPreparing to unpack .../libstdc++6_12.3.0-1ubuntu1~22.04.3_amd64.deb ...
1642026-03-11T19:40:55.656ZUnpacking libstdc++6:amd64 (12.3.0-1ubuntu1~22.04.3) over (12-20220319-1ubuntu1) ...
1652026-03-11T19:40:55.720ZSetting up libstdc++6:amd64 (12.3.0-1ubuntu1~22.04.3) ...
1662026-03-11T19:40:55.794Z(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 63625 files and directories currently installed.)
1672026-03-11T19:40:55.798ZPreparing to unpack .../libgcc-s1_12.3.0-1ubuntu1~22.04.3_amd64.deb ...
1682026-03-11T19:40:55.815ZUnpacking libgcc-s1:amd64 (12.3.0-1ubuntu1~22.04.3) over (12-20220319-1ubuntu1) ...
1692026-03-11T19:40:55.879ZSetting up libgcc-s1:amd64 (12.3.0-1ubuntu1~22.04.3) ...
1702026-03-11T19:40:55.964Z(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 63625 files and directories currently installed.)
1712026-03-11T19:40:55.967ZPreparing to unpack .../libc6_2.35-0ubuntu3.13_amd64.deb ...
1722026-03-11T19:40:56.029Zdebconf: unable to initialize frontend: Dialog
1732026-03-11T19:40:56.029Zdebconf: (TERM is not set, so the dialog frontend is not usable.)
1742026-03-11T19:40:56.029Zdebconf: falling back to frontend: Readline
1752026-03-11T19:40:56.076ZUnpacking libc6:amd64 (2.35-0ubuntu3.13) over (2.35-0ubuntu3) ...
1762026-03-11T19:40:56.189ZSetting up libc6:amd64 (2.35-0ubuntu3.13) ...
1772026-03-11T19:40:56.260Zdebconf: unable to initialize frontend: Dialog
1782026-03-11T19:40:56.260Zdebconf: (TERM is not set, so the dialog frontend is not usable.)
1792026-03-11T19:40:56.260Zdebconf: falling back to frontend: Readline
1802026-03-11T19:40:56.488Z(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 63625 files and directories currently installed.)
1812026-03-11T19:40:56.490ZPreparing to unpack .../libtirpc-common_1.3.2-2ubuntu0.1_all.deb ...
1822026-03-11T19:40:56.512ZUnpacking libtirpc-common (1.3.2-2ubuntu0.1) over (1.3.2-2build1) ...
1832026-03-11T19:40:56.577ZSetting up libtirpc-common (1.3.2-2ubuntu0.1) ...
1842026-03-11T19:40:56.659Z(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 63625 files and directories currently installed.)
1852026-03-11T19:40:56.661ZPreparing to unpack .../libtirpc3_1.3.2-2ubuntu0.1_amd64.deb ...
1862026-03-11T19:40:56.688ZUnpacking libtirpc3:amd64 (1.3.2-2ubuntu0.1) over (1.3.2-2build1) ...
1872026-03-11T19:40:56.748ZSetting up libtirpc3:amd64 (1.3.2-2ubuntu0.1) ...
1882026-03-11T19:40:56.801ZSelecting previously unselected package libc-dev-bin.
1892026-03-11T19:40:56.808Z(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 63625 files and directories currently installed.)
1902026-03-11T19:40:56.812ZPreparing to unpack .../00-libc-dev-bin_2.35-0ubuntu3.13_amd64.deb ...
1912026-03-11T19:40:56.818ZUnpacking libc-dev-bin (2.35-0ubuntu3.13) ...
1922026-03-11T19:40:56.857ZSelecting previously unselected package linux-libc-dev:amd64.
1932026-03-11T19:40:56.865ZPreparing to unpack .../01-linux-libc-dev_5.15.0-171.181_amd64.deb ...
1942026-03-11T19:40:56.868ZUnpacking linux-libc-dev:amd64 (5.15.0-171.181) ...
1952026-03-11T19:40:57.001ZSelecting previously unselected package libcrypt-dev:amd64.
1962026-03-11T19:40:57.004ZPreparing to unpack .../02-libcrypt-dev_1%3a4.4.27-1_amd64.deb ...
1972026-03-11T19:40:57.009ZUnpacking libcrypt-dev:amd64 (1:4.4.27-1) ...
1982026-03-11T19:40:57.062ZSelecting previously unselected package rpcsvc-proto.
1992026-03-11T19:40:57.065ZPreparing to unpack .../03-rpcsvc-proto_1.4.2-0ubuntu6_amd64.deb ...
2002026-03-11T19:40:57.070ZUnpacking rpcsvc-proto (1.4.2-0ubuntu6) ...
2012026-03-11T19:40:57.118ZSelecting previously unselected package libtirpc-dev:amd64.
2022026-03-11T19:40:57.121ZPreparing to unpack .../04-libtirpc-dev_1.3.2-2ubuntu0.1_amd64.deb ...
2032026-03-11T19:40:57.129ZUnpacking libtirpc-dev:amd64 (1.3.2-2ubuntu0.1) ...
2042026-03-11T19:40:57.178ZSelecting previously unselected package libnsl-dev:amd64.
2052026-03-11T19:40:57.182ZPreparing to unpack .../05-libnsl-dev_1.3.0-2build2_amd64.deb ...
2062026-03-11T19:40:57.187ZUnpacking libnsl-dev:amd64 (1.3.0-2build2) ...
2072026-03-11T19:40:57.230ZSelecting previously unselected package libc6-dev:amd64.
2082026-03-11T19:40:57.233ZPreparing to unpack .../06-libc6-dev_2.35-0ubuntu3.13_amd64.deb ...
2092026-03-11T19:40:57.238ZUnpacking libc6-dev:amd64 (2.35-0ubuntu3.13) ...
2102026-03-11T19:40:57.376ZSelecting previously unselected package gcc-11-base:amd64.
2112026-03-11T19:40:57.379ZPreparing to unpack .../07-gcc-11-base_11.4.0-1ubuntu1~22.04.3_amd64.deb ...
2122026-03-11T19:40:57.382ZUnpacking gcc-11-base:amd64 (11.4.0-1ubuntu1~22.04.3) ...
2132026-03-11T19:40:57.399ZSelecting previously unselected package libisl23:amd64.
2142026-03-11T19:40:57.402ZPreparing to unpack .../08-libisl23_0.24-2build1_amd64.deb ...
2152026-03-11T19:40:57.405ZUnpacking libisl23:amd64 (0.24-2build1) ...
2162026-03-11T19:40:57.427ZSelecting previously unselected package libmpc3:amd64.
2172026-03-11T19:40:57.430ZPreparing to unpack .../09-libmpc3_1.2.1-2build1_amd64.deb ...
2182026-03-11T19:40:57.432ZUnpacking libmpc3:amd64 (1.2.1-2build1) ...
2192026-03-11T19:40:57.447ZSelecting previously unselected package cpp-11.
2202026-03-11T19:40:57.449ZPreparing to unpack .../10-cpp-11_11.4.0-1ubuntu1~22.04.3_amd64.deb ...
2212026-03-11T19:40:57.452ZUnpacking cpp-11 (11.4.0-1ubuntu1~22.04.3) ...
2222026-03-11T19:40:57.528ZSelecting previously unselected package cpp.
2232026-03-11T19:40:57.531ZPreparing to unpack .../11-cpp_4%3a11.2.0-1ubuntu1_amd64.deb ...
2242026-03-11T19:40:57.534ZUnpacking cpp (4:11.2.0-1ubuntu1) ...
2252026-03-11T19:40:57.552ZSelecting previously unselected package libcc1-0:amd64.
2262026-03-11T19:40:57.555ZPreparing to unpack .../12-libcc1-0_12.3.0-1ubuntu1~22.04.3_amd64.deb ...
2272026-03-11T19:40:57.557ZUnpacking libcc1-0:amd64 (12.3.0-1ubuntu1~22.04.3) ...
2282026-03-11T19:40:57.574ZSelecting previously unselected package libgomp1:amd64.
2292026-03-11T19:40:57.577ZPreparing to unpack .../13-libgomp1_12.3.0-1ubuntu1~22.04.3_amd64.deb ...
2302026-03-11T19:40:57.579ZUnpacking libgomp1:amd64 (12.3.0-1ubuntu1~22.04.3) ...
2312026-03-11T19:40:57.598ZSelecting previously unselected package libitm1:amd64.
2322026-03-11T19:40:57.600ZPreparing to unpack .../14-libitm1_12.3.0-1ubuntu1~22.04.3_amd64.deb ...
2332026-03-11T19:40:57.603ZUnpacking libitm1:amd64 (12.3.0-1ubuntu1~22.04.3) ...
2342026-03-11T19:40:57.620ZSelecting previously unselected package libatomic1:amd64.
2352026-03-11T19:40:57.623ZPreparing to unpack .../15-libatomic1_12.3.0-1ubuntu1~22.04.3_amd64.deb ...
2362026-03-11T19:40:57.626ZUnpacking libatomic1:amd64 (12.3.0-1ubuntu1~22.04.3) ...
2372026-03-11T19:40:57.643ZSelecting previously unselected package libasan6:amd64.
2382026-03-11T19:40:57.645ZPreparing to unpack .../16-libasan6_11.4.0-1ubuntu1~22.04.3_amd64.deb ...
2392026-03-11T19:40:57.648ZUnpacking libasan6:amd64 (11.4.0-1ubuntu1~22.04.3) ...
2402026-03-11T19:40:57.680ZSelecting previously unselected package liblsan0:amd64.
2412026-03-11T19:40:57.683ZPreparing to unpack .../17-liblsan0_12.3.0-1ubuntu1~22.04.3_amd64.deb ...
2422026-03-11T19:40:57.686ZUnpacking liblsan0:amd64 (12.3.0-1ubuntu1~22.04.3) ...
2432026-03-11T19:40:57.710ZSelecting previously unselected package libtsan0:amd64.
2442026-03-11T19:40:57.713ZPreparing to unpack .../18-libtsan0_11.4.0-1ubuntu1~22.04.3_amd64.deb ...
2452026-03-11T19:40:57.716ZUnpacking libtsan0:amd64 (11.4.0-1ubuntu1~22.04.3) ...
2462026-03-11T19:40:57.785ZSelecting previously unselected package libubsan1:amd64.
2472026-03-11T19:40:57.788ZPreparing to unpack .../19-libubsan1_12.3.0-1ubuntu1~22.04.3_amd64.deb ...
2482026-03-11T19:40:57.793ZUnpacking libubsan1:amd64 (12.3.0-1ubuntu1~22.04.3) ...
2492026-03-11T19:40:57.858ZSelecting previously unselected package libquadmath0:amd64.
2502026-03-11T19:40:57.861ZPreparing to unpack .../20-libquadmath0_12.3.0-1ubuntu1~22.04.3_amd64.deb ...
2512026-03-11T19:40:57.872ZUnpacking libquadmath0:amd64 (12.3.0-1ubuntu1~22.04.3) ...
2522026-03-11T19:40:57.911ZSelecting previously unselected package libgcc-11-dev:amd64.
2532026-03-11T19:40:57.914ZPreparing to unpack .../21-libgcc-11-dev_11.4.0-1ubuntu1~22.04.3_amd64.deb ...
2542026-03-11T19:40:57.918ZUnpacking libgcc-11-dev:amd64 (11.4.0-1ubuntu1~22.04.3) ...
2552026-03-11T19:40:57.995ZSelecting previously unselected package gcc-11.
2562026-03-11T19:40:57.998ZPreparing to unpack .../22-gcc-11_11.4.0-1ubuntu1~22.04.3_amd64.deb ...
2572026-03-11T19:40:58.004ZUnpacking gcc-11 (11.4.0-1ubuntu1~22.04.3) ...
2582026-03-11T19:40:58.191ZSelecting previously unselected package gcc.
2592026-03-11T19:40:58.194ZPreparing to unpack .../23-gcc_4%3a11.2.0-1ubuntu1_amd64.deb ...
2602026-03-11T19:40:58.201ZUnpacking gcc (4:11.2.0-1ubuntu1) ...
2612026-03-11T19:40:58.248ZSelecting previously unselected package libstdc++-11-dev:amd64.
2622026-03-11T19:40:58.251ZPreparing to unpack .../24-libstdc++-11-dev_11.4.0-1ubuntu1~22.04.3_amd64.deb ...
2632026-03-11T19:40:58.256ZUnpacking libstdc++-11-dev:amd64 (11.4.0-1ubuntu1~22.04.3) ...
2642026-03-11T19:40:58.371ZSelecting previously unselected package g++-11.
2652026-03-11T19:40:58.374ZPreparing to unpack .../25-g++-11_11.4.0-1ubuntu1~22.04.3_amd64.deb ...
2662026-03-11T19:40:58.379ZUnpacking g++-11 (11.4.0-1ubuntu1~22.04.3) ...
2672026-03-11T19:40:58.502ZSelecting previously unselected package g++.
2682026-03-11T19:40:58.505ZPreparing to unpack .../26-g++_4%3a11.2.0-1ubuntu1_amd64.deb ...
2692026-03-11T19:40:58.510ZUnpacking g++ (4:11.2.0-1ubuntu1) ...
2702026-03-11T19:40:58.549ZSelecting previously unselected package make.
2712026-03-11T19:40:58.552ZPreparing to unpack .../27-make_4.3-4.1build1_amd64.deb ...
2722026-03-11T19:40:58.561ZUnpacking make (4.3-4.1build1) ...
2732026-03-11T19:40:58.605ZSelecting previously unselected package libdpkg-perl.
2742026-03-11T19:40:58.608ZPreparing to unpack .../28-libdpkg-perl_1.21.1ubuntu2.6_all.deb ...
2752026-03-11T19:40:58.613ZUnpacking libdpkg-perl (1.21.1ubuntu2.6) ...
2762026-03-11T19:40:58.667ZSelecting previously unselected package bzip2.
2772026-03-11T19:40:58.670ZPreparing to unpack .../29-bzip2_1.0.8-5build1_amd64.deb ...
2782026-03-11T19:40:58.676ZUnpacking bzip2 (1.0.8-5build1) ...
2792026-03-11T19:40:58.723ZSelecting previously unselected package lto-disabled-list.
2802026-03-11T19:40:58.726ZPreparing to unpack .../30-lto-disabled-list_24_all.deb ...
2812026-03-11T19:40:58.732ZUnpacking lto-disabled-list (24) ...
2822026-03-11T19:40:58.776ZSelecting previously unselected package dpkg-dev.
2832026-03-11T19:40:58.779ZPreparing to unpack .../31-dpkg-dev_1.21.1ubuntu2.6_all.deb ...
2842026-03-11T19:40:58.787ZUnpacking dpkg-dev (1.21.1ubuntu2.6) ...
2852026-03-11T19:40:58.865ZSelecting previously unselected package build-essential.
2862026-03-11T19:40:58.868ZPreparing to unpack .../32-build-essential_12.9ubuntu3_amd64.deb ...
2872026-03-11T19:40:58.877ZUnpacking build-essential (12.9ubuntu3) ...
2882026-03-11T19:40:58.928ZSelecting previously unselected package libfakeroot:amd64.
2892026-03-11T19:40:58.931ZPreparing to unpack .../33-libfakeroot_1.28-1ubuntu1_amd64.deb ...
2902026-03-11T19:40:58.936ZUnpacking libfakeroot:amd64 (1.28-1ubuntu1) ...
2912026-03-11T19:40:58.989ZSelecting previously unselected package fakeroot.
2922026-03-11T19:40:58.993ZPreparing to unpack .../34-fakeroot_1.28-1ubuntu1_amd64.deb ...
2932026-03-11T19:40:59.002ZUnpacking fakeroot (1.28-1ubuntu1) ...
2942026-03-11T19:40:59.049ZSelecting previously unselected package fonts-dejavu-core.
2952026-03-11T19:40:59.053ZPreparing to unpack .../35-fonts-dejavu-core_2.37-2build1_all.deb ...
2962026-03-11T19:40:59.057ZUnpacking fonts-dejavu-core (2.37-2build1) ...
2972026-03-11T19:40:59.165ZSelecting previously unselected package fontconfig-config.
2982026-03-11T19:40:59.169ZPreparing to unpack .../36-fontconfig-config_2.13.1-4.2ubuntu5_all.deb ...
2992026-03-11T19:40:59.173ZUnpacking fontconfig-config (2.13.1-4.2ubuntu5) ...
3002026-03-11T19:40:59.212ZSelecting previously unselected package libalgorithm-diff-perl.
3012026-03-11T19:40:59.215ZPreparing to unpack .../37-libalgorithm-diff-perl_1.201-1_all.deb ...
3022026-03-11T19:40:59.221ZUnpacking libalgorithm-diff-perl (1.201-1) ...
3032026-03-11T19:40:59.270ZSelecting previously unselected package libalgorithm-diff-xs-perl.
3042026-03-11T19:40:59.272ZPreparing to unpack .../38-libalgorithm-diff-xs-perl_0.04-6build3_amd64.deb ...
3052026-03-11T19:40:59.280ZUnpacking libalgorithm-diff-xs-perl (0.04-6build3) ...
3062026-03-11T19:40:59.320ZSelecting previously unselected package libalgorithm-merge-perl.
3072026-03-11T19:40:59.323ZPreparing to unpack .../39-libalgorithm-merge-perl_0.08-3_all.deb ...
3082026-03-11T19:40:59.327ZUnpacking libalgorithm-merge-perl (0.08-3) ...
3092026-03-11T19:40:59.386ZSelecting previously unselected package libfontconfig1:amd64.
3102026-03-11T19:40:59.391ZPreparing to unpack .../40-libfontconfig1_2.13.1-4.2ubuntu5_amd64.deb ...
3112026-03-11T19:40:59.393ZUnpacking libfontconfig1:amd64 (2.13.1-4.2ubuntu5) ...
3122026-03-11T19:40:59.448ZSelecting previously unselected package libjpeg-turbo8:amd64.
3132026-03-11T19:40:59.451ZPreparing to unpack .../41-libjpeg-turbo8_2.1.2-0ubuntu1_amd64.deb ...
3142026-03-11T19:40:59.456ZUnpacking libjpeg-turbo8:amd64 (2.1.2-0ubuntu1) ...
3152026-03-11T19:40:59.505ZSelecting previously unselected package libjpeg8:amd64.
3162026-03-11T19:40:59.509ZPreparing to unpack .../42-libjpeg8_8c-2ubuntu10_amd64.deb ...
3172026-03-11T19:40:59.513ZUnpacking libjpeg8:amd64 (8c-2ubuntu10) ...
3182026-03-11T19:40:59.569ZSelecting previously unselected package libdeflate0:amd64.
3192026-03-11T19:40:59.572ZPreparing to unpack .../43-libdeflate0_1.10-2_amd64.deb ...
3202026-03-11T19:40:59.578ZUnpacking libdeflate0:amd64 (1.10-2) ...
3212026-03-11T19:40:59.633ZSelecting previously unselected package libjbig0:amd64.
3222026-03-11T19:40:59.636ZPreparing to unpack .../44-libjbig0_2.1-3.1ubuntu0.22.04.1_amd64.deb ...
3232026-03-11T19:40:59.641ZUnpacking libjbig0:amd64 (2.1-3.1ubuntu0.22.04.1) ...
3242026-03-11T19:40:59.694ZSelecting previously unselected package libwebp7:amd64.
3252026-03-11T19:40:59.698ZPreparing to unpack .../45-libwebp7_1.2.2-2ubuntu0.22.04.2_amd64.deb ...
3262026-03-11T19:40:59.702ZUnpacking libwebp7:amd64 (1.2.2-2ubuntu0.22.04.2) ...
3272026-03-11T19:40:59.757ZSelecting previously unselected package libtiff5:amd64.
3282026-03-11T19:40:59.760ZPreparing to unpack .../46-libtiff5_4.3.0-6ubuntu0.12_amd64.deb ...
3292026-03-11T19:40:59.766ZUnpacking libtiff5:amd64 (4.3.0-6ubuntu0.12) ...
3302026-03-11T19:40:59.825ZSelecting previously unselected package libxpm4:amd64.
3312026-03-11T19:40:59.828ZPreparing to unpack .../47-libxpm4_1%3a3.5.12-1ubuntu0.22.04.2_amd64.deb ...
3322026-03-11T19:40:59.834ZUnpacking libxpm4:amd64 (1:3.5.12-1ubuntu0.22.04.2) ...
3332026-03-11T19:40:59.889ZSelecting previously unselected package libgd3:amd64.
3342026-03-11T19:40:59.892ZPreparing to unpack .../48-libgd3_2.3.0-2ubuntu2.3_amd64.deb ...
3352026-03-11T19:40:59.897ZUnpacking libgd3:amd64 (2.3.0-2ubuntu2.3) ...
3362026-03-11T19:40:59.940ZSelecting previously unselected package libc-devtools.
3372026-03-11T19:40:59.942ZPreparing to unpack .../49-libc-devtools_2.35-0ubuntu3.13_amd64.deb ...
3382026-03-11T19:40:59.949ZUnpacking libc-devtools (2.35-0ubuntu3.13) ...
3392026-03-11T19:40:59.992ZSelecting previously unselected package libfile-fcntllock-perl.
3402026-03-11T19:40:59.995ZPreparing to unpack .../50-libfile-fcntllock-perl_0.22-3build7_amd64.deb ...
3412026-03-11T19:41:00.000ZUnpacking libfile-fcntllock-perl (0.22-3build7) ...
3422026-03-11T19:41:00.046ZSelecting previously unselected package manpages-dev.
3432026-03-11T19:41:00.050ZPreparing to unpack .../51-manpages-dev_5.10-1ubuntu1_all.deb ...
3442026-03-11T19:41:00.053ZUnpacking manpages-dev (5.10-1ubuntu1) ...
3452026-03-11T19:41:00.242ZSelecting previously unselected package sysvbanner.
3462026-03-11T19:41:00.245ZPreparing to unpack .../52-sysvbanner_1%3a1.0-17fakesync1_amd64.deb ...
3472026-03-11T19:41:00.254ZUnpacking sysvbanner (1:1.0-17fakesync1) ...
3482026-03-11T19:41:00.307ZSetting up gcc-11-base:amd64 (11.4.0-1ubuntu1~22.04.3) ...
3492026-03-11T19:41:00.331ZSetting up manpages-dev (5.10-1ubuntu1) ...
3502026-03-11T19:41:00.350ZSetting up lto-disabled-list (24) ...
3512026-03-11T19:41:00.368ZSetting up libxpm4:amd64 (1:3.5.12-1ubuntu0.22.04.2) ...
3522026-03-11T19:41:00.388ZSetting up libfile-fcntllock-perl (0.22-3build7) ...
3532026-03-11T19:41:00.405ZSetting up libalgorithm-diff-perl (1.201-1) ...
3542026-03-11T19:41:00.426ZSetting up libdeflate0:amd64 (1.10-2) ...
3552026-03-11T19:41:00.447ZSetting up linux-libc-dev:amd64 (5.15.0-171.181) ...
3562026-03-11T19:41:00.464ZSetting up libgomp1:amd64 (12.3.0-1ubuntu1~22.04.3) ...
3572026-03-11T19:41:00.484ZSetting up bzip2 (1.0.8-5build1) ...
3582026-03-11T19:41:00.502ZSetting up libjbig0:amd64 (2.1-3.1ubuntu0.22.04.1) ...
3592026-03-11T19:41:00.523ZSetting up libfakeroot:amd64 (1.28-1ubuntu1) ...
3602026-03-11T19:41:00.548ZSetting up libasan6:amd64 (11.4.0-1ubuntu1~22.04.3) ...
3612026-03-11T19:41:00.566ZSetting up fakeroot (1.28-1ubuntu1) ...
3622026-03-11T19:41:00.595Zupdate-alternatives: using /usr/bin/fakeroot-sysv to provide /usr/bin/fakeroot (fakeroot) in auto mode
3632026-03-11T19:41:00.614ZSetting up libtirpc-dev:amd64 (1.3.2-2ubuntu0.1) ...
3642026-03-11T19:41:00.633ZSetting up rpcsvc-proto (1.4.2-0ubuntu6) ...
3652026-03-11T19:41:00.652ZSetting up make (4.3-4.1build1) ...
3662026-03-11T19:41:00.670ZSetting up libquadmath0:amd64 (12.3.0-1ubuntu1~22.04.3) ...
3672026-03-11T19:41:00.691ZSetting up libmpc3:amd64 (1.2.1-2build1) ...
3682026-03-11T19:41:00.713ZSetting up libatomic1:amd64 (12.3.0-1ubuntu1~22.04.3) ...
3692026-03-11T19:41:00.731ZSetting up fonts-dejavu-core (2.37-2build1) ...
3702026-03-11T19:41:00.831ZSetting up libjpeg-turbo8:amd64 (2.1.2-0ubuntu1) ...
3712026-03-11T19:41:00.851ZSetting up libdpkg-perl (1.21.1ubuntu2.6) ...
3722026-03-11T19:41:00.870ZSetting up libwebp7:amd64 (1.2.2-2ubuntu0.22.04.2) ...
3732026-03-11T19:41:00.891ZSetting up libubsan1:amd64 (12.3.0-1ubuntu1~22.04.3) ...
3742026-03-11T19:41:00.910ZSetting up libnsl-dev:amd64 (1.3.0-2build2) ...
3752026-03-11T19:41:00.927ZSetting up libcrypt-dev:amd64 (1:4.4.27-1) ...
3762026-03-11T19:41:00.946ZSetting up sysvbanner (1:1.0-17fakesync1) ...
3772026-03-11T19:41:00.966ZSetting up libisl23:amd64 (0.24-2build1) ...
3782026-03-11T19:41:00.984ZSetting up libc-dev-bin (2.35-0ubuntu3.13) ...
3792026-03-11T19:41:01.004ZSetting up libalgorithm-diff-xs-perl (0.04-6build3) ...
3802026-03-11T19:41:01.022ZSetting up libcc1-0:amd64 (12.3.0-1ubuntu1~22.04.3) ...
3812026-03-11T19:41:01.042ZSetting up liblsan0:amd64 (12.3.0-1ubuntu1~22.04.3) ...
3822026-03-11T19:41:01.061ZSetting up libitm1:amd64 (12.3.0-1ubuntu1~22.04.3) ...
3832026-03-11T19:41:01.079ZSetting up libalgorithm-merge-perl (0.08-3) ...
3842026-03-11T19:41:01.099ZSetting up libtsan0:amd64 (11.4.0-1ubuntu1~22.04.3) ...
3852026-03-11T19:41:01.117ZSetting up libjpeg8:amd64 (8c-2ubuntu10) ...
3862026-03-11T19:41:01.138ZSetting up cpp-11 (11.4.0-1ubuntu1~22.04.3) ...
3872026-03-11T19:41:01.158ZSetting up fontconfig-config (2.13.1-4.2ubuntu5) ...
3882026-03-11T19:41:01.431ZSetting up dpkg-dev (1.21.1ubuntu2.6) ...
3892026-03-11T19:41:01.463ZSetting up libgcc-11-dev:amd64 (11.4.0-1ubuntu1~22.04.3) ...
3902026-03-11T19:41:01.485ZSetting up gcc-11 (11.4.0-1ubuntu1~22.04.3) ...
3912026-03-11T19:41:01.503ZSetting up cpp (4:11.2.0-1ubuntu1) ...
3922026-03-11T19:41:01.530ZSetting up libc6-dev:amd64 (2.35-0ubuntu3.13) ...
3932026-03-11T19:41:01.552ZSetting up libtiff5:amd64 (4.3.0-6ubuntu0.12) ...
3942026-03-11T19:41:01.570ZSetting up libfontconfig1:amd64 (2.13.1-4.2ubuntu5) ...
3952026-03-11T19:41:01.590ZSetting up gcc (4:11.2.0-1ubuntu1) ...
3962026-03-11T19:41:01.623ZSetting up libgd3:amd64 (2.3.0-2ubuntu2.3) ...
3972026-03-11T19:41:01.645ZSetting up libstdc++-11-dev:amd64 (11.4.0-1ubuntu1~22.04.3) ...
3982026-03-11T19:41:01.664ZSetting up libc-devtools (2.35-0ubuntu3.13) ...
3992026-03-11T19:41:01.686ZSetting up g++-11 (11.4.0-1ubuntu1~22.04.3) ...
4002026-03-11T19:41:01.706ZSetting up g++ (4:11.2.0-1ubuntu1) ...
4012026-03-11T19:41:01.730Zupdate-alternatives: using /usr/bin/g++ to provide /usr/bin/c++ (c++) in auto mode
4022026-03-11T19:41:01.743ZSetting up build-essential (12.9ubuntu3) ...
4032026-03-11T19:41:01.762ZProcessing triggers for man-db (2.10.2-1) ...
4042026-03-11T19:41:02.159ZProcessing triggers for libc-bin (2.35-0ubuntu3) ...
4052026-03-11T19:41:02.778Z
4062026-03-11T19:41:02.778ZRunning kernel seems to be up-to-date.
4072026-03-11T19:41:02.778Z
4082026-03-11T19:41:02.778ZServices to be restarted:
4092026-03-11T19:41:02.781Z systemctl restart acpid.service
4102026-03-11T19:41:02.784Z systemctl restart buildomat-agent.service
4112026-03-11T19:41:02.787Z systemctl restart chrony.service
4122026-03-11T19:41:02.789Z systemctl restart cron.service
4132026-03-11T19:41:02.794Z systemctl restart irqbalance.service
4142026-03-11T19:41:02.799Z systemctl restart multipathd.service
4152026-03-11T19:41:02.803Z systemctl restart packagekit.service
4162026-03-11T19:41:02.806Z systemctl restart polkit.service
4172026-03-11T19:41:02.809Z systemctl restart rsyslog.service
4182026-03-11T19:41:02.812Z systemctl restart serial-getty@ttyS0.service
4192026-03-11T19:41:02.812Z systemctl restart snapd.service
4202026-03-11T19:41:02.815Z systemctl restart ssh.service
4212026-03-11T19:41:02.817Z systemctl restart systemd-fsckd.service
4222026-03-11T19:41:02.820Z systemctl restart systemd-hostnamed.service
4232026-03-11T19:41:02.820Z systemctl restart systemd-journald.service
4242026-03-11T19:41:02.824Z systemctl restart systemd-networkd.service
4252026-03-11T19:41:02.827Z systemctl restart systemd-resolved.service
4262026-03-11T19:41:02.830Z systemctl restart systemd-timedated.service
4272026-03-11T19:41:02.833Z systemctl restart systemd-udevd.service
4282026-03-11T19:41:02.836Z systemctl restart udisks2.service
4292026-03-11T19:41:02.836Z
4302026-03-11T19:41:02.836ZService restarts being deferred:
4312026-03-11T19:41:02.836Z systemctl restart ModemManager.service
4322026-03-11T19:41:02.836Z /etc/needrestart/restart.d/dbus.service
4332026-03-11T19:41:02.836Z systemctl restart getty@tty1.service
4342026-03-11T19:41:02.836Z systemctl restart networkd-dispatcher.service
4352026-03-11T19:41:02.836Z systemctl restart systemd-logind.service
4362026-03-11T19:41:02.836Z systemctl restart unattended-upgrades.service
4372026-03-11T19:41:02.836Z
4382026-03-11T19:41:02.836ZNo containers need to be restarted.
4392026-03-11T19:41:02.836Z
4402026-03-11T19:41:02.836ZNo user sessions are running outdated binaries.
4412026-03-11T19:41:02.836Z
4422026-03-11T19:41:02.836ZNo VM guests are running outdated hypervisor (qemu) binaries on this host.
4432026-03-11T19:41:05.118Z+ groupadd -g 12345 build
4442026-03-11T19:41:05.141Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build build
4452026-03-11T19:41:05.182Z+ echo 'build ALL=(ALL:ALL) NOPASSWD:ALL'
4462026-03-11T19:41:05.184Z+ chmod 0440 /etc/sudoers.d/build
4472026-03-11T19:41:05.184Z+ cat
4482026-03-11T19:41:05.184Z+ chmod 0755 /bin/pfexec
4492026-03-11T19:41:05.185Z+ cat
4502026-03-11T19:41:05.185Z+ chmod 0755 /bin/ptime
4512026-03-11T19:41:05.185Z+ [[ ! -L /bin ]]
4522026-03-11T19:41:05.185Z+ mkdir -p /work
4532026-03-11T19:41:05.185Z+ mkdir -p /home/build
4542026-03-11T19:41:05.185Z+ chown build:build /home/build /work
4552026-03-11T19:41:05.185Z+ chmod 0700 /home/build /work
4562026-03-11T19:41:05.188Zprocess exited: duration 19101 ms, exit code 0
 
4572026-03-11T19:41:05.193Zstarting task 1: "authentication"
4582026-03-11T19:41:05.249Zprocess exited: duration 5 ms, exit code 0
 
4592026-03-11T19:41:05.255Zstarting task 2: "clone repository"
4602026-03-11T19:41:05.258Z+ mkdir -p /work/oxidecomputer/cockroach
4612026-03-11T19:41:05.258Z+ git clone https://github.com/oxidecomputer/cockroach /work/oxidecomputer/cockroach
4622026-03-11T19:41:05.270ZCloning into '/work/oxidecomputer/cockroach'...
4632026-03-11T19:41:36.497Z+ cd /work/oxidecomputer/cockroach
4642026-03-11T19:41:36.497Z+ git fetch origin 76d728e7724724234d36ada4d402a921657c3724
4652026-03-11T19:41:36.714ZFrom https://github.com/oxidecomputer/cockroach
4662026-03-11T19:41:36.714Z * branch 76d728e7724724234d36ada4d402a921657c3724 -> FETCH_HEAD
4672026-03-11T19:41:36.718Z+ [[ -n iliana/kms ]]
4682026-03-11T19:41:36.721Z++ git branch --show-current
4692026-03-11T19:41:36.721Z+ current=release-22.1-oxide
4702026-03-11T19:41:36.721Z+ [[ release-22.1-oxide != iliana/kms ]]
4712026-03-11T19:41:36.721Z+ git branch -f iliana/kms 76d728e7724724234d36ada4d402a921657c3724
4722026-03-11T19:41:36.721Z+ git checkout -f iliana/kms
4732026-03-11T19:41:37.261ZSwitched to branch 'iliana/kms'
4742026-03-11T19:41:37.266Z+ git reset --hard 76d728e7724724234d36ada4d402a921657c3724
4752026-03-11T19:41:37.285ZHEAD is now at 76d728e772 remove unused KMS abstraction from pkg/cloud
4762026-03-11T19:41:37.288Zprocess exited: duration 32030 ms, exit code 0
 
4772026-03-11T19:41:37.294Zstarting task 3: "build"
4782026-03-11T19:41:37.297Z+ source .github/buildomat/linux-setup.sh
4792026-03-11T19:41:37.297Z++ source .github/buildomat/versions.sh
4802026-03-11T19:41:37.297Z+++ export GOTOOLCHAIN=local
4812026-03-11T19:41:37.297Z+++ GOTOOLCHAIN=local
4822026-03-11T19:41:37.297Z+++ GO_VERSION=1.22.11
4832026-03-11T19:41:37.297Z+++ NODE_VERSION=20.11.1
4842026-03-11T19:41:37.297Z+++ YARN_VERSION=1.22.22
4852026-03-11T19:41:37.297Z++ sudo apt-get install -y --no-install-recommends build-essential autoconf cmake libedit-dev ncurses-dev
4862026-03-11T19:41:37.341ZReading package lists...
4872026-03-11T19:41:37.458ZBuilding dependency tree...
4882026-03-11T19:41:37.462ZReading state information...
4892026-03-11T19:41:37.551Zbuild-essential is already the newest version (12.9ubuntu3).
4902026-03-11T19:41:37.552ZThe following additional packages will be installed:
4912026-03-11T19:41:37.555Z cmake-data dh-elpa-helper emacsen-common libbsd-dev libjsoncpp25 libmd-dev
4922026-03-11T19:41:37.555Z libncurses6 libncursesw6 librhash0 libtinfo6 m4
4932026-03-11T19:41:37.555ZSuggested packages:
4942026-03-11T19:41:37.555Z autoconf-archive gnu-standards autoconf-doc libtool gettext cmake-doc
4952026-03-11T19:41:37.555Z ninja-build cmake-format ncurses-doc m4-doc
4962026-03-11T19:41:37.555ZRecommended packages:
4972026-03-11T19:41:37.555Z automake | automaken
4982026-03-11T19:41:37.580ZThe following NEW packages will be installed:
4992026-03-11T19:41:37.584Z autoconf cmake cmake-data dh-elpa-helper emacsen-common libbsd-dev
5002026-03-11T19:41:37.584Z libedit-dev libjsoncpp25 libmd-dev libncurses-dev librhash0 m4
5012026-03-11T19:41:37.584ZThe following packages will be upgraded:
5022026-03-11T19:41:37.584Z libncurses6 libncursesw6 libtinfo6
5032026-03-11T19:41:37.597Z3 upgraded, 12 newly installed, 0 to remove and 300 not upgraded.
5042026-03-11T19:41:37.623ZNeed to get 8,757 kB of archives.
5052026-03-11T19:41:37.623ZAfter this operation, 38.1 MB of additional disk space will be used.
5062026-03-11T19:41:37.623ZGet:1 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libncursesw6 amd64 6.3-2ubuntu0.1 [147 kB]
5072026-03-11T19:41:37.635ZGet:2 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libncurses6 amd64 6.3-2ubuntu0.1 [111 kB]
5082026-03-11T19:41:37.639ZGet:3 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libtinfo6 amd64 6.3-2ubuntu0.1 [105 kB]
5092026-03-11T19:41:37.643ZGet:4 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 m4 amd64 1.4.18-5ubuntu2 [199 kB]
5102026-03-11T19:41:37.649ZGet:5 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 autoconf all 2.71-2 [338 kB]
5112026-03-11T19:41:37.661ZGet:6 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 libjsoncpp25 amd64 1.9.5-3 [80.0 kB]
5122026-03-11T19:41:37.664ZGet:7 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 librhash0 amd64 1.4.2-1ubuntu1 [125 kB]
5132026-03-11T19:41:37.664ZGet:8 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 dh-elpa-helper all 2.0.9ubuntu1 [7,610 B]
5142026-03-11T19:41:37.667ZGet:9 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 emacsen-common all 3.0.4 [14.9 kB]
5152026-03-11T19:41:37.667ZGet:10 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 cmake-data all 3.22.1-1ubuntu1.22.04.2 [1,913 kB]
5162026-03-11T19:41:37.693ZGet:11 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 cmake amd64 3.22.1-1ubuntu1.22.04.2 [5,010 kB]
5172026-03-11T19:41:37.733ZGet:12 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 libmd-dev amd64 1.0.4-1build1 [40.0 kB]
5182026-03-11T19:41:37.736ZGet:13 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 libbsd-dev amd64 0.11.5-1 [168 kB]
5192026-03-11T19:41:37.739ZGet:14 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libncurses-dev amd64 6.3-2ubuntu0.1 [381 kB]
5202026-03-11T19:41:37.743ZGet:15 http://us-west-2.ec2.archive.ubuntu.com/ubuntu jammy/main amd64 libedit-dev amd64 3.1-20210910-1build1 [118 kB]
5212026-03-11T19:41:37.835Zdebconf: unable to initialize frontend: Dialog
5222026-03-11T19:41:37.835Zdebconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.)
5232026-03-11T19:41:37.836Zdebconf: falling back to frontend: Readline
5242026-03-11T19:41:37.839Zdebconf: unable to initialize frontend: Readline
5252026-03-11T19:41:37.839Zdebconf: (This frontend requires a controlling tty.)
5262026-03-11T19:41:37.839Zdebconf: falling back to frontend: Teletype
5272026-03-11T19:41:37.842Zdpkg-preconfigure: unable to re-open stdin:
5282026-03-11T19:41:37.855ZFetched 8,757 kB in 0s (65.2 MB/s)
5292026-03-11T19:41:37.892Z(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 69365 files and directories currently installed.)
5302026-03-11T19:41:37.896ZPreparing to unpack .../libncursesw6_6.3-2ubuntu0.1_amd64.deb ...
5312026-03-11T19:41:37.915ZUnpacking libncursesw6:amd64 (6.3-2ubuntu0.1) over (6.3-2) ...
5322026-03-11T19:41:37.970ZPreparing to unpack .../libncurses6_6.3-2ubuntu0.1_amd64.deb ...
5332026-03-11T19:41:37.985ZUnpacking libncurses6:amd64 (6.3-2ubuntu0.1) over (6.3-2) ...
5342026-03-11T19:41:38.038ZPreparing to unpack .../libtinfo6_6.3-2ubuntu0.1_amd64.deb ...
5352026-03-11T19:41:38.053ZUnpacking libtinfo6:amd64 (6.3-2ubuntu0.1) over (6.3-2) ...
5362026-03-11T19:41:38.100ZSetting up libtinfo6:amd64 (6.3-2ubuntu0.1) ...
5372026-03-11T19:41:38.140ZSelecting previously unselected package m4.
5382026-03-11T19:41:38.150Z(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 69365 files and directories currently installed.)
5392026-03-11T19:41:38.152ZPreparing to unpack .../00-m4_1.4.18-5ubuntu2_amd64.deb ...
5402026-03-11T19:41:38.156ZUnpacking m4 (1.4.18-5ubuntu2) ...
5412026-03-11T19:41:38.206ZSelecting previously unselected package autoconf.
5422026-03-11T19:41:38.209ZPreparing to unpack .../01-autoconf_2.71-2_all.deb ...
5432026-03-11T19:41:38.213ZUnpacking autoconf (2.71-2) ...
5442026-03-11T19:41:38.265ZSelecting previously unselected package libjsoncpp25:amd64.
5452026-03-11T19:41:38.268ZPreparing to unpack .../02-libjsoncpp25_1.9.5-3_amd64.deb ...
5462026-03-11T19:41:38.273ZUnpacking libjsoncpp25:amd64 (1.9.5-3) ...
5472026-03-11T19:41:38.319ZSelecting previously unselected package librhash0:amd64.
5482026-03-11T19:41:38.322ZPreparing to unpack .../03-librhash0_1.4.2-1ubuntu1_amd64.deb ...
5492026-03-11T19:41:38.326ZUnpacking librhash0:amd64 (1.4.2-1ubuntu1) ...
5502026-03-11T19:41:38.362ZSelecting previously unselected package dh-elpa-helper.
5512026-03-11T19:41:38.365ZPreparing to unpack .../04-dh-elpa-helper_2.0.9ubuntu1_all.deb ...
5522026-03-11T19:41:38.369ZUnpacking dh-elpa-helper (2.0.9ubuntu1) ...
5532026-03-11T19:41:38.413ZSelecting previously unselected package emacsen-common.
5542026-03-11T19:41:38.416ZPreparing to unpack .../05-emacsen-common_3.0.4_all.deb ...
5552026-03-11T19:41:38.425ZUnpacking emacsen-common (3.0.4) ...
5562026-03-11T19:41:38.472ZSelecting previously unselected package cmake-data.
5572026-03-11T19:41:38.475ZPreparing to unpack .../06-cmake-data_3.22.1-1ubuntu1.22.04.2_all.deb ...
5582026-03-11T19:41:38.482ZUnpacking cmake-data (3.22.1-1ubuntu1.22.04.2) ...
5592026-03-11T19:41:38.696ZSelecting previously unselected package cmake.
5602026-03-11T19:41:38.699ZPreparing to unpack .../07-cmake_3.22.1-1ubuntu1.22.04.2_amd64.deb ...
5612026-03-11T19:41:38.704ZUnpacking cmake (3.22.1-1ubuntu1.22.04.2) ...
5622026-03-11T19:41:38.792ZSelecting previously unselected package libmd-dev:amd64.
5632026-03-11T19:41:38.795ZPreparing to unpack .../08-libmd-dev_1.0.4-1build1_amd64.deb ...
5642026-03-11T19:41:38.799ZUnpacking libmd-dev:amd64 (1.0.4-1build1) ...
5652026-03-11T19:41:38.837ZSelecting previously unselected package libbsd-dev:amd64.
5662026-03-11T19:41:38.840ZPreparing to unpack .../09-libbsd-dev_0.11.5-1_amd64.deb ...
5672026-03-11T19:41:38.844ZUnpacking libbsd-dev:amd64 (0.11.5-1) ...
5682026-03-11T19:41:38.890ZSelecting previously unselected package libncurses-dev:amd64.
5692026-03-11T19:41:38.893ZPreparing to unpack .../10-libncurses-dev_6.3-2ubuntu0.1_amd64.deb ...
5702026-03-11T19:41:38.898ZUnpacking libncurses-dev:amd64 (6.3-2ubuntu0.1) ...
5712026-03-11T19:41:38.942ZSelecting previously unselected package libedit-dev:amd64.
5722026-03-11T19:41:38.945ZPreparing to unpack .../11-libedit-dev_3.1-20210910-1build1_amd64.deb ...
5732026-03-11T19:41:38.949ZUnpacking libedit-dev:amd64 (3.1-20210910-1build1) ...
5742026-03-11T19:41:38.993ZSetting up m4 (1.4.18-5ubuntu2) ...
5752026-03-11T19:41:39.010ZSetting up emacsen-common (3.0.4) ...
5762026-03-11T19:41:39.038ZSetting up dh-elpa-helper (2.0.9ubuntu1) ...
5772026-03-11T19:41:39.053ZSetting up libncurses6:amd64 (6.3-2ubuntu0.1) ...
5782026-03-11T19:41:39.069ZSetting up libjsoncpp25:amd64 (1.9.5-3) ...
5792026-03-11T19:41:39.083ZSetting up libncursesw6:amd64 (6.3-2ubuntu0.1) ...
5802026-03-11T19:41:39.098ZSetting up autoconf (2.71-2) ...
5812026-03-11T19:41:39.117ZSetting up librhash0:amd64 (1.4.2-1ubuntu1) ...
5822026-03-11T19:41:39.132ZSetting up libmd-dev:amd64 (1.0.4-1build1) ...
5832026-03-11T19:41:39.146ZSetting up cmake-data (3.22.1-1ubuntu1.22.04.2) ...
5842026-03-11T19:41:39.176ZSetting up libbsd-dev:amd64 (0.11.5-1) ...
5852026-03-11T19:41:39.191ZSetting up libncurses-dev:amd64 (6.3-2ubuntu0.1) ...
5862026-03-11T19:41:39.208ZSetting up libedit-dev:amd64 (3.1-20210910-1build1) ...
5872026-03-11T19:41:39.223ZSetting up cmake (3.22.1-1ubuntu1.22.04.2) ...
5882026-03-11T19:41:39.237ZProcessing triggers for libc-bin (2.35-0ubuntu3) ...
5892026-03-11T19:41:39.255ZProcessing triggers for man-db (2.10.2-1) ...
5902026-03-11T19:41:39.419ZProcessing triggers for install-info (6.8-4build1) ...
5912026-03-11T19:41:39.684Z
5922026-03-11T19:41:39.685ZRunning kernel seems to be up-to-date.
5932026-03-11T19:41:39.685Z
5942026-03-11T19:41:39.685ZServices to be restarted:
5952026-03-11T19:41:39.688Z systemctl restart acpid.service
5962026-03-11T19:41:39.691Z systemctl restart buildomat-agent.service
5972026-03-11T19:41:39.694Z systemctl restart chrony.service
5982026-03-11T19:41:39.694Z systemctl restart cron.service
5992026-03-11T19:41:39.701Z systemctl restart irqbalance.service
6002026-03-11T19:41:39.706Z systemctl restart multipathd.service
6012026-03-11T19:41:39.710Z systemctl restart packagekit.service
6022026-03-11T19:41:39.713Z systemctl restart polkit.service
6032026-03-11T19:41:39.716Z systemctl restart rsyslog.service
6042026-03-11T19:41:39.719Z systemctl restart serial-getty@ttyS0.service
6052026-03-11T19:41:39.719Z systemctl restart snapd.service
6062026-03-11T19:41:39.722Z systemctl restart ssh.service
6072026-03-11T19:41:39.725Z systemctl restart systemd-journald.service
6082026-03-11T19:41:39.728Z systemctl restart systemd-networkd.service
6092026-03-11T19:41:39.731Z systemctl restart systemd-resolved.service
6102026-03-11T19:41:39.734Z systemctl restart systemd-udevd.service
6112026-03-11T19:41:39.737Z systemctl restart udisks2.service
6122026-03-11T19:41:39.740Z
6132026-03-11T19:41:39.740ZService restarts being deferred:
6142026-03-11T19:41:39.740Z systemctl restart ModemManager.service
6152026-03-11T19:41:39.740Z /etc/needrestart/restart.d/dbus.service
6162026-03-11T19:41:39.740Z systemctl restart getty@tty1.service
6172026-03-11T19:41:39.740Z systemctl restart networkd-dispatcher.service
6182026-03-11T19:41:39.740Z systemctl restart systemd-logind.service
6192026-03-11T19:41:39.740Z systemctl restart unattended-upgrades.service
6202026-03-11T19:41:39.740Z
6212026-03-11T19:41:39.740ZNo containers need to be restarted.
6222026-03-11T19:41:39.740Z
6232026-03-11T19:41:39.740ZNo user sessions are running outdated binaries.
6242026-03-11T19:41:39.740Z
6252026-03-11T19:41:39.740ZNo VM guests are running outdated hypervisor (qemu) binaries on this host.
6262026-03-11T19:41:42.025Z++ pushd /work
6272026-03-11T19:41:42.025Z/work /work/oxidecomputer/cockroach
6282026-03-11T19:41:42.025Z++ curl -sSfL --retry 10 -O https://go.dev/dl/go1.22.11.linux-amd64.tar.gz
6292026-03-11T19:41:42.864Z++ curl -sSfL --retry 10 -O https://nodejs.org/dist/v20.11.1/node-v20.11.1-linux-x64.tar.xz
6302026-03-11T19:41:43.077Z++ curl -sSfL --retry 10 -O https://github.com/yarnpkg/yarn/releases/download/v1.22.22/yarn-1.22.22.js
6312026-03-11T19:41:43.445Z++ sha256sum -c /work/oxidecomputer/cockroach/.github/buildomat/SHA256SUMS.linux
6322026-03-11T19:41:43.568Zgo1.22.11.linux-amd64.tar.gz: OK
6332026-03-11T19:41:43.613Znode-v20.11.1-linux-x64.tar.xz: OK
6342026-03-11T19:41:43.622Zyarn-1.22.22.js: OK
6352026-03-11T19:41:43.625Z++ tar xf go1.22.11.linux-amd64.tar.gz
6362026-03-11T19:41:44.560Z++ tar xf node-v20.11.1-linux-x64.tar.xz
6372026-03-11T19:41:45.725Z++ mv yarn-1.22.22.js node-v20.11.1-linux-x64/bin/yarn
6382026-03-11T19:41:45.728Z++ chmod a+x node-v20.11.1-linux-x64/bin/yarn
6392026-03-11T19:41:45.728Z++ popd
6402026-03-11T19:41:45.728Z++ export PATH=/work/go/bin:/work/node-v20.11.1-linux-x64/bin:/home/build/.cargo/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/ooce/bin:/opt/ooce/sbin
6412026-03-11T19:41:45.728Z/work/oxidecomputer/cockroach
6422026-03-11T19:41:45.729Z++ PATH=/work/go/bin:/work/node-v20.11.1-linux-x64/bin:/home/build/.cargo/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/ooce/bin:/opt/ooce/sbin
6432026-03-11T19:41:45.729Z++ nproc
6442026-03-11T19:41:45.732Z+ gmake -j16 test
6452026-03-11T19:41:45.735ZWarning: 'ccache' not found, consider installing it for faster builds
6462026-03-11T19:41:45.738ZWarning: 'bazel' not found (`brew install bazelisk` for macs)
6472026-03-11T19:41:45.784ZDetected change in build system. Rebooting make.
6482026-03-11T19:41:45.795ZWarning: 'ccache' not found, consider installing it for faster builds
6492026-03-11T19:41:45.798ZWarning: 'bazel' not found (`brew install bazelisk` for macs)
6502026-03-11T19:41:45.846Zgitdir=$(git rev-parse --git-dir 2>/dev/null || true); \
6512026-03-11T19:41:45.846Zif test -n "$gitdir"; then \
6522026-03-11T19:41:45.846Z git submodule update --init --recursive; \
6532026-03-11T19:41:45.846Zfi
6542026-03-11T19:41:45.860ZSubmodule 'c-deps/geos' (https://github.com/oxidecomputer/geos.git) registered for path 'c-deps/geos'
6552026-03-11T19:41:45.863ZSubmodule 'c-deps/jemalloc' (https://github.com/cockroachdb/jemalloc.git) registered for path 'c-deps/jemalloc'
6562026-03-11T19:41:45.863ZSubmodule 'c-deps/proj' (https://github.com/oxidecomputer/PROJ.git) registered for path 'c-deps/proj'
6572026-03-11T19:41:45.866Zmkdir -p pkg/sql/parser/gen
6582026-03-11T19:41:45.866ZCloning into '/work/oxidecomputer/cockroach/c-deps/geos'...
6592026-03-11T19:41:45.869Zset -euo pipefail; \
6602026-03-11T19:41:45.869Zawk '/func.*sqlSymUnion/ {print $(NF - 1)}' pkg/sql/parser/sql.y | \
6612026-03-11T19:41:45.869Zsed -e 's/[]\/$*.^|[]/\\&/g' | \
6622026-03-11T19:41:45.869Zsed -e "s/^/s_(type|token) <(/" | \
6632026-03-11T19:41:45.870Zawk '{print $0")>_\\1 <union> /* <\\2> */_"}' > pkg/sql/parser/gen/types_regex.tmp; \
6642026-03-11T19:41:45.870Zsed -E -f pkg/sql/parser/gen/types_regex.tmp < pkg/sql/parser/sql.y | \
6652026-03-11T19:41:45.870Zawk -f pkg/sql/parser/replace_help_rules.awk | \
6662026-03-11T19:41:45.870Zsed -Ee 's,//.*$,,g;s,/[*]([^*]|[*][^/])*[*]/, ,g;s/ +$//g' > pkg/sql/parser/gen/sql-gen.y.tmp || rm pkg/sql/parser/gen/sql-gen.y.tmp
6672026-03-11T19:41:45.938Zmv -f pkg/sql/parser/gen/sql-gen.y.tmp pkg/sql/parser/gen/sql-gen.y
6682026-03-11T19:41:45.941Zrm pkg/sql/parser/gen/types_regex.tmp
6692026-03-11T19:41:47.682ZCloning into '/work/oxidecomputer/cockroach/c-deps/jemalloc'...
6702026-03-11T19:41:48.511ZCloning into '/work/oxidecomputer/cockroach/c-deps/proj'...
6712026-03-11T19:41:55.011ZSubmodule path 'c-deps/geos': checked out '8bfb91ff2cfee16b29316894fd65d0f2a8a09607'
6722026-03-11T19:41:55.029ZSubmodule path 'c-deps/jemalloc': checked out 'fd00c883c119539844b3ffd5ad8640968e0cb1a3'
6732026-03-11T19:41:55.152ZSubmodule path 'c-deps/proj': checked out '47f32de4d4b683af4891844d6a6aefce5801a082'
6742026-03-11T19:41:55.155Zmkdir -p bin
6752026-03-11T19:41:55.158Ztouch bin/.submodules-initialized
6762026-03-11T19:41:55.158Zcd /work/oxidecomputer/cockroach/c-deps/jemalloc && autoconf
6772026-03-11T19:41:55.158Zrm -rf /work/oxidecomputer/cockroach/lib/build/geos
6782026-03-11T19:41:55.158Zrm -rf /work/oxidecomputer/cockroach/lib/build/proj
6792026-03-11T19:41:55.158Zmkdir -p /work/oxidecomputer/cockroach/lib/build/geos
6802026-03-11T19:41:55.158Zmkdir -p /work/oxidecomputer/cockroach/lib/build/proj
6812026-03-11T19:41:55.161Zcd /work/oxidecomputer/cockroach/lib/build/proj && cmake -DCMAKE_TARGET_MESSAGES=OFF /work/oxidecomputer/cockroach/c-deps/proj -DCMAKE_BUILD_TYPE=Release -DBUILD_LIBPROJ_SHARED=OFF
6822026-03-11T19:41:55.162Zcd /work/oxidecomputer/cockroach/lib/build/geos && \
6832026-03-11T19:41:55.162Z cmake -DCMAKE_TARGET_MESSAGES=OFF /work/oxidecomputer/cockroach/c-deps/geos -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_FLAGS=-fPIC -DCMAKE_CXX_FLAGS=-fPIC
6842026-03-11T19:41:55.164Z-- GEOS: Build type: Release
6852026-03-11T19:41:55.164Z-- GEOS: Run-time output: /work/oxidecomputer/cockroach/lib/build/geos/bin
6862026-03-11T19:41:55.164Z-- GEOS: Archives output: /work/oxidecomputer/cockroach/lib/build/geos/lib
6872026-03-11T19:41:55.201Z-- The C compiler identification is GNU 11.4.0
6882026-03-11T19:41:55.204Z-- The C compiler identification is GNU 11.4.0
6892026-03-11T19:41:55.207Z-- Detecting C compiler ABI info
6902026-03-11T19:41:55.227Z-- The CXX compiler identification is GNU 11.4.0
6912026-03-11T19:41:55.230Z-- Detecting C compiler ABI info
6922026-03-11T19:41:55.235Z-- Detecting C compiler ABI info - done
6932026-03-11T19:41:55.239Z-- Check for working C compiler: /usr/bin/cc - skipped
6942026-03-11T19:41:55.242Z-- Detecting C compile features
6952026-03-11T19:41:55.242Z-- Detecting C compile features - done
6962026-03-11T19:41:55.242Z--
6972026-03-11T19:41:55.242Z-- Configuring PROJ:
6982026-03-11T19:41:55.242Z--
6992026-03-11T19:41:55.242Z-- PROJ_VERSION = 4.9.3
7002026-03-11T19:41:55.242Z-- Looking for include file dlfcn.h
7012026-03-11T19:41:55.258Z-- Detecting C compiler ABI info - done
7022026-03-11T19:41:55.261Z-- Check for working C compiler: /usr/bin/cc - skipped
7032026-03-11T19:41:55.265Z-- Detecting C compile features
7042026-03-11T19:41:55.265Z-- Detecting C compile features - done
7052026-03-11T19:41:55.265Z-- Detecting CXX compiler ABI info
7062026-03-11T19:41:55.271Z-- Looking for include file dlfcn.h - found
7072026-03-11T19:41:55.274Z-- Looking for include file inttypes.h
7082026-03-11T19:41:55.296Z-- Detecting CXX compiler ABI info - done
7092026-03-11T19:41:55.299Z-- Looking for include file inttypes.h - found
7102026-03-11T19:41:55.302Z-- Looking for include file jni.h
7112026-03-11T19:41:55.302Z-- Check for working CXX compiler: /usr/bin/c++ - skipped
7122026-03-11T19:41:55.302Z-- Detecting CXX compile features
7132026-03-11T19:41:55.302Z-- Detecting CXX compile features - done
7142026-03-11T19:41:55.302Z-- GEOS: Version 3.8.1
7152026-03-11T19:41:55.302Z-- GEOS: C API Version 1.13.3
7162026-03-11T19:41:55.302Z-- GEOS: JTS port 1.13.0
7172026-03-11T19:41:55.302Z-- GEOS: Require C++11
7182026-03-11T19:41:55.302Zrm -rf /work/oxidecomputer/cockroach/lib/build/jemalloc
7192026-03-11T19:41:55.302Zmkdir -p /work/oxidecomputer/cockroach/lib/build/jemalloc
7202026-03-11T19:41:55.306Zcd /work/oxidecomputer/cockroach/lib/build/jemalloc && /work/oxidecomputer/cockroach/c-deps/jemalloc/configure --enable-prof
7212026-03-11T19:41:55.310Z-- Looking for include file jni.h - not found
7222026-03-11T19:41:55.313Z-- Looking for include file memory.h
7232026-03-11T19:41:55.336Z-- GEOS: Configured 'dist' target
7242026-03-11T19:41:55.339Z-- GEOS: Configured 'distcheck' target
7252026-03-11T19:41:55.339Z-- Configuring done
7262026-03-11T19:41:55.339Z-- Looking for include file memory.h - found
7272026-03-11T19:41:55.339Z-- Looking for include file stdint.h
7282026-03-11T19:41:55.344Zchecking for xsltproc... false
7292026-03-11T19:41:55.344Zchecking for gcc... gcc
7302026-03-11T19:41:55.366Zchecking whether the C compiler works... -- Looking for include file stdint.h - found
7312026-03-11T19:41:55.369Z-- Looking for include file stdlib.h
7322026-03-11T19:41:55.369Zyes
7332026-03-11T19:41:55.369Zchecking for C compiler default output file name... a.out
7342026-03-11T19:41:55.377Zchecking for suffix of executables... -- Generating done
7352026-03-11T19:41:55.380Z-- Build files have been written to: /work/oxidecomputer/cockroach/lib/build/geos
7362026-03-11T19:41:55.383Z
7372026-03-11T19:41:55.386Zchecking whether we are cross compiling... mkdir /work/oxidecomputer/cockroach/lib/build/geos/capi/geos
7382026-03-11T19:41:55.386Zcp /work/oxidecomputer/cockroach/c-deps/geos/include/geos/export.h /work/oxidecomputer/cockroach/lib/build/geos/capi/geos
7392026-03-11T19:41:55.399Z-- Looking for include file stdlib.h - found
7402026-03-11T19:41:55.402Zno
7412026-03-11T19:41:55.402Z-- Looking for include file string.h
7422026-03-11T19:41:55.406Zchecking for suffix of object files... o
7432026-03-11T19:41:55.413Zchecking whether the compiler supports GNU C... yes
7442026-03-11T19:41:55.420Zchecking whether gcc accepts -g... yes
7452026-03-11T19:41:55.427Zchecking for gcc option to enable C11 features... -- Looking for include file string.h - found
7462026-03-11T19:41:55.430Z-- Looking for include file sys/stat.h
7472026-03-11T19:41:55.434Znone needed
7482026-03-11T19:41:55.440Zchecking whether compiler is cray... no
7492026-03-11T19:41:55.447Zchecking whether compiler supports -std=gnu11... yes
7502026-03-11T19:41:55.453Zchecking whether compiler supports -Wall... yes
7512026-03-11T19:41:55.456Zchecking whether compiler supports -Werror=declaration-after-statement... -- Looking for include file sys/stat.h - found
7522026-03-11T19:41:55.456Z-- Looking for include file sys/types.h
7532026-03-11T19:41:55.460Zyes
7542026-03-11T19:41:55.469Zchecking whether compiler supports -Wshorten-64-to-32... no
7552026-03-11T19:41:55.476Zchecking whether compiler supports -Wsign-compare... yes
7562026-03-11T19:41:55.481Zchecking whether compiler supports -pipe... -- Looking for include file sys/types.h - found
7572026-03-11T19:41:55.481Zyes
7582026-03-11T19:41:55.484Z-- Looking for include file unistd.h
7592026-03-11T19:41:55.487Zchecking whether compiler supports -g3... yes
7602026-03-11T19:41:55.496Zchecking how to run the C preprocessor... gcc -E
7612026-03-11T19:41:55.509Zchecking for stdio.h... -- Looking for include file unistd.h - found
7622026-03-11T19:41:55.512Z-- Looking for 4 include files stdlib.h, ..., float.h
7632026-03-11T19:41:55.513Zyes
7642026-03-11T19:41:55.516Zchecking for stdlib.h... yes
7652026-03-11T19:41:55.523Zchecking for string.h... yes
7662026-03-11T19:41:55.531Zchecking for inttypes.h... yes
7672026-03-11T19:41:55.538Zchecking for stdint.h... -- Looking for 4 include files stdlib.h, ..., float.h - found
7682026-03-11T19:41:55.541Zyes
7692026-03-11T19:41:55.541Z-- Looking for localeconv
7702026-03-11T19:41:55.546Zchecking for strings.h... yes
7712026-03-11T19:41:55.554Zchecking for sys/stat.h... yes
7722026-03-11T19:41:55.562Zchecking for sys/types.h... yes
7732026-03-11T19:41:55.567Zchecking for unistd.h... -- Looking for localeconv - found
7742026-03-11T19:41:55.570Z-- Looking for ceil in m
7752026-03-11T19:41:55.573Zyes
7762026-03-11T19:41:55.592Zchecking whether byte ordering is bigendian... no
7772026-03-11T19:41:55.597Zchecking size of void *... -- Looking for ceil in m - found
7782026-03-11T19:41:55.600Z-- Looking for pthread.h
7792026-03-11T19:41:55.611Z8
7802026-03-11T19:41:55.626Zchecking size of int... -- Looking for pthread.h - found
7812026-03-11T19:41:55.629Z-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
7822026-03-11T19:41:55.633Z4
7832026-03-11T19:41:55.650Zchecking size of long... 8
7842026-03-11T19:41:55.655Zchecking size of long long... -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
7852026-03-11T19:41:55.658Z-- Found Threads: TRUE
7862026-03-11T19:41:55.658Z-- Performing Test HAVE_PTHREAD_MUTEX_RECURSIVE
7872026-03-11T19:41:55.669Z8
7882026-03-11T19:41:55.684Zchecking size of intmax_t... -- Performing Test HAVE_PTHREAD_MUTEX_RECURSIVE - Success
7892026-03-11T19:41:55.688Z-- PROJ_PLATFORM_NAME = x64
7902026-03-11T19:41:55.688Z-- PROJ_COMPILER_NAME = gcc-11
7912026-03-11T19:41:55.688Z-- PROJ4_TESTS = ON
7922026-03-11T19:41:55.691Z8
7932026-03-11T19:41:55.706Zchecking build system type... --
7942026-03-11T19:41:55.709Z-- Configuring proj library:
7952026-03-11T19:41:55.709Z--
7962026-03-11T19:41:55.713Zx86_64-pc-linux-gnu
7972026-03-11T19:41:55.713Zchecking host system type... x86_64-pc-linux-gnu
7982026-03-11T19:41:55.713Zchecking whether pause instruction is compilable... -- JNI_SUPPORT = OFF
7992026-03-11T19:41:55.713Z-- PROJ_CORE_TARGET = proj
8002026-03-11T19:41:55.713Z-- PROJ_CORE_TARGET_OUTPUT_NAME = proj
8012026-03-11T19:41:55.713Z-- PROJ_LIBRARY_TYPE = STATIC
8022026-03-11T19:41:55.713Z-- PROJ_LIBRARIES = proj
8032026-03-11T19:41:55.716Z-- Configuring done
8042026-03-11T19:41:55.724Zyes
8052026-03-11T19:41:55.727Zchecking for ar... ar
8062026-03-11T19:41:55.727Zchecking for malloc.h... -- Generating done
8072026-03-11T19:41:55.727Z-- Build files have been written to: /work/oxidecomputer/cockroach/lib/build/proj
8082026-03-11T19:41:55.733Zyes
8092026-03-11T19:41:55.742Zchecking whether malloc_usable_size definition can use const argument... no
8102026-03-11T19:41:55.776Zchecking for library containing log... -lm
8112026-03-11T19:41:55.791Zchecking whether __attribute__ syntax is compilable... yes
8122026-03-11T19:41:55.796Zchecking whether compiler supports -fvisibility=hidden... yes
8132026-03-11T19:41:55.803Zchecking whether compiler supports -Werror... yes
8142026-03-11T19:41:55.809Zchecking whether compiler supports -herror_on_warning... yes
8152026-03-11T19:41:55.824Zchecking whether tls_model attribute is compilable... yes
8162026-03-11T19:41:55.830Zchecking whether compiler supports -Werror... yes
8172026-03-11T19:41:55.836Zchecking whether compiler supports -herror_on_warning... yes
8182026-03-11T19:41:55.853Zchecking whether alloc_size attribute is compilable... yes
8192026-03-11T19:41:55.858Zchecking whether compiler supports -Werror... yes
8202026-03-11T19:41:55.865Zchecking whether compiler supports -herror_on_warning... yes
8212026-03-11T19:41:55.882Zchecking whether format(gnu_printf, ...) attribute is compilable... yes
8222026-03-11T19:41:55.887Zchecking whether compiler supports -Werror... yes
8232026-03-11T19:41:55.893Zchecking whether compiler supports -herror_on_warning... yes
8242026-03-11T19:41:55.910Zchecking whether format(printf, ...) attribute is compilable... yes
8252026-03-11T19:41:55.915Zchecking for a BSD-compatible install... /usr/bin/install -c
8262026-03-11T19:41:55.918Zchecking for ranlib... ranlib
8272026-03-11T19:41:55.918Zchecking for ld... /usr/bin/ld
8282026-03-11T19:41:55.918Zchecking for autoconf... /usr/bin/autoconf
8292026-03-11T19:41:55.932Zchecking for memalign... yes
8302026-03-11T19:41:55.947Zchecking for valloc... yes
8312026-03-11T19:41:55.954Zchecking whether compiler supports -O3... yes
8322026-03-11T19:41:55.960Zchecking whether compiler supports -funroll-loops... yes
8332026-03-11T19:41:55.973Zchecking for unwind.h... yes
8342026-03-11T19:41:55.988Zchecking for _Unwind_Backtrace in -lgcc... yes
8352026-03-11T19:41:55.991Zchecking configured backtracing method... libgcc
8362026-03-11T19:41:56.005Zchecking for sbrk... yes
8372026-03-11T19:41:56.017Zchecking whether utrace(2) is compilable... no
8382026-03-11T19:41:56.024Zchecking whether valgrind is compilable... no
8392026-03-11T19:41:56.039Zchecking whether a program using __builtin_unreachable is compilable... yes
8402026-03-11T19:41:56.058Zchecking whether a program using __builtin_ffsl is compilable... yes
8412026-03-11T19:41:56.081Zchecking LG_PAGE... 12
8422026-03-11T19:41:56.090ZMissing VERSION file, and unable to generate it; creating bogus VERSION
8432026-03-11T19:41:56.112Zchecking for pthread.h... yes
8442026-03-11T19:41:56.128Zchecking for pthread_create in -lpthread... yes
8452026-03-11T19:41:56.146Zchecking whether pthread_atfork(3) is compilable... yes
8462026-03-11T19:41:56.163Zchecking for library containing clock_gettime... none required
8472026-03-11T19:41:56.179Zchecking whether clock_gettime(CLOCK_MONOTONIC_COARSE, ...) is compilable... yes
8482026-03-11T19:41:56.199Zchecking whether clock_gettime(CLOCK_MONOTONIC, ...) is compilable... yes
8492026-03-11T19:41:56.206Zchecking whether mach_absolute_time() is compilable... no
8502026-03-11T19:41:56.213Zchecking whether compiler supports -Werror... yes
8512026-03-11T19:41:56.233Zchecking whether syscall(2) is compilable... yes
8522026-03-11T19:41:56.248Zchecking for secure_getenv... yes
8532026-03-11T19:41:56.266Zchecking for issetugid... no
8542026-03-11T19:41:56.286Zchecking for _malloc_thread_cleanup... no
8552026-03-11T19:41:56.294Zchecking whether Darwin OSSpin*() is compilable... no
8562026-03-11T19:41:56.312Zchecking for _pthread_mutex_init_calloc_cb... no
8572026-03-11T19:41:56.318Zchecking for TLS... yes
8582026-03-11T19:41:56.337Zchecking whether C11 atomics is compilable... yes
8592026-03-11T19:41:56.345Zchecking whether atomic(9) is compilable... no
8602026-03-11T19:41:56.352Zchecking whether Darwin OSAtomic*() is compilable... no
8612026-03-11T19:41:56.369Zchecking whether madvise(2) is compilable... yes
8622026-03-11T19:41:56.385Zchecking whether madvise(..., MADV_FREE) is compilable... yes
8632026-03-11T19:41:56.402Zchecking whether madvise(..., MADV_DONTNEED) is compilable... yes
8642026-03-11T19:41:56.418Zchecking whether madvise(..., MADV_[NO]HUGEPAGE) is compilable... yes
8652026-03-11T19:41:56.427Zchecking whether to force 32-bit __sync_{add,sub}_and_fetch()... no
8662026-03-11T19:41:56.434Zchecking whether to force 64-bit __sync_{add,sub}_and_fetch()... no
8672026-03-11T19:41:56.452Zchecking for __builtin_clz... yes
8682026-03-11T19:41:56.459Zchecking whether Darwin os_unfair_lock_*() is compilable... no
8692026-03-11T19:41:56.478Zchecking whether glibc malloc hook is compilable... no
8702026-03-11T19:41:56.496Zchecking whether glibc memalign hook is compilable... no
8712026-03-11T19:41:56.515Zchecking whether pthreads adaptive mutexes is compilable... yes
8722026-03-11T19:41:56.542Zchecking for _Bool... yes
8732026-03-11T19:41:56.553Zchecking for stdbool.h that conforms to C99... no
8742026-03-11T19:41:56.562Zconfigure: creating ./config.status
8752026-03-11T19:41:56.599Zconfig.status: creating Makefile
8762026-03-11T19:41:56.604Zconfig.status: creating jemalloc.pc
8772026-03-11T19:41:56.609Zconfig.status: creating doc/html.xsl
8782026-03-11T19:41:56.618Zconfig.status: creating doc/manpages.xsl
8792026-03-11T19:41:56.624Zconfig.status: creating doc/jemalloc.xml
8802026-03-11T19:41:56.632Zconfig.status: creating include/jemalloc/jemalloc_macros.h
8812026-03-11T19:41:56.639Zconfig.status: creating include/jemalloc/jemalloc_protos.h
8822026-03-11T19:41:56.646Zconfig.status: creating include/jemalloc/jemalloc_typedefs.h
8832026-03-11T19:41:56.652Zconfig.status: creating include/jemalloc/internal/jemalloc_internal.h
8842026-03-11T19:41:56.660Zconfig.status: creating test/test.sh
8852026-03-11T19:41:56.668Zconfig.status: creating test/include/test/jemalloc_test.h
8862026-03-11T19:41:56.674Zconfig.status: creating config.stamp
8872026-03-11T19:41:56.679Zconfig.status: creating bin/jemalloc-config
8882026-03-11T19:41:56.685Zconfig.status: creating bin/jemalloc.sh
8892026-03-11T19:41:56.691Zconfig.status: creating bin/jeprof
8902026-03-11T19:41:56.701Zconfig.status: creating include/jemalloc/jemalloc_defs.h
8912026-03-11T19:41:56.706Zconfig.status: creating include/jemalloc/internal/jemalloc_internal_defs.h
8922026-03-11T19:41:56.711Zconfig.status: creating test/include/test/jemalloc_test_defs.h
8932026-03-11T19:41:56.718Zconfig.status: executing include/jemalloc/internal/private_namespace.h commands
8942026-03-11T19:41:56.722Zconfig.status: executing include/jemalloc/internal/private_unnamespace.h commands
8952026-03-11T19:41:56.726Zconfig.status: executing include/jemalloc/internal/public_symbols.txt commands
8962026-03-11T19:41:56.730Zconfig.status: executing include/jemalloc/internal/public_namespace.h commands
8972026-03-11T19:41:56.752Zconfig.status: executing include/jemalloc/internal/public_unnamespace.h commands
8982026-03-11T19:41:56.772Zconfig.status: executing include/jemalloc/internal/size_classes.h commands
8992026-03-11T19:41:56.858Zconfig.status: executing include/jemalloc/jemalloc_protos_jet.h commands
9002026-03-11T19:41:56.862Zconfig.status: executing include/jemalloc/jemalloc_rename.h commands
9012026-03-11T19:41:56.901Zconfig.status: executing include/jemalloc/jemalloc_mangle.h commands
9022026-03-11T19:41:56.942Zconfig.status: executing include/jemalloc/jemalloc_mangle_jet.h commands
9032026-03-11T19:41:56.981Zconfig.status: executing include/jemalloc/jemalloc.h commands
9042026-03-11T19:41:56.987Z===============================================================================
9052026-03-11T19:41:56.987Zjemalloc version : 0.0.0-0-g0000000000000000000000000000000000000000
9062026-03-11T19:41:56.990Zlibrary revision : 2
9072026-03-11T19:41:56.990Z
9082026-03-11T19:41:56.990ZCONFIG : --enable-prof CFLAGS=-g1 LDFLAGS=
9092026-03-11T19:41:56.990ZCC : gcc
9102026-03-11T19:41:56.990ZCONFIGURE_CFLAGS : -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops
9112026-03-11T19:41:56.990ZSPECIFIED_CFLAGS : -g1
9122026-03-11T19:41:56.990ZEXTRA_CFLAGS :
9132026-03-11T19:41:56.990ZCPPFLAGS : -D_GNU_SOURCE -D_REENTRANT
9142026-03-11T19:41:56.990ZLDFLAGS :
9152026-03-11T19:41:56.990ZEXTRA_LDFLAGS :
9162026-03-11T19:41:56.990ZLIBS : -lm -lgcc -lm -lpthread
9172026-03-11T19:41:56.990ZRPATH_EXTRA :
9182026-03-11T19:41:56.990Z
9192026-03-11T19:41:56.990ZXSLTPROC : false
9202026-03-11T19:41:56.990ZXSLROOT :
9212026-03-11T19:41:56.990Z
9222026-03-11T19:41:56.990ZPREFIX : /usr/local
9232026-03-11T19:41:56.990ZBINDIR : /usr/local/bin
9242026-03-11T19:41:56.990ZDATADIR : /usr/local/share
9252026-03-11T19:41:56.990ZINCLUDEDIR : /usr/local/include
9262026-03-11T19:41:56.990ZLIBDIR : /usr/local/lib
9272026-03-11T19:41:56.990ZMANDIR : /usr/local/share/man
9282026-03-11T19:41:56.990Z
9292026-03-11T19:41:56.990Zsrcroot : /work/oxidecomputer/cockroach/c-deps/jemalloc/
9302026-03-11T19:41:56.990Zabs_srcroot : /work/oxidecomputer/cockroach/c-deps/jemalloc/
9312026-03-11T19:41:56.990Zobjroot :
9322026-03-11T19:41:56.990Zabs_objroot : /work/oxidecomputer/cockroach/lib/build/jemalloc/
9332026-03-11T19:41:56.990Z
9342026-03-11T19:41:56.990ZJEMALLOC_PREFIX :
9352026-03-11T19:41:56.990ZJEMALLOC_PRIVATE_NAMESPACE
9362026-03-11T19:41:56.990Z : je_
9372026-03-11T19:41:56.990Zinstall_suffix :
9382026-03-11T19:41:56.991Zmalloc_conf :
9392026-03-11T19:41:56.991Zautogen : 0
9402026-03-11T19:41:56.991Zcc-silence : 1
9412026-03-11T19:41:56.991Zdebug : 0
9422026-03-11T19:41:56.991Zcode-coverage : 0
9432026-03-11T19:41:56.991Zstats : 1
9442026-03-11T19:41:56.991Zprof : 1
9452026-03-11T19:41:56.991Zprof-libunwind : 0
9462026-03-11T19:41:56.991Zprof-libgcc : 1
9472026-03-11T19:41:56.991Zprof-gcc : 0
9482026-03-11T19:41:56.991Ztcache : 1
9492026-03-11T19:41:56.991Zthp : 1
9502026-03-11T19:41:56.991Zfill : 1
9512026-03-11T19:41:56.991Zutrace : 0
9522026-03-11T19:41:56.991Zvalgrind : 0
9532026-03-11T19:41:56.991Zxmalloc : 0
9542026-03-11T19:41:56.991Zmunmap : 0
9552026-03-11T19:41:56.991Zlazy_lock : 0
9562026-03-11T19:41:56.991Ztls : 1
9572026-03-11T19:41:56.991Zcache-oblivious : 1
9582026-03-11T19:41:56.991Z===============================================================================
9592026-03-11T19:41:56.996Zgo mod download
9602026-03-11T19:42:06.747Zgo mod vendor
9612026-03-11T19:42:07.808ZGOBIN='/work/oxidecomputer/cockroach/bin' GOFLAGS= go install -mod=vendor -v github.com/goware/modvendor
9622026-03-11T19:42:07.881Zinternal/goarch
9632026-03-11T19:42:07.884Zinternal/unsafeheader
9642026-03-11T19:42:07.884Zinternal/cpu
9652026-03-11T19:42:07.884Zinternal/itoa
9662026-03-11T19:42:07.884Zmath/bits
9672026-03-11T19:42:07.884Zinternal/coverage/rtcov
9682026-03-11T19:42:07.884Zinternal/godebugs
9692026-03-11T19:42:07.884Zunicode/utf8
9702026-03-11T19:42:07.884Zinternal/race
9712026-03-11T19:42:07.884Zsync/atomic
9722026-03-11T19:42:07.884Zencoding
9732026-03-11T19:42:07.884Zinternal/goexperiment
9742026-03-11T19:42:07.884Zinternal/goos
9752026-03-11T19:42:07.885Zruntime/internal/syscall
9762026-03-11T19:42:07.885Zruntime/internal/atomic
9772026-03-11T19:42:07.885Zunicode
9782026-03-11T19:42:07.888Zcmp
9792026-03-11T19:42:07.888Zinternal/abi
9802026-03-11T19:42:07.888Zinternal/chacha8rand
9812026-03-11T19:42:07.889Zruntime/internal/math
9822026-03-11T19:42:07.892Zruntime/internal/sys
9832026-03-11T19:42:07.895Zslices
9842026-03-11T19:42:07.899Zinternal/bytealg
9852026-03-11T19:42:07.901Zmath
9862026-03-11T19:42:07.943Zruntime
9872026-03-11T19:42:08.891Zinternal/reflectlite
9882026-03-11T19:42:08.894Zsync
9892026-03-11T19:42:08.920Zinternal/testlog
9902026-03-11T19:42:08.937Zerrors
9912026-03-11T19:42:08.940Zsort
9922026-03-11T19:42:08.946Zinternal/oserror
9932026-03-11T19:42:08.949Zinternal/safefilepath
9942026-03-11T19:42:08.949Zpath
9952026-03-11T19:42:08.949Zio
9962026-03-11T19:42:08.949Zstrconv
9972026-03-11T19:42:08.952Zsyscall
9982026-03-11T19:42:08.970Zstrings
9992026-03-11T19:42:08.970Zbytes
10002026-03-11T19:42:09.001Zreflect
10012026-03-11T19:42:09.012Zbufio
10022026-03-11T19:42:09.014Zregexp/syntax
10032026-03-11T19:42:09.068Zinternal/syscall/execenv
10042026-03-11T19:42:09.071Zinternal/syscall/unix
10052026-03-11T19:42:09.071Ztime
10062026-03-11T19:42:09.086Zregexp
10072026-03-11T19:42:09.176Zio/fs
10082026-03-11T19:42:09.180Zinternal/poll
10092026-03-11T19:42:09.206Zinternal/fmtsort
10102026-03-11T19:42:09.227Zos
10112026-03-11T19:42:09.308Zpath/filepath
10122026-03-11T19:42:09.309Zfmt
10132026-03-11T19:42:09.381Zflag
10142026-03-11T19:42:09.381Zgithub.com/mattn/go-zglob/fastwalk
10152026-03-11T19:42:09.394Zgithub.com/mattn/go-zglob
10162026-03-11T19:42:09.415Zgithub.com/goware/modvendor
10172026-03-11T19:42:09.500Zmodvendor -copy="**/*.c **/*.h **/*.proto" -include 'github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis/google/api,github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis/google/rpc,github.com/prometheus/client_model'
10182026-03-11T19:42:09.796Zfor patch in patches/*.patch; do patch -p1 <$patch; done
10192026-03-11T19:42:09.799Zpatching file vendor/github.com/knz/go-libedit/unix/editline_unix.go
10202026-03-11T19:42:09.799Zpatching file vendor/github.com/elastic/gosigar/sigar_stub.go
10212026-03-11T19:42:09.802Zpatching file vendor/github.com/elastic/gosigar/sigar_illumos.go
10222026-03-11T19:42:09.802Zpatching file vendor/github.com/elastic/gosigar/sigar_unix_common.go
10232026-03-11T19:42:09.802Zgo install -v ./pkg/cmd/prereqs
10242026-03-11T19:42:09.802Zregenerating pkg/cli/zcgo_flags.go
10252026-03-11T19:42:09.802Zregenerating pkg/cli/clisqlshell/zcgo_flags.go
10262026-03-11T19:42:09.802Zregenerating pkg/server/status/zcgo_flags.go
10272026-03-11T19:42:09.802Zregenerating vendor/github.com/knz/go-libedit/unix/zcgo_flags.go
10282026-03-11T19:42:09.802Zregenerating pkg/cli/zcgo_flags_x86_64_linux_gnu.go
10292026-03-11T19:42:09.802Zregenerating pkg/geo/geoproj/zcgo_flags.go
10302026-03-11T19:42:09.802Zregenerating pkg/cli/clisqlshell/zcgo_flags_x86_64_linux_gnu.go
10312026-03-11T19:42:09.802Zregenerating pkg/geo/geoproj/zcgo_flags_x86_64_linux_gnu.go
10322026-03-11T19:42:09.802Zregenerating pkg/server/status/zcgo_flags_x86_64_linux_gnu.go
10332026-03-11T19:42:09.802Zregenerating vendor/github.com/knz/go-libedit/unix/zcgo_flags_x86_64_linux_gnu.go
10342026-03-11T19:42:09.803Zregenerating vendor/github.com/knz/go-libedit/unix/zcgo_flags_extra.go
10352026-03-11T19:42:09.839Zgo: downloading github.com/bufbuild/buf v0.56.0
10362026-03-11T19:42:10.010Zunicode/utf16
10372026-03-11T19:42:10.010Zinternal/gover
10382026-03-11T19:42:10.010Zinternal/goversion
10392026-03-11T19:42:10.014Zhash/maphash
10402026-03-11T19:42:10.014Zinternal/platform
10412026-03-11T19:42:10.014Zlog/internal
10422026-03-11T19:42:10.014Zgolang.org/x/tools/internal/packagesinternal
10432026-03-11T19:42:10.017Zgolang.org/x/mod/semver
10442026-03-11T19:42:10.017Zinternal/types/errors
10452026-03-11T19:42:10.017Zinternal/bisect
10462026-03-11T19:42:10.017Zhash
10472026-03-11T19:42:10.017Zgo/build/constraint
10482026-03-11T19:42:10.020Zcontainer/heap
10492026-03-11T19:42:10.020Zcontext
10502026-03-11T19:42:10.020Zencoding/binary
10512026-03-11T19:42:10.020Zgo/version
10522026-03-11T19:42:10.020Zinternal/lazyregexp
10532026-03-11T19:42:10.020Zgo/token
10542026-03-11T19:42:10.020Zinternal/buildcfg
10552026-03-11T19:42:10.023Zgo/doc/comment
10562026-03-11T19:42:10.023Zgolang.org/x/tools/internal/event/label
10572026-03-11T19:42:10.023Zgolang.org/x/tools/internal/stdlib
10582026-03-11T19:42:10.039Zlog
10592026-03-11T19:42:10.039Zcrypto
10602026-03-11T19:42:10.043Zgolang.org/x/tools/internal/event/keys
10612026-03-11T19:42:10.043Zinternal/godebug
10622026-03-11T19:42:10.049Zgolang.org/x/sync/errgroup
10632026-03-11T19:42:10.052Zgo/scanner
10642026-03-11T19:42:10.052Zmath/rand
10652026-03-11T19:42:10.052Zos/exec
10662026-03-11T19:42:10.068Zgolang.org/x/tools/internal/event/core
10672026-03-11T19:42:10.073Zencoding/base64
10682026-03-11T19:42:10.074Zcrypto/md5
10692026-03-11T19:42:10.076Zgo/ast
10702026-03-11T19:42:10.083Zmath/big
10712026-03-11T19:42:10.086Zinternal/goroot
10722026-03-11T19:42:10.089Zgolang.org/x/tools/internal/event
10732026-03-11T19:42:10.091Zencoding/json
10742026-03-11T19:42:10.150Zgo/internal/typeparams
10752026-03-11T19:42:10.153Zgo/doc
10762026-03-11T19:42:10.155Zgo/parser
10772026-03-11T19:42:10.213Zgolang.org/x/tools/internal/gocommand
10782026-03-11T19:42:10.236Zgo/constant
10792026-03-11T19:42:10.250Zgo/build
10802026-03-11T19:42:10.276Zgolang.org/x/tools/internal/pkgbits
10812026-03-11T19:42:10.279Zgo/types
10822026-03-11T19:42:10.773Zgolang.org/x/tools/internal/aliases
10832026-03-11T19:42:10.773Zgolang.org/x/tools/internal/versions
10842026-03-11T19:42:10.784Zgolang.org/x/tools/internal/typeparams
10852026-03-11T19:42:10.811Zgolang.org/x/tools/go/types/typeutil
10862026-03-11T19:42:10.838Zgolang.org/x/tools/internal/typesinternal
10872026-03-11T19:42:10.870Zgolang.org/x/tools/go/types/objectpath
10882026-03-11T19:42:10.894Zgolang.org/x/tools/internal/gcimporter
10892026-03-11T19:42:11.004Zgolang.org/x/tools/go/gcexportdata
10902026-03-11T19:42:11.018Zgolang.org/x/tools/go/packages
10912026-03-11T19:42:11.090Zgithub.com/cockroachdb/cockroach/pkg/cmd/prereqs
10922026-03-11T19:42:11.221Zgo install -v uptodate
10932026-03-11T19:42:11.224ZGOFLAGS= bin/prereqs ./pkg/cmd/uptodate > bin/uptodate.d.tmp
10942026-03-11T19:42:11.224Zgo install -v protoc-gen-gogoroach
10952026-03-11T19:42:11.224ZGOFLAGS= bin/prereqs ./pkg/cmd/protoc-gen-gogoroach > bin/protoc-gen-gogoroach.d.tmp
10962026-03-11T19:42:11.227Zgo install -v terraformgen
10972026-03-11T19:42:11.227ZGOFLAGS= bin/prereqs ./pkg/roachprod/vm/aws/terraformgen > bin/terraformgen.d.tmp
10982026-03-11T19:42:11.230Zgo install -v optgen
10992026-03-11T19:42:11.230ZGOFLAGS= bin/prereqs ./pkg/sql/opt/optgen/cmd/optgen > bin/optgen.d.tmp
11002026-03-11T19:42:11.242Zmv -f bin/uptodate.d.tmp bin/uptodate.d
11012026-03-11T19:42:11.245Zmv -f bin/terraformgen.d.tmp bin/terraformgen.d
11022026-03-11T19:42:11.245ZGOBIN='/work/oxidecomputer/cockroach/bin' GOFLAGS= go install -mod=vendor -tags 'make x86_64_linux_gnu crdb_test' -ldflags '-X github.com/cockroachdb/cockroach/pkg/build.typ=development -extldflags "" -X "github.com/cockroachdb/cockroach/pkg/build.tag=v22.1.22-59-g76d728e772" -X "github.com/cockroachdb/cockroach/pkg/build.rev=76d728e7724724234d36ada4d402a921657c3724" -X "github.com/cockroachdb/cockroach/pkg/build.cgoTargetTriple=x86_64-linux-gnu" ' -v ./pkg/cmd/uptodate
11032026-03-11T19:42:11.245ZGOBIN='/work/oxidecomputer/cockroach/bin' GOFLAGS= go install -mod=vendor -tags 'make x86_64_linux_gnu crdb_test' -ldflags '-X github.com/cockroachdb/cockroach/pkg/build.typ=development -extldflags "" -X "github.com/cockroachdb/cockroach/pkg/build.tag=v22.1.22-59-g76d728e772" -X "github.com/cockroachdb/cockroach/pkg/build.rev=76d728e7724724234d36ada4d402a921657c3724" -X "github.com/cockroachdb/cockroach/pkg/build.cgoTargetTriple=x86_64-linux-gnu" ' -v ./pkg/roachprod/vm/aws/terraformgen
11042026-03-11T19:42:11.248Zmv -f bin/protoc-gen-gogoroach.d.tmp bin/protoc-gen-gogoroach.d
11052026-03-11T19:42:11.248Zmv -f bin/optgen.d.tmp bin/optgen.d
11062026-03-11T19:42:11.248ZGOBIN='/work/oxidecomputer/cockroach/bin' GOFLAGS= go install -mod=vendor -tags 'make x86_64_linux_gnu crdb_test' -ldflags '-X github.com/cockroachdb/cockroach/pkg/build.typ=development -extldflags "" -X "github.com/cockroachdb/cockroach/pkg/build.tag=v22.1.22-59-g76d728e772" -X "github.com/cockroachdb/cockroach/pkg/build.rev=76d728e7724724234d36ada4d402a921657c3724" -X "github.com/cockroachdb/cockroach/pkg/build.cgoTargetTriple=x86_64-linux-gnu" ' -v ./pkg/cmd/protoc-gen-gogoroach
11072026-03-11T19:42:11.248ZGOBIN='/work/oxidecomputer/cockroach/bin' GOFLAGS= go install -mod=vendor -tags 'make x86_64_linux_gnu crdb_test' -ldflags '-X github.com/cockroachdb/cockroach/pkg/build.typ=development -extldflags "" -X "github.com/cockroachdb/cockroach/pkg/build.tag=v22.1.22-59-g76d728e772" -X "github.com/cockroachdb/cockroach/pkg/build.rev=76d728e7724724234d36ada4d402a921657c3724" -X "github.com/cockroachdb/cockroach/pkg/build.cgoTargetTriple=x86_64-linux-gnu" ' -v ./pkg/sql/opt/optgen/cmd/optgen
11082026-03-11T19:42:11.284Zinternal/nettrace
11092026-03-11T19:42:11.288Zvendor/golang.org/x/net/dns/dnsmessage
11102026-03-11T19:42:11.291Zinternal/singleflight
11112026-03-11T19:42:11.291Zinternal/intern
11122026-03-11T19:42:11.291Zgithub.com/cockroachdb/redact/internal/rfmt/fmtsort
11132026-03-11T19:42:11.291Zruntime/cgo
11142026-03-11T19:42:11.291Zgithub.com/cockroachdb/redact/internal/fmtforward
11152026-03-11T19:42:11.291Zgithub.com/cockroachdb/redact/interfaces
11162026-03-11T19:42:11.291Zencoding/hex
11172026-03-11T19:42:11.291Zencoding/csv
11182026-03-11T19:42:11.291Znet/url
11192026-03-11T19:42:11.294Ztext/template/parse
11202026-03-11T19:42:11.299Znet/netip
11212026-03-11T19:42:11.302Zgithub.com/cockroachdb/redact/internal/markers
11222026-03-11T19:42:11.305Zgithub.com/cockroachdb/redact/internal/redact
11232026-03-11T19:42:11.308Zgithub.com/cockroachdb/redact/internal/escape
11242026-03-11T19:42:11.315Zgithub.com/cockroachdb/redact/internal/buffer
11252026-03-11T19:42:11.344Zgithub.com/cockroachdb/redact/internal/rfmt
11262026-03-11T19:42:11.347Zgithub.com/cockroachdb/errors/stdstrings
11272026-03-11T19:42:11.347Zcrypto/internal/alias
11282026-03-11T19:42:11.347Zcontainer/list
11292026-03-11T19:42:11.351Zcrypto/internal/alias
11302026-03-11T19:42:11.356Zcrypto/subtle
11312026-03-11T19:42:11.359Zcrypto/internal/boring/sig
11322026-03-11T19:42:11.362Zcrypto/subtle
11332026-03-11T19:42:11.362Zvendor/golang.org/x/crypto/cryptobyte/asn1
11342026-03-11T19:42:11.362Zvendor/golang.org/x/crypto/internal/alias
11352026-03-11T19:42:11.362Zcrypto/internal/boring/sig
11362026-03-11T19:42:11.367Ztext/tabwriter
11372026-03-11T19:42:11.370Zhash/crc32
11382026-03-11T19:42:11.370Zio/ioutil
11392026-03-11T19:42:11.370Zgithub.com/gogo/protobuf/protoc-gen-gogo/generator/internal/remap
11402026-03-11T19:42:11.377Zcompress/flate
11412026-03-11T19:42:11.377Zgithub.com/gogo/protobuf/proto
11422026-03-11T19:42:11.377Zgithub.com/gogo/protobuf/sortkeys
11432026-03-11T19:42:11.380Zhash/crc32
11442026-03-11T19:42:11.380Zcrypto/rc4
11452026-03-11T19:42:11.380Zgithub.com/kr/text
11462026-03-11T19:42:11.380Ztext/tabwriter
11472026-03-11T19:42:11.380Zcrypto/cipher
11482026-03-11T19:42:11.387Ztext/template
11492026-03-11T19:42:11.391Zcrypto/internal/randutil
11502026-03-11T19:42:11.391Zgithub.com/rogpeppe/go-internal/fmtsort
11512026-03-11T19:42:11.391Zencoding/pem
11522026-03-11T19:42:11.391Zvendor/golang.org/x/text/transform
11532026-03-11T19:42:11.391Znet/http/internal/ascii
11542026-03-11T19:42:11.392Zgo/printer
11552026-03-11T19:42:11.395Zgithub.com/gogo/protobuf/proto
11562026-03-11T19:42:11.406Zcompress/flate
11572026-03-11T19:42:11.409Zgithub.com/pkg/errors
11582026-03-11T19:42:11.409Zgithub.com/cockroachdb/logtags
11592026-03-11T19:42:11.409Zcrypto/internal/boring
11602026-03-11T19:42:11.409Zgithub.com/cockroachdb/redact/internal/rfmt
11612026-03-11T19:42:11.409Zembed
11622026-03-11T19:42:11.409Zgithub.com/kr/pretty
11632026-03-11T19:42:11.409Zvendor/golang.org/x/sys/cpu
11642026-03-11T19:42:11.409Zcrypto/internal/bigmod
11652026-03-11T19:42:11.422Zruntime/cgo
11662026-03-11T19:42:11.422Zgithub.com/cockroachdb/redact/builder
11672026-03-11T19:42:11.422Zcrypto/internal/nistec/fiat
11682026-03-11T19:42:11.422Zencoding/asn1
11692026-03-11T19:42:11.426Zcompress/gzip
11702026-03-11T19:42:11.430Zcrypto/dsa
11712026-03-11T19:42:11.430Zvendor/golang.org/x/text/unicode/bidi
11722026-03-11T19:42:11.433Zvendor/golang.org/x/text/unicode/norm
11732026-03-11T19:42:11.436Zcrypto/des
11742026-03-11T19:42:11.441Zcrypto/sha256
11752026-03-11T19:42:11.445Zcrypto/internal/edwards25519/field
11762026-03-11T19:42:11.448Zgithub.com/cockroachdb/redact
11772026-03-11T19:42:11.458Zcrypto/internal/edwards25519/field
11782026-03-11T19:42:11.473Zcrypto/aes
11792026-03-11T19:42:11.473Zcrypto/des
11802026-03-11T19:42:11.473Zcrypto/sha512
11812026-03-11T19:42:11.473Zcrypto/hmac
11822026-03-11T19:42:11.473Zcrypto/sha1
11832026-03-11T19:42:11.473Zcrypto/sha256
11842026-03-11T19:42:11.473Zvendor/golang.org/x/crypto/chacha20
11852026-03-11T19:42:11.473Zcrypto/internal/nistec/fiat
11862026-03-11T19:42:11.473Zvendor/golang.org/x/crypto/internal/poly1305
11872026-03-11T19:42:11.473Zembed
11882026-03-11T19:42:11.480Zvendor/golang.org/x/crypto/chacha20
11892026-03-11T19:42:11.483Zgithub.com/cockroachdb/cockroach/pkg/cli/exit
11902026-03-11T19:42:11.497Zvendor/golang.org/x/crypto/hkdf
11912026-03-11T19:42:11.509Zvendor/golang.org/x/crypto/internal/poly1305
11922026-03-11T19:42:11.512Zvendor/golang.org/x/net/http2/hpack
11932026-03-11T19:42:11.518Zvendor/golang.org/x/crypto/cryptobyte
11942026-03-11T19:42:11.521Zcrypto/rand
11952026-03-11T19:42:11.527Zcrypto/aes
11962026-03-11T19:42:11.530Zcrypto/internal/boring/bbig
11972026-03-11T19:42:11.533Zcrypto/sha512
11982026-03-11T19:42:11.533Zgithub.com/MichaelTJones/walk
11992026-03-11T19:42:11.536Zgithub.com/gogo/protobuf/proto
12002026-03-11T19:42:11.536Zcrypto/sha1
12012026-03-11T19:42:11.544Zcrypto/sha256
12022026-03-11T19:42:11.544Zcrypto/x509/pkix
12032026-03-11T19:42:11.556Zcrypto/internal/edwards25519
12042026-03-11T19:42:11.559Zmime
12052026-03-11T19:42:11.562Zvendor/golang.org/x/text/unicode/bidi
12062026-03-11T19:42:11.562Zmime
12072026-03-11T19:42:11.562Zmime/quotedprintable
12082026-03-11T19:42:11.565Zvendor/golang.org/x/text/unicode/norm
12092026-03-11T19:42:11.573Zruntime/cgo
12102026-03-11T19:42:11.578Znet/http/internal
12112026-03-11T19:42:11.578Zgithub.com/getsentry/sentry-go/internal/crypto/randutil
12122026-03-11T19:42:11.584Zcrypto/internal/boring/bbig
12132026-03-11T19:42:11.584Zvendor/golang.org/x/crypto/cryptobyte
12142026-03-11T19:42:11.587Zmime/quotedprintable
12152026-03-11T19:42:11.591Zvendor/golang.org/x/crypto/chacha20poly1305
12162026-03-11T19:42:11.594Zcrypto/x509/pkix
12172026-03-11T19:42:11.594Zgo/format
12182026-03-11T19:42:11.594Znet/http/internal
12192026-03-11T19:42:11.600Zvendor/golang.org/x/crypto/chacha20poly1305
12202026-03-11T19:42:11.607Zgithub.com/getsentry/sentry-go/internal/crypto/randutil
12212026-03-11T19:42:11.607Zcrypto/rsa
12222026-03-11T19:42:11.607Zgolang.org/x/sys/execabs
12232026-03-11T19:42:11.607Zgolang.org/x/sys/execabs
12242026-03-11T19:42:11.610Zvendor/golang.org/x/text/secure/bidirule
12252026-03-11T19:42:11.621Zruntime/debug
12262026-03-11T19:42:11.635Zruntime/debug
12272026-03-11T19:42:11.638Zgo/format
12282026-03-11T19:42:11.649Zcrypto/rsa
12292026-03-11T19:42:11.661Zvendor/golang.org/x/net/idna
12302026-03-11T19:42:11.684Zcrypto/ed25519
12312026-03-11T19:42:11.692Zcrypto/ed25519
12322026-03-11T19:42:11.692Zcrypto/internal/nistec
12332026-03-11T19:42:11.757Zcrypto/internal/nistec
12342026-03-11T19:42:11.766Zvendor/golang.org/x/net/idna
12352026-03-11T19:42:11.820Znet
12362026-03-11T19:42:11.830Zcrypto/ecdh
12372026-03-11T19:42:11.833Zcrypto/elliptic
12382026-03-11T19:42:11.893Zcrypto/elliptic
12392026-03-11T19:42:11.896Zcrypto/ecdsa
12402026-03-11T19:42:11.936Zcrypto/ecdsa
12412026-03-11T19:42:12.022Znet
12422026-03-11T19:42:12.026Znet
12432026-03-11T19:42:12.037Zgithub.com/gogo/protobuf/protoc-gen-gogo/descriptor
12442026-03-11T19:42:12.074Zgithub.com/gogo/protobuf/protoc-gen-gogo/descriptor
12452026-03-11T19:42:12.077Zgithub.com/gogo/protobuf/types
12462026-03-11T19:42:12.095Zgithub.com/gogo/protobuf/protoc-gen-gogo/descriptor
12472026-03-11T19:42:12.097Zgithub.com/gogo/protobuf/types
12482026-03-11T19:42:12.136Zgo: downloading google.golang.org/protobuf v1.27.1
12492026-03-11T19:42:12.136Zgo: downloading go.uber.org/multierr v1.7.0
12502026-03-11T19:42:12.139Zgo: downloading github.com/pkg/profile v1.6.0
12512026-03-11T19:42:12.139Zgo: downloading go.uber.org/zap v1.19.0
12522026-03-11T19:42:12.139Zgo: downloading github.com/twitchtv/twirp v8.1.0+incompatible
12532026-03-11T19:42:12.139Zgo: downloading google.golang.org/grpc v1.41.0-dev.0.20210907181116-2f3355d2244e
12542026-03-11T19:42:12.139Zgo: downloading golang.org/x/term v0.0.0-20210615171337-6886f2dfbf5b
12552026-03-11T19:42:12.147Zgithub.com/gogo/protobuf/protoc-gen-gogo/plugin
12562026-03-11T19:42:12.147Zgithub.com/gogo/protobuf/gogoproto
12572026-03-11T19:42:12.152Zgo: downloading github.com/jhump/protoreflect v1.9.1-0.20210817181203-db1a327a393e
12582026-03-11T19:42:12.159Zgo: downloading github.com/bgentry/go-netrc v0.0.0-20140422174119-9fd32a8b3d3d
12592026-03-11T19:42:12.170Zgithub.com/gogo/protobuf/vanity
12602026-03-11T19:42:12.172Zgithub.com/gogo/protobuf/protoc-gen-gogo/generator
12612026-03-11T19:42:12.185Zgo: downloading golang.org/x/sync v0.0.0-20210220032951-036812b2e83c
12622026-03-11T19:42:12.239Zgo: downloading github.com/gofrs/uuid v4.0.0+incompatible
12632026-03-11T19:42:12.250Zgo: downloading github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e
12642026-03-11T19:42:12.253Zgo: downloading gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b
12652026-03-11T19:42:12.253Zgo: downloading github.com/gofrs/flock v0.8.1
12662026-03-11T19:42:12.263Zgo: downloading github.com/klauspost/compress v1.13.5
12672026-03-11T19:42:12.276Zgo: downloading github.com/klauspost/pgzip v1.2.5
12682026-03-11T19:42:12.276Zgo: downloading go.uber.org/atomic v1.7.0
12692026-03-11T19:42:12.276Zgo: downloading golang.org/x/sys v0.0.0-20210908143011-c212e7322662
12702026-03-11T19:42:12.276Zgo: downloading github.com/golang/protobuf v1.5.2
12712026-03-11T19:42:12.296Zgo: downloading google.golang.org/genproto v0.0.0-20210903162649-d08c68adba83
12722026-03-11T19:42:12.312Zgo: downloading golang.org/x/net v0.0.0-20210907225631-ff17edfbf26d
12732026-03-11T19:42:12.371Zgithub.com/gogo/protobuf/plugin/defaultcheck
12742026-03-11T19:42:12.371Zgithub.com/gogo/protobuf/plugin/embedcheck
12752026-03-11T19:42:12.371Zgithub.com/gogo/protobuf/plugin/enumstringer
12762026-03-11T19:42:12.371Zgithub.com/gogo/protobuf/plugin/marshalto
12772026-03-11T19:42:12.374Zgithub.com/gogo/protobuf/plugin/testgen
12782026-03-11T19:42:12.374Zgithub.com/gogo/protobuf/plugin/oneofcheck
12792026-03-11T19:42:12.374Zgithub.com/gogo/protobuf/plugin/populate
12802026-03-11T19:42:12.374Zgithub.com/gogo/protobuf/plugin/unmarshal
12812026-03-11T19:42:12.374Zgithub.com/gogo/protobuf/protoc-gen-gogo/grpc
12822026-03-11T19:42:12.396Zgithub.com/cockroachdb/errors/errorspb
12832026-03-11T19:42:12.412Zgo: downloading golang.org/x/text v0.3.7
12842026-03-11T19:42:12.418Zgithub.com/cockroachdb/errors/errorspb
12852026-03-11T19:42:12.508Zgithub.com/cockroachdb/errors/errbase
12862026-03-11T19:42:12.514Zgithub.com/cockroachdb/errors/errbase
12872026-03-11T19:42:12.559Zgithub.com/gogo/protobuf/plugin/compare
12882026-03-11T19:42:12.562Zgithub.com/gogo/protobuf/plugin/face
12892026-03-11T19:42:12.562Zgithub.com/gogo/protobuf/plugin/stringer
12902026-03-11T19:42:12.562Zgithub.com/gogo/protobuf/plugin/size
12912026-03-11T19:42:12.562Zgithub.com/gogo/protobuf/plugin/gostring
12922026-03-11T19:42:12.562Zgithub.com/gogo/protobuf/plugin/union
12932026-03-11T19:42:12.562Zgithub.com/gogo/protobuf/plugin/description
12942026-03-11T19:42:12.562Zgithub.com/gogo/protobuf/plugin/equal
12952026-03-11T19:42:12.583Zgithub.com/cockroachdb/errors/markers
12962026-03-11T19:42:12.583Zgithub.com/cockroachdb/errors/barriers
12972026-03-11T19:42:12.583Zgithub.com/cockroachdb/errors/contexttags
12982026-03-11T19:42:12.583Zgithub.com/cockroachdb/errors/secondary
12992026-03-11T19:42:12.586Zgithub.com/cockroachdb/errors/telemetrykeys
13002026-03-11T19:42:12.589Zgithub.com/cockroachdb/errors/safedetails
13012026-03-11T19:42:12.589Zgithub.com/cockroachdb/errors/hintdetail
13022026-03-11T19:42:12.593Zgithub.com/cockroachdb/errors/markers
13032026-03-11T19:42:12.593Zgithub.com/cockroachdb/errors/barriers
13042026-03-11T19:42:12.593Zgithub.com/cockroachdb/errors/contexttags
13052026-03-11T19:42:12.605Zgithub.com/spf13/pflag
13062026-03-11T19:42:12.615Zgithub.com/cockroachdb/errors/secondary
13072026-03-11T19:42:12.615Zgithub.com/cockroachdb/errors/hintdetail
13082026-03-11T19:42:12.615Zgithub.com/cockroachdb/errors/safedetails
13092026-03-11T19:42:12.615Zgithub.com/cockroachdb/errors/telemetrykeys
13102026-03-11T19:42:12.630Zgithub.com/cockroachdb/errors/assert
13112026-03-11T19:42:12.630Zgithub.com/cockroachdb/errors/issuelink
13122026-03-11T19:42:12.630Zgithub.com/cockroachdb/errors/assert
13132026-03-11T19:42:12.633Zgithub.com/cockroachdb/errors/issuelink
13142026-03-11T19:42:12.636Zgithub.com/cockroachdb/errors/domains
13152026-03-11T19:42:12.639Zgithub.com/cockroachdb/errors/domains
13162026-03-11T19:42:12.764Zgithub.com/gogo/protobuf/vanity/command
13172026-03-11T19:42:12.779Zgithub.com/cockroachdb/cockroach/pkg/cmd/protoc-gen-gogoroach
13182026-03-11T19:42:12.783Zvendor/golang.org/x/net/http/httpproxy
13192026-03-11T19:42:12.788Znet/textproto
13202026-03-11T19:42:12.788Zcrypto/x509
13212026-03-11T19:42:12.788Zgithub.com/spf13/pflag
13222026-03-11T19:42:12.830Zvendor/golang.org/x/net/http/httpguts
13232026-03-11T19:42:12.830Zmime/multipart
13242026-03-11T19:42:12.849Zgithub.com/spf13/cobra
13252026-03-11T19:42:12.880Zcrypto/x509
13262026-03-11T19:42:12.942Zgithub.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/terraformgen
13272026-03-11T19:42:12.962Zcrypto/tls
13282026-03-11T19:42:13.021Zcrypto/tls
13292026-03-11T19:42:13.277Znet/http/httptrace
13302026-03-11T19:42:13.290Znet/http
13312026-03-11T19:42:13.319Znet/http
13322026-03-11T19:42:13.406Zgoogle.golang.org/protobuf/internal/flags
13332026-03-11T19:42:13.406Zgoogle.golang.org/protobuf/internal/set
13342026-03-11T19:42:13.406Zgithub.com/golang/groupcache/lru
13352026-03-11T19:42:13.406Zgo.opencensus.io
13362026-03-11T19:42:13.409Zgo.opencensus.io/trace/internal
13372026-03-11T19:42:13.409Zgithub.com/twitchtv/twirp/internal/contextkeys
13382026-03-11T19:42:13.409Zgoogle.golang.org/grpc/serviceconfig
13392026-03-11T19:42:13.409Zgolang.org/x/sys/internal/unsafeheader
13402026-03-11T19:42:13.409Zgo.opencensus.io/internal/tagencoding
13412026-03-11T19:42:13.412Zgithub.com/klauspost/compress
13422026-03-11T19:42:13.415Zgoogle.golang.org/protobuf/internal/pragma
13432026-03-11T19:42:13.415Zgoogle.golang.org/grpc/internal/buffer
13442026-03-11T19:42:13.415Zhash/fnv
13452026-03-11T19:42:13.416Zgoogle.golang.org/grpc/encoding
13462026-03-11T19:42:13.416Zgoogle.golang.org/grpc/internal/grpcsync
13472026-03-11T19:42:13.416Zhtml
13482026-03-11T19:42:13.416Zgo.uber.org/zap/buffer
13492026-03-11T19:42:13.416Zgoogle.golang.org/grpc/backoff
13502026-03-11T19:42:13.416Zgithub.com/klauspost/compress/internal/snapref
13512026-03-11T19:42:13.416Zgithub.com/klauspost/compress/zstd/internal/xxhash
13522026-03-11T19:42:13.416Zgoogle.golang.org/grpc/internal/grpcrand
13532026-03-11T19:42:13.418Zgithub.com/bufbuild/buf/private/usage
13542026-03-11T19:42:13.418Zos/signal
13552026-03-11T19:42:13.418Zgopkg.in/yaml.v3
13562026-03-11T19:42:13.418Zgo.uber.org/atomic
13572026-03-11T19:42:13.419Zgoogle.golang.org/protobuf/internal/version
13582026-03-11T19:42:13.419Zgo.opencensus.io/trace/tracestate
13592026-03-11T19:42:13.419Zgo.opencensus.io/internal
13602026-03-11T19:42:13.419Zruntime/trace
13612026-03-11T19:42:13.422Zgo.uber.org/zap/internal/color
13622026-03-11T19:42:13.425Zgithub.com/bufbuild/buf/private/bufpkg/bufanalysis
13632026-03-11T19:42:13.425Zgithub.com/bufbuild/buf/private/pkg/stringutil
13642026-03-11T19:42:13.427Zgithub.com/bufbuild/buf/private/pkg/protoversion
13652026-03-11T19:42:13.427Zgo.uber.org/zap/internal/exit
13662026-03-11T19:42:13.432Znet/http
13672026-03-11T19:42:13.435Zgithub.com/bufbuild/buf/private/pkg/diff
13682026-03-11T19:42:13.435Zgo.uber.org/zap/internal/bufferpool
13692026-03-11T19:42:13.437Zgithub.com/bufbuild/buf/private/pkg/netextended
13702026-03-11T19:42:13.438Zgithub.com/bufbuild/buf/private/pkg/normalpath
13712026-03-11T19:42:13.440Zdatabase/sql/driver
13722026-03-11T19:42:13.443Zgithub.com/bufbuild/buf/private/pkg/osextended
13732026-03-11T19:42:13.443Zgithub.com/bgentry/go-netrc/netrc
13742026-03-11T19:42:13.443Zgoogle.golang.org/protobuf/internal/detrand
13752026-03-11T19:42:13.443Zos/user
13762026-03-11T19:42:13.443Zgithub.com/bufbuild/buf/private/pkg/interrupt
13772026-03-11T19:42:13.447Zgo.uber.org/multierr
13782026-03-11T19:42:13.449Zgithub.com/klauspost/compress/flate
13792026-03-11T19:42:13.452Zgoogle.golang.org/protobuf/internal/errors
13802026-03-11T19:42:13.452Zgithub.com/klauspost/compress/fse
13812026-03-11T19:42:13.452Zgithub.com/jhump/protoreflect/desc/protoparse/ast
13822026-03-11T19:42:13.456Ztext/scanner
13832026-03-11T19:42:13.456Zgithub.com/bufbuild/buf/private/pkg/storage/storageutil
13842026-03-11T19:42:13.456Zgithub.com/bufbuild/buf/private/bufpkg/bufmodule/internal
13852026-03-11T19:42:13.459Zgo.opencensus.io/trace
13862026-03-11T19:42:13.462Zgolang.org/x/net/internal/timeseries
13872026-03-11T19:42:13.465Zgithub.com/bufbuild/buf/private/pkg/ioextended
13882026-03-11T19:42:13.470Zgithub.com/bufbuild/buf/private/buf/bufcheck
13892026-03-11T19:42:13.473Zgoogle.golang.org/protobuf/encoding/protowire
13902026-03-11T19:42:13.473Zgithub.com/bufbuild/buf/private/pkg/thread
13912026-03-11T19:42:13.480Zgo.uber.org/zap/zapcore
13922026-03-11T19:42:13.483Zgithub.com/gofrs/uuid
13932026-03-11T19:42:13.483Zgithub.com/bufbuild/buf/private/pkg/app
13942026-03-11T19:42:13.487Zgithub.com/bufbuild/buf/private/pkg/filepathextended
13952026-03-11T19:42:13.490Zgoogle.golang.org/protobuf/reflect/protoreflect
13962026-03-11T19:42:13.493Zhtml/template
13972026-03-11T19:42:13.496Zgithub.com/bufbuild/buf/private/pkg/storage
13982026-03-11T19:42:13.496Zgoogle.golang.org/grpc/internal/grpclog
13992026-03-11T19:42:13.496Zgithub.com/klauspost/compress/huff0
14002026-03-11T19:42:13.502Zgoogle.golang.org/grpc/attributes
14012026-03-11T19:42:13.508Zgoogle.golang.org/grpc/metadata
14022026-03-11T19:42:13.522Zgithub.com/bufbuild/buf/private/pkg/netrc
14032026-03-11T19:42:13.525Zgithub.com/bufbuild/buf/private/pkg/uuidutil
14042026-03-11T19:42:13.532Zgoogle.golang.org/grpc/grpclog
14052026-03-11T19:42:13.539Zgoogle.golang.org/grpc/codes
14062026-03-11T19:42:13.548Zgoogle.golang.org/grpc/internal/backoff
14072026-03-11T19:42:13.557Zgoogle.golang.org/grpc/internal/balancerload
14082026-03-11T19:42:13.558Zgithub.com/bufbuild/buf/private/pkg/tmp
14092026-03-11T19:42:13.598Zgoogle.golang.org/grpc/internal/envconfig
14102026-03-11T19:42:13.602Zgolang.org/x/text/transform
14112026-03-11T19:42:13.602Zgithub.com/bufbuild/buf/private/pkg/storage/storagemem
14122026-03-11T19:42:13.602Zgoogle.golang.org/protobuf/internal/encoding/messageset
14132026-03-11T19:42:13.602Zgoogle.golang.org/protobuf/internal/genid
14142026-03-11T19:42:13.602Zgolang.org/x/sys/unix
14152026-03-11T19:42:13.606Zgoogle.golang.org/protobuf/internal/order
14162026-03-11T19:42:13.615Zgoogle.golang.org/protobuf/internal/strs
14172026-03-11T19:42:13.618Zgoogle.golang.org/protobuf/runtime/protoiface
14182026-03-11T19:42:13.618Zgoogle.golang.org/protobuf/internal/descfmt
14192026-03-11T19:42:13.618Zgoogle.golang.org/protobuf/reflect/protoregistry
14202026-03-11T19:42:13.624Zgoogle.golang.org/protobuf/internal/descopts
14212026-03-11T19:42:13.627Zgithub.com/bufbuild/buf/private/pkg/storage/storageos
14222026-03-11T19:42:13.630Zgoogle.golang.org/grpc/connectivity
14232026-03-11T19:42:13.630Zgithub.com/klauspost/compress/zstd
14242026-03-11T19:42:13.633Zgoogle.golang.org/grpc/internal/credentials
14252026-03-11T19:42:13.636Zgolang.org/x/text/unicode/bidi
14262026-03-11T19:42:13.639Zgolang.org/x/text/unicode/norm
14272026-03-11T19:42:13.641Zgithub.com/bufbuild/buf/private/gen/data/datawkt
14282026-03-11T19:42:13.648Zgolang.org/x/net/http2/hpack
14292026-03-11T19:42:13.654Zgoogle.golang.org/grpc/keepalive
14302026-03-11T19:42:13.664Zgoogle.golang.org/protobuf/internal/encoding/text
14312026-03-11T19:42:13.683Zgoogle.golang.org/protobuf/internal/encoding/json
14322026-03-11T19:42:13.686Zgoogle.golang.org/grpc/stats
14332026-03-11T19:42:13.704Zgoogle.golang.org/grpc/tap
14342026-03-11T19:42:13.713Zgithub.com/klauspost/compress/zip
14352026-03-11T19:42:13.720Zgithub.com/klauspost/pgzip
14362026-03-11T19:42:13.722Zgolang.org/x/text/secure/bidirule
14372026-03-11T19:42:13.722Zgithub.com/bufbuild/buf/private/pkg/rpc
14382026-03-11T19:42:13.728Zgoogle.golang.org/protobuf/proto
14392026-03-11T19:42:13.728Zgoogle.golang.org/protobuf/internal/encoding/defval
14402026-03-11T19:42:13.737Zgithub.com/bufbuild/buf/private/pkg/rpc/rpcheader
14412026-03-11T19:42:13.743Zgo.opencensus.io/resource
14422026-03-11T19:42:13.768Zgithub.com/bufbuild/buf/private/pkg/encoding
14432026-03-11T19:42:13.774Zruntime/pprof
14442026-03-11T19:42:13.777Zgoogle.golang.org/grpc/encoding/gzip
14452026-03-11T19:42:13.781Zgithub.com/gofrs/flock
14462026-03-11T19:42:13.786Zgithub.com/bufbuild/buf/private/pkg/verbose
14472026-03-11T19:42:13.789Zgithub.com/bufbuild/buf/private/bufpkg/bufrpc
14482026-03-11T19:42:13.792Zgithub.com/bufbuild/buf/private/bufpkg/buftransport
14492026-03-11T19:42:13.803Zgithub.com/spf13/pflag
14502026-03-11T19:42:13.803Zgithub.com/bufbuild/buf/private/pkg/app/appverbose
14512026-03-11T19:42:13.806Zgolang.org/x/sync/errgroup
14522026-03-11T19:42:13.809Zgo.opencensus.io/metric/metricdata
14532026-03-11T19:42:13.812Zgolang.org/x/net/idna
14542026-03-11T19:42:13.817Zgo.opencensus.io/metric/metricproducer
14552026-03-11T19:42:13.825Zgithub.com/bufbuild/buf/private/pkg/app/appname
14562026-03-11T19:42:13.828Zgithub.com/bufbuild/buf/private/bufpkg/buflock
14572026-03-11T19:42:13.828Zgithub.com/bufbuild/buf/private/pkg/filelock
14582026-03-11T19:42:13.854Zgithub.com/pkg/profile
14592026-03-11T19:42:13.854Zgo.opencensus.io/tag
14602026-03-11T19:42:13.857Zgoogle.golang.org/protobuf/encoding/prototext
14612026-03-11T19:42:13.857Zgoogle.golang.org/protobuf/internal/filedesc
14622026-03-11T19:42:13.883Zgolang.org/x/net/http/httpguts
14632026-03-11T19:42:13.900Zgo.opencensus.io/stats/internal
14642026-03-11T19:42:13.908Zgo.opencensus.io/stats
14652026-03-11T19:42:13.913Zgithub.com/bufbuild/buf/private/pkg/cert/certclient
14662026-03-11T19:42:13.938Zgo.opencensus.io/stats/view
14672026-03-11T19:42:13.957Zgithub.com/bufbuild/buf/private/buf/bufapp
14682026-03-11T19:42:14.015Zgolang.org/x/term
14692026-03-11T19:42:14.019Zgoogle.golang.org/grpc/internal/syscall
14702026-03-11T19:42:14.019Zarchive/tar
14712026-03-11T19:42:14.037Zgithub.com/spf13/cobra
14722026-03-11T19:42:14.085Zgoogle.golang.org/protobuf/encoding/protojson
14732026-03-11T19:42:14.088Zgoogle.golang.org/protobuf/internal/encoding/tag
14742026-03-11T19:42:14.123Zgoogle.golang.org/protobuf/internal/impl
14752026-03-11T19:42:14.141Zgithub.com/bufbuild/buf/private/pkg/storage/storagearchive
14762026-03-11T19:42:14.145Zgithub.com/bufbuild/buf/private/pkg/app/appcmd
14772026-03-11T19:42:14.222Znet/http/httputil
14782026-03-11T19:42:14.225Zgithub.com/getsentry/sentry-go/internal/ratelimit
14792026-03-11T19:42:14.234Znet/http/httputil
14802026-03-11T19:42:14.234Zgithub.com/getsentry/sentry-go/internal/ratelimit
14812026-03-11T19:42:14.274Zgithub.com/getsentry/sentry-go/internal/debug
14822026-03-11T19:42:14.286Zgithub.com/getsentry/sentry-go
14832026-03-11T19:42:14.290Zgithub.com/getsentry/sentry-go
14842026-03-11T19:42:14.351Zgithub.com/bufbuild/buf/private/pkg/httpauth
14852026-03-11T19:42:14.357Zgo.uber.org/zap
14862026-03-11T19:42:14.357Zgithub.com/twitchtv/twirp/ctxsetters
14872026-03-11T19:42:14.357Zgithub.com/twitchtv/twirp
14882026-03-11T19:42:14.357Zgolang.org/x/net/trace
14892026-03-11T19:42:14.357Zgithub.com/bufbuild/buf/private/pkg/rpc/rpchttp
14902026-03-11T19:42:14.357Zgo.opencensus.io/trace/propagation
14912026-03-11T19:42:14.357Zgithub.com/bufbuild/buf/private/pkg/rpc/rpcauth
14922026-03-11T19:42:14.357Zgolang.org/x/net/http2
14932026-03-11T19:42:14.361Zgo.opencensus.io/plugin/ochttp/propagation/b3
14942026-03-11T19:42:14.375Zgo.opencensus.io/plugin/ochttp
14952026-03-11T19:42:14.387Zgithub.com/bufbuild/buf/private/pkg/rpc/rpctwirp
14962026-03-11T19:42:14.416Zgithub.com/bufbuild/buf/private/pkg/transport/twirp/twirpclient
14972026-03-11T19:42:14.430Zgithub.com/cockroachdb/errors/withstack
14982026-03-11T19:42:14.448Zgithub.com/cockroachdb/errors/withstack
14992026-03-11T19:42:14.453Zgithub.com/cockroachdb/errors/report
15002026-03-11T19:42:14.453Zgithub.com/cockroachdb/errors/errutil
15012026-03-11T19:42:14.465Zgithub.com/bufbuild/buf/private/pkg/git
15022026-03-11T19:42:14.465Zgithub.com/bufbuild/buf/private/pkg/zaputil
15032026-03-11T19:42:14.465Zgithub.com/cockroachdb/errors/report
15042026-03-11T19:42:14.467Zgithub.com/cockroachdb/errors/errutil
15052026-03-11T19:42:14.473Zgithub.com/bufbuild/buf/private/pkg/observability
15062026-03-11T19:42:14.477Zgithub.com/bufbuild/buf/private/pkg/app/applog
15072026-03-11T19:42:14.482Zgithub.com/cockroachdb/errors
15082026-03-11T19:42:14.485Zgithub.com/bufbuild/buf/private/pkg/transport/http/httpclient
15092026-03-11T19:42:14.485Zgithub.com/bufbuild/buf/private/pkg/observability/observabilityzap
15102026-03-11T19:42:14.491Zgithub.com/cockroachdb/errors
15112026-03-11T19:42:14.494Zgithub.com/bufbuild/buf/private/pkg/app/appflag
15122026-03-11T19:42:14.508Zgithub.com/cockroachdb/errors/oserror
15132026-03-11T19:42:14.515Zgithub.com/cockroachdb/cockroach/pkg/sql/opt/optgen/lang
15142026-03-11T19:42:14.518Zgithub.com/cockroachdb/cockroach/pkg/cmd/uptodate
15152026-03-11T19:42:14.564Zgoogle.golang.org/protobuf/internal/filetype
15162026-03-11T19:42:14.589Zgoogle.golang.org/protobuf/runtime/protoimpl
15172026-03-11T19:42:14.597Zgoogle.golang.org/protobuf/types/known/durationpb
15182026-03-11T19:42:14.597Zgoogle.golang.org/protobuf/types/known/anypb
15192026-03-11T19:42:14.597Zgoogle.golang.org/protobuf/types/known/emptypb
15202026-03-11T19:42:14.597Zgoogle.golang.org/protobuf/types/known/structpb
15212026-03-11T19:42:14.597Zgoogle.golang.org/protobuf/types/known/wrapperspb
15222026-03-11T19:42:14.597Zgoogle.golang.org/protobuf/types/known/sourcecontextpb
15232026-03-11T19:42:14.597Zgoogle.golang.org/protobuf/types/descriptorpb
15242026-03-11T19:42:14.597Zgoogle.golang.org/protobuf/types/dynamicpb
15252026-03-11T19:42:14.597Zgoogle.golang.org/protobuf/types/known/timestamppb
15262026-03-11T19:42:14.597Zgoogle.golang.org/protobuf/types/known/fieldmaskpb
15272026-03-11T19:42:14.614Zgithub.com/golang/protobuf/ptypes/empty
15282026-03-11T19:42:14.617Zgithub.com/golang/protobuf/ptypes/any
15292026-03-11T19:42:14.617Zgoogle.golang.org/genproto/googleapis/rpc/status
15302026-03-11T19:42:14.617Zgithub.com/bufbuild/buf/private/pkg/prototime
15312026-03-11T19:42:14.617Zgithub.com/bufbuild/buf/private/gen/proto/go/buf/alpha/module/v1alpha1
15322026-03-11T19:42:14.617Zgithub.com/golang/protobuf/ptypes/timestamp
15332026-03-11T19:42:14.620Zgithub.com/golang/protobuf/ptypes/duration
15342026-03-11T19:42:14.620Zgithub.com/golang/protobuf/ptypes/wrappers
15352026-03-11T19:42:14.622Zgoogle.golang.org/genproto/protobuf/field_mask
15362026-03-11T19:42:14.625Zgoogle.golang.org/protobuf/types/known/typepb
15372026-03-11T19:42:14.625Zgoogle.golang.org/genproto/protobuf/source_context
15382026-03-11T19:42:14.625Zgithub.com/cockroachdb/cockroach/pkg/sql/opt/optgen/cmd/optgen
15392026-03-11T19:42:14.625Zgithub.com/golang/protobuf/ptypes/struct
15402026-03-11T19:42:14.649Zgoogle.golang.org/protobuf/types/known/apipb
15412026-03-11T19:42:14.649Zgoogle.golang.org/genproto/protobuf/ptype
15422026-03-11T19:42:14.664Zgoogle.golang.org/protobuf/types/pluginpb
15432026-03-11T19:42:14.664Zgithub.com/bufbuild/buf/private/gen/proto/go/buf/alpha/image/v1
15442026-03-11T19:42:14.666Zgoogle.golang.org/protobuf/reflect/protodesc
15452026-03-11T19:42:14.666Zgithub.com/golang/protobuf/protoc-gen-go/descriptor
15462026-03-11T19:42:14.667Zgoogle.golang.org/genproto/protobuf/api
15472026-03-11T19:42:14.684Zgithub.com/bufbuild/buf/private/pkg/protodescriptor
15482026-03-11T19:42:14.687Zgithub.com/golang/protobuf/protoc-gen-go/plugin
15492026-03-11T19:42:14.699Zgithub.com/bufbuild/buf/private/bufpkg/bufmodule
15502026-03-11T19:42:14.699Zgithub.com/bufbuild/buf/private/pkg/protosource
15512026-03-11T19:42:14.726Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/arena.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/arena.c
15522026-03-11T19:42:14.729Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/jemalloc.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/jemalloc.c
15532026-03-11T19:42:14.729Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/atomic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/atomic.c
15542026-03-11T19:42:14.729Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/base.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/base.c
15552026-03-11T19:42:14.729Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/bitmap.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/bitmap.c
15562026-03-11T19:42:14.729Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/chunk.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/chunk.c
15572026-03-11T19:42:14.729Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/chunk_dss.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/chunk_dss.c
15582026-03-11T19:42:14.729Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/chunk_mmap.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/chunk_mmap.c
15592026-03-11T19:42:14.729Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/ckh.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/ckh.c
15602026-03-11T19:42:14.735Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/ctl.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/ctl.c
15612026-03-11T19:42:14.738Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/extent.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/extent.c
15622026-03-11T19:42:14.738Zgithub.com/golang/protobuf/proto
15632026-03-11T19:42:14.738Zgithub.com/bufbuild/buf/private/pkg/protoencoding
15642026-03-11T19:42:14.746Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/hash.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/hash.c
15652026-03-11T19:42:14.774Zgithub.com/bufbuild/buf/private/bufpkg/bufimage
15662026-03-11T19:42:14.777Zgithub.com/bufbuild/buf/private/bufpkg/bufmodule/bufmoduleconfig
15672026-03-11T19:42:14.777Zgithub.com/bufbuild/buf/private/bufpkg/bufmodule/bufmodulecache
15682026-03-11T19:42:14.777Zgithub.com/bufbuild/buf/private/buf/buffetch/internal
15692026-03-11T19:42:14.777Zgithub.com/bufbuild/buf/private/pkg/app/appproto
15702026-03-11T19:42:14.808Zgithub.com/bufbuild/buf/private/bufpkg/bufmodule/bufmodulebuild
15712026-03-11T19:42:14.815Z[ 1%] Building C object src/CMakeFiles/proj.dir/nad_init.c.o
15722026-03-11T19:42:14.818Zgithub.com/bufbuild/buf/private/bufpkg/bufimage/bufimagemodify
15732026-03-11T19:42:14.831Z[ 1%] Building CXX object CMakeFiles/geos.dir/src/algorithm/Angle.cpp.o
15742026-03-11T19:42:14.871Zgithub.com/bufbuild/buf/private/pkg/app/appproto/appprotoexec
15752026-03-11T19:42:14.874Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/huge.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/huge.c
15762026-03-11T19:42:14.878Z[ 1%] Building C object src/CMakeFiles/proj.dir/PJ_aea.c.o
15772026-03-11T19:42:14.882Zgithub.com/bufbuild/buf/private/buf/bufcheck/internal
15782026-03-11T19:42:14.884Zgithub.com/bufbuild/buf/private/bufpkg/bufimage/bufimageutil
15792026-03-11T19:42:14.890Z[ 2%] Building C object src/CMakeFiles/proj.dir/PJ_aeqd.c.o
15802026-03-11T19:42:14.901Zgithub.com/jhump/protoreflect/internal
15812026-03-11T19:42:14.901Zgithub.com/jhump/protoreflect/internal/codec
15822026-03-11T19:42:14.901Zgithub.com/golang/protobuf/jsonpb
15832026-03-11T19:42:14.901Zgoogle.golang.org/grpc/credentials
15842026-03-11T19:42:14.904Zgoogle.golang.org/grpc/encoding/proto
15852026-03-11T19:42:14.904Zgithub.com/golang/protobuf/ptypes
15862026-03-11T19:42:14.904Zgoogle.golang.org/grpc/binarylog/grpc_binarylog_v1
15872026-03-11T19:42:14.907Z[ 1%] Building CXX object CMakeFiles/geos.dir/src/algorithm/Area.cpp.o
15882026-03-11T19:42:14.907Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/mb.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/mb.c
15892026-03-11T19:42:14.913Zgithub.com/bufbuild/buf/private/pkg/app/appproto/appprotoos
15902026-03-11T19:42:14.919Z[ 2%] Building C object src/CMakeFiles/proj.dir/PJ_airy.c.o
15912026-03-11T19:42:14.922Zgithub.com/bufbuild/buf/private/buf/bufcheck/buflint/internal/buflintcheck
15922026-03-11T19:42:14.925Zgithub.com/bufbuild/buf/private/buf/bufcheck/bufbreaking/internal/bufbreakingcheck
15932026-03-11T19:42:14.950Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/mutex.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/mutex.c
15942026-03-11T19:42:14.955Zgoogle.golang.org/grpc/internal/status
15952026-03-11T19:42:14.955Zgithub.com/jhump/protoreflect/desc/internal
15962026-03-11T19:42:14.958Zgoogle.golang.org/grpc/resolver
15972026-03-11T19:42:14.958Zgoogle.golang.org/grpc/internal/channelz
15982026-03-11T19:42:14.958Zgoogle.golang.org/grpc/peer
15992026-03-11T19:42:14.958Z[ 3%] Building C object src/CMakeFiles/proj.dir/PJ_august.c.o
16002026-03-11T19:42:14.961Z[ 4%] Building C object src/CMakeFiles/proj.dir/PJ_aitoff.c.o
16012026-03-11T19:42:14.973Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/nstime.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/nstime.c
16022026-03-11T19:42:14.977Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/pages.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/pages.c
16032026-03-11T19:42:14.980Z[ 4%] Building C object src/CMakeFiles/proj.dir/PJ_bacon.c.o
16042026-03-11T19:42:14.983Zgoogle.golang.org/grpc/internal
16052026-03-11T19:42:14.983Zgoogle.golang.org/grpc/status
16062026-03-11T19:42:14.983Zgoogle.golang.org/grpc/internal/grpcutil
16072026-03-11T19:42:14.983Zgoogle.golang.org/grpc/balancer/grpclb/state
16082026-03-11T19:42:14.983Zgoogle.golang.org/grpc/internal/resolver/passthrough
16092026-03-11T19:42:14.983Zgoogle.golang.org/grpc/internal/transport/networktype
16102026-03-11T19:42:14.983Zgoogle.golang.org/grpc/internal/metadata
16112026-03-11T19:42:14.994Zgithub.com/jhump/protoreflect/desc
16122026-03-11T19:42:15.013Zgoogle.golang.org/grpc/internal/resolver/unix
16132026-03-11T19:42:15.028Zgoogle.golang.org/grpc/internal/binarylog
16142026-03-11T19:42:15.037Zgo.opencensus.io/plugin/ocgrpc
16152026-03-11T19:42:15.065Zgoogle.golang.org/grpc/internal/resolver/dns
16162026-03-11T19:42:15.097Zgoogle.golang.org/grpc/balancer
16172026-03-11T19:42:15.097Zgithub.com/bufbuild/buf/private/buf/bufcheck/buflint/internal/buflintbuild
16182026-03-11T19:42:15.102Zgithub.com/bufbuild/buf/private/buf/bufcheck/bufbreaking/internal/bufbreakingbuild
16192026-03-11T19:42:15.102Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/prng.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/prng.c
16202026-03-11T19:42:15.102Zgoogle.golang.org/grpc/internal/transport
16212026-03-11T19:42:15.105Z[ 5%] Building C object src/CMakeFiles/proj.dir/PJ_bipc.c.o
16222026-03-11T19:42:15.107Zgithub.com/jhump/protoreflect/codec
16232026-03-11T19:42:15.111Z[ 1%] Building CXX object CMakeFiles/geos.dir/src/algorithm/BoundaryNodeRule.cpp.o
16242026-03-11T19:42:15.114Z[ 5%] Building C object src/CMakeFiles/proj.dir/PJ_boggs.c.o
16252026-03-11T19:42:15.114Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/prof.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/prof.c
16262026-03-11T19:42:15.124Zgoogle.golang.org/grpc/balancer/base
16272026-03-11T19:42:15.124Zgoogle.golang.org/grpc/internal/serviceconfig
16282026-03-11T19:42:15.127Zgithub.com/bufbuild/buf/private/buf/bufcheck/bufbreaking/internal/bufbreakingv1
16292026-03-11T19:42:15.129Z[ 1%] Building CXX object CMakeFiles/geos.dir/src/algorithm/CGAlgorithmsDD.cpp.o
16302026-03-11T19:42:15.129Zgithub.com/bufbuild/buf/private/buf/bufcheck/bufbreaking/internal/bufbreakingv1beta1
16312026-03-11T19:42:15.133Z[ 1%] Building CXX object CMakeFiles/geos.dir/src/algorithm/Centroid.cpp.o
16322026-03-11T19:42:15.135Zoptgen -out pkg/sql/opt/memo/expr.og.go exprs pkg/sql/opt/ops/*.opt
16332026-03-11T19:42:15.135Zgoogle.golang.org/grpc/internal/resolver
16342026-03-11T19:42:15.136Z[ 6%] Building C object src/CMakeFiles/proj.dir/PJ_bonne.c.o
16352026-03-11T19:42:15.146Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/quarantine.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/quarantine.c
16362026-03-11T19:42:15.156Zgoogle.golang.org/grpc/balancer/roundrobin
16372026-03-11T19:42:15.156Zgithub.com/bufbuild/buf/private/buf/bufcheck/buflint/internal/buflintv1
16382026-03-11T19:42:15.156Zgithub.com/bufbuild/buf/private/buf/bufcheck/buflint/internal/buflintv1beta1
16392026-03-11T19:42:15.159Z[ 6%] Building C object src/CMakeFiles/proj.dir/PJ_calcofi.c.o
16402026-03-11T19:42:15.159Zgithub.com/jhump/protoreflect/dynamic
16412026-03-11T19:42:15.165Zgithub.com/bufbuild/buf/private/buf/bufcheck/buflint/buflintconfig
16422026-03-11T19:42:15.168Zgithub.com/bufbuild/buf/private/buf/bufcheck/bufbreaking/bufbreakingconfig
16432026-03-11T19:42:15.176Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/rtree.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/rtree.c
16442026-03-11T19:42:15.179Z[ 7%] Building C object src/CMakeFiles/proj.dir/PJ_cass.c.o
16452026-03-11T19:42:15.193Z[ 7%] Building C object src/CMakeFiles/proj.dir/PJ_cc.c.o
16462026-03-11T19:42:15.197Zgithub.com/bufbuild/buf/private/buf/bufcheck/bufbreaking
16472026-03-11T19:42:15.207Zoptgen -out pkg/sql/opt/operator.og.go ops pkg/sql/opt/ops/*.opt
16482026-03-11T19:42:15.221Zgithub.com/bufbuild/buf/private/buf/bufcheck/buflint
16492026-03-11T19:42:15.221Zgithub.com/bufbuild/buf/private/buf/bufconfig
16502026-03-11T19:42:15.224Zoptgen -out pkg/sql/opt/xform/explorer.og.go explorer pkg/sql/opt/ops/*.opt pkg/sql/opt/xform/rules/*.opt
16512026-03-11T19:42:15.224Zoptgen -out pkg/sql/opt/norm/factory.og.go factory pkg/sql/opt/ops/*.opt pkg/sql/opt/norm/rules/*.opt
16522026-03-11T19:42:15.227Zoptgen -out pkg/sql/opt/rule_name.og.go rulenames pkg/sql/opt/ops/*.opt pkg/sql/opt/norm/rules/*.opt pkg/sql/opt/xform/rules/*.opt
16532026-03-11T19:42:15.244Zoptgen -out pkg/sql/opt/exec/factory.og.go execfactory pkg/sql/opt/exec/factory.opt
16542026-03-11T19:42:15.257Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/config/internal
16552026-03-11T19:42:15.257Zgithub.com/bufbuild/buf/private/buf/bufwork
16562026-03-11T19:42:15.260Zoptgen -out pkg/sql/opt/exec/explain/explain_factory.og.go execexplain pkg/sql/opt/exec/factory.opt
16572026-03-11T19:42:15.263Zoptgen -out pkg/sql/opt/exec/explain/plan_gist_factory.og.go execplangist pkg/sql/opt/exec/factory.opt
16582026-03-11T19:42:15.279Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/stats.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/stats.c
16592026-03-11T19:42:15.281Z[ 3%] Building CXX object CMakeFiles/geos.dir/src/algorithm/ConvexHull.cpp.o
16602026-03-11T19:42:15.297Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/config/configlsbreakingrules
16612026-03-11T19:42:15.300Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/config/configlslintrules
16622026-03-11T19:42:15.305Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/spin.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/spin.c
16632026-03-11T19:42:15.308Z[ 8%] Building C object src/CMakeFiles/proj.dir/PJ_cea.c.o
16642026-03-11T19:42:15.311Zgithub.com/bufbuild/buf/private/buf/buffetch
16652026-03-11T19:42:15.317Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/tcache.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/tcache.c
16662026-03-11T19:42:15.320Z[ 3%] Building CXX object CMakeFiles/geos.dir/src/algorithm/HCoordinate.cpp.o
16672026-03-11T19:42:15.322Z[ 3%] Building CXX object CMakeFiles/geos.dir/src/algorithm/InteriorPointArea.cpp.o
16682026-03-11T19:42:15.326Z[ 3%] Building CXX object CMakeFiles/geos.dir/src/algorithm/Distance.cpp.o
16692026-03-11T19:42:15.331Z[ 9%] Building C object src/CMakeFiles/proj.dir/PJ_chamb.c.o
16702026-03-11T19:42:15.400Z[ 9%] Building C object src/CMakeFiles/proj.dir/PJ_collg.c.o
16712026-03-11T19:42:15.407Zgithub.com/jhump/protoreflect/desc/protoparse
16722026-03-11T19:42:15.407Zgoogle.golang.org/grpc
16732026-03-11T19:42:15.407Z[ 10%] Building C object src/CMakeFiles/proj.dir/PJ_comill.c.o
16742026-03-11T19:42:15.415Z[ 3%] Building CXX object CMakeFiles/geos.dir/src/algorithm/InteriorPointLine.cpp.o
16752026-03-11T19:42:15.425Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/ticker.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/ticker.c
16762026-03-11T19:42:15.438Z[ 10%] Building C object src/CMakeFiles/proj.dir/PJ_crast.c.o
16772026-03-11T19:42:15.476Z[ 11%] Building C object src/CMakeFiles/proj.dir/PJ_denoy.c.o
16782026-03-11T19:42:15.494Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/tsd.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/tsd.c
16792026-03-11T19:42:15.505Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/util.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c
16802026-03-11T19:42:15.558Z[ 11%] Building C object src/CMakeFiles/proj.dir/PJ_eck1.c.o
16812026-03-11T19:42:15.586Z[ 4%] Building CXX object CMakeFiles/geos.dir/src/algorithm/InteriorPointPoint.cpp.o
16822026-03-11T19:42:15.600Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/witness.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/witness.c
16832026-03-11T19:42:15.603Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/jemalloc.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/jemalloc.c
16842026-03-11T19:42:15.603Z/work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c: In function 'je_malloc_vsnprintf':
16852026-03-11T19:42:15.603Z/work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c:359:9: warning: case label value exceeds maximum value for type [-Wswitch-outside-range]
16862026-03-11T19:42:15.603Z 359 | case '?' | 0x80: \
16872026-03-11T19:42:15.603Z | ^~~~
16882026-03-11T19:42:15.603Z/work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c:569:33: note: in expansion of macro 'GET_ARG_NUMERIC'
16892026-03-11T19:42:15.603Z 569 | GET_ARG_NUMERIC(val, 'p');
16902026-03-11T19:42:15.604Z | ^~~~~~~~~~~~~~~
16912026-03-11T19:42:15.604Z/work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c:377:9: warning: case label value exceeds maximum value for type [-Wswitch-outside-range]
16922026-03-11T19:42:15.604Z 377 | case 'j' | 0x80: \
16932026-03-11T19:42:15.604Z | ^~~~
16942026-03-11T19:42:15.604Z/work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c:569:33: note: in expansion of macro 'GET_ARG_NUMERIC'
16952026-03-11T19:42:15.604Z 569 | GET_ARG_NUMERIC(val, 'p');
16962026-03-11T19:42:15.604Z | ^~~~~~~~~~~~~~~
16972026-03-11T19:42:15.604Z/work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c:365:9: warning: case label value exceeds maximum value for type [-Wswitch-outside-range]
16982026-03-11T19:42:15.604Z 365 | case 'l' | 0x80: \
16992026-03-11T19:42:15.604Z | ^~~~
17002026-03-11T19:42:15.604Z/work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c:569:33: note: in expansion of macro 'GET_ARG_NUMERIC'
17012026-03-11T19:42:15.604Z 569 | GET_ARG_NUMERIC(val, 'p');
17022026-03-11T19:42:15.604Z | ^~~~~~~~~~~~~~~
17032026-03-11T19:42:15.604Z/work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c:371:9: warning: case label value exceeds maximum value for type [-Wswitch-outside-range]
17042026-03-11T19:42:15.604Z 371 | case 'q' | 0x80: \
17052026-03-11T19:42:15.604Z | ^~~~
17062026-03-11T19:42:15.604Z/work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c:569:33: note: in expansion of macro 'GET_ARG_NUMERIC'
17072026-03-11T19:42:15.604Z 569 | GET_ARG_NUMERIC(val, 'p');
17082026-03-11T19:42:15.604Z | ^~~~~~~~~~~~~~~
17092026-03-11T19:42:15.604Z/work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c:386:9: warning: case label value exceeds maximum value for type [-Wswitch-outside-range]
17102026-03-11T19:42:15.604Z 386 | case 'z' | 0x80: \
17112026-03-11T19:42:15.604Z | ^~~~
17122026-03-11T19:42:15.604Z/work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c:569:33: note: in expansion of macro 'GET_ARG_NUMERIC'
17132026-03-11T19:42:15.604Z 569 | GET_ARG_NUMERIC(val, 'p');
17142026-03-11T19:42:15.604Z | ^~~~~~~~~~~~~~~
17152026-03-11T19:42:15.612Z[ 4%] Building CXX object CMakeFiles/geos.dir/src/algorithm/Intersection.cpp.o
17162026-03-11T19:42:15.616Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/arena.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/arena.c
17172026-03-11T19:42:15.665Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/atomic.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/atomic.c
17182026-03-11T19:42:15.668Z[ 13%] Building C object src/CMakeFiles/proj.dir/PJ_eck2.c.o
17192026-03-11T19:42:15.679Z[ 13%] Building C object src/CMakeFiles/proj.dir/PJ_eck3.c.o
17202026-03-11T19:42:15.689Z[ 4%] Building CXX object CMakeFiles/geos.dir/src/algorithm/Length.cpp.o
17212026-03-11T19:42:15.747Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/base.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/base.c
17222026-03-11T19:42:16.753Z[ 14%] Building C object src/CMakeFiles/proj.dir/PJ_eck4.c.o
17232026-03-11T19:42:16.758Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/bitmap.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/bitmap.c
17242026-03-11T19:42:16.758Zgithub.com/bufbuild/buf/private/gen/proto/go/buf/alpha/registry/v1alpha1
17252026-03-11T19:42:16.758Zgithub.com/bufbuild/buf/private/pkg/rpc/rpcgrpc
17262026-03-11T19:42:16.758Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/chunk.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/chunk.c
17272026-03-11T19:42:16.758Z[ 4%] Building CXX object CMakeFiles/geos.dir/src/algorithm/LineIntersector.cpp.o
17282026-03-11T19:42:16.758Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/chunk_dss.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/chunk_dss.c
17292026-03-11T19:42:16.758Z[ 14%] Building C object src/CMakeFiles/proj.dir/PJ_eck5.c.o
17302026-03-11T19:42:16.758Zgithub.com/bufbuild/buf/private/pkg/transport/grpc/grpcclient
17312026-03-11T19:42:16.758Zgithub.com/bufbuild/buf/private/bufpkg/bufmodule/bufmoduleprotoparse
17322026-03-11T19:42:16.758Z[ 15%] Building C object src/CMakeFiles/proj.dir/PJ_eqc.c.o
17332026-03-11T19:42:16.758Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/chunk_mmap.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/chunk_mmap.c
17342026-03-11T19:42:16.758Zgithub.com/bufbuild/buf/private/bufpkg/bufimage/bufimagebuild
17352026-03-11T19:42:16.758Z[ 4%] Building CXX object CMakeFiles/geos.dir/src/algorithm/MinimumBoundingCircle.cpp.o
17362026-03-11T19:42:16.758Z[ 15%] Building C object src/CMakeFiles/proj.dir/PJ_fahey.c.o
17372026-03-11T19:42:16.758Z[ 16%] Building C object src/CMakeFiles/proj.dir/PJ_eqdc.c.o
17382026-03-11T19:42:16.758Z[ 6%] Building CXX object CMakeFiles/geos.dir/src/algorithm/MinimumDiameter.cpp.o
17392026-03-11T19:42:16.758Z[ 17%] Building C object src/CMakeFiles/proj.dir/PJ_fouc_s.c.o
17402026-03-11T19:42:16.758Zgithub.com/bufbuild/buf/private/buf/bufwire
17412026-03-11T19:42:16.758Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/ckh.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/ckh.c
17422026-03-11T19:42:16.758Z[ 6%] Building CXX object CMakeFiles/geos.dir/src/algorithm/NotRepresentableException.cpp.o
17432026-03-11T19:42:16.758Z[ 17%] Building C object src/CMakeFiles/proj.dir/PJ_gall.c.o
17442026-03-11T19:42:16.758Z[ 6%] Building CXX object CMakeFiles/geos.dir/src/algorithm/Orientation.cpp.o
17452026-03-11T19:42:16.758Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/ctl.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/ctl.c
17462026-03-11T19:42:16.758Z[ 18%] Building C object src/CMakeFiles/proj.dir/PJ_geos.c.o
17472026-03-11T19:42:16.758Z[ 6%] Building CXX object CMakeFiles/geos.dir/src/algorithm/PointLocation.cpp.o
17482026-03-11T19:42:16.758Z[ 18%] Building C object src/CMakeFiles/proj.dir/PJ_gins8.c.o
17492026-03-11T19:42:16.758Z[ 19%] Building C object src/CMakeFiles/proj.dir/PJ_gnom.c.o
17502026-03-11T19:42:16.758Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/extent.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/extent.c
17512026-03-11T19:42:16.758Z[ 19%] Building C object src/CMakeFiles/proj.dir/PJ_gn_sinu.c.o
17522026-03-11T19:42:16.758Z[ 20%] Building C object src/CMakeFiles/proj.dir/PJ_goode.c.o
17532026-03-11T19:42:16.758Z[ 21%] Building C object src/CMakeFiles/proj.dir/PJ_gstmerc.c.o
17542026-03-11T19:42:16.758Z[ 21%] Building C object src/CMakeFiles/proj.dir/PJ_hammer.c.o
17552026-03-11T19:42:16.758Z[ 6%] Building CXX object CMakeFiles/geos.dir/src/algorithm/PointLocator.cpp.o
17562026-03-11T19:42:16.758Z[ 22%] Building C object src/CMakeFiles/proj.dir/PJ_hatano.c.o
17572026-03-11T19:42:16.758Z[ 8%] Building CXX object CMakeFiles/geos.dir/src/algorithm/RayCrossingCounter.cpp.o
17582026-03-11T19:42:16.759Z[ 23%] Building C object src/CMakeFiles/proj.dir/PJ_isea.c.o
17592026-03-11T19:42:16.759Z[ 23%] Building C object src/CMakeFiles/proj.dir/PJ_igh.c.o
17602026-03-11T19:42:16.759Z[ 23%] Building C object src/CMakeFiles/proj.dir/PJ_imw_p.c.o
17612026-03-11T19:42:16.759Z[ 8%] Building CXX object CMakeFiles/geos.dir/src/algorithm/RayCrossingCounterDD.cpp.o
17622026-03-11T19:42:16.759Z[ 8%] Building CXX object CMakeFiles/geos.dir/src/algorithm/RobustDeterminant.cpp.o
17632026-03-11T19:42:16.759Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/hash.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/hash.c
17642026-03-11T19:42:16.759Z[ 25%] Building C object src/CMakeFiles/proj.dir/PJ_krovak.c.o
17652026-03-11T19:42:16.759Z[ 25%] Building C object src/CMakeFiles/proj.dir/PJ_labrd.c.o
17662026-03-11T19:42:16.759Z[ 26%] Building C object src/CMakeFiles/proj.dir/PJ_laea.c.o
17672026-03-11T19:42:16.759Z[ 27%] Building C object src/CMakeFiles/proj.dir/PJ_larr.c.o
17682026-03-11T19:42:16.759Z[ 27%] Building C object src/CMakeFiles/proj.dir/PJ_lagrng.c.o
17692026-03-11T19:42:16.759Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/huge.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/huge.c
17702026-03-11T19:42:16.759Z[ 8%] Building CXX object CMakeFiles/geos.dir/src/algorithm/SimplePointInRing.cpp.o
17712026-03-11T19:42:16.759Z[ 28%] Building C object src/CMakeFiles/proj.dir/PJ_lask.c.o
17722026-03-11T19:42:16.759Z[ 28%] Building C object src/CMakeFiles/proj.dir/PJ_lcca.c.o
17732026-03-11T19:42:16.759Z[ 29%] Building C object src/CMakeFiles/proj.dir/PJ_lcc.c.o
17742026-03-11T19:42:16.759Z[ 8%] Building CXX object CMakeFiles/geos.dir/src/algorithm/distance/DiscreteFrechetDistance.cpp.o
17752026-03-11T19:42:16.759Z[ 29%] Building C object src/CMakeFiles/proj.dir/PJ_loxim.c.o
17762026-03-11T19:42:16.759Z[ 9%] Building CXX object CMakeFiles/geos.dir/src/algorithm/distance/DiscreteHausdorffDistance.cpp.o
17772026-03-11T19:42:16.759Z[ 9%] Building CXX object CMakeFiles/geos.dir/src/algorithm/distance/DistanceToPoint.cpp.o
17782026-03-11T19:42:16.759Z[ 30%] Building C object src/CMakeFiles/proj.dir/PJ_lsat.c.o
17792026-03-11T19:42:16.759Z[ 30%] Building C object src/CMakeFiles/proj.dir/PJ_misrsom.c.o
17802026-03-11T19:42:16.759Z[ 31%] Building C object src/CMakeFiles/proj.dir/PJ_mbt_fps.c.o
17812026-03-11T19:42:16.759Z[ 9%] Building CXX object CMakeFiles/geos.dir/src/algorithm/locate/IndexedPointInAreaLocator.cpp.o
17822026-03-11T19:42:16.759Z[ 31%] Building C object src/CMakeFiles/proj.dir/PJ_mbtfpp.c.o
17832026-03-11T19:42:16.759Z[ 32%] Building C object src/CMakeFiles/proj.dir/PJ_mbtfpq.c.o
17842026-03-11T19:42:16.759Z[ 9%] Building CXX object CMakeFiles/geos.dir/src/algorithm/locate/PointOnGeometryLocator.cpp.o
17852026-03-11T19:42:16.759Z[ 33%] Building C object src/CMakeFiles/proj.dir/PJ_merc.c.o
17862026-03-11T19:42:16.759Z[ 9%] Building CXX object CMakeFiles/geos.dir/src/algorithm/locate/SimplePointInAreaLocator.cpp.o
17872026-03-11T19:42:16.759Z[ 34%] Building C object src/CMakeFiles/proj.dir/PJ_mod_ster.c.o
17882026-03-11T19:42:16.759Z[ 34%] Building C object src/CMakeFiles/proj.dir/PJ_mill.c.o
17892026-03-11T19:42:16.759Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/mb.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/mb.c
17902026-03-11T19:42:16.759Z[ 34%] Building C object src/CMakeFiles/proj.dir/PJ_moll.c.o
17912026-03-11T19:42:16.759Z[ 11%] Building CXX object CMakeFiles/geos.dir/src/geom/Coordinate.cpp.o
17922026-03-11T19:42:16.759Z[ 35%] Building C object src/CMakeFiles/proj.dir/PJ_natearth.c.o
17932026-03-11T19:42:16.759Z[ 35%] Building C object src/CMakeFiles/proj.dir/PJ_natearth2.c.o
17942026-03-11T19:42:16.759Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/mutex.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/mutex.c
17952026-03-11T19:42:16.759Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/nstime.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/nstime.c
17962026-03-11T19:42:16.759Z[ 36%] Building C object src/CMakeFiles/proj.dir/PJ_nell.c.o
17972026-03-11T19:42:16.760Z[ 11%] Building CXX object CMakeFiles/geos.dir/src/geom/CoordinateArraySequence.cpp.o
17982026-03-11T19:42:16.760Z[ 36%] Building C object src/CMakeFiles/proj.dir/PJ_nell_h.c.o
17992026-03-11T19:42:16.760Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/pages.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/pages.c
18002026-03-11T19:42:16.760Z[ 38%] Building C object src/CMakeFiles/proj.dir/PJ_nocol.c.o
18012026-03-11T19:42:16.760Z[ 38%] Building C object src/CMakeFiles/proj.dir/PJ_nsper.c.o
18022026-03-11T19:42:16.760Z[ 39%] Building C object src/CMakeFiles/proj.dir/PJ_nzmg.c.o
18032026-03-11T19:42:16.760Z[ 11%] Building CXX object CMakeFiles/geos.dir/src/geom/CoordinateArraySequenceFactory.cpp.o
18042026-03-11T19:42:16.760Z[ 40%] Building C object src/CMakeFiles/proj.dir/PJ_ob_tran.c.o
18052026-03-11T19:42:16.760Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/prng.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/prng.c
18062026-03-11T19:42:16.760Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/prof.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/prof.c
18072026-03-11T19:42:16.760Z[ 40%] Building C object src/CMakeFiles/proj.dir/PJ_ocea.c.o
18082026-03-11T19:42:16.760Z[ 41%] Building C object src/CMakeFiles/proj.dir/PJ_oea.c.o
18092026-03-11T19:42:16.760Z[ 11%] Building CXX object CMakeFiles/geos.dir/src/geom/CoordinateSequence.cpp.o
18102026-03-11T19:42:16.760Z[ 41%] Building C object src/CMakeFiles/proj.dir/PJ_omerc.c.o
18112026-03-11T19:42:16.760Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/quarantine.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/quarantine.c
18122026-03-11T19:42:16.760Z[ 41%] Building C object src/CMakeFiles/proj.dir/PJ_patterson.c.o
18132026-03-11T19:42:16.760Z[ 42%] Building C object src/CMakeFiles/proj.dir/PJ_ortho.c.o
18142026-03-11T19:42:16.760Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/rtree.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/rtree.c
18152026-03-11T19:42:16.760Z[ 43%] Building C object src/CMakeFiles/proj.dir/PJ_poly.c.o
18162026-03-11T19:42:16.760Z[ 11%] Building CXX object CMakeFiles/geos.dir/src/geom/DefaultCoordinateSequenceFactory.cpp.o
18172026-03-11T19:42:16.760Z[ 13%] Building CXX object CMakeFiles/geos.dir/src/geom/Dimension.cpp.o
18182026-03-11T19:42:16.760Z[ 43%] Building C object src/CMakeFiles/proj.dir/PJ_putp2.c.o
18192026-03-11T19:42:16.760Z[ 44%] Building C object src/CMakeFiles/proj.dir/PJ_putp3.c.o
18202026-03-11T19:42:16.760Z[ 45%] Building C object src/CMakeFiles/proj.dir/PJ_putp4p.c.o
18212026-03-11T19:42:16.760Z[ 13%] Building CXX object CMakeFiles/geos.dir/src/geom/Envelope.cpp.o
18222026-03-11T19:42:16.760Z[ 45%] Building C object src/CMakeFiles/proj.dir/PJ_putp5.c.o
18232026-03-11T19:42:16.764Z[ 45%] Building C object src/CMakeFiles/proj.dir/PJ_qsc.c.o
18242026-03-11T19:42:16.764Z[ 46%] Building C object src/CMakeFiles/proj.dir/PJ_putp6.c.o
18252026-03-11T19:42:16.764Z[ 47%] Building C object src/CMakeFiles/proj.dir/PJ_robin.c.o
18262026-03-11T19:42:16.764Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/stats.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/stats.c
18272026-03-11T19:42:16.764Z[ 13%] Building CXX object CMakeFiles/geos.dir/src/geom/Geometry.cpp.o
18282026-03-11T19:42:16.764Z[ 47%] Building C object src/CMakeFiles/proj.dir/PJ_rpoly.c.o
18292026-03-11T19:42:16.764Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/spin.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/spin.c
18302026-03-11T19:42:16.764Z[ 48%] Building C object src/CMakeFiles/proj.dir/PJ_sconics.c.o
18312026-03-11T19:42:16.764Z[ 48%] Building C object src/CMakeFiles/proj.dir/PJ_sch.c.o
18322026-03-11T19:42:16.764Z[ 50%] Building C object src/CMakeFiles/proj.dir/PJ_somerc.c.o
18332026-03-11T19:42:16.764Z[ 50%] Building C object src/CMakeFiles/proj.dir/PJ_sterea.c.o
18342026-03-11T19:42:16.764Z[ 51%] Building C object src/CMakeFiles/proj.dir/PJ_stere.c.o
18352026-03-11T19:42:16.764Z[ 52%] Building C object src/CMakeFiles/proj.dir/PJ_sts.c.o
18362026-03-11T19:42:16.764Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/tcache.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/tcache.c
18372026-03-11T19:42:16.764Z[ 52%] Building C object src/CMakeFiles/proj.dir/PJ_tcc.c.o
18382026-03-11T19:42:16.764Z[ 53%] Building C object src/CMakeFiles/proj.dir/PJ_tcea.c.o
18392026-03-11T19:42:16.775Z[ 53%] Building C object src/CMakeFiles/proj.dir/PJ_times.c.o
18402026-03-11T19:42:16.781Z[ 54%] Building C object src/CMakeFiles/proj.dir/PJ_tmerc.c.o
18412026-03-11T19:42:16.785Z[ 54%] Building C object src/CMakeFiles/proj.dir/PJ_tpeqd.c.o
18422026-03-11T19:42:16.788Z[ 13%] Building CXX object CMakeFiles/geos.dir/src/geom/GeometryCollection.cpp.o
18432026-03-11T19:42:16.791Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/ticker.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/ticker.c
18442026-03-11T19:42:16.794Z[ 55%] Building C object src/CMakeFiles/proj.dir/PJ_urm5.c.o
18452026-03-11T19:42:16.794Z[ 55%] Building C object src/CMakeFiles/proj.dir/PJ_urmfps.c.o
18462026-03-11T19:42:16.816Z[ 13%] Building CXX object CMakeFiles/geos.dir/src/geom/GeometryComponentFilter.cpp.o
18472026-03-11T19:42:16.821Z[ 56%] Building C object src/CMakeFiles/proj.dir/PJ_vandg2.c.o
18482026-03-11T19:42:16.821Z[ 56%] Building C object src/CMakeFiles/proj.dir/PJ_vandg.c.o
18492026-03-11T19:42:16.826Z[ 57%] Building C object src/CMakeFiles/proj.dir/PJ_vandg4.c.o
18502026-03-11T19:42:16.831Z[ 58%] Building C object src/CMakeFiles/proj.dir/PJ_wag2.c.o
18512026-03-11T19:42:16.835Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/tsd.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/tsd.c
18522026-03-11T19:42:16.853Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/util.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c
18532026-03-11T19:42:16.858Zgcc -std=gnu11 -Wall -Werror=declaration-after-statement -Wsign-compare -pipe -g3 -fvisibility=hidden -O3 -funroll-loops -g1 -fPIC -DPIC -c -D_GNU_SOURCE -D_REENTRANT -I/work/oxidecomputer/cockroach/c-deps/jemalloc/include -Iinclude -o src/witness.pic.o /work/oxidecomputer/cockroach/c-deps/jemalloc/src/witness.c
18542026-03-11T19:42:16.858Z[ 58%] Building C object src/CMakeFiles/proj.dir/PJ_wag3.c.o
18552026-03-11T19:42:16.858Z[ 59%] Building C object src/CMakeFiles/proj.dir/PJ_wag7.c.o
18562026-03-11T19:42:16.861Z[ 60%] Building C object src/CMakeFiles/proj.dir/PJ_wink2.c.o
18572026-03-11T19:42:16.861Z[ 60%] Building C object src/CMakeFiles/proj.dir/PJ_wink1.c.o
18582026-03-11T19:42:16.886Z[ 60%] Building C object src/CMakeFiles/proj.dir/proj_etmerc.c.o
18592026-03-11T19:42:16.890Z[ 14%] Building CXX object CMakeFiles/geos.dir/src/geom/GeometryFactory.cpp.o
18602026-03-11T19:42:16.892Z[ 61%] Building C object src/CMakeFiles/proj.dir/aasincos.c.o
18612026-03-11T19:42:16.896Z[ 14%] Building CXX object CMakeFiles/geos.dir/src/geom/IntersectionMatrix.cpp.o
18622026-03-11T19:42:16.899Z/work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c: In function 'je_malloc_vsnprintf':
18632026-03-11T19:42:16.899Z/work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c:359:9: warning: case label value exceeds maximum value for type [-Wswitch-outside-range]
18642026-03-11T19:42:16.899Z 359 | case '?' | 0x80: \
18652026-03-11T19:42:16.899Z | ^~~~
18662026-03-11T19:42:16.899Z/work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c:569:33: note: in expansion of macro 'GET_ARG_NUMERIC'
18672026-03-11T19:42:16.899Z 569 | GET_ARG_NUMERIC(val, 'p');
18682026-03-11T19:42:16.899Z | ^~~~~~~~~~~~~~~
18692026-03-11T19:42:16.899Z/work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c:377:9: warning: case label value exceeds maximum value for type [-Wswitch-outside-range]
18702026-03-11T19:42:16.899Z 377 | case 'j' | 0x80: \
18712026-03-11T19:42:16.899Z | ^~~~
18722026-03-11T19:42:16.900Z/work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c:569:33: note: in expansion of macro 'GET_ARG_NUMERIC'
18732026-03-11T19:42:16.900Z 569 | GET_ARG_NUMERIC(val, 'p');
18742026-03-11T19:42:16.900Z | ^~~~~~~~~~~~~~~
18752026-03-11T19:42:16.900Z/work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c:365:9: warning: case label value exceeds maximum value for type [-Wswitch-outside-range]
18762026-03-11T19:42:16.900Z 365 | case 'l' | 0x80: \
18772026-03-11T19:42:16.900Z | ^~~~
18782026-03-11T19:42:16.900Z/work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c:569:33: note: in expansion of macro 'GET_ARG_NUMERIC'
18792026-03-11T19:42:16.900Z 569 | GET_ARG_NUMERIC(val, 'p');
18802026-03-11T19:42:16.900Z | ^~~~~~~~~~~~~~~
18812026-03-11T19:42:16.900Z/work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c:371:9: warning: case label value exceeds maximum value for type [-Wswitch-outside-range]
18822026-03-11T19:42:16.900Z 371 | case 'q' | 0x80: \
18832026-03-11T19:42:16.900Z | ^~~~
18842026-03-11T19:42:16.900Z/work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c:569:33: note: in expansion of macro 'GET_ARG_NUMERIC'
18852026-03-11T19:42:16.900Z 569 | GET_ARG_NUMERIC(val, 'p');
18862026-03-11T19:42:16.900Z | ^~~~~~~~~~~~~~~
18872026-03-11T19:42:16.900Z/work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c:386:9: warning: case label value exceeds maximum value for type [-Wswitch-outside-range]
18882026-03-11T19:42:16.900Z 386 | case 'z' | 0x80: \
18892026-03-11T19:42:16.900Z | ^~~~
18902026-03-11T19:42:16.900Z/work/oxidecomputer/cockroach/c-deps/jemalloc/src/util.c:569:33: note: in expansion of macro 'GET_ARG_NUMERIC'
18912026-03-11T19:42:16.900Z 569 | GET_ARG_NUMERIC(val, 'p');
18922026-03-11T19:42:16.900Z | ^~~~~~~~~~~~~~~
18932026-03-11T19:42:16.903Zar crus lib/libjemalloc.a src/jemalloc.o src/arena.o src/atomic.o src/base.o src/bitmap.o src/chunk.o src/chunk_dss.o src/chunk_mmap.o src/ckh.o src/ctl.o src/extent.o src/hash.o src/huge.o src/mb.o src/mutex.o src/nstime.o src/pages.o src/prng.o src/prof.o src/quarantine.o src/rtree.o src/stats.o src/spin.o src/tcache.o src/ticker.o src/tsd.o src/util.o src/witness.o
18942026-03-11T19:42:16.909Zar: `u' modifier ignored since `D' is the default (see `U')
18952026-03-11T19:42:16.921Z[ 63%] Building C object src/CMakeFiles/proj.dir/bch2bps.c.o
18962026-03-11T19:42:16.928Z[ 63%] Building C object src/CMakeFiles/proj.dir/adjlon.c.o
18972026-03-11T19:42:16.928Z[ 14%] Building CXX object CMakeFiles/geos.dir/src/geom/LineSegment.cpp.o
18982026-03-11T19:42:16.930Z[ 14%] Building CXX object CMakeFiles/geos.dir/src/geom/LineString.cpp.o
18992026-03-11T19:42:16.935Z[ 64%] Building C object src/CMakeFiles/proj.dir/bchgen.c.o
19002026-03-11T19:42:16.985Z[ 64%] Building C object src/CMakeFiles/proj.dir/biveval.c.o
19012026-03-11T19:42:16.989Z[ 65%] Building C object src/CMakeFiles/proj.dir/dmstor.c.o
19022026-03-11T19:42:17.004Z[ 65%] Building C object src/CMakeFiles/proj.dir/emess.c.o
19032026-03-11T19:42:17.010Z[ 66%] Building C object src/CMakeFiles/proj.dir/geocent.c.o
19042026-03-11T19:42:17.037Z[ 14%] Building CXX object CMakeFiles/geos.dir/src/geom/LinearRing.cpp.o
19052026-03-11T19:42:17.041Z[ 66%] Building C object src/CMakeFiles/proj.dir/geodesic.c.o
19062026-03-11T19:42:17.053Z[ 16%] Building CXX object CMakeFiles/geos.dir/src/geom/Location.cpp.o
19072026-03-11T19:42:17.057Z[ 67%] Building C object src/CMakeFiles/proj.dir/mk_cheby.c.o
19082026-03-11T19:42:17.062Z[ 68%] Building C object src/CMakeFiles/proj.dir/nad_intr.c.o
19092026-03-11T19:42:17.064Z[ 16%] Building CXX object CMakeFiles/geos.dir/src/geom/MultiLineString.cpp.o
19102026-03-11T19:42:17.067Z[ 68%] Building C object src/CMakeFiles/proj.dir/nad_cvt.c.o
19112026-03-11T19:42:17.071Z[ 16%] Building CXX object CMakeFiles/geos.dir/src/geom/MultiPoint.cpp.o
19122026-03-11T19:42:17.097Zar crus lib/libjemalloc_pic.a src/jemalloc.pic.o src/arena.pic.o src/atomic.pic.o src/base.pic.o src/bitmap.pic.o src/chunk.pic.o src/chunk_dss.pic.o src/chunk_mmap.pic.o src/ckh.pic.o src/ctl.pic.o src/extent.pic.o src/hash.pic.o src/huge.pic.o src/mb.pic.o src/mutex.pic.o src/nstime.pic.o src/pages.pic.o src/prng.pic.o src/prof.pic.o src/quarantine.pic.o src/rtree.pic.o src/stats.pic.o src/spin.pic.o src/tcache.pic.o src/ticker.pic.o src/tsd.pic.o src/util.pic.o src/witness.pic.o
19132026-03-11T19:42:17.114Zar: `u' modifier ignored since `D' is the default (see `U')
19142026-03-11T19:42:17.134Z[ 68%] Building C object src/CMakeFiles/proj.dir/pj_apply_gridshift.c.o
19152026-03-11T19:42:17.166Z[ 69%] Building C object src/CMakeFiles/proj.dir/pj_apply_vgridshift.c.o
19162026-03-11T19:42:17.171Z[ 70%] Building C object src/CMakeFiles/proj.dir/pj_auth.c.o
19172026-03-11T19:42:17.175Z[ 16%] Building CXX object CMakeFiles/geos.dir/src/geom/MultiPolygon.cpp.o
19182026-03-11T19:42:17.193Z[ 70%] Building C object src/CMakeFiles/proj.dir/pj_ctx.c.o
19192026-03-11T19:42:17.212Z[ 71%] Building C object src/CMakeFiles/proj.dir/pj_fileapi.c.o
19202026-03-11T19:42:17.219Z[ 71%] Building C object src/CMakeFiles/proj.dir/pj_datum_set.c.o
19212026-03-11T19:42:17.222Z[ 18%] Building CXX object CMakeFiles/geos.dir/src/geom/Polygon.cpp.o
19222026-03-11T19:42:17.225Z[ 18%] Building CXX object CMakeFiles/geos.dir/src/geom/Point.cpp.o
19232026-03-11T19:42:17.225Z[ 72%] Building C object src/CMakeFiles/proj.dir/pj_datums.c.o
19242026-03-11T19:42:17.246Z[ 72%] Building C object src/CMakeFiles/proj.dir/pj_deriv.c.o
19252026-03-11T19:42:17.252Z[ 73%] Building C object src/CMakeFiles/proj.dir/pj_ell_set.c.o
19262026-03-11T19:42:17.279Z[ 18%] Building CXX object CMakeFiles/geos.dir/src/geom/PrecisionModel.cpp.o
19272026-03-11T19:42:17.283Z[ 18%] Building CXX object CMakeFiles/geos.dir/src/geom/Triangle.cpp.o
19282026-03-11T19:42:17.308Z[ 73%] Building C object src/CMakeFiles/proj.dir/pj_ellps.c.o
19292026-03-11T19:42:17.320Z[ 75%] Building C object src/CMakeFiles/proj.dir/pj_errno.c.o
19302026-03-11T19:42:17.348Z[ 18%] Building CXX object CMakeFiles/geos.dir/src/geom/prep/AbstractPreparedPolygonContains.cpp.o
19312026-03-11T19:42:17.353Z[ 18%] Building CXX object CMakeFiles/geos.dir/src/geom/prep/BasicPreparedGeometry.cpp.o
19322026-03-11T19:42:17.358Z[ 76%] Building C object src/CMakeFiles/proj.dir/pj_factors.c.o
19332026-03-11T19:42:17.365Z[ 19%] Building CXX object CMakeFiles/geos.dir/src/geom/prep/PreparedGeometry.cpp.o
19342026-03-11T19:42:17.375Z[ 76%] Building C object src/CMakeFiles/proj.dir/pj_fwd.c.o
19352026-03-11T19:42:17.411Z[ 19%] Building CXX object CMakeFiles/geos.dir/src/geom/prep/PreparedGeometryFactory.cpp.o
19362026-03-11T19:42:17.411Z[ 77%] Building C object src/CMakeFiles/proj.dir/pj_fwd3d.c.o
19372026-03-11T19:42:17.414Z[ 19%] Building CXX object CMakeFiles/geos.dir/src/geom/prep/PreparedLineString.cpp.o
19382026-03-11T19:42:17.417Z[ 19%] Building CXX object CMakeFiles/geos.dir/src/geom/prep/PreparedLineStringIntersects.cpp.o
19392026-03-11T19:42:17.425Z[ 77%] Building C object src/CMakeFiles/proj.dir/pj_gauss.c.o
19402026-03-11T19:42:17.440Z[ 77%] Building C object src/CMakeFiles/proj.dir/pj_generic_selftest.c.o
19412026-03-11T19:42:17.452Z[ 78%] Building C object src/CMakeFiles/proj.dir/pj_gc_reader.c.o
19422026-03-11T19:42:17.510Z[ 19%] Building CXX object CMakeFiles/geos.dir/src/geom/prep/PreparedPoint.cpp.o
19432026-03-11T19:42:17.510Z[ 79%] Building C object src/CMakeFiles/proj.dir/pj_geocent.c.o
19442026-03-11T19:42:17.513Z[ 79%] Building C object src/CMakeFiles/proj.dir/pj_gridcatalog.c.o
19452026-03-11T19:42:17.521Z[ 21%] Building CXX object CMakeFiles/geos.dir/src/geom/prep/PreparedPolygon.cpp.o
19462026-03-11T19:42:17.567Z[ 21%] Building CXX object CMakeFiles/geos.dir/src/geom/prep/PreparedPolygonContains.cpp.o
19472026-03-11T19:42:17.577Z[ 80%] Building C object src/CMakeFiles/proj.dir/pj_gridinfo.c.o
19482026-03-11T19:42:17.581Z[ 80%] Building C object src/CMakeFiles/proj.dir/pj_gridlist.c.o
19492026-03-11T19:42:17.581Z[ 21%] Building CXX object CMakeFiles/geos.dir/src/geom/prep/PreparedPolygonContainsProperly.cpp.o
19502026-03-11T19:42:17.586Z[ 21%] Building CXX object CMakeFiles/geos.dir/src/geom/prep/PreparedPolygonCovers.cpp.o
19512026-03-11T19:42:17.629Z[ 21%] Building CXX object CMakeFiles/geos.dir/src/geom/prep/PreparedPolygonIntersects.cpp.o
19522026-03-11T19:42:17.635Z[ 81%] Building C object src/CMakeFiles/proj.dir/PJ_healpix.c.o
19532026-03-11T19:42:17.676Z[ 82%] Building C object src/CMakeFiles/proj.dir/pj_init.c.o
19542026-03-11T19:42:17.684Z[ 22%] Building CXX object CMakeFiles/geos.dir/src/geom/prep/PreparedPolygonPredicate.cpp.o
19552026-03-11T19:42:17.700Z[ 22%] Building CXX object CMakeFiles/geos.dir/src/geom/util/ComponentCoordinateExtracter.cpp.o
19562026-03-11T19:42:17.732Z[ 22%] Building CXX object CMakeFiles/geos.dir/src/geom/util/CoordinateOperation.cpp.o
19572026-03-11T19:42:17.732Z[ 82%] Building C object src/CMakeFiles/proj.dir/pj_initcache.c.o
19582026-03-11T19:42:17.745Z[ 22%] Building CXX object CMakeFiles/geos.dir/src/geom/util/Densifier.cpp.o
19592026-03-11T19:42:17.756Z[ 22%] Building CXX object CMakeFiles/geos.dir/src/geom/util/GeometryCombiner.cpp.o
19602026-03-11T19:42:17.804Z[ 83%] Building C object src/CMakeFiles/proj.dir/pj_inv.c.o
19612026-03-11T19:42:17.807Z[ 24%] Building CXX object CMakeFiles/geos.dir/src/geom/util/GeometryEditor.cpp.o
19622026-03-11T19:42:17.807Z[ 84%] Building C object src/CMakeFiles/proj.dir/pj_latlong.c.o
19632026-03-11T19:42:17.810Z[ 84%] Building C object src/CMakeFiles/proj.dir/pj_list.c.o
19642026-03-11T19:42:17.812Z[ 24%] Building CXX object CMakeFiles/geos.dir/src/geom/util/GeometryTransformer.cpp.o
19652026-03-11T19:42:17.812Z[ 84%] Building C object src/CMakeFiles/proj.dir/pj_inv3d.c.o
19662026-03-11T19:42:17.824Z[ 24%] Building CXX object CMakeFiles/geos.dir/src/geom/util/LinearComponentExtracter.cpp.o
19672026-03-11T19:42:17.836Z[ 85%] Building C object src/CMakeFiles/proj.dir/pj_log.c.o
19682026-03-11T19:42:17.890Z[ 85%] Building C object src/CMakeFiles/proj.dir/pj_malloc.c.o
19692026-03-11T19:42:17.893Z[ 24%] Building CXX object CMakeFiles/geos.dir/src/geom/util/PolygonExtracter.cpp.o
19702026-03-11T19:42:17.894Z[ 86%] Building C object src/CMakeFiles/proj.dir/pj_msfn.c.o
19712026-03-11T19:42:17.896Z[ 24%] Building CXX object CMakeFiles/geos.dir/src/geom/util/PointExtracter.cpp.o
19722026-03-11T19:42:17.901Z[ 88%] Building C object src/CMakeFiles/proj.dir/pj_mlfn.c.o
19732026-03-11T19:42:17.929Z[ 88%] Building C object src/CMakeFiles/proj.dir/pj_mutex.c.o
19742026-03-11T19:42:17.942Z[ 89%] Building C object src/CMakeFiles/proj.dir/pj_open_lib.c.o
19752026-03-11T19:42:17.945Z[ 26%] Building CXX object CMakeFiles/geos.dir/src/geom/util/ShortCircuitedGeometryVisitor.cpp.o
19762026-03-11T19:42:17.948Z[ 89%] Building C object src/CMakeFiles/proj.dir/pj_param.c.o
19772026-03-11T19:42:17.952Z[ 26%] Building CXX object CMakeFiles/geos.dir/src/geom/util/SineStarFactory.cpp.o
19782026-03-11T19:42:17.954Z[ 90%] Building C object src/CMakeFiles/proj.dir/pj_phi2.c.o
19792026-03-11T19:42:17.957Z[ 90%] Building C object src/CMakeFiles/proj.dir/pj_pr_list.c.o
19802026-03-11T19:42:17.960Z[ 91%] Building C object src/CMakeFiles/proj.dir/pj_qsfn.c.o
19812026-03-11T19:42:17.977Z[ 91%] Building C object src/CMakeFiles/proj.dir/pj_release.c.o
19822026-03-11T19:42:17.980Z[ 26%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/Depth.cpp.o
19832026-03-11T19:42:17.983Z[ 92%] Building C object src/CMakeFiles/proj.dir/pj_run_selftests.c.o
19842026-03-11T19:42:17.997Z[ 92%] Building C object src/CMakeFiles/proj.dir/pj_strerrno.c.o
19852026-03-11T19:42:18.002Z[ 94%] Building C object src/CMakeFiles/proj.dir/pj_tsfn.c.o
19862026-03-11T19:42:18.004Z[ 94%] Building C object src/CMakeFiles/proj.dir/pj_transform.c.o
19872026-03-11T19:42:18.016Z[ 94%] Building C object src/CMakeFiles/proj.dir/pj_units.c.o
19882026-03-11T19:42:18.023Z[ 95%] Building C object src/CMakeFiles/proj.dir/pj_utils.c.o
19892026-03-11T19:42:18.038Z[ 95%] Building C object src/CMakeFiles/proj.dir/pj_zpoly1.c.o
19902026-03-11T19:42:18.041Z[ 96%] Building C object src/CMakeFiles/proj.dir/proj_mdist.c.o
19912026-03-11T19:42:18.044Z[ 26%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/DirectedEdge.cpp.o
19922026-03-11T19:42:18.055Z[ 26%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/DirectedEdgeStar.cpp.o
19932026-03-11T19:42:18.060Z[ 96%] Building C object src/CMakeFiles/proj.dir/proj_rouss.c.o
19942026-03-11T19:42:18.067Z[ 97%] Building C object src/CMakeFiles/proj.dir/rtodms.c.o
19952026-03-11T19:42:18.103Z[ 98%] Building C object src/CMakeFiles/proj.dir/pj_strtod.c.o
19962026-03-11T19:42:18.108Z[ 98%] Building C object src/CMakeFiles/proj.dir/vector1.c.o
19972026-03-11T19:42:18.108Z[ 27%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/Edge.cpp.o
19982026-03-11T19:42:18.108Z[ 27%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/EdgeEnd.cpp.o
19992026-03-11T19:42:18.122Z[ 27%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/EdgeEndStar.cpp.o
20002026-03-11T19:42:18.125Z[ 27%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/EdgeList.cpp.o
20012026-03-11T19:42:18.128Z[ 29%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/EdgeNodingValidator.cpp.o
20022026-03-11T19:42:18.131Z[ 29%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/EdgeIntersectionList.cpp.o
20032026-03-11T19:42:18.134Z[ 29%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/EdgeRing.cpp.o
20042026-03-11T19:42:18.139Z[ 29%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/GeometryGraph.cpp.o
20052026-03-11T19:42:18.143Z[100%] Linking C static library ../lib/libproj.a
20062026-03-11T19:42:18.146Z[ 29%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/GraphComponent.cpp.o
20072026-03-11T19:42:18.153Z[ 29%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/Label.cpp.o
20082026-03-11T19:42:18.186Z[ 31%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/Node.cpp.o
20092026-03-11T19:42:18.211Zgithub.com/bufbuild/buf/private/bufpkg/bufplugin
20102026-03-11T19:42:18.215Zgithub.com/bufbuild/buf/private/gen/proto/api/buf/alpha/registry/v1alpha1/registryv1alpha1api
20112026-03-11T19:42:18.223Z[ 31%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/NodeFactory.cpp.o
20122026-03-11T19:42:18.246Z[ 31%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/NodeMap.cpp.o
20132026-03-11T19:42:18.256Zgithub.com/bufbuild/buf/private/gen/proto/apiclient/buf/alpha/registry/v1alpha1/registryv1alpha1apiclient
20142026-03-11T19:42:18.279Zgithub.com/bufbuild/buf/private/gen/proto/apiclientgrpc/buf/alpha/registry/v1alpha1/registryv1alpha1apiclientgrpc
20152026-03-11T19:42:18.279Zgithub.com/bufbuild/buf/private/bufpkg/bufapimodule
20162026-03-11T19:42:18.279Zgithub.com/bufbuild/buf/private/gen/proto/apiclienttwirp/buf/alpha/registry/v1alpha1/registryv1alpha1apiclienttwirp
20172026-03-11T19:42:18.282Zgithub.com/bufbuild/buf/private/buf/bufgen
20182026-03-11T19:42:18.282Zgithub.com/bufbuild/buf/private/buf/bufprint
20192026-03-11T19:42:18.282Z[ 31%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/PlanarGraph.cpp.o
20202026-03-11T19:42:18.341Z[ 31%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/Position.cpp.o
20212026-03-11T19:42:18.388Z[ 32%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/Quadrant.cpp.o
20222026-03-11T19:42:18.391Zgithub.com/bufbuild/buf/private/buf/bufmigrate
20232026-03-11T19:42:18.414Z[ 32%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/TopologyLocation.cpp.o
20242026-03-11T19:42:18.417Z[ 32%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/index/MonotoneChainEdge.cpp.o
20252026-03-11T19:42:18.420Zgithub.com/bufbuild/buf/private/bufpkg/bufapiclient
20262026-03-11T19:42:18.437Z[ 32%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/index/MonotoneChainIndexer.cpp.o
20272026-03-11T19:42:18.440Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/config/configmigratev1beta1
20282026-03-11T19:42:18.455Z[ 32%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/index/SegmentIntersector.cpp.o
20292026-03-11T19:42:18.461Zgithub.com/bufbuild/buf/private/buf/bufcli
20302026-03-11T19:42:18.476Z[ 34%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/index/SimpleEdgeSetIntersector.cpp.o
20312026-03-11T19:42:18.498Z[ 34%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/index/SimpleSweepLineIntersector.cpp.o
20322026-03-11T19:42:18.501Z[ 34%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/index/SimpleMCSweepLineIntersector.cpp.o
20332026-03-11T19:42:18.505Z[ 34%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/index/SweepLineEvent.cpp.o
20342026-03-11T19:42:18.518Z[ 34%] Building CXX object CMakeFiles/geos.dir/src/geomgraph/index/SweepLineSegment.cpp.o
20352026-03-11T19:42:18.522Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/alpha/registry/token/tokencreate
20362026-03-11T19:42:18.522Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/alpha/registry/token/tokendelete
20372026-03-11T19:42:18.522Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/alpha/registry/token/tokenget
20382026-03-11T19:42:18.522Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/beta/registry/commit/commitlist
20392026-03-11T19:42:18.522Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/beta/registry/organization/organizationdelete
20402026-03-11T19:42:18.522Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/beta/registry/organization/organizationget
20412026-03-11T19:42:18.522Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/beta/registry/plugin/plugincreate
20422026-03-11T19:42:18.522Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/beta/registry/commit/commitget
20432026-03-11T19:42:18.522Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/beta/registry/organization/organizationcreate
20442026-03-11T19:42:18.522Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/beta/registry/plugin/pluginversion/pluginversionlist
20452026-03-11T19:42:18.522Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/beta/registry/plugin/pluginlist
20462026-03-11T19:42:18.523Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/beta/registry/repository/repositorycreate
20472026-03-11T19:42:18.523Z[ 36%] Building CXX object CMakeFiles/geos.dir/src/index/bintree/Bintree.cpp.o
20482026-03-11T19:42:18.523Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/beta/registry/plugin/plugindelete
20492026-03-11T19:42:18.523Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/beta/registry/repository/repositorydelete
20502026-03-11T19:42:18.523Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/beta/registry/repository/repositoryget
20512026-03-11T19:42:18.523Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/alpha/registry/token/tokenlist
20522026-03-11T19:42:18.544Z[ 36%] Building CXX object CMakeFiles/geos.dir/src/index/bintree/Interval.cpp.o
20532026-03-11T19:42:18.554Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/beta/registry/repository/repositorylist
20542026-03-11T19:42:18.556Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/beta/registry/tag/tagcreate
20552026-03-11T19:42:18.559Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/beta/registry/tag/taglist
20562026-03-11T19:42:18.562Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/beta/registry/template/templatecreate
20572026-03-11T19:42:18.562Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/beta/registry/template/templatedelete
20582026-03-11T19:42:18.565Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/beta/registry/template/templatelist
20592026-03-11T19:42:18.565Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/beta/registry/template/templateversion/templateversioncreate
20602026-03-11T19:42:18.565Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/beta/registry/template/templateversion/templateversionlist
20612026-03-11T19:42:18.569Z[ 36%] Building CXX object CMakeFiles/geos.dir/src/index/bintree/Key.cpp.o
20622026-03-11T19:42:18.592Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/build
20632026-03-11T19:42:18.595Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/convert
20642026-03-11T19:42:18.595Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/breaking
20652026-03-11T19:42:18.595Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/export
20662026-03-11T19:42:18.595Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/generate
20672026-03-11T19:42:18.595Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/lint
20682026-03-11T19:42:18.598Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/login
20692026-03-11T19:42:18.598Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/logout
20702026-03-11T19:42:18.601Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/lsfiles
20712026-03-11T19:42:18.604Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/mod/modclearcache
20722026-03-11T19:42:18.604Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/mod/modinit
20732026-03-11T19:42:18.610Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/mod/modprune
20742026-03-11T19:42:18.614Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/mod/modupdate
20752026-03-11T19:42:18.614Z[ 36%] Building CXX object CMakeFiles/geos.dir/src/index/bintree/Node.cpp.o
20762026-03-11T19:42:18.617Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/protoc
20772026-03-11T19:42:18.620Zgithub.com/bufbuild/buf/private/buf/cmd/buf/command/push
20782026-03-11T19:42:18.625Z[ 36%] Building CXX object CMakeFiles/geos.dir/src/index/bintree/NodeBase.cpp.o
20792026-03-11T19:42:18.655Z[ 37%] Building CXX object CMakeFiles/geos.dir/src/index/bintree/Root.cpp.o
20802026-03-11T19:42:18.671Z[ 37%] Building CXX object CMakeFiles/geos.dir/src/index/chain/MonotoneChain.cpp.o
20812026-03-11T19:42:18.685Zgithub.com/bufbuild/buf/private/buf/cmd/buf
20822026-03-11T19:42:18.688Z[ 37%] Building CXX object CMakeFiles/geos.dir/src/index/chain/MonotoneChainBuilder.cpp.o
20832026-03-11T19:42:18.688Z[ 37%] Building CXX object CMakeFiles/geos.dir/src/index/chain/MonotoneChainOverlapAction.cpp.o
20842026-03-11T19:42:18.691Z[ 37%] Building CXX object CMakeFiles/geos.dir/src/index/chain/MonotoneChainSelectAction.cpp.o
20852026-03-11T19:42:18.709Zgithub.com/bufbuild/buf/cmd/buf
20862026-03-11T19:42:18.712Z[ 39%] Building CXX object CMakeFiles/geos.dir/src/index/intervalrtree/IntervalRTreeBranchNode.cpp.o
20872026-03-11T19:42:18.715Z[ 39%] Building CXX object CMakeFiles/geos.dir/src/index/intervalrtree/IntervalRTreeLeafNode.cpp.o
20882026-03-11T19:42:18.731Z[ 39%] Building CXX object CMakeFiles/geos.dir/src/index/intervalrtree/SortedPackedIntervalRTree.cpp.o
20892026-03-11T19:42:18.756Z[ 39%] Building CXX object CMakeFiles/geos.dir/src/index/quadtree/DoubleBits.cpp.o
20902026-03-11T19:42:18.759Z[ 39%] Building CXX object CMakeFiles/geos.dir/src/index/quadtree/IntervalSize.cpp.o
20912026-03-11T19:42:18.773Z[ 40%] Building CXX object CMakeFiles/geos.dir/src/index/quadtree/Key.cpp.o
20922026-03-11T19:42:18.776Z[ 40%] Building CXX object CMakeFiles/geos.dir/src/index/quadtree/NodeBase.cpp.o
20932026-03-11T19:42:18.776Z[ 40%] Building CXX object CMakeFiles/geos.dir/src/index/quadtree/Quadtree.cpp.o
20942026-03-11T19:42:18.776Z[ 40%] Building CXX object CMakeFiles/geos.dir/src/index/quadtree/Node.cpp.o
20952026-03-11T19:42:18.789Z[ 40%] Building CXX object CMakeFiles/geos.dir/src/index/quadtree/Root.cpp.o
20962026-03-11T19:42:18.792Z[ 42%] Building CXX object CMakeFiles/geos.dir/src/index/strtree/AbstractSTRtree.cpp.o
20972026-03-11T19:42:18.840Z[ 42%] Building CXX object CMakeFiles/geos.dir/src/index/strtree/BoundablePair.cpp.o
20982026-03-11T19:42:18.854Z[ 42%] Building CXX object CMakeFiles/geos.dir/src/index/strtree/EnvelopeUtil.cpp.o
20992026-03-11T19:42:18.870Z[ 42%] Building CXX object CMakeFiles/geos.dir/src/index/strtree/GeometryItemDistance.cpp.o
21002026-03-11T19:42:18.873Z[ 42%] Building CXX object CMakeFiles/geos.dir/src/index/strtree/Interval.cpp.o
21012026-03-11T19:42:18.885Z[ 44%] Building CXX object CMakeFiles/geos.dir/src/index/strtree/SIRtree.cpp.o
21022026-03-11T19:42:18.885Z[ 44%] Building CXX object CMakeFiles/geos.dir/src/index/strtree/STRtree.cpp.o
21032026-03-11T19:42:18.914Z[ 44%] Building CXX object CMakeFiles/geos.dir/src/index/sweepline/SweepLineEvent.cpp.o
21042026-03-11T19:42:18.931Z[ 44%] Building CXX object CMakeFiles/geos.dir/src/index/sweepline/SweepLineInterval.cpp.o
21052026-03-11T19:42:18.934Z[ 45%] Building CXX object CMakeFiles/geos.dir/src/inlines.cpp.o
21062026-03-11T19:42:18.944Z[ 45%] Building CXX object CMakeFiles/geos.dir/src/index/sweepline/SweepLineIndex.cpp.o
21072026-03-11T19:42:18.944Z[ 45%] Building CXX object CMakeFiles/geos.dir/src/io/ByteOrderDataInStream.cpp.o
21082026-03-11T19:42:18.952Z[ 45%] Building CXX object CMakeFiles/geos.dir/src/io/ByteOrderValues.cpp.o
21092026-03-11T19:42:18.956Z[ 45%] Building CXX object CMakeFiles/geos.dir/src/io/CLocalizer.cpp.o
21102026-03-11T19:42:18.998Z[ 47%] Building CXX object CMakeFiles/geos.dir/src/io/StringTokenizer.cpp.o
21112026-03-11T19:42:18.998Z[ 47%] Building CXX object CMakeFiles/geos.dir/src/io/ParseException.cpp.o
21122026-03-11T19:42:19.022Z[ 47%] Building CXX object CMakeFiles/geos.dir/src/io/WKBReader.cpp.o
21132026-03-11T19:42:19.025Z[ 47%] Building CXX object CMakeFiles/geos.dir/src/io/WKBWriter.cpp.o
21142026-03-11T19:42:19.030Z[ 47%] Building CXX object CMakeFiles/geos.dir/src/io/Unload.cpp.o
21152026-03-11T19:42:19.048Z[ 49%] Building CXX object CMakeFiles/geos.dir/src/io/WKTWriter.cpp.o
21162026-03-11T19:42:19.049Z[ 49%] Building CXX object CMakeFiles/geos.dir/src/io/WKTReader.cpp.o
21172026-03-11T19:42:19.064Z[ 49%] Building CXX object CMakeFiles/geos.dir/src/io/Writer.cpp.o
21182026-03-11T19:42:19.067Z[ 49%] Building CXX object CMakeFiles/geos.dir/src/linearref/ExtractLineByLocation.cpp.o
21192026-03-11T19:42:19.098Z[ 49%] Building CXX object CMakeFiles/geos.dir/src/linearref/LengthIndexOfPoint.cpp.o
21202026-03-11T19:42:19.105Z[ 50%] Building CXX object CMakeFiles/geos.dir/src/linearref/LengthLocationMap.cpp.o
21212026-03-11T19:42:19.113Z[ 50%] Building CXX object CMakeFiles/geos.dir/src/linearref/LinearGeometryBuilder.cpp.o
21222026-03-11T19:42:19.117Z[ 50%] Building CXX object CMakeFiles/geos.dir/src/linearref/LengthIndexedLine.cpp.o
21232026-03-11T19:42:19.133Z[ 50%] Building CXX object CMakeFiles/geos.dir/src/linearref/LinearIterator.cpp.o
21242026-03-11T19:42:19.186Z[ 50%] Building CXX object CMakeFiles/geos.dir/src/linearref/LinearLocation.cpp.o
21252026-03-11T19:42:19.256Z[ 50%] Building CXX object CMakeFiles/geos.dir/src/linearref/LocationIndexOfLine.cpp.o
21262026-03-11T19:42:19.280Z[ 52%] Building CXX object CMakeFiles/geos.dir/src/linearref/LocationIndexOfPoint.cpp.o
21272026-03-11T19:42:19.325Z[ 52%] Building CXX object CMakeFiles/geos.dir/src/noding/BasicSegmentString.cpp.o
21282026-03-11T19:42:19.345Z[ 52%] Building CXX object CMakeFiles/geos.dir/src/noding/FastNodingValidator.cpp.o
21292026-03-11T19:42:19.357Z[ 52%] Building CXX object CMakeFiles/geos.dir/src/noding/FastSegmentSetIntersectionFinder.cpp.o
21302026-03-11T19:42:19.365Z[ 52%] Building CXX object CMakeFiles/geos.dir/src/noding/GeometryNoder.cpp.o
21312026-03-11T19:42:19.378Z[ 52%] Building CXX object CMakeFiles/geos.dir/src/noding/IntersectionFinderAdder.cpp.o
21322026-03-11T19:42:19.382Z[ 54%] Building CXX object CMakeFiles/geos.dir/src/noding/IntersectionAdder.cpp.o
21332026-03-11T19:42:19.414Z[ 54%] Building CXX object CMakeFiles/geos.dir/src/noding/IteratedNoder.cpp.o
21342026-03-11T19:42:19.420Z[ 54%] Building CXX object CMakeFiles/geos.dir/src/noding/MCIndexNoder.cpp.o
21352026-03-11T19:42:19.468Z[ 54%] Building CXX object CMakeFiles/geos.dir/src/noding/MCIndexSegmentSetMutualIntersector.cpp.o
21362026-03-11T19:42:19.476Z[ 55%] Building CXX object CMakeFiles/geos.dir/src/noding/NodedSegmentString.cpp.o
21372026-03-11T19:42:19.498Z[ 55%] Building CXX object CMakeFiles/geos.dir/src/noding/NodingIntersectionFinder.cpp.o
21382026-03-11T19:42:19.515Z[ 55%] Building CXX object CMakeFiles/geos.dir/src/noding/NodingValidator.cpp.o
21392026-03-11T19:42:19.529Z[ 55%] Building CXX object CMakeFiles/geos.dir/src/noding/Octant.cpp.o
21402026-03-11T19:42:19.556Z[ 55%] Building CXX object CMakeFiles/geos.dir/src/noding/OrientedCoordinateArray.cpp.o
21412026-03-11T19:42:19.566Z[ 57%] Building CXX object CMakeFiles/geos.dir/src/noding/ScaledNoder.cpp.o
21422026-03-11T19:42:19.571Z[ 57%] Building CXX object CMakeFiles/geos.dir/src/noding/SegmentIntersectionDetector.cpp.o
21432026-03-11T19:42:19.584Z[ 57%] Building CXX object CMakeFiles/geos.dir/src/noding/SegmentNode.cpp.o
21442026-03-11T19:42:19.605Z[ 57%] Building CXX object CMakeFiles/geos.dir/src/noding/SegmentNodeList.cpp.o
21452026-03-11T19:42:19.621Zgithub.com/cockroachdb/gostdlib/x/tools/internal/semver
21462026-03-11T19:42:19.624Zgo.mongodb.org/mongo-driver/bson/bsonoptions
21472026-03-11T19:42:19.641Zgo.mongodb.org/mongo-driver/bson/bsontype
21482026-03-11T19:42:19.645Zgoogle.golang.org/protobuf/internal/flags
21492026-03-11T19:42:19.645Zgithub.com/grpc-ecosystem/grpc-gateway/internal/casing
21502026-03-11T19:42:19.649Zgithub.com/pseudomuto/protoc-gen-doc/extensions
21512026-03-11T19:42:19.649Zmaps
21522026-03-11T19:42:19.649Zhonnef.co/go/tools/internal/sync
21532026-03-11T19:42:19.649Zhonnef.co/go/tools/sarif
21542026-03-11T19:42:19.656Z[ 57%] Building CXX object CMakeFiles/geos.dir/src/noding/SegmentString.cpp.o
21552026-03-11T19:42:19.659Zgoogle.golang.org/protobuf/internal/set
21562026-03-11T19:42:19.663Zgithub.com/josharian/intern
21572026-03-11T19:42:19.663Zinternal/saferio
21582026-03-11T19:42:19.663Zgolang.org/x/text/transform
21592026-03-11T19:42:19.677Zgithub.com/hashicorp/hcl/hcl/strconv
21602026-03-11T19:42:19.678Zhash/crc64
21612026-03-11T19:42:19.678Zhash/adler32
21622026-03-11T19:42:19.681Zgolang.org/x/sys/unix
21632026-03-11T19:42:19.681Zgithub.com/golang/glog/internal/stackdump
21642026-03-11T19:42:19.681Zgoogle.golang.org/protobuf/internal/pragma
21652026-03-11T19:42:19.681Zgoogle.golang.org/protobuf/internal/editiondefaults
21662026-03-11T19:42:19.686Zgithub.com/cockroachdb/gostdlib/x/tools/internal/fastwalk
21672026-03-11T19:42:19.689Zgithub.com/cockroachdb/gostdlib/x/tools/internal/module
21682026-03-11T19:42:19.689Zgithub.com/cockroachdb/gostdlib/go/printer
21692026-03-11T19:42:19.689Zgithub.com/cockroachdb/gostdlib/x/tools/go/ast/astutil
21702026-03-11T19:42:19.689Zgithub.com/cockroachdb/crlfmt/internal/parser
21712026-03-11T19:42:19.689Zgithub.com/cockroachdb/gostdlib/x/tools/go/internal/packagesdriver
21722026-03-11T19:42:19.692Zgithub.com/client9/misspell
21732026-03-11T19:42:19.692Z[ 59%] Building CXX object CMakeFiles/geos.dir/src/noding/SegmentStringUtil.cpp.o
21742026-03-11T19:42:19.702Zgithub.com/cockroachdb/gostdlib/x/tools/internal/span
21752026-03-11T19:42:19.705Zgithub.com/cockroachdb/ttycolor
21762026-03-11T19:42:19.710Zgithub.com/cockroachdb/stress
21772026-03-11T19:42:19.716Zgithub.com/cockroachdb/tools/cmd/stringer
21782026-03-11T19:42:19.716Zgithub.com/go-openapi/analysis/internal
21792026-03-11T19:42:19.717Zgithub.com/mailru/easyjson/jlexer
21802026-03-11T19:42:19.717Zgithub.com/mailru/easyjson/buffer
21812026-03-11T19:42:19.726Zgithub.com/cockroachdb/gostdlib/x/tools/internal/gopathwalk
21822026-03-11T19:42:19.729Zgopkg.in/yaml.v2
21832026-03-11T19:42:19.729Z[ 59%] Building CXX object CMakeFiles/geos.dir/src/noding/SimpleNoder.cpp.o
21842026-03-11T19:42:19.729Zencoding/gob
21852026-03-11T19:42:19.732Zgithub.com/PuerkitoBio/urlesc
21862026-03-11T19:42:19.732Zgithub.com/cockroachdb/crlfmt/internal/render
21872026-03-11T19:42:19.735Zgolang.org/x/text/unicode/bidi
21882026-03-11T19:42:19.750Zgolang.org/x/text/unicode/norm
21892026-03-11T19:42:19.762Zgithub.com/mailru/easyjson/jwriter
21902026-03-11T19:42:19.767Zgolang.org/x/text/width
21912026-03-11T19:42:19.775Zgithub.com/cockroachdb/gostdlib/x/tools/go/packages
21922026-03-11T19:42:19.779Zgithub.com/asaskevich/govalidator
21932026-03-11T19:42:19.792Z[ 59%] Building CXX object CMakeFiles/geos.dir/src/noding/snapround/MCIndexSnapRounder.cpp.o
21942026-03-11T19:42:19.797Zgithub.com/go-openapi/errors
21952026-03-11T19:42:19.800Z[ 59%] Building CXX object CMakeFiles/geos.dir/src/noding/snapround/MCIndexPointSnapper.cpp.o
21962026-03-11T19:42:19.804Z[ 60%] Building CXX object CMakeFiles/geos.dir/src/noding/snapround/SimpleSnapRounder.cpp.o
21972026-03-11T19:42:19.807Z[ 60%] Building CXX object CMakeFiles/geos.dir/src/noding/snapround/HotPixel.cpp.o
21982026-03-11T19:42:19.809Zgithub.com/mitchellh/mapstructure
21992026-03-11T19:42:19.826Zgithub.com/oklog/ulid
22002026-03-11T19:42:19.843Zgo.mongodb.org/mongo-driver/bson/primitive
22012026-03-11T19:42:19.846Zgithub.com/go-stack/stack
22022026-03-11T19:42:19.849Z[ 60%] Building CXX object CMakeFiles/geos.dir/src/operation/GeometryGraphOperation.cpp.o
22032026-03-11T19:42:19.869Znet/mail
22042026-03-11T19:42:19.872Z[ 60%] Building CXX object CMakeFiles/geos.dir/src/operation/IsSimpleOp.cpp.o
22052026-03-11T19:42:19.889Zgolang.org/x/text/secure/bidirule
22062026-03-11T19:42:19.898Zencoding/xml
22072026-03-11T19:42:19.917Z[ 60%] Building CXX object CMakeFiles/geos.dir/src/operation/buffer/BufferBuilder.cpp.o
22082026-03-11T19:42:19.920Zgithub.com/go-openapi/runtime/logger
22092026-03-11T19:42:19.921Zgithub.com/go-openapi/runtime/middleware/denco
22102026-03-11T19:42:19.926Zgithub.com/go-openapi/runtime/middleware/header
22112026-03-11T19:42:19.932Zgo.mongodb.org/mongo-driver/x/bsonx/bsoncore
22122026-03-11T19:42:19.941Zgithub.com/cockroachdb/gostdlib/go/format
22132026-03-11T19:42:19.944Zgithub.com/cockroachdb/gostdlib/cmd/gofmt
22142026-03-11T19:42:19.947Zgolang.org/x/tools/go/ast/astutil
22152026-03-11T19:42:19.961Z[ 62%] Building CXX object CMakeFiles/geos.dir/src/operation/buffer/BufferOp.cpp.o
22162026-03-11T19:42:19.972Z[ 62%] Building CXX object CMakeFiles/geos.dir/src/operation/buffer/BufferInputLineSimplifier.cpp.o
22172026-03-11T19:42:19.994Zgithub.com/go-openapi/inflect
22182026-03-11T19:42:19.997Z[ 62%] Building CXX object CMakeFiles/geos.dir/src/operation/buffer/BufferParameters.cpp.o
22192026-03-11T19:42:20.000Zgithub.com/hashicorp/hcl/hcl/token
22202026-03-11T19:42:20.009Zgithub.com/client9/misspell/cmd/misspell
22212026-03-11T19:42:20.013Zgithub.com/magiconair/properties
22222026-03-11T19:42:20.016Zgithub.com/pelletier/go-toml
22232026-03-11T19:42:20.024Z[ 62%] Building CXX object CMakeFiles/geos.dir/src/operation/buffer/OffsetCurveBuilder.cpp.o
22242026-03-11T19:42:20.030Zgithub.com/cockroachdb/gostdlib/x/tools/internal/imports
22252026-03-11T19:42:20.033Zgithub.com/spf13/afero/mem
22262026-03-11T19:42:20.040Z[ 62%] Building CXX object CMakeFiles/geos.dir/src/operation/buffer/BufferSubgraph.cpp.o
22272026-03-11T19:42:20.050Zgolang.org/x/net/idna
22282026-03-11T19:42:20.095Zgithub.com/spf13/cast
22292026-03-11T19:42:20.106Zgithub.com/spf13/afero
22302026-03-11T19:42:20.114Zgithub.com/spf13/jwalterweatherman
22312026-03-11T19:42:20.117Zgithub.com/hashicorp/hcl/hcl/ast
22322026-03-11T19:42:20.121Zgithub.com/hashicorp/hcl/hcl/scanner
22332026-03-11T19:42:20.134Zgithub.com/hashicorp/hcl/json/token
22342026-03-11T19:42:20.150Zgithub.com/subosito/gotenv
22352026-03-11T19:42:20.164Zgopkg.in/ini.v1
22362026-03-11T19:42:20.168Z[ 62%] Building CXX object CMakeFiles/geos.dir/src/operation/buffer/OffsetCurveSetBuilder.cpp.o
22372026-03-11T19:42:20.173Zgolang.org/x/mod/internal/lazyregexp
22382026-03-11T19:42:20.181Z[ 62%] Building CXX object CMakeFiles/geos.dir/src/operation/buffer/RightmostEdgeFinder.cpp.o
22392026-03-11T19:42:20.183Zgolang.org/x/tools/internal/gopathwalk
22402026-03-11T19:42:20.186Z[ 63%] Building CXX object CMakeFiles/geos.dir/src/operation/buffer/OffsetSegmentGenerator.cpp.o
22412026-03-11T19:42:20.191Zinternal/sysinfo
22422026-03-11T19:42:20.194Zgo.mongodb.org/mongo-driver/bson/bsonrw
22432026-03-11T19:42:20.200Zgithub.com/hashicorp/hcl/hcl/parser
22442026-03-11T19:42:20.205Zgithub.com/hashicorp/hcl/json/scanner
22452026-03-11T19:42:20.231Zgolang.org/x/mod/module
22462026-03-11T19:42:20.235Zgithub.com/PuerkitoBio/purell
22472026-03-11T19:42:20.238Ztesting
22482026-03-11T19:42:20.274Zgithub.com/felixge/httpsnoop
22492026-03-11T19:42:20.280Zgithub.com/toqueteos/webbrowser
22502026-03-11T19:42:20.286Zgithub.com/hashicorp/hcl/json/parser
22512026-03-11T19:42:20.294Zgithub.com/golang/mock/mockgen/model
22522026-03-11T19:42:20.309Z[ 63%] Building CXX object CMakeFiles/geos.dir/src/operation/buffer/SubgraphDepthLocater.cpp.o
22532026-03-11T19:42:20.320Zgithub.com/go-openapi/swag
22542026-03-11T19:42:20.323Zcompress/zlib
22552026-03-11T19:42:20.327Zgithub.com/hashicorp/hcl/hcl/printer
22562026-03-11T19:42:20.350Zgithub.com/hashicorp/hcl
22572026-03-11T19:42:20.357Zdebug/dwarf
22582026-03-11T19:42:20.360Zinternal/zstd
22592026-03-11T19:42:20.360Zinternal/pkgbits
22602026-03-11T19:42:20.381Zgithub.com/fsnotify/fsnotify
22612026-03-11T19:42:20.389Zgithub.com/jessevdk/go-flags
22622026-03-11T19:42:20.406Zgo/internal/srcimporter
22632026-03-11T19:42:20.417Zgithub.com/gorilla/handlers
22642026-03-11T19:42:20.443Zgolang.org/x/mod/modfile
22652026-03-11T19:42:20.446Zgithub.com/golang/glog/internal/logsink
22662026-03-11T19:42:20.464Z[ 63%] Building CXX object CMakeFiles/geos.dir/src/operation/distance/ConnectedElementLocationFilter.cpp.o
22672026-03-11T19:42:20.466Zgoogle.golang.org/protobuf/internal/detrand
22682026-03-11T19:42:20.467Zgoogle.golang.org/protobuf/internal/version
22692026-03-11T19:42:20.470Zgo/internal/gcimporter
22702026-03-11T19:42:20.470Z[ 63%] Building CXX object CMakeFiles/geos.dir/src/operation/distance/ConnectedElementPointFilter.cpp.o
22712026-03-11T19:42:20.501Zgithub.com/golang/glog
22722026-03-11T19:42:20.516Zgo.mongodb.org/mongo-driver/bson/bsoncodec
22732026-03-11T19:42:20.519Zgithub.com/ghodss/yaml
22742026-03-11T19:42:20.522Zgithub.com/grpc-ecosystem/grpc-gateway/utilities
22752026-03-11T19:42:20.525Zgithub.com/cockroachdb/gostdlib/x/tools/imports
22762026-03-11T19:42:20.532Zgithub.com/cockroachdb/gostdlib/x/tools/cmd/goimports
22772026-03-11T19:42:20.532Zgoogle.golang.org/protobuf/internal/errors
22782026-03-11T19:42:20.532Zgolang.org/x/tools/internal/modindex
22792026-03-11T19:42:20.536Zencoding/base32
22802026-03-11T19:42:20.539Zgolang.org/x/tools/go/analysis
22812026-03-11T19:42:20.565Zgithub.com/go-openapi/jsonpointer
22822026-03-11T19:42:20.582Zgithub.com/cockroachdb/crlfmt
22832026-03-11T19:42:20.585Zgithub.com/go-openapi/loads/fmts
22842026-03-11T19:42:20.589Zgoogle.golang.org/protobuf/encoding/protowire
22852026-03-11T19:42:20.589Zgithub.com/spf13/viper
22862026-03-11T19:42:20.597Zgithub.com/go-openapi/jsonreference
22872026-03-11T19:42:20.600Zgithub.com/kisielk/errcheck/errcheck
22882026-03-11T19:42:20.604Zgithub.com/kevinburke/go-bindata
22892026-03-11T19:42:20.607Z[ 65%] Building CXX object CMakeFiles/geos.dir/src/operation/distance/DistanceOp.cpp.o
22902026-03-11T19:42:20.630Zgolang.org/x/tools/cover
22912026-03-11T19:42:20.635Zgithub.com/mibk/dupl/suffixtree
22922026-03-11T19:42:20.660Zgoogle.golang.org/protobuf/reflect/protoreflect
22932026-03-11T19:42:20.660Zgithub.com/go-openapi/spec
22942026-03-11T19:42:20.663Zgithub.com/mmatczuk/go_generics/globals
22952026-03-11T19:42:20.669Zgithub.com/mattn/goveralls
22962026-03-11T19:42:20.674Z[ 65%] Building CXX object CMakeFiles/geos.dir/src/operation/distance/FacetSequenceTreeBuilder.cpp.o
22972026-03-11T19:42:20.679Zgithub.com/kevinburke/go-bindata/go-bindata
22982026-03-11T19:42:20.682Z[ 65%] Building CXX object CMakeFiles/geos.dir/src/operation/distance/FacetSequence.cpp.o
22992026-03-11T19:42:20.690Zgolang.org/x/tools/internal/imports
23002026-03-11T19:42:20.693Zgithub.com/Masterminds/goutils
23012026-03-11T19:42:20.708Zgithub.com/mibk/dupl/syntax
23022026-03-11T19:42:20.711Zdebug/elf
23032026-03-11T19:42:20.712Zinternal/xcoff
23042026-03-11T19:42:20.731Zgithub.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway/httprule
23052026-03-11T19:42:20.731Zgithub.com/kisielk/errcheck
23062026-03-11T19:42:20.739Zgithub.com/mmatczuk/go_generics/cmd/go_generics
23072026-03-11T19:42:20.747Z[ 65%] Building CXX object CMakeFiles/geos.dir/src/operation/distance/GeometryLocation.cpp.o
23082026-03-11T19:42:20.786Zgithub.com/Masterminds/semver
23092026-03-11T19:42:20.793Zgithub.com/mibk/dupl/syntax/golang
23102026-03-11T19:42:20.809Zgithub.com/mibk/dupl/output
23112026-03-11T19:42:20.824Zgithub.com/google/uuid
23122026-03-11T19:42:20.840Zgoogle.golang.org/protobuf/internal/encoding/messageset
23132026-03-11T19:42:20.843Zgoogle.golang.org/protobuf/internal/strs
23142026-03-11T19:42:20.847Zgoogle.golang.org/protobuf/internal/genid
23152026-03-11T19:42:20.851Zgoogle.golang.org/protobuf/internal/order
23162026-03-11T19:42:20.854Z[ 65%] Building CXX object CMakeFiles/geos.dir/src/operation/distance/IndexedFacetDistance.cpp.o
23172026-03-11T19:42:20.866Zgoogle.golang.org/protobuf/runtime/protoiface
23182026-03-11T19:42:20.880Zgoogle.golang.org/protobuf/internal/descfmt
23192026-03-11T19:42:20.894Zgoogle.golang.org/protobuf/internal/encoding/text
23202026-03-11T19:42:20.901Z[ 67%] Building CXX object CMakeFiles/geos.dir/src/operation/intersection/Rectangle.cpp.o
23212026-03-11T19:42:20.918Zgoogle.golang.org/protobuf/internal/descopts
23222026-03-11T19:42:20.922Zgoogle.golang.org/protobuf/internal/encoding/json
23232026-03-11T19:42:20.925Zgoogle.golang.org/protobuf/reflect/protoregistry
23242026-03-11T19:42:20.935Zgo/internal/gccgoimporter
23252026-03-11T19:42:20.942Zgithub.com/mibk/dupl/job
23262026-03-11T19:42:20.963Z[ 67%] Building CXX object CMakeFiles/geos.dir/src/operation/intersection/RectangleIntersection.cpp.o
23272026-03-11T19:42:20.977Zgithub.com/huandu/xstrings
23282026-03-11T19:42:20.980Zgithub.com/imdario/mergo
23292026-03-11T19:42:20.998Zgolang.org/x/tools/imports
23302026-03-11T19:42:21.001Zgithub.com/go-swagger/go-swagger/cmd/swagger/commands/diff
23312026-03-11T19:42:21.012Zgithub.com/go-swagger/go-swagger/codescan
23322026-03-11T19:42:21.017Zgithub.com/go-swagger/go-swagger/cmd/swagger/commands/initcmd
23332026-03-11T19:42:21.017Zgoogle.golang.org/protobuf/internal/encoding/defval
23342026-03-11T19:42:21.017Zgithub.com/mitchellh/reflectwalk
23352026-03-11T19:42:21.022Z[ 67%] Building CXX object CMakeFiles/geos.dir/src/operation/intersection/RectangleIntersectionBuilder.cpp.o
23362026-03-11T19:42:21.022Zgithub.com/mibk/dupl
23372026-03-11T19:42:21.022Zgoogle.golang.org/protobuf/proto
23382026-03-11T19:42:21.025Zgo.mongodb.org/mongo-driver/bson
23392026-03-11T19:42:21.036Z[ 67%] Building CXX object CMakeFiles/geos.dir/src/operation/linemerge/EdgeString.cpp.o
23402026-03-11T19:42:21.040Zgolang.org/x/crypto/pbkdf2
23412026-03-11T19:42:21.066Zgithub.com/mwitkow/go-proto-validators
23422026-03-11T19:42:21.066Zgithub.com/wadey/gocovmerge
23432026-03-11T19:42:21.071Zgithub.com/mitchellh/copystructure
23442026-03-11T19:42:21.078Zgolang.org/x/perf/internal/stats
23452026-03-11T19:42:21.081Zgolang.org/x/perf/storage/benchfmt
23462026-03-11T19:42:21.081Zgolang.org/x/tools/cmd/goyacc
23472026-03-11T19:42:21.081Zgolang.org/x/tools/internal/astutil/edge
23482026-03-11T19:42:21.087Zgolang.org/x/tools/internal/analysisinternal
23492026-03-11T19:42:21.087Zgolang.org/x/crypto/scrypt
23502026-03-11T19:42:21.090Z[ 67%] Building CXX object CMakeFiles/geos.dir/src/operation/linemerge/LineMergeDirectedEdge.cpp.o
23512026-03-11T19:42:21.110Zgo/importer
23522026-03-11T19:42:21.114Zgolang.org/x/tools/go/analysis/internal/analysisflags
23532026-03-11T19:42:21.158Zgolang.org/x/tools/go/analysis/internal
23542026-03-11T19:42:21.161Z[ 68%] Building CXX object CMakeFiles/geos.dir/src/operation/linemerge/LineMergeEdge.cpp.o
23552026-03-11T19:42:21.164Zgithub.com/Masterminds/sprig
23562026-03-11T19:42:21.170Zgolang.org/x/tools/go/ast/inspector
23572026-03-11T19:42:21.173Zgolang.org/x/tools/internal/diff/lcs
23582026-03-11T19:42:21.173Zgithub.com/golang/mock/mockgen
23592026-03-11T19:42:21.182Zgithub.com/go-openapi/strfmt
23602026-03-11T19:42:21.186Zgoogle.golang.org/protobuf/encoding/prototext
23612026-03-11T19:42:21.190Z[ 68%] Building CXX object CMakeFiles/geos.dir/src/operation/linemerge/LineMergeGraph.cpp.o
23622026-03-11T19:42:21.198Zgoogle.golang.org/protobuf/internal/filedesc
23632026-03-11T19:42:21.232Zgolang.org/x/perf/benchstat
23642026-03-11T19:42:21.245Z[ 68%] Building CXX object CMakeFiles/geos.dir/src/operation/linemerge/LineMerger.cpp.o
23652026-03-11T19:42:21.251Zgolang.org/x/tools/go/analysis/passes/internal/analysisutil
23662026-03-11T19:42:21.271Zgolang.org/x/tools/go/analysis/passes/inspect
23672026-03-11T19:42:21.274Zgolang.org/x/tools/internal/diff
23682026-03-11T19:42:21.309Zgolang.org/x/tools/internal/facts
23692026-03-11T19:42:21.312Zgolang.org/x/tools/go/analysis/checker
23702026-03-11T19:42:21.313Zgolang.org/x/tools/go/buildutil
23712026-03-11T19:42:21.326Zhonnef.co/go/tools/analysis/facts/tokenfile
23722026-03-11T19:42:21.349Zgithub.com/BurntSushi/toml/internal
23732026-03-11T19:42:21.356Zgithub.com/BurntSushi/toml
23742026-03-11T19:42:21.361Z[ 70%] Building CXX object CMakeFiles/geos.dir/src/operation/overlay/ElevationMatrix.cpp.o
23752026-03-11T19:42:21.364Zdebug/macho
23762026-03-11T19:42:21.364Z[ 70%] Building CXX object CMakeFiles/geos.dir/src/operation/linemerge/LineSequencer.cpp.o
23772026-03-11T19:42:21.367Z[ 70%] Building CXX object CMakeFiles/geos.dir/src/operation/overlay/EdgeSetNoder.cpp.o
23782026-03-11T19:42:21.370Zgolang.org/x/tools/go/analysis/passes/shadow
23792026-03-11T19:42:21.370Z[ 70%] Building CXX object CMakeFiles/geos.dir/src/operation/overlay/ElevationMatrixCell.cpp.o
23802026-03-11T19:42:21.373Zhonnef.co/go/tools/analysis/lint
23812026-03-11T19:42:21.376Zgolang.org/x/perf/cmd/benchstat
23822026-03-11T19:42:21.376Zhonnef.co/go/tools/internal/robustio
23832026-03-11T19:42:21.389Zhonnef.co/go/tools/analysis/facts/generated
23842026-03-11T19:42:21.401Zgolang.org/x/exp/typeparams
23852026-03-11T19:42:21.410Zhonnef.co/go/tools/go/ast/astutil
23862026-03-11T19:42:21.423Zgolang.org/x/tools/go/internal/cgo
23872026-03-11T19:42:21.431Zgolang.org/x/tools/go/analysis/unitchecker
23882026-03-11T19:42:21.436Zhonnef.co/go/tools/internal/renameio
23892026-03-11T19:42:21.439Zhonnef.co/go/tools/knowledge
23902026-03-11T19:42:21.441Zgithub.com/go-openapi/analysis
23912026-03-11T19:42:21.466Zgithub.com/go-openapi/runtime
23922026-03-11T19:42:21.472Zgolang.org/x/tools/go/analysis/internal/checker
23932026-03-11T19:42:21.475Z[ 70%] Building CXX object CMakeFiles/geos.dir/src/operation/overlay/LineBuilder.cpp.o
23942026-03-11T19:42:21.487Zhonnef.co/go/tools/go/types/typeutil
23952026-03-11T19:42:21.493Zhonnef.co/go/tools/lintcmd/cache
23962026-03-11T19:42:21.497Z[ 70%] Building CXX object CMakeFiles/geos.dir/src/operation/overlay/MaximalEdgeRing.cpp.o
23972026-03-11T19:42:21.500Zgoogle.golang.org/protobuf/internal/encoding/tag
23982026-03-11T19:42:21.503Zgoogle.golang.org/protobuf/encoding/protojson
23992026-03-11T19:42:21.509Z[ 70%] Building CXX object CMakeFiles/geos.dir/src/operation/overlay/MinimalEdgeRing.cpp.o
24002026-03-11T19:42:21.512Zgolang.org/x/tools/go/loader
24012026-03-11T19:42:21.514Zhonnef.co/go/tools/pattern
24022026-03-11T19:42:21.517Zhonnef.co/go/tools/analysis/facts/directives
24032026-03-11T19:42:21.524Zhonnef.co/go/tools/go/buildid
24042026-03-11T19:42:21.558Zhonnef.co/go/tools/lintcmd/version
24052026-03-11T19:42:21.561Zhonnef.co/go/tools/analysis/facts/deprecated
24062026-03-11T19:42:21.564Zgolang.org/x/tools/go/analysis/singlechecker
24072026-03-11T19:42:21.579Z[ 70%] Building CXX object CMakeFiles/geos.dir/src/operation/overlay/OverlayOp.cpp.o
24082026-03-11T19:42:21.584Z[ 72%] Building CXX object CMakeFiles/geos.dir/src/operation/overlay/OverlayNodeFactory.cpp.o
24092026-03-11T19:42:21.584Z[ 72%] Building CXX object CMakeFiles/geos.dir/src/operation/overlay/PolygonBuilder.cpp.o
24102026-03-11T19:42:21.596Zhonnef.co/go/tools/config
24112026-03-11T19:42:21.601Zhonnef.co/go/tools/staticcheck/fakereflect
24122026-03-11T19:42:21.604Zgoogle.golang.org/protobuf/internal/impl
24132026-03-11T19:42:21.607Zhonnef.co/go/tools/printf
24142026-03-11T19:42:21.618Zhonnef.co/go/tools/go/ir
24152026-03-11T19:42:21.621Z[ 72%] Building CXX object CMakeFiles/geos.dir/src/operation/overlay/PointBuilder.cpp.o
24162026-03-11T19:42:21.624Zgolang.org/x/tools/go/analysis/passes/shadow/cmd/shadow
24172026-03-11T19:42:21.636Zgithub.com/go-openapi/runtime/security
24182026-03-11T19:42:21.663Zhonnef.co/go/tools/staticcheck/fakejson
24192026-03-11T19:42:21.666Zhonnef.co/go/tools/staticcheck/fakexml
24202026-03-11T19:42:21.691Z[ 72%] Building CXX object CMakeFiles/geos.dir/src/operation/overlay/snap/GeometrySnapper.cpp.o
24212026-03-11T19:42:21.696Zhonnef.co/go/tools/go/loader
24222026-03-11T19:42:21.771Zgithub.com/go-openapi/loads
24232026-03-11T19:42:21.778Zhonnef.co/go/tools/analysis/edit
24242026-03-11T19:42:21.816Zgithub.com/go-openapi/runtime/middleware/untyped
24252026-03-11T19:42:21.819Zgithub.com/go-openapi/validate
24262026-03-11T19:42:21.828Z[ 73%] Building CXX object CMakeFiles/geos.dir/src/operation/overlay/snap/LineStringSnapper.cpp.o
24272026-03-11T19:42:21.924Z[ 73%] Building CXX object CMakeFiles/geos.dir/src/operation/overlay/snap/SnapIfNeededOverlayOp.cpp.o
24282026-03-11T19:42:21.937Z[ 73%] Building CXX object CMakeFiles/geos.dir/src/operation/overlay/validate/OffsetPointGenerator.cpp.o
24292026-03-11T19:42:21.942Z[ 73%] Building CXX object CMakeFiles/geos.dir/src/operation/overlay/snap/SnapOverlayOp.cpp.o
24302026-03-11T19:42:21.948Z[ 73%] Building CXX object CMakeFiles/geos.dir/src/operation/overlay/validate/FuzzyPointLocator.cpp.o
24312026-03-11T19:42:21.954Z[ 75%] Building CXX object CMakeFiles/geos.dir/src/operation/overlay/validate/OverlayResultValidator.cpp.o
24322026-03-11T19:42:21.969Z[ 75%] Building CXX object CMakeFiles/geos.dir/src/operation/polygonize/BuildArea.cpp.o
24332026-03-11T19:42:22.001Z[ 75%] Building CXX object CMakeFiles/geos.dir/src/operation/polygonize/EdgeRing.cpp.o
24342026-03-11T19:42:22.025Z[ 75%] Building CXX object CMakeFiles/geos.dir/src/operation/polygonize/HoleAssigner.cpp.o
24352026-03-11T19:42:22.034Z[ 75%] Building CXX object CMakeFiles/geos.dir/src/operation/polygonize/PolygonizeDirectedEdge.cpp.o
24362026-03-11T19:42:22.098Zgithub.com/go-openapi/runtime/middleware
24372026-03-11T19:42:22.101Zgithub.com/go-swagger/go-swagger/generator
24382026-03-11T19:42:22.173Z[ 77%] Building CXX object CMakeFiles/geos.dir/src/operation/polygonize/PolygonizeEdge.cpp.o
24392026-03-11T19:42:22.203Z[ 77%] Building CXX object CMakeFiles/geos.dir/src/operation/polygonize/PolygonizeGraph.cpp.o
24402026-03-11T19:42:22.237Z[ 77%] Building CXX object CMakeFiles/geos.dir/src/operation/polygonize/Polygonizer.cpp.o
24412026-03-11T19:42:22.273Z[ 77%] Building CXX object CMakeFiles/geos.dir/src/operation/predicate/RectangleContains.cpp.o
24422026-03-11T19:42:22.304Z[ 77%] Building CXX object CMakeFiles/geos.dir/src/operation/predicate/RectangleIntersects.cpp.o
24432026-03-11T19:42:22.316Z[ 78%] Building CXX object CMakeFiles/geos.dir/src/operation/predicate/SegmentIntersectionTester.cpp.o
24442026-03-11T19:42:22.335Z[ 78%] Building CXX object CMakeFiles/geos.dir/src/operation/relate/EdgeEndBuilder.cpp.o
24452026-03-11T19:42:22.363Z[ 78%] Building CXX object CMakeFiles/geos.dir/src/operation/relate/EdgeEndBundleStar.cpp.o
24462026-03-11T19:42:22.368Z[ 80%] Building CXX object CMakeFiles/geos.dir/src/operation/relate/RelateComputer.cpp.o
24472026-03-11T19:42:22.376Z[ 80%] Building CXX object CMakeFiles/geos.dir/src/operation/relate/EdgeEndBundle.cpp.o
24482026-03-11T19:42:22.381Zgoogle.golang.org/protobuf/internal/filetype
24492026-03-11T19:42:22.385Z[ 80%] Building CXX object CMakeFiles/geos.dir/src/operation/relate/RelateNode.cpp.o
24502026-03-11T19:42:22.388Z[ 80%] Building CXX object CMakeFiles/geos.dir/src/operation/relate/RelateNodeFactory.cpp.o
24512026-03-11T19:42:22.429Zgoogle.golang.org/protobuf/runtime/protoimpl
24522026-03-11T19:42:22.435Zhonnef.co/go/tools/internal/passes/buildir
24532026-03-11T19:42:22.435Zhonnef.co/go/tools/go/ir/irutil
24542026-03-11T19:42:22.469Z[ 80%] Building CXX object CMakeFiles/geos.dir/src/operation/relate/RelateNodeGraph.cpp.o
24552026-03-11T19:42:22.472Zgoogle.golang.org/protobuf/types/descriptorpb
24562026-03-11T19:42:22.472Zgoogle.golang.org/protobuf/types/known/durationpb
24572026-03-11T19:42:22.472Zgoogle.golang.org/protobuf/types/known/timestamppb
24582026-03-11T19:42:22.517Zhonnef.co/go/tools/analysis/facts/nilness
24592026-03-11T19:42:22.541Z[ 80%] Building CXX object CMakeFiles/geos.dir/src/operation/relate/RelateOp.cpp.o
24602026-03-11T19:42:22.576Z[ 81%] Building CXX object CMakeFiles/geos.dir/src/operation/sharedpaths/SharedPathsOp.cpp.o
24612026-03-11T19:42:22.582Zhonnef.co/go/tools/analysis/facts/purity
24622026-03-11T19:42:22.582Zhonnef.co/go/tools/analysis/facts/typedness
24632026-03-11T19:42:22.585Z[ 81%] Building CXX object CMakeFiles/geos.dir/src/operation/union/CascadedPolygonUnion.cpp.o
24642026-03-11T19:42:22.627Zhonnef.co/go/tools/analysis/code
24652026-03-11T19:42:22.634Z[ 81%] Building CXX object CMakeFiles/geos.dir/src/operation/union/CascadedUnion.cpp.o
24662026-03-11T19:42:22.643Z[ 81%] Building CXX object CMakeFiles/geos.dir/src/operation/union/CoverageUnion.cpp.o
24672026-03-11T19:42:22.646Z[ 81%] Building CXX object CMakeFiles/geos.dir/src/operation/union/OverlapUnion.cpp.o
24682026-03-11T19:42:22.650Zgoogle.golang.org/protobuf/types/gofeaturespb
24692026-03-11T19:42:22.650Zgithub.com/golang/protobuf/protoc-gen-go/descriptor
24702026-03-11T19:42:22.650Zgoogle.golang.org/genproto/googleapis/api/annotations
24712026-03-11T19:42:22.650Zgithub.com/envoyproxy/protoc-gen-validate/validate
24722026-03-11T19:42:22.650Zgoogle.golang.org/protobuf/types/pluginpb
24732026-03-11T19:42:22.666Zhonnef.co/go/tools/analysis/report
24742026-03-11T19:42:22.679Z[ 83%] Building CXX object CMakeFiles/geos.dir/src/operation/union/PointGeometryUnion.cpp.o
24752026-03-11T19:42:22.682Zgoogle.golang.org/protobuf/reflect/protodesc
24762026-03-11T19:42:22.682Zgithub.com/golang/protobuf/protoc-gen-go/plugin
24772026-03-11T19:42:22.685Zgithub.com/go-swagger/go-swagger/cmd/swagger/commands/generate
24782026-03-11T19:42:22.696Z[ 83%] Building CXX object CMakeFiles/geos.dir/src/operation/union/UnaryUnionOp.cpp.o
24792026-03-11T19:42:22.701Zgithub.com/pseudomuto/protoc-gen-doc/extensions/google_api_http
24802026-03-11T19:42:22.711Z[ 83%] Building CXX object CMakeFiles/geos.dir/src/operation/valid/ConnectedInteriorTester.cpp.o
24812026-03-11T19:42:22.734Zhonnef.co/go/tools/quickfix/qf1002
24822026-03-11T19:42:22.734Zhonnef.co/go/tools/unused
24832026-03-11T19:42:22.737Zhonnef.co/go/tools/quickfix/qf1003
24842026-03-11T19:42:22.737Zhonnef.co/go/tools/quickfix/qf1004
24852026-03-11T19:42:22.737Zhonnef.co/go/tools/quickfix/qf1005
24862026-03-11T19:42:22.737Zhonnef.co/go/tools/quickfix/qf1006
24872026-03-11T19:42:22.737Zhonnef.co/go/tools/quickfix/qf1007
24882026-03-11T19:42:22.737Zhonnef.co/go/tools/quickfix/qf1008
24892026-03-11T19:42:22.737Zhonnef.co/go/tools/quickfix/qf1009
24902026-03-11T19:42:22.748Z[ 83%] Building CXX object CMakeFiles/geos.dir/src/operation/valid/ConsistentAreaTester.cpp.o
24912026-03-11T19:42:22.752Zhonnef.co/go/tools/quickfix/qf1010
24922026-03-11T19:42:22.752Zhonnef.co/go/tools/quickfix/qf1001
24932026-03-11T19:42:22.761Zhonnef.co/go/tools/internal/sharedcheck
24942026-03-11T19:42:22.761Zgithub.com/go-swagger/go-swagger/cmd/swagger/commands
24952026-03-11T19:42:22.761Zhonnef.co/go/tools/quickfix/qf1012
24962026-03-11T19:42:22.761Zgithub.com/pseudomuto/protoc-gen-doc/extensions/envoyproxy_validate
24972026-03-11T19:42:22.773Z[ 83%] Building CXX object CMakeFiles/geos.dir/src/operation/valid/IndexedNestedRingTester.cpp.o
24982026-03-11T19:42:22.791Zhonnef.co/go/tools/simple/s1000
24992026-03-11T19:42:22.791Zhonnef.co/go/tools/simple/s1001
25002026-03-11T19:42:22.791Zhonnef.co/go/tools/simple/s1002
25012026-03-11T19:42:22.796Z[ 85%] Building CXX object CMakeFiles/geos.dir/src/operation/valid/IsValidOp.cpp.o
25022026-03-11T19:42:22.799Zhonnef.co/go/tools/simple/s1003
25032026-03-11T19:42:22.799Zhonnef.co/go/tools/simple/s1004
25042026-03-11T19:42:22.799Zhonnef.co/go/tools/simple/s1005
25052026-03-11T19:42:22.799Zhonnef.co/go/tools/simple/s1006
25062026-03-11T19:42:22.802Zhonnef.co/go/tools/simple/s1007
25072026-03-11T19:42:22.802Zhonnef.co/go/tools/simple/s1008
25082026-03-11T19:42:22.806Zhonnef.co/go/tools/simple/s1009
25092026-03-11T19:42:22.806Zgithub.com/pseudomuto/protoc-gen-doc/extensions/lyft_validate
25102026-03-11T19:42:22.814Zhonnef.co/go/tools/simple/s1010
25112026-03-11T19:42:22.819Zgithub.com/go-swagger/go-swagger/cmd/swagger
25122026-03-11T19:42:22.838Zhonnef.co/go/tools/simple/s1011
25132026-03-11T19:42:22.841Zhonnef.co/go/tools/simple/s1012
25142026-03-11T19:42:22.844Zhonnef.co/go/tools/simple/s1016
25152026-03-11T19:42:22.844Zhonnef.co/go/tools/simple/s1017
25162026-03-11T19:42:22.844Z[ 85%] Building CXX object CMakeFiles/geos.dir/src/operation/valid/MakeValid.cpp.o
25172026-03-11T19:42:22.848Zhonnef.co/go/tools/simple/s1018
25182026-03-11T19:42:22.848Zhonnef.co/go/tools/simple/s1019
25192026-03-11T19:42:22.848Zhonnef.co/go/tools/simple/s1020
25202026-03-11T19:42:22.848Zhonnef.co/go/tools/simple/s1021
25212026-03-11T19:42:22.851Zhonnef.co/go/tools/simple/s1023
25222026-03-11T19:42:22.854Zhonnef.co/go/tools/simple/s1024
25232026-03-11T19:42:22.854Zhonnef.co/go/tools/simple/s1025
25242026-03-11T19:42:22.854Z[ 85%] Building CXX object CMakeFiles/geos.dir/src/operation/valid/QuadtreeNestedRingTester.cpp.o
25252026-03-11T19:42:22.860Zhonnef.co/go/tools/lintcmd/runner
25262026-03-11T19:42:22.880Zhonnef.co/go/tools/simple/s1028
25272026-03-11T19:42:22.881Zhonnef.co/go/tools/simple/s1030
25282026-03-11T19:42:22.881Z[ 85%] Building CXX object CMakeFiles/geos.dir/src/operation/valid/RepeatedPointRemover.cpp.o
25292026-03-11T19:42:22.881Zhonnef.co/go/tools/simple/s1031
25302026-03-11T19:42:22.884Zhonnef.co/go/tools/simple/s1032
25312026-03-11T19:42:22.884Zhonnef.co/go/tools/simple/s1033
25322026-03-11T19:42:22.886Zhonnef.co/go/tools/simple/s1034
25332026-03-11T19:42:22.891Zhonnef.co/go/tools/quickfix/qf1011
25342026-03-11T19:42:22.897Zhonnef.co/go/tools/simple/s1029
25352026-03-11T19:42:22.906Zhonnef.co/go/tools/simple/s1035
25362026-03-11T19:42:22.906Zhonnef.co/go/tools/simple/s1036
25372026-03-11T19:42:22.915Zhonnef.co/go/tools/simple/s1037
25382026-03-11T19:42:22.932Zhonnef.co/go/tools/simple/s1038
25392026-03-11T19:42:22.936Zhonnef.co/go/tools/simple/s1039
25402026-03-11T19:42:22.936Zhonnef.co/go/tools/simple/s1040
25412026-03-11T19:42:22.936Zhonnef.co/go/tools/analysis/callcheck
25422026-03-11T19:42:22.936Zhonnef.co/go/tools/staticcheck/sa1001
25432026-03-11T19:42:22.936Zhonnef.co/go/tools/staticcheck/sa1004
25442026-03-11T19:42:22.936Zhonnef.co/go/tools/staticcheck/sa1005
25452026-03-11T19:42:22.939Zhonnef.co/go/tools/staticcheck/sa1006
25462026-03-11T19:42:22.946Zhonnef.co/go/tools/staticcheck/sa1008
25472026-03-11T19:42:22.949Zgithub.com/golang/protobuf/proto
25482026-03-11T19:42:22.949Zhonnef.co/go/tools/staticcheck/sa1012
25492026-03-11T19:42:22.963Z[ 85%] Building CXX object CMakeFiles/geos.dir/src/operation/valid/RepeatedPointTester.cpp.o
25502026-03-11T19:42:22.966Zhonnef.co/go/tools/staticcheck/sa1013
25512026-03-11T19:42:22.966Zhonnef.co/go/tools/staticcheck/sa1015
25522026-03-11T19:42:22.966Zhonnef.co/go/tools/staticcheck/sa1016
25532026-03-11T19:42:22.969Zhonnef.co/go/tools/quickfix
25542026-03-11T19:42:22.969Zhonnef.co/go/tools/staticcheck/sa1019
25552026-03-11T19:42:22.969Zhonnef.co/go/tools/lintcmd
25562026-03-11T19:42:22.969Zhonnef.co/go/tools/staticcheck/sa1023
25572026-03-11T19:42:22.973Zhonnef.co/go/tools/simple
25582026-03-11T19:42:22.976Zhonnef.co/go/tools/staticcheck/sa1025
25592026-03-11T19:42:22.979Zhonnef.co/go/tools/staticcheck/sa2000
25602026-03-11T19:42:22.982Zhonnef.co/go/tools/staticcheck/sa2001
25612026-03-11T19:42:22.986Zhonnef.co/go/tools/staticcheck/sa2002
25622026-03-11T19:42:22.998Zhonnef.co/go/tools/staticcheck/sa2003
25632026-03-11T19:42:23.002Zhonnef.co/go/tools/staticcheck/sa3000
25642026-03-11T19:42:23.014Zhonnef.co/go/tools/staticcheck/sa3001
25652026-03-11T19:42:23.018Zhonnef.co/go/tools/staticcheck/sa4000
25662026-03-11T19:42:23.018Zhonnef.co/go/tools/staticcheck/sa4001
25672026-03-11T19:42:23.018Zhonnef.co/go/tools/staticcheck/sa1000
25682026-03-11T19:42:23.024Zhonnef.co/go/tools/staticcheck/sa1002
25692026-03-11T19:42:23.024Zhonnef.co/go/tools/staticcheck/sa1003
25702026-03-11T19:42:23.029Zhonnef.co/go/tools/staticcheck/sa1007
25712026-03-11T19:42:23.033Zhonnef.co/go/tools/staticcheck/sa1010
25722026-03-11T19:42:23.041Zhonnef.co/go/tools/staticcheck/sa1011
25732026-03-11T19:42:23.057Zhonnef.co/go/tools/staticcheck/sa1017
25742026-03-11T19:42:23.057Zhonnef.co/go/tools/staticcheck/sa1014
25752026-03-11T19:42:23.060Zhonnef.co/go/tools/staticcheck/sa1018
25762026-03-11T19:42:23.060Zhonnef.co/go/tools/staticcheck/sa1020
25772026-03-11T19:42:23.061Zhonnef.co/go/tools/staticcheck/sa1021
25782026-03-11T19:42:23.064Zhonnef.co/go/tools/staticcheck/sa1024
25792026-03-11T19:42:23.067Zhonnef.co/go/tools/staticcheck/sa1026
25802026-03-11T19:42:23.074Zhonnef.co/go/tools/staticcheck/sa1027
25812026-03-11T19:42:23.077Z[ 86%] Building CXX object CMakeFiles/geos.dir/src/operation/valid/SimpleNestedRingTester.cpp.o
25822026-03-11T19:42:23.080Zhonnef.co/go/tools/staticcheck/sa1028
25832026-03-11T19:42:23.083Zhonnef.co/go/tools/staticcheck/sa1029
25842026-03-11T19:42:23.083Zhonnef.co/go/tools/staticcheck/sa1030
25852026-03-11T19:42:23.100Zhonnef.co/go/tools/staticcheck/sa1031
25862026-03-11T19:42:23.107Zhonnef.co/go/tools/staticcheck/sa1032
25872026-03-11T19:42:23.119Zhonnef.co/go/tools/staticcheck/sa4003
25882026-03-11T19:42:23.130Zhonnef.co/go/tools/staticcheck/sa4004
25892026-03-11T19:42:23.130Zhonnef.co/go/tools/staticcheck/sa4005
25902026-03-11T19:42:23.133Zhonnef.co/go/tools/staticcheck/sa4006
25912026-03-11T19:42:23.136Zgithub.com/grpc-ecosystem/grpc-gateway/codegenerator
25922026-03-11T19:42:23.139Zgithub.com/golang/protobuf/jsonpb
25932026-03-11T19:42:23.148Zgithub.com/pseudomuto/protokit
25942026-03-11T19:42:23.154Zgithub.com/pseudomuto/protoc-gen-doc/extensions/validator_field
25952026-03-11T19:42:23.161Zhonnef.co/go/tools/staticcheck/sa4008
25962026-03-11T19:42:23.162Zhonnef.co/go/tools/staticcheck/sa4009
25972026-03-11T19:42:23.164Zhonnef.co/go/tools/staticcheck/sa4010
25982026-03-11T19:42:23.168Zhonnef.co/go/tools/staticcheck/sa4011
25992026-03-11T19:42:23.173Zhonnef.co/go/tools/staticcheck/sa4012
26002026-03-11T19:42:23.180Zhonnef.co/go/tools/staticcheck/sa4013
26012026-03-11T19:42:23.180Zhonnef.co/go/tools/staticcheck/sa4014
26022026-03-11T19:42:23.183Zhonnef.co/go/tools/staticcheck/sa4015
26032026-03-11T19:42:23.183Zhonnef.co/go/tools/staticcheck/sa4016
26042026-03-11T19:42:23.186Zhonnef.co/go/tools/staticcheck/sa4017
26052026-03-11T19:42:23.189Zhonnef.co/go/tools/staticcheck/sa4018
26062026-03-11T19:42:23.192Zhonnef.co/go/tools/staticcheck/sa4019
26072026-03-11T19:42:23.195Zhonnef.co/go/tools/staticcheck/sa4020
26082026-03-11T19:42:23.218Zhonnef.co/go/tools/staticcheck/sa4021
26092026-03-11T19:42:23.221Zhonnef.co/go/tools/staticcheck/sa4022
26102026-03-11T19:42:23.221Zhonnef.co/go/tools/staticcheck/sa4023
26112026-03-11T19:42:23.224Zhonnef.co/go/tools/staticcheck/sa4024
26122026-03-11T19:42:23.224Zhonnef.co/go/tools/staticcheck/sa4025
26132026-03-11T19:42:23.233Zhonnef.co/go/tools/staticcheck/sa4026
26142026-03-11T19:42:23.236Zhonnef.co/go/tools/staticcheck/sa4027
26152026-03-11T19:42:23.236Zhonnef.co/go/tools/staticcheck/sa4028
26162026-03-11T19:42:23.236Zhonnef.co/go/tools/staticcheck/sa4029
26172026-03-11T19:42:23.241Zhonnef.co/go/tools/staticcheck/sa4030
26182026-03-11T19:42:23.244Zhonnef.co/go/tools/staticcheck/sa4032
26192026-03-11T19:42:23.251Zhonnef.co/go/tools/staticcheck/sa5000
26202026-03-11T19:42:23.254Zhonnef.co/go/tools/staticcheck/sa4031
26212026-03-11T19:42:23.254Zgithub.com/pseudomuto/protoc-gen-doc
26222026-03-11T19:42:23.254Zhonnef.co/go/tools/staticcheck/sa5001
26232026-03-11T19:42:23.257Zgithub.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway/descriptor
26242026-03-11T19:42:23.266Zhonnef.co/go/tools/staticcheck/sa5002
26252026-03-11T19:42:23.266Zhonnef.co/go/tools/staticcheck/sa5003
26262026-03-11T19:42:23.266Zhonnef.co/go/tools/staticcheck/sa5004
26272026-03-11T19:42:23.270Zhonnef.co/go/tools/staticcheck/sa5005
26282026-03-11T19:42:23.273Z[ 86%] Building CXX object CMakeFiles/geos.dir/src/operation/valid/SweeplineNestedRingTester.cpp.o
26292026-03-11T19:42:23.273Zhonnef.co/go/tools/staticcheck/sa5007
26302026-03-11T19:42:23.273Z[ 86%] Building CXX object CMakeFiles/geos.dir/src/operation/valid/TopologyValidationError.cpp.o
26312026-03-11T19:42:23.278Zhonnef.co/go/tools/staticcheck/sa5008
26322026-03-11T19:42:23.285Zhonnef.co/go/tools/staticcheck/sa5009
26332026-03-11T19:42:23.290Zhonnef.co/go/tools/staticcheck/sa5010
26342026-03-11T19:42:23.293Zhonnef.co/go/tools/staticcheck/sa5011
26352026-03-11T19:42:23.293Z[ 86%] Building CXX object CMakeFiles/geos.dir/src/planargraph/DirectedEdge.cpp.o
26362026-03-11T19:42:23.293Zhonnef.co/go/tools/staticcheck/sa5012
26372026-03-11T19:42:23.296Zhonnef.co/go/tools/staticcheck/sa6000
26382026-03-11T19:42:23.296Zhonnef.co/go/tools/staticcheck/sa6001
26392026-03-11T19:42:23.300Zhonnef.co/go/tools/staticcheck/sa6002
26402026-03-11T19:42:23.300Zhonnef.co/go/tools/staticcheck/sa6003
26412026-03-11T19:42:23.300Zhonnef.co/go/tools/staticcheck/sa6005
26422026-03-11T19:42:23.306Zhonnef.co/go/tools/staticcheck/sa6006
26432026-03-11T19:42:23.317Zhonnef.co/go/tools/staticcheck/sa9001
26442026-03-11T19:42:23.322Zhonnef.co/go/tools/staticcheck/sa9002
26452026-03-11T19:42:23.341Z[ 88%] Building CXX object CMakeFiles/geos.dir/src/planargraph/Edge.cpp.o
26462026-03-11T19:42:23.345Zhonnef.co/go/tools/staticcheck/sa9003
26472026-03-11T19:42:23.345Zhonnef.co/go/tools/staticcheck/sa9004
26482026-03-11T19:42:23.347Zhonnef.co/go/tools/staticcheck/sa9005
26492026-03-11T19:42:23.350Z[ 88%] Building CXX object CMakeFiles/geos.dir/src/planargraph/DirectedEdgeStar.cpp.o
26502026-03-11T19:42:23.350Zhonnef.co/go/tools/staticcheck/sa9006
26512026-03-11T19:42:23.350Zhonnef.co/go/tools/staticcheck/sa9007
26522026-03-11T19:42:23.359Zhonnef.co/go/tools/staticcheck/sa9008
26532026-03-11T19:42:23.362Zhonnef.co/go/tools/staticcheck/sa9009
26542026-03-11T19:42:23.366Zhonnef.co/go/tools/stylecheck/st1000
26552026-03-11T19:42:23.370Zgithub.com/pseudomuto/protoc-gen-doc/cmd/protoc-gen-doc
26562026-03-11T19:42:23.373Zhonnef.co/go/tools/stylecheck/st1001
26572026-03-11T19:42:23.376Zhonnef.co/go/tools/stylecheck/st1003
26582026-03-11T19:42:23.379Zhonnef.co/go/tools/stylecheck/st1005
26592026-03-11T19:42:23.379Zhonnef.co/go/tools/stylecheck/st1006
26602026-03-11T19:42:23.383Z[ 88%] Building CXX object CMakeFiles/geos.dir/src/planargraph/Node.cpp.o
26612026-03-11T19:42:23.386Zhonnef.co/go/tools/stylecheck/st1008
26622026-03-11T19:42:23.390Zhonnef.co/go/tools/stylecheck/st1011
26632026-03-11T19:42:23.393Zhonnef.co/go/tools/stylecheck/st1012
26642026-03-11T19:42:23.396Zhonnef.co/go/tools/stylecheck/st1013
26652026-03-11T19:42:23.396Zhonnef.co/go/tools/stylecheck/st1015
26662026-03-11T19:42:23.396Zhonnef.co/go/tools/stylecheck/st1016
26672026-03-11T19:42:23.399Zhonnef.co/go/tools/stylecheck/st1017
26682026-03-11T19:42:23.404Zhonnef.co/go/tools/stylecheck/st1018
26692026-03-11T19:42:23.407Zgithub.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway/generator
26702026-03-11T19:42:23.414Zhonnef.co/go/tools/stylecheck/st1019
26712026-03-11T19:42:23.417Z[ 88%] Building CXX object CMakeFiles/geos.dir/src/planargraph/NodeMap.cpp.o
26722026-03-11T19:42:23.417Zhonnef.co/go/tools/stylecheck/st1020
26732026-03-11T19:42:23.427Zhonnef.co/go/tools/stylecheck/st1021
26742026-03-11T19:42:23.430Zhonnef.co/go/tools/staticcheck
26752026-03-11T19:42:23.430Zhonnef.co/go/tools/stylecheck/st1022
26762026-03-11T19:42:23.433Zgithub.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway/internal/gengateway
26772026-03-11T19:42:23.437Zhonnef.co/go/tools/stylecheck/st1023
26782026-03-11T19:42:23.483Zhonnef.co/go/tools/stylecheck
26792026-03-11T19:42:23.486Zgithub.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway
26802026-03-11T19:42:23.517Z[ 88%] Building CXX object CMakeFiles/geos.dir/src/planargraph/PlanarGraph.cpp.o
26812026-03-11T19:42:23.522Zhonnef.co/go/tools/cmd/staticcheck
26822026-03-11T19:42:23.529Z[ 90%] Building CXX object CMakeFiles/geos.dir/src/planargraph/algorithm/ConnectedSubgraphFinder.cpp.o
26832026-03-11T19:42:23.532Z[ 90%] Building CXX object CMakeFiles/geos.dir/src/planargraph/Subgraph.cpp.o
26842026-03-11T19:42:23.535Z[ 90%] Building CXX object CMakeFiles/geos.dir/src/precision/CommonBits.cpp.o
26852026-03-11T19:42:23.602Z[ 90%] Building CXX object CMakeFiles/geos.dir/src/precision/CommonBitsOp.cpp.o
26862026-03-11T19:42:23.607Z[ 90%] Building CXX object CMakeFiles/geos.dir/src/precision/CommonBitsRemover.cpp.o
26872026-03-11T19:42:23.613Z[ 91%] Building CXX object CMakeFiles/geos.dir/src/precision/GeometryPrecisionReducer.cpp.o
26882026-03-11T19:42:23.622Z[ 91%] Building CXX object CMakeFiles/geos.dir/src/precision/MinimumClearance.cpp.o
26892026-03-11T19:42:23.627Z[ 91%] Building CXX object CMakeFiles/geos.dir/src/precision/EnhancedPrecisionOp.cpp.o
26902026-03-11T19:42:23.673Z[ 91%] Building CXX object CMakeFiles/geos.dir/src/precision/SimpleGeometryPrecisionReducer.cpp.o
26912026-03-11T19:42:23.676Z[ 91%] Building CXX object CMakeFiles/geos.dir/src/precision/PrecisionReducerCoordinateOperation.cpp.o
26922026-03-11T19:42:23.702Z[ 91%] Building CXX object CMakeFiles/geos.dir/src/simplify/DouglasPeuckerLineSimplifier.cpp.o
26932026-03-11T19:42:23.705Z[ 93%] Building CXX object CMakeFiles/geos.dir/src/simplify/DouglasPeuckerSimplifier.cpp.o
26942026-03-11T19:42:23.708Z[ 93%] Building CXX object CMakeFiles/geos.dir/src/simplify/LineSegmentIndex.cpp.o
26952026-03-11T19:42:23.741Z[ 93%] Building CXX object CMakeFiles/geos.dir/src/simplify/TaggedLineSegment.cpp.o
26962026-03-11T19:42:23.753Z[ 93%] Building CXX object CMakeFiles/geos.dir/src/simplify/TaggedLineString.cpp.o
26972026-03-11T19:42:23.811Z[ 93%] Building CXX object CMakeFiles/geos.dir/src/simplify/TaggedLineStringSimplifier.cpp.o
26982026-03-11T19:42:23.814Z[ 95%] Building CXX object CMakeFiles/geos.dir/src/simplify/TaggedLinesSimplifier.cpp.o
26992026-03-11T19:42:23.852Z[ 95%] Building CXX object CMakeFiles/geos.dir/src/simplify/TopologyPreservingSimplifier.cpp.o
27002026-03-11T19:42:23.865Z[ 95%] Building CXX object CMakeFiles/geos.dir/src/triangulate/DelaunayTriangulationBuilder.cpp.o
27012026-03-11T19:42:23.889Z[ 95%] Building CXX object CMakeFiles/geos.dir/src/triangulate/VoronoiDiagramBuilder.cpp.o
27022026-03-11T19:42:23.892Z[ 95%] Building CXX object CMakeFiles/geos.dir/src/triangulate/IncrementalDelaunayTriangulator.cpp.o
27032026-03-11T19:42:23.902Z[ 96%] Building CXX object CMakeFiles/geos.dir/src/triangulate/quadedge/LastFoundQuadEdgeLocator.cpp.o
27042026-03-11T19:42:23.963Z[ 96%] Building CXX object CMakeFiles/geos.dir/src/triangulate/quadedge/LocateFailureException.cpp.o
27052026-03-11T19:42:23.963Z[ 96%] Building CXX object CMakeFiles/geos.dir/src/triangulate/quadedge/QuadEdge.cpp.o
27062026-03-11T19:42:23.963Z[ 96%] Building CXX object CMakeFiles/geos.dir/src/triangulate/quadedge/QuadEdgeSubdivision.cpp.o
27072026-03-11T19:42:23.968Z[ 96%] Building CXX object CMakeFiles/geos.dir/src/triangulate/quadedge/TrianglePredicate.cpp.o
27082026-03-11T19:42:23.973Z[ 98%] Building CXX object CMakeFiles/geos.dir/src/triangulate/quadedge/Vertex.cpp.o
27092026-03-11T19:42:23.977Z[ 98%] Building CXX object CMakeFiles/geos.dir/src/util/Assert.cpp.o
27102026-03-11T19:42:23.985Ztouch bin/.bootstrap
27112026-03-11T19:42:23.988Zfind ./pkg -name node_modules -prune -o -type f -name '*.pb.go' -exec rm {} +
27122026-03-11T19:42:23.997Zset -e; for dir in ./pkg/acceptance/cluster/ ./pkg/blobs/blobspb/ ./pkg/build/ ./pkg/clusterversion/ ./pkg/config/ ./pkg/config/zonepb/ ./pkg/geo/geoindex/ ./pkg/geo/geopb/ ./pkg/gossip/ ./pkg/jobs/jobspb/ ./pkg/kv/kvnemesis/ ./pkg/kv/kvserver/ ./pkg/kv/kvserver/closedts/ctpb/ ./pkg/kv/kvserver/concurrency/lock/ ./pkg/kv/kvserver/concurrency/poison/ ./pkg/kv/kvserver/kvserverpb/ ./pkg/kv/kvserver/liveness/livenesspb/ ./pkg/kv/kvserver/loqrecovery/loqrecoverypb/ ./pkg/kv/kvserver/protectedts/ptpb/ ./pkg/kv/kvserver/protectedts/ptstorage/ ./pkg/kv/kvserver/readsummary/rspb/ ./pkg/roachpb/ ./pkg/rpc/ ./pkg/server/diagnostics/diagnosticspb/ ./pkg/server/serverpb/ ./pkg/server/status/statuspb/ ./pkg/settings/ ./pkg/sql/catalog/catpb/ ./pkg/sql/catalog/descpb/ ./pkg/sql/contentionpb/ ./pkg/sql/execinfrapb/ ./pkg/sql/inverted/ ./pkg/sql/lex/ ./pkg/sql/pgwire/pgerror/ ./pkg/sql/protoreflect/test/ ./pkg/sql/rowenc/rowencpb/ ./pkg/sql/schemachanger/scpb/ ./pkg/sql/sessiondatapb/ ./pkg/sql/sqlstats/persistedsqlstats/ ./pkg/sql/stats/ ./pkg/sql/types/ ./pkg/startupmigrations/leasemanager/ ./pkg/storage/enginepb/ ./pkg/testutils/grpcutils/ ./pkg/ts/catalog/ ./pkg/ts/tspb/ ./pkg/util/ ./pkg/util/duration/ ./pkg/util/hlc/ ./pkg/util/log/eventpb/ ./pkg/util/log/logpb/ ./pkg/util/metric/ ./pkg/util/optional/ ./pkg/util/protoutil/ ./pkg/util/timeutil/pgdate/ ./pkg/util/tracing/tracingpb/ ./pkg/util/tracing/tracingservicepb/; do \
27132026-03-11T19:42:23.997Z buf protoc -Ipkg -I./vendor/github.com/gogo/protobuf -I./vendor/go.etcd.io -I./vendor/github.com/prometheus/client_model -I./vendor/github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis -I./vendor/github.com/cockroachdb/errors --gogoroach_out=Mgoogle/api/annotations.proto=google.golang.org/genproto/googleapis/api/annotations,Mgoogle/protobuf/timestamp.proto=github.com/gogo/protobuf/types,Mgoogle/protobuf/any.proto=github.com/gogo/protobuf/types,Mgoogle/protobuf/duration.proto=github.com/gogo/protobuf/types,plugins=grpc,import_prefix=github.com/cockroachdb/cockroach/pkg/:./pkg $dir/*.proto; \
27142026-03-11T19:42:23.997Zdone
27152026-03-11T19:42:24.015Z[ 98%] Building CXX object CMakeFiles/geos.dir/src/util/GeometricShapeFactory.cpp.o
27162026-03-11T19:42:24.052Z[ 98%] Building CXX object CMakeFiles/geos.dir/src/util/Interrupt.cpp.o
27172026-03-11T19:42:24.086Z[ 98%] Building CXX object CMakeFiles/geos.dir/src/util/Profiler.cpp.o
27182026-03-11T19:42:24.093Zfind ./pkg -name node_modules -prune -o -type f -name '*.pb.gw.go' -exec rm {} +
27192026-03-11T19:42:25.106Z[100%] Building CXX object CMakeFiles/geos.dir/src/util/math.cpp.o
27202026-03-11T19:42:25.109Zbuf protoc -Ipkg -I./vendor/github.com/gogo/protobuf -I./vendor/github.com/cockroachdb/errors -I./vendor/go.etcd.io -I./vendor/github.com/prometheus/client_model -I./vendor/github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis --grpc-gateway_out=logtostderr=true,request_context=true:./pkg ./pkg/server/serverpb/admin.proto ./pkg/server/serverpb/status.proto ./pkg/server/serverpb/authentication.proto
27212026-03-11T19:42:25.109Z(cd pkg/roachprod/vm/aws && go generate)
27222026-03-11T19:42:25.109Z(cd pkg/security/securitytest && go generate)
27232026-03-11T19:42:25.109Zgo run -mod=vendor pkg/util/log/gen/main.go pkg/util/log/logpb/log.proto channel.go pkg/util/log/channel/channel_generated.go.tmp || { rm -f pkg/util/log/channel/channel_generated.go.tmp; exit 1; }
27242026-03-11T19:42:25.109Zset -euo pipefail; \
27252026-03-11T19:42:25.109Z ret=$(cd pkg/sql/parser/gen && goyacc -p sql -o sql.go.tmp sql-gen.y); \
27262026-03-11T19:42:25.109Z if expr "$ret" : ".*conflicts" >/dev/null; then \
27272026-03-11T19:42:25.109Z echo "$ret"; exit 1; \
27282026-03-11T19:42:25.109Z fi
27292026-03-11T19:42:25.109Zmv -f pkg/sql/parser/helpmap_test.go.tmp pkg/sql/parser/helpmap_test.go
27302026-03-11T19:42:25.109Zgofmt -s -w pkg/sql/parser/helpmap_test.go
27312026-03-11T19:42:25.109Zawk -f pkg/sql/parser/help.awk < pkg/sql/parser/sql.y > pkg/sql/parser/help_messages.go.tmp || rm pkg/sql/parser/help_messages.go.tmp
27322026-03-11T19:42:25.109Zgo run -mod=vendor -tags all-keywords pkg/sql/lexbase/allkeywords/main.go < pkg/sql/parser/sql.y > pkg/sql/lexbase/keywords.go.tmp || rm pkg/sql/lexbase/keywords.go.tmp
27332026-03-11T19:42:25.109Zawk -f pkg/sql/parser/reserved_keywords.awk < pkg/sql/parser/sql.y > pkg/sql/lexbase/reserved_keywords.go.tmp || rm pkg/sql/lexbase/reserved_keywords.go.tmp
27342026-03-11T19:42:25.109Zmv -f pkg/sql/parser/help_messages.go.tmp pkg/sql/parser/help_messages.go
27352026-03-11T19:42:25.109Zgofmt -s -w pkg/sql/parser/help_messages.go
27362026-03-11T19:42:25.109Zmv -f pkg/sql/lexbase/reserved_keywords.go.tmp pkg/sql/lexbase/reserved_keywords.go
27372026-03-11T19:42:25.109Zgofmt -s -w pkg/sql/lexbase/reserved_keywords.go
27382026-03-11T19:42:25.109Zstringer -output=pkg/sql/opt/rule_name_string.go -type=RuleName pkg/sql/opt/rule_name.go pkg/sql/opt/rule_name.og.go
27392026-03-11T19:42:25.109Zbuf protoc -Ipkg -I./vendor/github.com/gogo/protobuf -I./vendor/github.com/cockroachdb/errors -I./vendor/go.etcd.io -I./vendor/github.com/prometheus/client_model -I./vendor/github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis --grpc-gateway_out=logtostderr=true,request_context=true:./pkg ./pkg/ts/tspb/timeseries.proto
27402026-03-11T19:42:25.109Zgofmt -s -w ./pkg/server/serverpb/admin.pb.gw.go ./pkg/server/serverpb/status.pb.gw.go ./pkg/server/serverpb/authentication.pb.gw.go ./pkg/ts/tspb/timeseries.pb.gw.go
27412026-03-11T19:42:25.109Zmv -f pkg/sql/lexbase/keywords.go.tmp pkg/sql/lexbase/keywords.go
27422026-03-11T19:42:25.109Zgofmt -s -w pkg/sql/lexbase/keywords.go
27432026-03-11T19:42:25.109Zgoimports -w ./pkg/server/serverpb/admin.pb.gw.go ./pkg/server/serverpb/status.pb.gw.go ./pkg/server/serverpb/authentication.pb.gw.go ./pkg/ts/tspb/timeseries.pb.gw.go
27442026-03-11T19:42:25.110Ztouch bin/.gw_protobuf_sources
27452026-03-11T19:42:25.110Zmv -f pkg/util/log/channel/channel_generated.go.tmp pkg/util/log/channel/channel_generated.go
27462026-03-11T19:42:25.110Z[100%] Linking CXX shared library lib/libgeos.so
27472026-03-11T19:42:25.110Z[100%] Building CXX object CMakeFiles/geos_c.dir/capi/geos_ts_c.cpp.o
27482026-03-11T19:42:25.110Z[100%] Building CXX object CMakeFiles/geos_c.dir/capi/geos_c.cpp.o
27492026-03-11T19:42:26.007Z(echo "// Code generated by goyacc. DO NOT EDIT."; \
27502026-03-11T19:42:26.007Z echo "// GENERATED FILE DO NOT EDIT"; \
27512026-03-11T19:42:26.007Z cat pkg/sql/parser/gen/sql.go.tmp | \
27522026-03-11T19:42:26.007Z sed -E 's/^const ([A-Z][_A-Z0-9]*) =.*$/const \1 = lexbase.\1/g') > pkg/sql/parser/sql.go.tmp || rm pkg/sql/parser/sql.go.tmp
27532026-03-11T19:42:26.010Z(echo "// Code generated by make. DO NOT EDIT."; \
27542026-03-11T19:42:26.010Z echo "// GENERATED FILE DO NOT EDIT"; \
27552026-03-11T19:42:26.010Z echo; \
27562026-03-11T19:42:26.010Z echo "package lexbase"; \
27572026-03-11T19:42:26.010Z echo; \
27582026-03-11T19:42:26.010Z grep '^const [A-Z][_A-Z0-9]* ' pkg/sql/parser/gen/sql.go.tmp) > pkg/sql/lexbase/tokens.go.tmp || rm pkg/sql/lexbase/tokens.go.tmp
27592026-03-11T19:42:26.011Zmv -f pkg/sql/lexbase/tokens.go.tmp pkg/sql/lexbase/tokens.go
27602026-03-11T19:42:26.014Zmv -f pkg/sql/parser/sql.go.tmp pkg/sql/parser/sql.go
27612026-03-11T19:42:26.014Zgoimports -w pkg/sql/parser/sql.go
27622026-03-11T19:42:26.843Z[100%] Linking CXX shared library lib/libgeos_c.so
27632026-03-11T19:42:26.886Zmkdir -p lib
27642026-03-11T19:42:26.888Zrm -f lib/lib{geos,geos_c}.so
27652026-03-11T19:42:26.888Zcp /work/oxidecomputer/cockroach/lib/build/geos/lib/lib{geos,geos_c}.so lib
27662026-03-11T19:42:28.742Zgofmt -s -w ./pkg/acceptance/cluster/testconfig.pb.go ./pkg/blobs/blobspb/blobs.pb.go ./pkg/build/info.pb.go ./pkg/clusterversion/cluster_version.pb.go ./pkg/config/system.pb.go ./pkg/config/zonepb/zone.pb.go ./pkg/geo/geoindex/config.pb.go ./pkg/geo/geopb/geopb.pb.go ./pkg/gossip/gossip.pb.go ./pkg/jobs/jobspb/jobs.pb.go ./pkg/jobs/jobspb/schedule.pb.go ./pkg/kv/kvnemesis/operations.pb.go ./pkg/kv/kvserver/api.pb.go ./pkg/kv/kvserver/closedts/ctpb/service.pb.go ./pkg/kv/kvserver/concurrency/lock/lock_waiter.pb.go ./pkg/kv/kvserver/concurrency/lock/locking.pb.go ./pkg/kv/kvserver/concurrency/poison/error.pb.go ./pkg/kv/kvserver/concurrency/poison/policy.pb.go ./pkg/kv/kvserver/kvserverpb/lease_status.pb.go ./pkg/kv/kvserver/kvserverpb/proposer_kv.pb.go ./pkg/kv/kvserver/kvserverpb/raft.pb.go ./pkg/kv/kvserver/kvserverpb/range_log.pb.go ./pkg/kv/kvserver/kvserverpb/state.pb.go ./pkg/kv/kvserver/liveness/livenesspb/liveness.pb.go ./pkg/kv/kvserver/loqrecovery/loqrecoverypb/recovery.pb.go ./pkg/kv/kvserver/protectedts/ptpb/protectedts.pb.go ./pkg/kv/kvserver/protectedts/ptstorage/storage.pb.go ./pkg/kv/kvserver/readsummary/rspb/summary.pb.go ./pkg/kv/kvserver/storage_services.pb.go ./pkg/roachpb/api.pb.go ./pkg/roachpb/app_stats.pb.go ./pkg/roachpb/data.pb.go ./pkg/roachpb/errors.pb.go ./pkg/roachpb/index_usage_stats.pb.go ./pkg/roachpb/internal.pb.go ./pkg/roachpb/internal_raft.pb.go ./pkg/roachpb/io-formats.pb.go ./pkg/roachpb/metadata.pb.go ./pkg/roachpb/span_config.pb.go ./pkg/rpc/heartbeat.pb.go ./pkg/server/diagnostics/diagnosticspb/diagnostics.pb.go ./pkg/server/serverpb/admin.pb.go ./pkg/server/serverpb/authentication.pb.go ./pkg/server/serverpb/init.pb.go ./pkg/server/serverpb/migration.pb.go ./pkg/server/serverpb/status.pb.go ./pkg/server/status/statuspb/status.pb.go ./pkg/settings/encoding.pb.go ./pkg/sql/catalog/catpb/catalog.pb.go ./pkg/sql/catalog/catpb/privilege.pb.go ./pkg/sql/catalog/descpb/encoded_datum.pb.go ./pkg/sql/catalog/descpb/index_fetch.pb.go ./pkg/sql/catalog/descpb/join_type.pb.go ./pkg/sql/catalog/descpb/locking.pb.go ./pkg/sql/catalog/descpb/structured.pb.go ./pkg/sql/catalog/descpb/tenant.pb.go ./pkg/sql/contentionpb/contention.pb.go ./pkg/sql/execinfrapb/api.pb.go ./pkg/sql/execinfrapb/component_stats.pb.go ./pkg/sql/execinfrapb/data.pb.go ./pkg/sql/execinfrapb/processors.pb.go ./pkg/sql/execinfrapb/processors_base.pb.go ./pkg/sql/execinfrapb/processors_bulk_io.pb.go ./pkg/sql/execinfrapb/processors_changefeeds.pb.go ./pkg/sql/execinfrapb/processors_sql.pb.go ./pkg/sql/execinfrapb/processors_table_stats.pb.go ./pkg/sql/inverted/span_expression.pb.go ./pkg/sql/lex/encode.pb.go ./pkg/sql/pgwire/pgerror/errors.pb.go ./pkg/sql/protoreflect/test/test.pb.go ./pkg/sql/rowenc/rowencpb/index_encoding.pb.go ./pkg/sql/schemachanger/scpb/elements.pb.go ./pkg/sql/schemachanger/scpb/scpb.pb.go ./pkg/sql/sessiondatapb/local_only_session_data.pb.go ./pkg/sql/sessiondatapb/session_data.pb.go ./pkg/sql/sessiondatapb/session_migration.pb.go ./pkg/sql/sessiondatapb/session_revival_token.pb.go ./pkg/sql/sqlstats/persistedsqlstats/sql_stats_compact.pb.go ./pkg/sql/stats/histogram.pb.go ./pkg/sql/stats/table_statistic.pb.go ./pkg/sql/types/types.pb.go ./pkg/startupmigrations/leasemanager/lease.pb.go ./pkg/storage/enginepb/engine.pb.go ./pkg/storage/enginepb/file_registry.pb.go ./pkg/storage/enginepb/mvcc.pb.go ./pkg/storage/enginepb/mvcc3.pb.go ./pkg/storage/enginepb/rocksdb.pb.go ./pkg/testutils/grpcutils/testservice.pb.go ./pkg/ts/catalog/chart_catalog.pb.go ./pkg/ts/tspb/timeseries.pb.go ./pkg/util/duration/duration.pb.go ./pkg/util/hlc/legacy_timestamp.pb.go ./pkg/util/hlc/timestamp.pb.go ./pkg/util/log/eventpb/cluster_events.pb.go ./pkg/util/log/eventpb/ddl_events.pb.go ./pkg/util/log/eventpb/debug_events.pb.go ./pkg/util/log/eventpb/events.pb.go ./pkg/util/log/eventpb/health_events.pb.go ./pkg/util/log/eventpb/job_events.pb.go ./pkg/util/log/eventpb/misc_sql_events.pb.go ./pkg/util/log/eventpb/privilege_events.pb.go ./pkg/util/log/eventpb/role_events.pb.go ./pkg/util/log/eventpb/session_events.pb.go ./pkg/util/log/eventpb/sql_audit_events.pb.go ./pkg/util/log/eventpb/telemetry.pb.go ./pkg/util/log/eventpb/zone_events.pb.go ./pkg/util/log/logpb/log.pb.go ./pkg/util/metric/metric.pb.go ./pkg/util/optional/optional.pb.go ./pkg/util/protoutil/clone.pb.go ./pkg/util/timeutil/pgdate/pgdate.pb.go ./pkg/util/tracing/tracingpb/recorded_span.pb.go ./pkg/util/tracing/tracingpb/tracing.pb.go ./pkg/util/tracing/tracingservicepb/tracing_service.pb.go ./pkg/util/unresolved_addr.pb.go
27672026-03-11T19:42:29.538Ztouch bin/.go_protobuf_sources
27682026-03-11T19:42:29.541ZGOFLAGS= go test -mod=vendor -tags 'make x86_64_linux_gnu crdb_test' -ldflags '-X github.com/cockroachdb/cockroach/pkg/build.typ=development -extldflags "" -X "github.com/cockroachdb/cockroach/pkg/build.tag=v22.1.22-59-g76d728e772" -X "github.com/cockroachdb/cockroach/pkg/build.rev=76d728e7724724234d36ada4d402a921657c3724" -X "github.com/cockroachdb/cockroach/pkg/build.cgoTargetTriple=x86_64-linux-gnu" ' -run "." -timeout 60m ./pkg/...
27692026-03-11T19:42:49.772Zok github.com/cockroachdb/cockroach/pkg/acceptance 0.012s
27702026-03-11T19:42:49.772Z? github.com/cockroachdb/cockroach/pkg/acceptance/cluster [no test files]
27712026-03-11T19:43:07.022Z? github.com/cockroachdb/cockroach/pkg/acceptance/localcluster [no test files]
27722026-03-11T19:43:07.022Z? github.com/cockroachdb/cockroach/pkg/acceptance/localcluster/tc [no test files]
27732026-03-11T19:43:07.635Zok github.com/cockroachdb/cockroach/pkg/base 0.036s
27742026-03-11T19:43:35.303Zok github.com/cockroachdb/cockroach/pkg/bench 0.023s [no tests to run]
27752026-03-11T19:43:35.406Z? github.com/cockroachdb/cockroach/pkg/bench/cmd/pgbenchsetup [no test files]
27762026-03-11T19:43:39.106Z? github.com/cockroachdb/cockroach/pkg/blobs/blobspb [no test files]
27772026-03-11T19:43:39.109Z? github.com/cockroachdb/cockroach/pkg/build [no test files]
27782026-03-11T19:43:39.109Z? github.com/cockroachdb/cockroach/pkg/build/bazel [no test files]
27792026-03-11T19:43:41.863Zok github.com/cockroachdb/cockroach/pkg/bench/rttanalysis 5.094s
27802026-03-11T19:43:41.867Zok github.com/cockroachdb/cockroach/pkg/bench/tpcc 0.050s
27812026-03-11T19:43:41.867Zok github.com/cockroachdb/cockroach/pkg/blobs 0.196s
27822026-03-11T19:43:41.867Zok github.com/cockroachdb/cockroach/pkg/build/starlarkutil 0.031s
27832026-03-11T19:43:41.867Zok github.com/cockroachdb/cockroach/pkg/build/util 0.027s
27842026-03-11T19:43:45.183Z? github.com/cockroachdb/cockroach/pkg/cli/clicfg [no test files]
27852026-03-11T19:43:45.186Z? github.com/cockroachdb/cockroach/pkg/cli/clierrorplus [no test files]
27862026-03-11T19:43:45.186Z? github.com/cockroachdb/cockroach/pkg/cli/cliflags [no test files]
27872026-03-11T19:43:45.186Z? github.com/cockroachdb/cockroach/pkg/cli/clisqlcfg [no test files]
27882026-03-11T19:43:45.568Z? github.com/cockroachdb/cockroach/pkg/cli/democluster/api [no test files]
27892026-03-11T19:43:45.572Z? github.com/cockroachdb/cockroach/pkg/cli/syncbench [no test files]
27902026-03-11T19:43:45.572Z? github.com/cockroachdb/cockroach/pkg/cloud [no test files]
27912026-03-11T19:43:46.352Z? github.com/cockroachdb/cockroach/pkg/cloud/cloudtestutils [no test files]
27922026-03-11T19:43:46.652Z? github.com/cockroachdb/cockroach/pkg/cloud/impl [no test files]
27932026-03-11T19:43:48.507Z? github.com/cockroachdb/cockroach/pkg/cloud/userfile/filetable [no test files]
27942026-03-11T19:43:48.904Z? github.com/cockroachdb/cockroach/pkg/cmd/allocsim [no test files]
27952026-03-11T19:43:48.907Z? github.com/cockroachdb/cockroach/pkg/cmd/cmdutil [no test files]
27962026-03-11T19:43:48.907Z? github.com/cockroachdb/cockroach/pkg/cmd/cmp-protocol [no test files]
27972026-03-11T19:43:48.907Z? github.com/cockroachdb/cockroach/pkg/cmd/cmp-protocol/pgconnect [no test files]
27982026-03-11T19:43:48.907Z? github.com/cockroachdb/cockroach/pkg/cmd/cmp-sql [no test files]
27992026-03-11T19:43:50.312Z? github.com/cockroachdb/cockroach/pkg/cmd/cockroach-short [no test files]
28002026-03-11T19:43:50.312Z? github.com/cockroachdb/cockroach/pkg/cmd/cockroach-sql [no test files]
28012026-03-11T19:43:50.315Z? github.com/cockroachdb/cockroach/pkg/cmd/cr2pg [no test files]
28022026-03-11T19:43:50.315Z? github.com/cockroachdb/cockroach/pkg/cmd/cr2pg/sqlstream [no test files]
28032026-03-11T19:43:50.315Z? github.com/cockroachdb/cockroach/pkg/cmd/dev/io/exec [no test files]
28042026-03-11T19:43:50.315Z? github.com/cockroachdb/cockroach/pkg/cmd/dev/io/os [no test files]
28052026-03-11T19:43:51.384Z? github.com/cockroachdb/cockroach/pkg/cmd/docgen [no test files]
28062026-03-11T19:43:51.387Z? github.com/cockroachdb/cockroach/pkg/cmd/fuzz [no test files]
28072026-03-11T19:43:51.387Z? github.com/cockroachdb/cockroach/pkg/cmd/generate-binary [no test files]
28082026-03-11T19:43:51.387Z? github.com/cockroachdb/cockroach/pkg/cmd/generate-distdir [no test files]
28092026-03-11T19:43:51.387Z? github.com/cockroachdb/cockroach/pkg/cmd/generate-metadata-tables [no test files]
28102026-03-11T19:43:51.387Z? github.com/cockroachdb/cockroach/pkg/cmd/generate-metadata-tables/rdbms [no test files]
28112026-03-11T19:43:51.387Z? github.com/cockroachdb/cockroach/pkg/cmd/generate-spatial-ref-sys [no test files]
28122026-03-11T19:43:51.387Z? github.com/cockroachdb/cockroach/pkg/cmd/generate-staticcheck [no test files]
28132026-03-11T19:43:51.387Z? github.com/cockroachdb/cockroach/pkg/cmd/generate-test-suites [no test files]
28142026-03-11T19:43:51.387Z? github.com/cockroachdb/cockroach/pkg/cmd/geoviz [no test files]
28152026-03-11T19:43:51.387Z? github.com/cockroachdb/cockroach/pkg/cmd/gossipsim [no test files]
28162026-03-11T19:43:51.387Z? github.com/cockroachdb/cockroach/pkg/cmd/import-tools [no test files]
28172026-03-11T19:43:52.824Z? github.com/cockroachdb/cockroach/pkg/cmd/protoc-gen-gogoroach [no test files]
28182026-03-11T19:43:52.827Z? github.com/cockroachdb/cockroach/pkg/cmd/reduce [no test files]
28192026-03-11T19:43:54.460Z? github.com/cockroachdb/cockroach/pkg/cmd/returncheck [no test files]
28202026-03-11T19:44:02.974Z? github.com/cockroachdb/cockroach/pkg/cmd/roachprod [no test files]
28212026-03-11T19:44:02.974Z? github.com/cockroachdb/cockroach/pkg/cmd/roachprod-stress [no test files]
28222026-03-11T19:44:07.106Z? github.com/cockroachdb/cockroach/pkg/cmd/roachtest/cluster [no test files]
28232026-03-11T19:44:07.106Z? github.com/cockroachdb/cockroach/pkg/cmd/roachtest/option [no test files]
28242026-03-11T19:44:07.702Z? github.com/cockroachdb/cockroach/pkg/cmd/roachtest/registry [no test files]
28252026-03-11T19:44:07.702Z? github.com/cockroachdb/cockroach/pkg/cmd/roachtest/spec [no test files]
28262026-03-11T19:44:07.705Z? github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test [no test files]
28272026-03-11T19:44:11.300Z? github.com/cockroachdb/cockroach/pkg/cmd/roachvet [no test files]
28282026-03-11T19:44:11.300Z? github.com/cockroachdb/cockroach/pkg/cmd/skip-test [no test files]
28292026-03-11T19:44:11.300Z? github.com/cockroachdb/cockroach/pkg/cmd/skiperrs [no test files]
28302026-03-11T19:44:11.300Z? github.com/cockroachdb/cockroach/pkg/cmd/skipped-tests [no test files]
28312026-03-11T19:44:11.300Z? github.com/cockroachdb/cockroach/pkg/cmd/smithcmp [no test files]
28322026-03-11T19:44:11.304Z? github.com/cockroachdb/cockroach/pkg/cmd/smithtest [no test files]
28332026-03-11T19:44:11.304Z? github.com/cockroachdb/cockroach/pkg/cmd/uptodate [no test files]
28342026-03-11T19:44:11.304Z? github.com/cockroachdb/cockroach/pkg/cmd/urlcheck [no test files]
28352026-03-11T19:44:11.304Z? github.com/cockroachdb/cockroach/pkg/cmd/urlcheck/lib/urlcheck [no test files]
28362026-03-11T19:44:11.304Z? github.com/cockroachdb/cockroach/pkg/cmd/workload [no test files]
28372026-03-11T19:44:11.304Z? github.com/cockroachdb/cockroach/pkg/cmd/wraprules [no test files]
28382026-03-11T19:44:11.304Z? github.com/cockroachdb/cockroach/pkg/cmd/zerosum [no test files]
28392026-03-11T19:44:12.193Z? github.com/cockroachdb/cockroach/pkg/col/coldatatestutils [no test files]
28402026-03-11T19:44:12.624Z? github.com/cockroachdb/cockroach/pkg/col/colserde/arrowserde [no test files]
28412026-03-11T19:44:12.627Z? github.com/cockroachdb/cockroach/pkg/col/typeconv [no test files]
28422026-03-11T19:44:12.627Z? github.com/cockroachdb/cockroach/pkg/compose [no test files]
28432026-03-11T19:44:12.627Z? github.com/cockroachdb/cockroach/pkg/compose/compare/compare [no test files]
28442026-03-11T19:44:12.627Z? github.com/cockroachdb/cockroach/pkg/docs [no test files]
28452026-03-11T19:44:12.627Z? github.com/cockroachdb/cockroach/pkg/featureflag [no test files]
28462026-03-11T19:44:12.627Z? github.com/cockroachdb/cockroach/pkg/gen/genbzl [no test files]
28472026-03-11T19:44:12.627Z? github.com/cockroachdb/cockroach/pkg/geo/geodist [no test files]
28482026-03-11T19:44:13.558Z? github.com/cockroachdb/cockroach/pkg/geo/geoprojbase/embeddedproj [no test files]
28492026-03-11T19:44:13.561Z? github.com/cockroachdb/cockroach/pkg/geo/geosegmentize [no test files]
28502026-03-11T19:44:14.271Z? github.com/cockroachdb/cockroach/pkg/gossip/simulation [no test files]
28512026-03-11T19:44:14.275Z? github.com/cockroachdb/cockroach/pkg/internal/reporoot [no test files]
28522026-03-11T19:44:29.849Z? github.com/cockroachdb/cockroach/pkg/jobs/jobspb [no test files]
28532026-03-11T19:44:29.852Z? github.com/cockroachdb/cockroach/pkg/jobs/jobstest [no test files]
28542026-03-11T19:44:29.852Z? github.com/cockroachdb/cockroach/pkg/keysbase [no test files]
28552026-03-11T19:44:51.552Z? github.com/cockroachdb/cockroach/pkg/kv/kvbase [no test files]
28562026-03-11T19:44:51.555Z? github.com/cockroachdb/cockroach/pkg/kv/kvclient [no test files]
28572026-03-11T19:44:52.340Z? github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvtenant [no test files]
28582026-03-11T19:44:52.348Z? github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangecache/rangecachemock [no test files]
28592026-03-11T19:45:18.525Zok github.com/cockroachdb/cockroach/pkg/cli 93.224s
28602026-03-11T19:45:18.525Zok github.com/cockroachdb/cockroach/pkg/cli/clierror 0.296s
28612026-03-11T19:45:18.525Zok github.com/cockroachdb/cockroach/pkg/cli/clisqlclient 0.685s
28622026-03-11T19:45:18.525Zok github.com/cockroachdb/cockroach/pkg/cli/clisqlexec 2.373s
28632026-03-11T19:45:18.525Zok github.com/cockroachdb/cockroach/pkg/cli/clisqlshell 2.185s
28642026-03-11T19:45:18.525Zinitialized metamorphic constant "max-batch-size" with value 3558
28652026-03-11T19:45:18.525Zinitialized metamorphic constant "max-batch-byte-size" with value 13230482
28662026-03-11T19:45:18.525Zinitialized metamorphic constant "row-container-rows-per-chunk-shift" with value 1
28672026-03-11T19:45:18.525Zinitialized metamorphic constant "split-scans-right-for-stats-first" with value true
28682026-03-11T19:45:18.525Zinitialized metamorphic constant "kv.rangefeed.use_dedicated_connection_class.enabled" with value true
28692026-03-11T19:45:18.526Zinitialized metamorphic constant "raft-log-truncation-clearrange-threshold" with value 128800
28702026-03-11T19:45:18.526Zinitialized metamorphic constant "spilling-queue-initial-len" with value 6
28712026-03-11T19:45:18.526Zinitialized metamorphic constant "merge-joiner-groups-buffer" with value 9
28722026-03-11T19:45:18.526Zinitialized metamorphic constant "ColIndexJoin-batch-size" with value 4177035
28732026-03-11T19:45:18.526Zinitialized metamorphic constant "inverted-joiner-batch-size" with value 1
28742026-03-11T19:45:18.526Zinitialized metamorphic constant "zig-zag-joiner-batch-size" with value 1
28752026-03-11T19:45:18.526Zinitialized metamorphic constant "resolve-membership-single-scan-enabled" with value false
28762026-03-11T19:45:18.526Zinitialized metamorphic constant "async-IE-result-channel-buffer-size" with value 18
28772026-03-11T19:45:18.526Ztest logs left over in: /tmp/logTestTransientClusterSimulateLatencies3816193631
28782026-03-11T19:45:18.526Z--- FAIL: TestTransientClusterSimulateLatencies (0.75s)
28792026-03-11T19:45:18.526Z test_log_scope.go:79: test logs captured to: /tmp/logTestTransientClusterSimulateLatencies3816193631
28802026-03-11T19:45:18.526Z test_log_scope.go:80: use -show-logs to present logs inline
28812026-03-11T19:45:18.526Z --- FAIL: TestTransientClusterSimulateLatencies/from_us-west1 (0.06s)
28822026-03-11T19:45:18.526Z demo_cluster_test.go:246:
28832026-03-11T19:45:18.526Z Error Trace: /work/oxidecomputer/cockroach/pkg/cli/democluster/demo_cluster_test.go:246
28842026-03-11T19:45:18.526Z Error: Should be true
28852026-03-11T19:45:18.526Z Test: TestTransientClusterSimulateLatencies/from_us-west1
28862026-03-11T19:45:18.526Z Messages: expected duration at least 73ms, got 63.717993ms
28872026-03-11T19:45:18.526Z --- FAIL: TestTransientClusterSimulateLatencies/from_europe-west1 (0.07s)
28882026-03-11T19:45:18.526Z demo_cluster_test.go:246:
28892026-03-11T19:45:18.526Z Error Trace: /work/oxidecomputer/cockroach/pkg/cli/democluster/demo_cluster_test.go:246
28902026-03-11T19:45:18.526Z Error: Should be true
28912026-03-11T19:45:18.526Z Test: TestTransientClusterSimulateLatencies/from_europe-west1
28922026-03-11T19:45:18.526Z Messages: expected duration at least 73ms, got 67.938207ms
28932026-03-11T19:45:18.526Z demo_cluster_test.go:255: -- test log scope end --
28942026-03-11T19:45:18.526ZFAIL
28952026-03-11T19:45:18.526ZFAIL github.com/cockroachdb/cockroach/pkg/cli/democluster 0.792s
28962026-03-11T19:45:18.526Zok github.com/cockroachdb/cockroach/pkg/cli/exit 0.002s [no tests to run]
28972026-03-11T19:45:18.526Zok github.com/cockroachdb/cockroach/pkg/cloud/amazon 0.483s
28982026-03-11T19:45:18.526Zok github.com/cockroachdb/cockroach/pkg/cloud/azure 0.022s
28992026-03-11T19:45:18.530Zok github.com/cockroachdb/cockroach/pkg/cloud/gcp 0.021s
29002026-03-11T19:45:18.530Zok github.com/cockroachdb/cockroach/pkg/cloud/httpsink 0.222s
29012026-03-11T19:45:18.530Zok github.com/cockroachdb/cockroach/pkg/cloud/impl/cloudimpltests 0.025s [no tests to run]
29022026-03-11T19:45:18.530Zok github.com/cockroachdb/cockroach/pkg/cloud/nodelocal 0.156s
29032026-03-11T19:45:18.530Zok github.com/cockroachdb/cockroach/pkg/cloud/nullsink 0.019s
29042026-03-11T19:45:18.530Zok github.com/cockroachdb/cockroach/pkg/cloud/userfile 1.788s
29052026-03-11T19:45:18.530Zok github.com/cockroachdb/cockroach/pkg/cloud/userfile/filetable/filetabletest 5.071s
29062026-03-11T19:45:18.530Zok github.com/cockroachdb/cockroach/pkg/clusterversion 0.017s
29072026-03-11T19:45:18.530Zok github.com/cockroachdb/cockroach/pkg/cmd/bazci 0.016s
29082026-03-11T19:45:18.530Zok github.com/cockroachdb/cockroach/pkg/cmd/cmpconn 0.083s
29092026-03-11T19:45:18.530Zok github.com/cockroachdb/cockroach/pkg/cmd/cockroach-oss 0.451s
29102026-03-11T19:45:18.530Zok github.com/cockroachdb/cockroach/pkg/cmd/dev 0.014s
29112026-03-11T19:45:18.530Zok github.com/cockroachdb/cockroach/pkg/cmd/docgen/extract 0.004s
29122026-03-11T19:45:18.530Zok github.com/cockroachdb/cockroach/pkg/cmd/docs-issue-generation 0.007s
29132026-03-11T19:45:18.530Zok github.com/cockroachdb/cockroach/pkg/cmd/internal/issues 0.024s
29142026-03-11T19:45:18.530Zok github.com/cockroachdb/cockroach/pkg/cmd/label-merged-pr 0.005s
29152026-03-11T19:45:18.530Zok github.com/cockroachdb/cockroach/pkg/cmd/mirror 0.014s
29162026-03-11T19:45:18.530Zok github.com/cockroachdb/cockroach/pkg/cmd/prereqs 0.367s
29172026-03-11T19:45:18.530Zok github.com/cockroachdb/cockroach/pkg/cmd/publish-artifacts 0.016s
29182026-03-11T19:45:18.530Zok github.com/cockroachdb/cockroach/pkg/cmd/publish-provisional-artifacts 0.050s
29192026-03-11T19:45:18.530Zok github.com/cockroachdb/cockroach/pkg/cmd/reduce/reduce 0.009s
29202026-03-11T19:45:18.531Zok github.com/cockroachdb/cockroach/pkg/cmd/reduce/reduce/reducesql 0.044s
29212026-03-11T19:45:18.531Zok github.com/cockroachdb/cockroach/pkg/cmd/release 0.036s
29222026-03-11T19:45:18.531Zok github.com/cockroachdb/cockroach/pkg/cmd/roachtest 0.118s
29232026-03-11T19:45:18.531Zok github.com/cockroachdb/cockroach/pkg/cmd/roachtest/prometheus 0.005s
29242026-03-11T19:45:18.531Zok github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests 0.106s
29252026-03-11T19:45:18.531Zok github.com/cockroachdb/cockroach/pkg/cmd/testfilter 0.017s
29262026-03-11T19:45:18.531Zok github.com/cockroachdb/cockroach/pkg/col/coldata 0.345s
29272026-03-11T19:45:18.531Zok github.com/cockroachdb/cockroach/pkg/col/coldataext 0.015s
29282026-03-11T19:45:18.531Zok github.com/cockroachdb/cockroach/pkg/col/colserde 0.775s
29292026-03-11T19:45:18.531Zok github.com/cockroachdb/cockroach/pkg/config 0.075s
29302026-03-11T19:45:18.531Zok github.com/cockroachdb/cockroach/pkg/config/zonepb 0.026s
29312026-03-11T19:45:18.531Zok github.com/cockroachdb/cockroach/pkg/geo 0.076s
29322026-03-11T19:45:18.531Zok github.com/cockroachdb/cockroach/pkg/geo/geogen 0.005s
29332026-03-11T19:45:18.531Zok github.com/cockroachdb/cockroach/pkg/geo/geogfn 0.104s
29342026-03-11T19:45:18.531Zok github.com/cockroachdb/cockroach/pkg/geo/geographiclib 0.021s
29352026-03-11T19:45:18.531Zok github.com/cockroachdb/cockroach/pkg/geo/geoindex 0.189s
29362026-03-11T19:45:18.531Zok github.com/cockroachdb/cockroach/pkg/geo/geomfn 0.114s
29372026-03-11T19:45:18.531Zok github.com/cockroachdb/cockroach/pkg/geo/geopb 0.002s
29382026-03-11T19:45:18.531Zok github.com/cockroachdb/cockroach/pkg/geo/geoproj 0.014s
29392026-03-11T19:45:18.531Zok github.com/cockroachdb/cockroach/pkg/geo/geoprojbase 0.112s
29402026-03-11T19:45:18.531Zok github.com/cockroachdb/cockroach/pkg/geo/geos 0.003s
29412026-03-11T19:45:18.531Zok github.com/cockroachdb/cockroach/pkg/geo/geotransform 0.003s
29422026-03-11T19:45:18.531Zok github.com/cockroachdb/cockroach/pkg/geo/twkb 0.002s
29432026-03-11T19:45:18.531Zok github.com/cockroachdb/cockroach/pkg/gossip 5.996s
29442026-03-11T19:45:18.531Zok github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher 0.216s
29452026-03-11T19:45:18.531Zok github.com/cockroachdb/cockroach/pkg/internal/rsg 0.103s
29462026-03-11T19:45:18.531Zok github.com/cockroachdb/cockroach/pkg/internal/rsg/yacc 0.024s
29472026-03-11T19:45:18.531Zok github.com/cockroachdb/cockroach/pkg/internal/sqlsmith 2.698s
29482026-03-11T19:45:18.531Zok github.com/cockroachdb/cockroach/pkg/internal/team 0.002s
29492026-03-11T19:45:18.531Zok github.com/cockroachdb/cockroach/pkg/jobs 41.157s
29502026-03-11T19:45:18.531Zok github.com/cockroachdb/cockroach/pkg/jobs/joberror 0.027s
29512026-03-11T19:45:18.531Zok github.com/cockroachdb/cockroach/pkg/jobs/jobsprotectedts 0.035s [no tests to run]
29522026-03-11T19:45:18.531Zok github.com/cockroachdb/cockroach/pkg/keys 0.030s
29532026-03-11T19:45:18.531Zok github.com/cockroachdb/cockroach/pkg/kv 21.750s
29542026-03-11T19:45:18.531Zok github.com/cockroachdb/cockroach/pkg/kv/bulk 1.503s
29552026-03-11T19:45:21.439Z? github.com/cockroachdb/cockroach/pkg/kv/kvserver/closedts/ctpb [no test files]
29562026-03-11T19:45:21.443Z? github.com/cockroachdb/cockroach/pkg/kv/kvserver/constraint [no test files]
29572026-03-11T19:45:21.443Z? github.com/cockroachdb/cockroach/pkg/kv/kvserver/diskmap [no test files]
29582026-03-11T19:45:34.941Z? github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvserverbase [no test files]
29592026-03-11T19:45:34.944Z? github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvserverpb [no test files]
29602026-03-11T19:45:41.977Z? github.com/cockroachdb/cockroach/pkg/kv/kvserver/liveness/livenesspb [no test files]
29612026-03-11T19:45:41.982Z? github.com/cockroachdb/cockroach/pkg/kv/kvserver/loqrecovery/loqrecoverypb [no test files]
29622026-03-11T19:45:41.982Z? github.com/cockroachdb/cockroach/pkg/kv/kvserver/protectedts/ptpb [no test files]
29632026-03-11T19:45:41.982Z? github.com/cockroachdb/cockroach/pkg/kv/kvserver/protectedts/ptprovider [no test files]
29642026-03-11T19:45:41.982Z? github.com/cockroachdb/cockroach/pkg/kv/kvserver/protectedts/ptutil [no test files]
29652026-03-11T19:45:42.857Z? github.com/cockroachdb/cockroach/pkg/kv/kvserver/readsummary [no test files]
29662026-03-11T19:45:42.857Z? github.com/cockroachdb/cockroach/pkg/kv/kvserver/readsummary/rspb [no test files]
29672026-03-11T19:45:46.421Z? github.com/cockroachdb/cockroach/pkg/migration [no test files]
29682026-03-11T19:45:50.488Z? github.com/cockroachdb/cockroach/pkg/migration/migrationjob [no test files]
29692026-03-11T19:45:51.744Z? github.com/cockroachdb/cockroach/pkg/migration/nodelivenesstest [no test files]
29702026-03-11T19:45:51.748Z? github.com/cockroachdb/cockroach/pkg/multitenant [no test files]
29712026-03-11T19:45:51.748Z? github.com/cockroachdb/cockroach/pkg/multitenant/tenantcostmodel [no test files]
29722026-03-11T19:45:51.748Z? github.com/cockroachdb/cockroach/pkg/release [no test files]
29732026-03-11T19:45:53.676Z? github.com/cockroachdb/cockroach/pkg/roachpb/gen [no test files]
29742026-03-11T19:45:53.676Z? github.com/cockroachdb/cockroach/pkg/roachpb/roachpbmock [no test files]
29752026-03-11T19:45:53.676Z? github.com/cockroachdb/cockroach/pkg/roachprod [no test files]
29762026-03-11T19:45:53.676Z? github.com/cockroachdb/cockroach/pkg/roachprod/errors [no test files]
29772026-03-11T19:45:53.676Z? github.com/cockroachdb/cockroach/pkg/roachprod/logger [no test files]
29782026-03-11T19:45:53.754Z? github.com/cockroachdb/cockroach/pkg/roachprod/ui [no test files]
29792026-03-11T19:45:54.167Z? github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws [no test files]
29802026-03-11T19:45:54.167Z? github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/terraformgen [no test files]
29812026-03-11T19:45:54.171Z? github.com/cockroachdb/cockroach/pkg/roachprod/vm/azure [no test files]
29822026-03-11T19:45:54.171Z? github.com/cockroachdb/cockroach/pkg/roachprod/vm/flagstub [no test files]
29832026-03-11T19:45:54.171Z? github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce [no test files]
29842026-03-11T19:45:54.171Z? github.com/cockroachdb/cockroach/pkg/roachprod/vm/local [no test files]
29852026-03-11T19:45:55.276Z? github.com/cockroachdb/cockroach/pkg/scheduledjobs [no test files]
29862026-03-11T19:46:09.635Zok github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord 77.274s
29872026-03-11T19:46:09.639Zok github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangecache 0.460s
29882026-03-11T19:46:09.639Zok github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed 4.199s
29892026-03-11T19:46:09.639Zok github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedbuffer 0.083s
29902026-03-11T19:46:09.639Zok github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache 0.504s
29912026-03-11T19:46:09.639Zok github.com/cockroachdb/cockroach/pkg/kv/kvnemesis 2.941s
29922026-03-11T19:46:09.639Zok github.com/cockroachdb/cockroach/pkg/kv/kvprober 3.578s
29932026-03-11T19:46:32.713Z? github.com/cockroachdb/cockroach/pkg/security/pprompt [no test files]
29942026-03-11T19:46:32.713Z? github.com/cockroachdb/cockroach/pkg/security/securitytest [no test files]
29952026-03-11T19:46:32.717Z? github.com/cockroachdb/cockroach/pkg/server/diagnostics/diagnosticspb [no test files]
29962026-03-11T19:46:42.536Z? github.com/cockroachdb/cockroach/pkg/server/status/statuspb [no test files]
29972026-03-11T19:46:42.539Z? github.com/cockroachdb/cockroach/pkg/server/systemconfigwatcher/systemconfigwatchertest [no test files]
29982026-03-11T19:46:50.281Z? github.com/cockroachdb/cockroach/pkg/settings/cluster [no test files]
29992026-03-11T19:46:50.284Z? github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigjob [no test files]
30002026-03-11T19:46:50.284Z? github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfiglimiter [no test files]
30012026-03-11T19:46:50.284Z? github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigsplitter [no test files]
30022026-03-11T19:46:54.604Z? github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigtestutils/spanconfigtestcluster [no test files]
30032026-03-11T19:47:00.456Z? github.com/cockroachdb/cockroach/pkg/sql/catalog/catconstants [no test files]
30042026-03-11T19:47:00.460Z? github.com/cockroachdb/cockroach/pkg/sql/catalog/colinfo/colinfotestutils [no test files]
30052026-03-11T19:47:00.461Z? github.com/cockroachdb/cockroach/pkg/sql/catalog/descbuilder [no test files]
30062026-03-11T19:47:00.461Z? github.com/cockroachdb/cockroach/pkg/sql/catalog/descidgen [no test files]
30072026-03-11T19:47:02.810Z? github.com/cockroachdb/cockroach/pkg/sql/catalog/desctestutils [no test files]
30082026-03-11T19:47:02.810Z? github.com/cockroachdb/cockroach/pkg/sql/catalog/ingesting [no test files]
30092026-03-11T19:47:02.810Z? github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/catkv [no test files]
30102026-03-11T19:47:19.075Z? github.com/cockroachdb/cockroach/pkg/sql/catalog/rewrite [no test files]
30112026-03-11T19:47:54.677Z? github.com/cockroachdb/cockroach/pkg/sql/descmetadata [no test files]
30122026-03-11T19:48:11.967Z? github.com/cockroachdb/cockroach/pkg/sql/faketreeeval [no test files]
30132026-03-11T19:48:27.224Z? github.com/cockroachdb/cockroach/pkg/sql/lexbase/allkeywords [no test files]
30142026-03-11T19:48:27.227Z? github.com/cockroachdb/cockroach/pkg/sql/memsize [no test files]
30152026-03-11T19:48:27.227Z? github.com/cockroachdb/cockroach/pkg/sql/mutations [no test files]
30162026-03-11T19:48:30.281Z? github.com/cockroachdb/cockroach/pkg/sql/opt/exec [no test files]
30172026-03-11T19:48:59.813Z? github.com/cockroachdb/cockroach/pkg/sql/opt/optgen/cmd/langgen [no test files]
30182026-03-11T19:49:03.481Z? github.com/cockroachdb/cockroach/pkg/sql/opt/testutils/testexpr [no test files]
30192026-03-11T19:49:13.265Z? github.com/cockroachdb/cockroach/pkg/sql/optionalnodeliveness [no test files]
30202026-03-11T19:49:13.268Z? github.com/cockroachdb/cockroach/pkg/sql/paramparse [no test files]
30212026-03-11T19:49:23.255Z? github.com/cockroachdb/cockroach/pkg/sql/parser/fuzz [no test files]
30222026-03-11T19:49:23.299Z? github.com/cockroachdb/cockroach/pkg/sql/pgwire/pgcode [no test files]
30232026-03-11T19:49:23.299Z? github.com/cockroachdb/cockroach/pkg/sql/pgwire/pgnotice [no test files]
30242026-03-11T19:49:23.299Z? github.com/cockroachdb/cockroach/pkg/sql/pgwire/pgwirebase [no test files]
30252026-03-11T19:50:02.042Z? github.com/cockroachdb/cockroach/pkg/sql/protoreflect/test [no test files]
30262026-03-11T19:50:02.047Z? github.com/cockroachdb/cockroach/pkg/sql/roleoption [no test files]
30272026-03-11T19:50:17.738Z? github.com/cockroachdb/cockroach/pkg/sql/rowenc/rowencpb [no test files]
30282026-03-11T19:50:22.924Z? github.com/cockroachdb/cockroach/pkg/sql/rowinfra [no test files]
30292026-03-11T19:50:37.059Z? github.com/cockroachdb/cockroach/pkg/sql/schemachanger/rel/internal/comparetest [no test files]
30302026-03-11T19:50:37.062Z? github.com/cockroachdb/cockroach/pkg/sql/schemachanger/rel/internal/cyclegraphtest [no test files]
30312026-03-11T19:50:37.062Z? github.com/cockroachdb/cockroach/pkg/sql/schemachanger/rel/internal/entitynodetest [no test files]
30322026-03-11T19:50:37.062Z? github.com/cockroachdb/cockroach/pkg/sql/schemachanger/rel/reltest [no test files]
30332026-03-11T19:50:37.062Z? github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scbackup [no test files]
30342026-03-11T19:50:38.596Z? github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scbuild/internal/scbuildstmt [no test files]
30352026-03-11T19:50:40.116Z? github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scdeps/sctestdeps [no test files]
30362026-03-11T19:50:40.116Z? github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scdeps/sctestutils [no test files]
30372026-03-11T19:50:40.120Z? github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scerrors [no test files]
30382026-03-11T19:50:40.933Z? github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scexec/scmutationexec [no test files]
30392026-03-11T19:50:40.933Z? github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scjob [no test files]
30402026-03-11T19:50:40.933Z? github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scop [no test files]
30412026-03-11T19:50:40.933Z? github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scpb [no test files]
30422026-03-11T19:50:43.276Z? github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scplan/internal/scgraphviz [no test files]
30432026-03-11T19:50:43.279Z? github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scplan/internal/scstage [no test files]
30442026-03-11T19:50:43.279Z? github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scrun [no test files]
30452026-03-11T19:50:43.279Z? github.com/cockroachdb/cockroach/pkg/sql/schemachanger/sctest [no test files]
30462026-03-11T19:50:43.279Z? github.com/cockroachdb/cockroach/pkg/sql/scrub [no test files]
30472026-03-11T19:51:04.098Z? github.com/cockroachdb/cockroach/pkg/sql/sem/catid [no test files]
30482026-03-11T19:51:04.099Z? github.com/cockroachdb/cockroach/pkg/sql/sem/transform [no test files]
30492026-03-11T19:51:16.133Z? github.com/cockroachdb/cockroach/pkg/sql/sem/tree/treebin [no test files]
30502026-03-11T19:51:16.133Z? github.com/cockroachdb/cockroach/pkg/sql/sem/tree/treecmp [no test files]
30512026-03-11T19:51:16.133Z? github.com/cockroachdb/cockroach/pkg/sql/sem/tree/treewindow [no test files]
30522026-03-11T19:51:16.137Z? github.com/cockroachdb/cockroach/pkg/sql/sessiondatapb [no test files]
30532026-03-11T19:51:16.137Z? github.com/cockroachdb/cockroach/pkg/sql/sessionphase [no test files]
30542026-03-11T19:51:16.137Z? github.com/cockroachdb/cockroach/pkg/sql/sqlerrors [no test files]
30552026-03-11T19:51:16.137Z? github.com/cockroachdb/cockroach/pkg/sql/sqlfsm [no test files]
30562026-03-11T19:51:16.137Z? github.com/cockroachdb/cockroach/pkg/sql/sqlinstance [no test files]
30572026-03-11T19:51:18.457Z? github.com/cockroachdb/cockroach/pkg/sql/sqlliveness [no test files]
30582026-03-11T19:51:18.460Z? github.com/cockroachdb/cockroach/pkg/sql/sqlliveness/slprovider [no test files]
30592026-03-11T19:51:18.568Z? github.com/cockroachdb/cockroach/pkg/sql/sqlliveness/sqllivenesstestutils [no test files]
30602026-03-11T19:51:18.568Z? github.com/cockroachdb/cockroach/pkg/sql/sqlstats [no test files]
30612026-03-11T19:51:19.687Z? github.com/cockroachdb/cockroach/pkg/sql/sqlstats/ssmemstorage [no test files]
30622026-03-11T19:51:19.687Z? github.com/cockroachdb/cockroach/pkg/sql/sqltelemetry [no test files]
30632026-03-11T19:51:19.691Z? github.com/cockroachdb/cockroach/pkg/sql/sqltestutils [no test files]
30642026-03-11T19:51:19.691Z? github.com/cockroachdb/cockroach/pkg/sql/sqlutil [no test files]
30652026-03-11T19:51:32.100Z? github.com/cockroachdb/cockroach/pkg/sql/storageparam [no test files]
30662026-03-11T19:51:32.103Z? github.com/cockroachdb/cockroach/pkg/sql/storageparam/indexstorageparam [no test files]
30672026-03-11T19:51:32.103Z? github.com/cockroachdb/cockroach/pkg/sql/storageparam/tablestorageparam [no test files]
30682026-03-11T19:51:32.103Z? github.com/cockroachdb/cockroach/pkg/sql/ttl/ttlschedule [no test files]
30692026-03-11T19:51:32.103Z? github.com/cockroachdb/cockroach/pkg/sql/vtable [no test files]
30702026-03-11T19:51:34.647Z? github.com/cockroachdb/cockroach/pkg/testutils/buildutil [no test files]
30712026-03-11T19:51:34.647Z? github.com/cockroachdb/cockroach/pkg/testutils/colcontainerutils [no test files]
30722026-03-11T19:51:34.647Z? github.com/cockroachdb/cockroach/pkg/testutils/diagutils [no test files]
30732026-03-11T19:51:34.647Z? github.com/cockroachdb/cockroach/pkg/testutils/distsqlutils [no test files]
30742026-03-11T19:51:34.653Z? github.com/cockroachdb/cockroach/pkg/testutils/docker [no test files]
30752026-03-11T19:51:34.697Z? github.com/cockroachdb/cockroach/pkg/testutils/docker/docker-fsnotify [no test files]
30762026-03-11T19:51:34.697Z? github.com/cockroachdb/cockroach/pkg/testutils/echotest [no test files]
30772026-03-11T19:51:34.803Z? github.com/cockroachdb/cockroach/pkg/testutils/gossiputil [no test files]
30782026-03-11T19:51:34.853Z? github.com/cockroachdb/cockroach/pkg/testutils/grpcutils [no test files]
30792026-03-11T19:51:34.853Z? github.com/cockroachdb/cockroach/pkg/testutils/jobutils [no test files]
30802026-03-11T19:51:35.677Z? github.com/cockroachdb/cockroach/pkg/testutils/kvclientutils [no test files]
30812026-03-11T19:51:35.677Z? github.com/cockroachdb/cockroach/pkg/testutils/lint [no test files]
30822026-03-11T19:51:35.677Z? github.com/cockroachdb/cockroach/pkg/testutils/lint/passes/descriptormarshal [no test files]
30832026-03-11T19:51:35.681Z? github.com/cockroachdb/cockroach/pkg/testutils/lint/passes/errcheck [no test files]
30842026-03-11T19:51:35.681Z? github.com/cockroachdb/cockroach/pkg/testutils/lint/passes/errcmp [no test files]
30852026-03-11T19:51:36.535Z? github.com/cockroachdb/cockroach/pkg/testutils/lint/passes/grpcclientconnclose [no test files]
30862026-03-11T19:51:36.539Z? github.com/cockroachdb/cockroach/pkg/testutils/lint/passes/grpcstatuswithdetails [no test files]
30872026-03-11T19:51:37.341Z? github.com/cockroachdb/cockroach/pkg/testutils/lint/passes/returncheck [no test files]
30882026-03-11T19:51:37.991Z? github.com/cockroachdb/cockroach/pkg/testutils/lint/passes/staticcheck [no test files]
30892026-03-11T19:51:38.267Z? github.com/cockroachdb/cockroach/pkg/testutils/localtestcluster [no test files]
30902026-03-11T19:51:38.267Z? github.com/cockroachdb/cockroach/pkg/testutils/metrictestutils [no test files]
30912026-03-11T19:51:38.267Z? github.com/cockroachdb/cockroach/pkg/testutils/pgtest [no test files]
30922026-03-11T19:51:38.267Z? github.com/cockroachdb/cockroach/pkg/testutils/physicalplanutils [no test files]
30932026-03-11T19:51:38.267Z? github.com/cockroachdb/cockroach/pkg/testutils/serverutils [no test files]
30942026-03-11T19:51:38.267Z? github.com/cockroachdb/cockroach/pkg/testutils/skip [no test files]
30952026-03-11T19:51:47.918Z? github.com/cockroachdb/cockroach/pkg/testutils/sstutil [no test files]
30962026-03-11T19:51:47.918Z? github.com/cockroachdb/cockroach/pkg/testutils/storageutils [no test files]
30972026-03-11T19:51:47.922Z? github.com/cockroachdb/cockroach/pkg/ts/catalog [no test files]
30982026-03-11T19:51:47.922Z? github.com/cockroachdb/cockroach/pkg/ts/tspb [no test files]
30992026-03-11T19:51:47.922Z? github.com/cockroachdb/cockroach/pkg/ui [no test files]
31002026-03-11T19:51:47.922Z? github.com/cockroachdb/cockroach/pkg/ui/distoss [no test files]
31012026-03-11T19:51:48.745Z? github.com/cockroachdb/cockroach/pkg/util/arith [no test files]
31022026-03-11T19:51:48.749Z? github.com/cockroachdb/cockroach/pkg/util/bufalloc [no test files]
31032026-03-11T19:51:49.616Z? github.com/cockroachdb/cockroach/pkg/util/cancelchecker [no test files]
31042026-03-11T19:51:53.077Z? github.com/cockroachdb/cockroach/pkg/util/encoding/encodingtype [no test files]
31052026-03-11T19:51:53.080Z? github.com/cockroachdb/cockroach/pkg/util/fileutil [no test files]
31062026-03-11T19:51:53.814Z? github.com/cockroachdb/cockroach/pkg/util/growstack [no test files]
31072026-03-11T19:52:24.581Z? github.com/cockroachdb/cockroach/pkg/util/interval/generic/internal [no test files]
31082026-03-11T19:52:24.582Z? github.com/cockroachdb/cockroach/pkg/util/ioctx [no test files]
31092026-03-11T19:52:24.584Z? github.com/cockroachdb/cockroach/pkg/util/iterutil [no test files]
31102026-03-11T19:52:24.585Z? github.com/cockroachdb/cockroach/pkg/util/jsonbytes [no test files]
31112026-03-11T19:52:24.585Z? github.com/cockroachdb/cockroach/pkg/util/keysutil [no test files]
31122026-03-11T19:52:24.585Z? github.com/cockroachdb/cockroach/pkg/util/leaktest [no test files]
31132026-03-11T19:52:31.093Z? github.com/cockroachdb/cockroach/pkg/util/log/channel [no test files]
31142026-03-11T19:52:31.100Z? github.com/cockroachdb/cockroach/pkg/util/log/gen [no test files]
31152026-03-11T19:53:15.451Z? github.com/cockroachdb/cockroach/pkg/util/log/logflags [no test files]
31162026-03-11T19:53:15.451Z? github.com/cockroachdb/cockroach/pkg/util/log/logpb [no test files]
31172026-03-11T19:53:15.455Z? github.com/cockroachdb/cockroach/pkg/util/log/logtestutils [no test files]
31182026-03-11T19:53:15.455Z? github.com/cockroachdb/cockroach/pkg/util/log/severity [no test files]
31192026-03-11T19:53:15.455Z? github.com/cockroachdb/cockroach/pkg/util/memzipper [no test files]
31202026-03-11T19:53:15.455Z? github.com/cockroachdb/cockroach/pkg/util/pprofutil [no test files]
31212026-03-11T19:53:16.624Z? github.com/cockroachdb/cockroach/pkg/util/sort [no test files]
31222026-03-11T19:53:17.882Z? github.com/cockroachdb/cockroach/pkg/util/stringencoding [no test files]
31232026-03-11T19:53:17.885Z? github.com/cockroachdb/cockroach/pkg/util/system [no test files]
31242026-03-11T19:53:17.885Z? github.com/cockroachdb/cockroach/pkg/util/targz/gen [no test files]
31252026-03-11T19:53:19.226Z? github.com/cockroachdb/cockroach/pkg/util/timeutil/gen [no test files]
31262026-03-11T19:55:02.662Z? github.com/cockroachdb/cockroach/pkg/util/tracing/tracingpb [no test files]
31272026-03-11T19:55:02.665Z? github.com/cockroachdb/cockroach/pkg/util/tracing/tracingservicepb [no test files]
31282026-03-11T19:55:02.665Z? github.com/cockroachdb/cockroach/pkg/util/tracing/tracingui [no test files]
31292026-03-11T19:55:02.665Z? github.com/cockroachdb/cockroach/pkg/util/tracing/zipper [no test files]
31302026-03-11T19:55:02.665Z? github.com/cockroachdb/cockroach/pkg/util/unaccent [no test files]
31312026-03-11T19:55:07.145Z? github.com/cockroachdb/cockroach/pkg/workload/bulkingest [no test files]
31322026-03-11T19:55:07.148Z? github.com/cockroachdb/cockroach/pkg/workload/connectionlatency [no test files]
31332026-03-11T19:55:07.148Z? github.com/cockroachdb/cockroach/pkg/workload/debug [no test files]
31342026-03-11T19:55:07.148Z? github.com/cockroachdb/cockroach/pkg/workload/examples [no test files]
31352026-03-11T19:55:07.148Z? github.com/cockroachdb/cockroach/pkg/workload/geospatial [no test files]
31362026-03-11T19:55:07.148Z? github.com/cockroachdb/cockroach/pkg/workload/indexes [no test files]
31372026-03-11T19:55:07.148Z? github.com/cockroachdb/cockroach/pkg/workload/jsonload [no test files]
31382026-03-11T19:55:07.148Z? github.com/cockroachdb/cockroach/pkg/workload/kv [no test files]
31392026-03-11T19:55:07.236Z? github.com/cockroachdb/cockroach/pkg/workload/ledger [no test files]
31402026-03-11T19:55:07.790Z? github.com/cockroachdb/cockroach/pkg/workload/querybench [no test files]
31412026-03-11T19:55:07.793Z? github.com/cockroachdb/cockroach/pkg/workload/querylog [no test files]
31422026-03-11T19:55:07.793Z? github.com/cockroachdb/cockroach/pkg/workload/queue [no test files]
31432026-03-11T19:55:07.793Z? github.com/cockroachdb/cockroach/pkg/workload/rand [no test files]
31442026-03-11T19:55:07.793Z? github.com/cockroachdb/cockroach/pkg/workload/schemachange [no test files]
31452026-03-11T19:55:07.793Z? github.com/cockroachdb/cockroach/pkg/workload/sqlsmith [no test files]
31462026-03-11T19:55:08.053Z? github.com/cockroachdb/cockroach/pkg/workload/tpccchecks [no test files]
31472026-03-11T19:55:08.053Z? github.com/cockroachdb/cockroach/pkg/workload/tpcds [no test files]
31482026-03-11T19:55:08.056Z? github.com/cockroachdb/cockroach/pkg/workload/ttllogger [no test files]
31492026-03-11T20:03:06.421Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver 1065.943s
31502026-03-11T20:03:06.421Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/abortspan 0.013s
31512026-03-11T20:03:06.421Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply 0.002s
31522026-03-11T20:03:06.421Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval 55.831s
31532026-03-11T20:03:06.421Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/batcheval/result 0.023s
31542026-03-11T20:03:06.421Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/closedts 0.014s
31552026-03-11T20:03:06.421Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/closedts/sidetransport 1.931s
31562026-03-11T20:03:06.421Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/closedts/tracker 2.017s
31572026-03-11T20:03:06.421Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency 19.229s
31582026-03-11T20:03:06.421Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock 0.017s
31592026-03-11T20:03:06.421Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/poison 0.012s
31602026-03-11T20:03:06.421Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/gc 1.286s
31612026-03-11T20:03:06.421Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/idalloc 0.178s
31622026-03-11T20:03:06.421Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver 0.165s
31632026-03-11T20:03:06.421Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/liveness 8.938s
31642026-03-11T20:03:06.421Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/loqrecovery 0.709s
31652026-03-11T20:03:06.421Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/protectedts 0.016s
31662026-03-11T20:03:06.421Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/protectedts/ptcache 2.023s
31672026-03-11T20:03:06.421Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/protectedts/ptreconcile 0.435s
31682026-03-11T20:03:06.422Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/protectedts/ptstorage 12.824s
31692026-03-11T20:03:06.422Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/raftentry 2.329s
31702026-03-11T20:03:06.422Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed 0.621s
31712026-03-11T20:03:06.422Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/rditer 0.076s
31722026-03-11T20:03:06.422Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/reports 7.918s
31732026-03-11T20:03:06.422Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanlatch 6.548s
31742026-03-11T20:03:06.422Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset 0.045s
31752026-03-11T20:03:06.422Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/split 0.031s
31762026-03-11T20:03:06.422Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/stateloader 0.012s
31772026-03-11T20:03:06.422Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/tenantrate 0.059s
31782026-03-11T20:03:06.422Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/tscache 3.264s
31792026-03-11T20:03:06.422Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/txnrecovery 0.026s
31802026-03-11T20:03:06.422Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/txnwait 0.029s
31812026-03-11T20:03:06.422Zok github.com/cockroachdb/cockroach/pkg/kv/kvserver/uncertainty 0.025s
31822026-03-11T20:03:06.422Zok github.com/cockroachdb/cockroach/pkg/migration/migrationcluster 1.119s
31832026-03-11T20:45:50.635Zinitialized metamorphic constant "max-batch-byte-size" with value 4570237
31842026-03-11T20:45:50.635Zinitialized metamorphic constant "coldata-batch-size" with value 2244
31852026-03-11T20:45:50.635Zinitialized metamorphic constant "span-reuse-rate" with value 18
31862026-03-11T20:45:50.635Zinitialized metamorphic constant "row-container-rows-per-chunk-shift" with value 1
31872026-03-11T20:45:50.635Zinitialized metamorphic constant "addsst-rewrite-concurrency" with value 5
31882026-03-11T20:45:50.635Zinitialized metamorphic constant "addsst-rewrite-forced" with value true
31892026-03-11T20:45:50.635Zinitialized metamorphic constant "kv.rangefeed.use_dedicated_connection_class.enabled" with value true
31902026-03-11T20:45:50.635Zinitialized metamorphic constant "raft-log-truncation-clearrange-threshold" with value 470822
31912026-03-11T20:45:50.635Zinitialized metamorphic constant "kv-batch-size" with value 1
31922026-03-11T20:45:50.635Zinitialized metamorphic constant "datum-row-converter-batch-size" with value 1
31932026-03-11T20:45:50.635Zinitialized metamorphic constant "spilling-queue-initial-len" with value 13
31942026-03-11T20:45:50.635Zinitialized metamorphic constant "ColIndexJoin-batch-size" with value 759651
31952026-03-11T20:45:50.635Zinitialized metamorphic constant "zig-zag-joiner-batch-size" with value 1
31962026-03-11T20:45:50.635Zinitialized metamorphic constant "resolve-membership-single-scan-enabled" with value false
31972026-03-11T20:45:50.635Zinitialized metamorphic constant "async-IE-result-channel-buffer-size" with value 1
31982026-03-11T20:45:50.635Zpanic: test timed out after 1h0m0s
31992026-03-11T20:45:50.635Zrunning tests:
32002026-03-11T20:45:50.635Z TestPauseMigration (59m57s)
32012026-03-11T20:45:50.635Z
32022026-03-11T20:45:50.635Zgoroutine 3707620 [running]:
32032026-03-11T20:45:50.643Ztesting.(*M).startAlarm.func1()
32042026-03-11T20:45:50.643Z /work/go/src/testing/testing.go:2366 +0x385
32052026-03-11T20:45:50.643Zcreated by time.goFunc
32062026-03-11T20:45:50.643Z /work/go/src/time/sleep.go:177 +0x2d
32072026-03-11T20:45:50.643Z
32082026-03-11T20:45:50.643Zgoroutine 1 [chan receive, 59 minutes]:
32092026-03-11T20:45:50.643Ztesting.(*T).Run(0xc000a436c0, {0x409571c?, 0x0?}, 0x42e1368)
32102026-03-11T20:45:50.643Z /work/go/src/testing/testing.go:1750 +0x3ab
32112026-03-11T20:45:50.643Ztesting.runTests.func1(0xc000a436c0)
32122026-03-11T20:45:50.643Z /work/go/src/testing/testing.go:2161 +0x37
32132026-03-11T20:45:50.643Ztesting.tRunner(0xc000a436c0, 0xc00073fc20)
32142026-03-11T20:45:50.643Z /work/go/src/testing/testing.go:1689 +0xfb
32152026-03-11T20:45:50.643Ztesting.runTests(0xc00078e6f0, {0x73ad400, 0x5, 0x5}, {0x41e1e5?, 0xc00073fcb0?, 0x76cce00?})
32162026-03-11T20:45:50.643Z /work/go/src/testing/testing.go:2159 +0x445
32172026-03-11T20:45:50.643Ztesting.(*M).Run(0xc000418f00)
32182026-03-11T20:45:50.643Z /work/go/src/testing/testing.go:2027 +0x68b
32192026-03-11T20:45:50.643Zgithub.com/cockroachdb/cockroach/pkg/migration/migrationmanager_test.TestMain(0xc0000bc708?)
32202026-03-11T20:45:50.643Z /work/oxidecomputer/cockroach/pkg/migration/migrationmanager/main_test.go:28 +0xe8
32212026-03-11T20:45:50.643Zmain.main()
32222026-03-11T20:45:50.643Z _testmain.go:59 +0x195
32232026-03-11T20:45:50.643Z
32242026-03-11T20:45:50.643Zgoroutine 12 [chan receive]:
32252026-03-11T20:45:50.644Zgithub.com/cockroachdb/cockroach/pkg/util/goschedstats.init.0.func1()
32262026-03-11T20:45:50.644Z /work/oxidecomputer/cockroach/pkg/util/goschedstats/runnable.go:165 +0x159
32272026-03-11T20:45:50.644Zcreated by github.com/cockroachdb/cockroach/pkg/util/goschedstats.init.0 in goroutine 1
32282026-03-11T20:45:50.644Z /work/oxidecomputer/cockroach/pkg/util/goschedstats/runnable.go:157 +0x1a
32292026-03-11T20:45:50.644Z
32302026-03-11T20:45:50.644Zgoroutine 13 [chan receive]:
32312026-03-11T20:45:50.644Zgithub.com/cockroachdb/cockroach/pkg/util/log.flushDaemon()
32322026-03-11T20:45:50.644Z /work/oxidecomputer/cockroach/pkg/util/log/log_flush.go:75 +0x4e
32332026-03-11T20:45:50.644Zcreated by github.com/cockroachdb/cockroach/pkg/util/log.init.5 in goroutine 1
32342026-03-11T20:45:50.644Z /work/oxidecomputer/cockroach/pkg/util/log/log_flush.go:41 +0x1a
32352026-03-11T20:45:50.644Z
32362026-03-11T20:45:50.644Zgoroutine 14 [chan receive, 60 minutes]:
32372026-03-11T20:45:50.644Zgithub.com/cockroachdb/cockroach/pkg/util/log.signalFlusher()
32382026-03-11T20:45:50.644Z /work/oxidecomputer/cockroach/pkg/util/log/log_flush.go:98 +0xd9
32392026-03-11T20:45:50.644Zcreated by github.com/cockroachdb/cockroach/pkg/util/log.init.5 in goroutine 1
32402026-03-11T20:45:50.644Z /work/oxidecomputer/cockroach/pkg/util/log/log_flush.go:42 +0x26
32412026-03-11T20:45:50.644Z
32422026-03-11T20:45:50.644Zgoroutine 22 [syscall, 60 minutes]:
32432026-03-11T20:45:50.644Zos/signal.signal_recv()
32442026-03-11T20:45:50.644Z /work/go/src/runtime/sigqueue.go:152 +0x29
32452026-03-11T20:45:50.644Zos/signal.loop()
32462026-03-11T20:45:50.644Z /work/go/src/os/signal/signal_unix.go:23 +0x13
32472026-03-11T20:45:50.644Zcreated by os/signal.Notify.func1.1 in goroutine 14
32482026-03-11T20:45:50.644Z /work/go/src/os/signal/signal.go:151 +0x1f
32492026-03-11T20:45:50.644Z
32502026-03-11T20:45:50.644Zgoroutine 9708 [chan receive, 59 minutes]:
32512026-03-11T20:45:50.644Zgithub.com/cockroachdb/cockroach/pkg/migration/migrationmanager_test.TestPauseMigration(0xc001afe000)
32522026-03-11T20:45:50.644Z /work/oxidecomputer/cockroach/pkg/migration/migrationmanager/manager_external_test.go:442 +0x506
32532026-03-11T20:45:50.644Ztesting.tRunner(0xc001afe000, 0x42e1368)
32542026-03-11T20:45:50.644Z /work/go/src/testing/testing.go:1689 +0xfb
32552026-03-11T20:45:50.644Zcreated by testing.(*T).Run in goroutine 1
32562026-03-11T20:45:50.644Z /work/go/src/testing/testing.go:1742 +0x390
32572026-03-11T20:45:50.644Z
32582026-03-11T20:45:50.644Zgoroutine 11039 [select, 59 minutes]:
32592026-03-11T20:45:50.644Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1({0x50b1338, 0xc004c69540})
32602026-03-11T20:45:50.645Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:116 +0x157
32612026-03-11T20:45:50.645Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.Group.GoCtx.func3()
32622026-03-11T20:45:50.645Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
32632026-03-11T20:45:50.645Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
32642026-03-11T20:45:50.645Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
32652026-03-11T20:45:50.645Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 11120
32662026-03-11T20:45:50.645Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
32672026-03-11T20:45:50.645Z
32682026-03-11T20:45:50.645Zgoroutine 9893 [select]:
32692026-03-11T20:45:50.645Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents(0xc003cd4dc0, {0x50b1338, 0xc000afc0f0}, 0xc0076599b0, 0xc0020f21e0)
32702026-03-11T20:45:50.645Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:331 +0xc5
32712026-03-11T20:45:50.645Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func2({0x50b1338?, 0xc000afc0f0?})
32722026-03-11T20:45:50.645Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:288 +0x32
32732026-03-11T20:45:50.645Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
32742026-03-11T20:45:50.645Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
32752026-03-11T20:45:50.646Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
32762026-03-11T20:45:50.646Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
32772026-03-11T20:45:50.646Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9957
32782026-03-11T20:45:50.646Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
32792026-03-11T20:45:50.646Z
32802026-03-11T20:45:50.646Zgoroutine 11055 [select, 59 minutes]:
32812026-03-11T20:45:50.646Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1({0x50b1338, 0xc0032344b0})
32822026-03-11T20:45:50.646Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:116 +0x157
32832026-03-11T20:45:50.646Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.Group.GoCtx.func3()
32842026-03-11T20:45:50.646Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
32852026-03-11T20:45:50.646Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
32862026-03-11T20:45:50.646Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
32872026-03-11T20:45:50.646Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 11053
32882026-03-11T20:45:50.646Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
32892026-03-11T20:45:50.646Z
32902026-03-11T20:45:50.646Zgoroutine 10082 [sync.Cond.Wait]:
32912026-03-11T20:45:50.646Zsync.runtime_notifyListWait(0xc001e22250, 0x3779)
32922026-03-11T20:45:50.646Z /work/go/src/runtime/sema.go:569 +0x159
32932026-03-11T20:45:50.646Zsync.(*Cond).Wait(0x3aac600?)
32942026-03-11T20:45:50.646Z /work/go/src/sync/cond.go:70 +0x85
32952026-03-11T20:45:50.646Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583a40})
32962026-03-11T20:45:50.646Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
32972026-03-11T20:45:50.646Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
32982026-03-11T20:45:50.646Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
32992026-03-11T20:45:50.646Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
33002026-03-11T20:45:50.646Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
33012026-03-11T20:45:50.646Z
33022026-03-11T20:45:50.646Zgoroutine 10127 [select]:
33032026-03-11T20:45:50.781Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*StoreRebalancer).Start.func1({0x50b1300, 0xc003ebc5d0})
33042026-03-11T20:45:50.781Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store_rebalancer.go:235 +0x1c5
33052026-03-11T20:45:50.781Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
33062026-03-11T20:45:50.781Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
33072026-03-11T20:45:50.781Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
33082026-03-11T20:45:50.781Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
33092026-03-11T20:45:50.781Z
33102026-03-11T20:45:50.781Zgoroutine 10929 [select, 59 minutes]:
33112026-03-11T20:45:50.781Zgoogle.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc007706bc0)
33122026-03-11T20:45:50.781Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:112 +0x6d
33132026-03-11T20:45:50.781Zcreated by google.golang.org/grpc.newCCBalancerWrapper in goroutine 10928
33142026-03-11T20:45:50.781Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:73 +0x22b
33152026-03-11T20:45:50.781Z
33162026-03-11T20:45:50.781Zgoroutine 10304 [select]:
33172026-03-11T20:45:50.781Zgoogle.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc006302000)
33182026-03-11T20:45:50.781Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1572 +0x137
33192026-03-11T20:45:50.781Zcreated by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 10302
33202026-03-11T20:45:50.781Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:360 +0x180b
33212026-03-11T20:45:50.781Z
33222026-03-11T20:45:50.781Zgoroutine 10151 [select, 59 minutes]:
33232026-03-11T20:45:50.781Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2({0x50b1300, 0xc009329c20})
33242026-03-11T20:45:50.781Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/queue.go:823 +0xff
33252026-03-11T20:45:50.781Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
33262026-03-11T20:45:50.781Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
33272026-03-11T20:45:50.781Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10124
33282026-03-11T20:45:50.781Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
33292026-03-11T20:45:50.781Z
33302026-03-11T20:45:50.781Zgoroutine 9993 [select, 59 minutes]:
33312026-03-11T20:45:50.781Zgoogle.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0034cbe00)
33322026-03-11T20:45:50.781Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:112 +0x6d
33332026-03-11T20:45:50.781Zcreated by google.golang.org/grpc.newCCBalancerWrapper in goroutine 9708
33342026-03-11T20:45:50.782Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:73 +0x22b
33352026-03-11T20:45:50.782Z
33362026-03-11T20:45:50.782Zgoroutine 11139 [select, 59 minutes]:
33372026-03-11T20:45:50.782Zgithub.com/cockroachdb/cockroach/pkg/util/tracing.newTracingClientStream.func2()
33382026-03-11T20:45:50.782Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:386 +0x67
33392026-03-11T20:45:50.782Zcreated by github.com/cockroachdb/cockroach/pkg/util/tracing.newTracingClientStream in goroutine 11041
33402026-03-11T20:45:50.782Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:385 +0x125
33412026-03-11T20:45:50.782Z
33422026-03-11T20:45:50.782Zgoroutine 9757 [select, 59 minutes]:
33432026-03-11T20:45:50.782Zgoogle.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc007b18180)
33442026-03-11T20:45:50.782Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:112 +0x6d
33452026-03-11T20:45:50.782Zcreated by google.golang.org/grpc.newCCBalancerWrapper in goroutine 9755
33462026-03-11T20:45:50.782Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:73 +0x22b
33472026-03-11T20:45:50.782Z
33482026-03-11T20:45:50.782Zgoroutine 10785 [sync.Cond.Wait, 59 minutes]:
33492026-03-11T20:45:50.782Zsync.runtime_notifyListWait(0xc0064c5050, 0x4c)
33502026-03-11T20:45:50.782Z /work/go/src/runtime/sema.go:569 +0x159
33512026-03-11T20:45:50.782Zsync.(*Cond).Wait(0x50f7930?)
33522026-03-11T20:45:50.782Z /work/go/src/sync/cond.go:70 +0x85
33532026-03-11T20:45:50.782Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd37d0})
33542026-03-11T20:45:50.782Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
33552026-03-11T20:45:50.782Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
33562026-03-11T20:45:50.782Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
33572026-03-11T20:45:50.782Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
33582026-03-11T20:45:50.782Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
33592026-03-11T20:45:50.782Z
33602026-03-11T20:45:50.782Zgoroutine 10075 [sync.Cond.Wait]:
33612026-03-11T20:45:50.782Zsync.runtime_notifyListWait(0xc001e22250, 0x3772)
33622026-03-11T20:45:50.782Z /work/go/src/runtime/sema.go:569 +0x159
33632026-03-11T20:45:50.782Zsync.(*Cond).Wait(0x3aac600?)
33642026-03-11T20:45:50.782Z /work/go/src/sync/cond.go:70 +0x85
33652026-03-11T20:45:50.782Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc0075838f0})
33662026-03-11T20:45:50.782Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
33672026-03-11T20:45:50.782Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
33682026-03-11T20:45:50.782Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
33692026-03-11T20:45:50.782Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
33702026-03-11T20:45:50.782Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
33712026-03-11T20:45:50.783Z
33722026-03-11T20:45:50.783Zgoroutine 9795 [select]:
33732026-03-11T20:45:50.783Zgithub.com/cockroachdb/cockroach/pkg/gossip.newInfoStore.func1({0xc00588c908?, 0xc0021b16e0?})
33742026-03-11T20:45:50.783Z /work/oxidecomputer/cockroach/pkg/gossip/infostore.go:192 +0x74
33752026-03-11T20:45:50.783Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
33762026-03-11T20:45:50.783Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
33772026-03-11T20:45:50.783Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
33782026-03-11T20:45:50.783Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
33792026-03-11T20:45:50.783Z
33802026-03-11T20:45:50.783Zgoroutine 9916 [chan receive, 59 minutes]:
33812026-03-11T20:45:50.783Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc002e5abd0?})
33822026-03-11T20:45:50.783Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
33832026-03-11T20:45:50.783Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x0?, 0x0?}}, 0xc00465a7a8)
33842026-03-11T20:45:50.783Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
33852026-03-11T20:45:50.783Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
33862026-03-11T20:45:50.783Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
33872026-03-11T20:45:50.783Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
33882026-03-11T20:45:50.783Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
33892026-03-11T20:45:50.783Z
33902026-03-11T20:45:50.783Zgoroutine 10051 [sync.Cond.Wait]:
33912026-03-11T20:45:50.783Zsync.runtime_notifyListWait(0xc001e22250, 0x3759)
33922026-03-11T20:45:50.783Z /work/go/src/runtime/sema.go:569 +0x159
33932026-03-11T20:45:50.783Zsync.(*Cond).Wait(0x3aac600?)
33942026-03-11T20:45:50.783Z /work/go/src/sync/cond.go:70 +0x85
33952026-03-11T20:45:50.783Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583470})
33962026-03-11T20:45:50.783Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
33972026-03-11T20:45:50.783Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
33982026-03-11T20:45:50.783Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
33992026-03-11T20:45:50.783Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
34002026-03-11T20:45:50.783Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
34012026-03-11T20:45:50.783Z
34022026-03-11T20:45:50.783Zgoroutine 10326 [select]:
34032026-03-11T20:45:50.792Zgithub.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func8({0x50b1300?, 0xc003457230?})
34042026-03-11T20:45:50.792Z /work/oxidecomputer/cockroach/pkg/jobs/registry.go:811 +0x192
34052026-03-11T20:45:50.792Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
34062026-03-11T20:45:50.792Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
34072026-03-11T20:45:50.792Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
34082026-03-11T20:45:50.792Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
34092026-03-11T20:45:50.792Z
34102026-03-11T20:45:50.792Zgoroutine 10875 [select]:
34112026-03-11T20:45:50.792Zgithub.com/cockroachdb/cockroach/pkg/ts.(*poller).start.func1({0x50b1300, 0xc002ff1860})
34122026-03-11T20:45:50.792Z /work/oxidecomputer/cockroach/pkg/ts/db.go:176 +0xea
34132026-03-11T20:45:50.792Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
34142026-03-11T20:45:50.792Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
34152026-03-11T20:45:50.792Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
34162026-03-11T20:45:50.792Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
34172026-03-11T20:45:50.792Z
34182026-03-11T20:45:50.792Zgoroutine 9786 [chan receive, 59 minutes]:
34192026-03-11T20:45:50.792Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc007582270?})
34202026-03-11T20:45:50.792Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
34212026-03-11T20:45:50.792Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x0?, 0xc0000bbfd0?}}, 0xc0000bbfa8)
34222026-03-11T20:45:50.792Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
34232026-03-11T20:45:50.792Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
34242026-03-11T20:45:50.792Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
34252026-03-11T20:45:50.792Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
34262026-03-11T20:45:50.793Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
34272026-03-11T20:45:50.793Z
34282026-03-11T20:45:50.793Zgoroutine 10457 [semacquire, 59 minutes]:
34292026-03-11T20:45:50.793Zsync.runtime_Semacquire(0xc006e565d0?)
34302026-03-11T20:45:50.793Z /work/go/src/runtime/sema.go:62 +0x25
34312026-03-11T20:45:50.793Zsync.(*WaitGroup).Wait(0x44e7a0?)
34322026-03-11T20:45:50.793Z /work/go/src/sync/waitgroup.go:116 +0x48
34332026-03-11T20:45:50.793Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc005ed4f00)
34342026-03-11T20:45:50.793Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
34352026-03-11T20:45:50.793Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc005ed4f00?, {0x50b1338?, 0xc007684230?}})
34362026-03-11T20:45:50.793Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
34372026-03-11T20:45:50.793Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait({0x50b1300?, 0xc007494e10?}, {0xc003bd3d78, 0x2, 0x4000000000000000?})
34382026-03-11T20:45:50.793Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:193 +0xff
34392026-03-11T20:45:50.793Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0xc0032834a0, {0x50b1300, 0xc007494e10}, 0xc007494c60)
34402026-03-11T20:45:50.793Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:291 +0x59c
34412026-03-11T20:45:50.793Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).Start.func2({0x50b1300, 0xc007494d50})
34422026-03-11T20:45:50.793Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:215 +0x2f3
34432026-03-11T20:45:50.793Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
34442026-03-11T20:45:50.793Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
34452026-03-11T20:45:50.793Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10391
34462026-03-11T20:45:50.793Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
34472026-03-11T20:45:50.793Z
34482026-03-11T20:45:50.793Zgoroutine 10125 [select]:
34492026-03-11T20:45:50.793Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).startLeaseRenewer.func1({0x50b1300, 0xc003ebc4e0})
34502026-03-11T20:45:50.793Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:2256 +0x2ac
34512026-03-11T20:45:50.793Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
34522026-03-11T20:45:50.793Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
34532026-03-11T20:45:50.793Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
34542026-03-11T20:45:50.793Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
34552026-03-11T20:45:50.793Z
34562026-03-11T20:45:50.793Zgoroutine 10126 [select]:
34572026-03-11T20:45:50.793Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).startRangefeedUpdater.func1({0x50b1300, 0xc003ebc510})
34582026-03-11T20:45:50.793Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:2307 +0x28a
34592026-03-11T20:45:50.794Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
34602026-03-11T20:45:50.794Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
34612026-03-11T20:45:50.794Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
34622026-03-11T20:45:50.794Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
34632026-03-11T20:45:50.794Z
34642026-03-11T20:45:50.794Zgoroutine 9810 [chan receive, 59 minutes]:
34652026-03-11T20:45:50.794Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc0036f8330?})
34662026-03-11T20:45:50.794Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
34672026-03-11T20:45:50.794Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0xc001feec60?, 0xc001feeca0?}}, 0xc0040547a8)
34682026-03-11T20:45:50.794Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
34692026-03-11T20:45:50.794Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
34702026-03-11T20:45:50.794Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
34712026-03-11T20:45:50.794Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
34722026-03-11T20:45:50.794Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
34732026-03-11T20:45:50.794Z
34742026-03-11T20:45:50.794Zgoroutine 10860 [select, 59 minutes]:
34752026-03-11T20:45:50.794Zgithub.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowScheduler).Start.func1({0xc003987b08?, 0x0?})
34762026-03-11T20:45:50.794Z /work/oxidecomputer/cockroach/pkg/sql/flowinfra/flow_scheduler.go:331 +0x25c
34772026-03-11T20:45:50.794Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
34782026-03-11T20:45:50.794Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
34792026-03-11T20:45:50.794Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
34802026-03-11T20:45:50.794Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
34812026-03-11T20:45:50.794Z
34822026-03-11T20:45:50.794Zgoroutine 10044 [sync.Cond.Wait]:
34832026-03-11T20:45:50.794Zsync.runtime_notifyListWait(0xc001e22250, 0x3757)
34842026-03-11T20:45:50.794Z /work/go/src/runtime/sema.go:569 +0x159
34852026-03-11T20:45:50.795Zsync.(*Cond).Wait(0x3aac600?)
34862026-03-11T20:45:50.795Z /work/go/src/sync/cond.go:70 +0x85
34872026-03-11T20:45:50.795Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583320})
34882026-03-11T20:45:50.795Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
34892026-03-11T20:45:50.795Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
34902026-03-11T20:45:50.795Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
34912026-03-11T20:45:50.795Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
34922026-03-11T20:45:50.795Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
34932026-03-11T20:45:50.795Z
34942026-03-11T20:45:50.795Zgoroutine 10683 [select, 59 minutes]:
34952026-03-11T20:45:50.795Zgithub.com/cockroachdb/cockroach/pkg/server.(*loopbackListener).Accept(0xc006e62780)
34962026-03-11T20:45:50.795Z /work/oxidecomputer/cockroach/pkg/server/loopback.go:49 +0x90
34972026-03-11T20:45:50.795Zgoogle.golang.org/grpc.(*Server).Serve(0xc0076b8000, {0x509c250, 0xc006e62780})
34982026-03-11T20:45:50.795Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:780 +0x469
34992026-03-11T20:45:50.795Zgithub.com/cockroachdb/cockroach/pkg/server.configureGRPCGateway.func2({0xc006e55808?, 0xc003a4b170?})
35002026-03-11T20:45:50.795Z /work/oxidecomputer/cockroach/pkg/server/grpc_gateway.go:94 +0x25
35012026-03-11T20:45:50.795Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
35022026-03-11T20:45:50.795Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
35032026-03-11T20:45:50.804Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
35042026-03-11T20:45:50.804Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
35052026-03-11T20:45:50.804Z
35062026-03-11T20:45:50.804Zgoroutine 10152 [select, 59 minutes]:
35072026-03-11T20:45:50.804Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2({0x50b1300, 0xc009329c50})
35082026-03-11T20:45:50.804Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/queue.go:823 +0xff
35092026-03-11T20:45:50.804Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
35102026-03-11T20:45:50.804Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
35112026-03-11T20:45:50.804Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10124
35122026-03-11T20:45:50.804Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
35132026-03-11T20:45:50.804Z
35142026-03-11T20:45:50.804Zgoroutine 10130 [select, 1 minutes]:
35152026-03-11T20:45:50.804Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).startGossiping.func1({0x50b1300, 0xc003ebc8d0})
35162026-03-11T20:45:50.804Z /work/oxidecomputer/cockroach/pkg/server/node.go:713 +0x1be
35172026-03-11T20:45:50.804Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
35182026-03-11T20:45:50.804Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
35192026-03-11T20:45:50.804Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
35202026-03-11T20:45:50.804Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
35212026-03-11T20:45:50.804Z
35222026-03-11T20:45:50.804Zgoroutine 10201 [select]:
35232026-03-11T20:45:50.804Zgithub.com/cockroachdb/cockroach/pkg/sql/sqlliveness/slinstance.(*Instance).heartbeatLoop(0xc003146240, {0x50b1338, 0xc001fb3a40})
35242026-03-11T20:45:50.804Z /work/oxidecomputer/cockroach/pkg/sql/sqlliveness/slinstance/slinstance.go:244 +0x187
35252026-03-11T20:45:50.804Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
35262026-03-11T20:45:50.804Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
35272026-03-11T20:45:50.804Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
35282026-03-11T20:45:50.805Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
35292026-03-11T20:45:50.805Z
35302026-03-11T20:45:50.805Zgoroutine 10202 [semacquire, 59 minutes]:
35312026-03-11T20:45:50.805Zsync.runtime_Semacquire(0xc007ee0378?)
35322026-03-11T20:45:50.805Z /work/go/src/runtime/sema.go:62 +0x25
35332026-03-11T20:45:50.805Zsync.(*WaitGroup).Wait(0x44e7a0?)
35342026-03-11T20:45:50.805Z /work/go/src/sync/waitgroup.go:116 +0x48
35352026-03-11T20:45:50.805Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc006042f40)
35362026-03-11T20:45:50.805Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
35372026-03-11T20:45:50.805Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc006042f40?, {0x50b1338?, 0xc005f39d60?}})
35382026-03-11T20:45:50.805Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
35392026-03-11T20:45:50.805Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait({0x50b1300?, 0xc00081b4d0?}, {0xc00597bd78, 0x2, 0xc00597bc70?})
35402026-03-11T20:45:50.805Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:193 +0xff
35412026-03-11T20:45:50.805Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0xc002326160, {0x50b1300, 0xc00081b4d0}, 0xc00081af90)
35422026-03-11T20:45:50.805Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:291 +0x59c
35432026-03-11T20:45:50.805Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).Start.func2({0x50b1300, 0xc00081b470})
35442026-03-11T20:45:50.805Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:215 +0x2f3
35452026-03-11T20:45:50.805Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
35462026-03-11T20:45:50.805Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
35472026-03-11T20:45:50.805Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
35482026-03-11T20:45:50.805Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
35492026-03-11T20:45:50.805Z
35502026-03-11T20:45:50.805Zgoroutine 9930 [select, 59 minutes]:
35512026-03-11T20:45:50.805Zgithub.com/cockroachdb/cockroach/pkg/sql/gcjob/gcjobnotifier.(*Notifier).run(0xc006b736e0, {0x26ca?, 0xc001194280?})
35522026-03-11T20:45:50.805Z /work/oxidecomputer/cockroach/pkg/sql/gcjob/gcjobnotifier/notifier.go:145 +0xc8
35532026-03-11T20:45:50.805Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
35542026-03-11T20:45:50.805Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
35552026-03-11T20:45:50.805Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
35562026-03-11T20:45:50.805Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
35572026-03-11T20:45:50.805Z
35582026-03-11T20:45:50.805Zgoroutine 10379 [select]:
35592026-03-11T20:45:50.805Zgithub.com/cockroachdb/cockroach/pkg/kv/kvprober.(*Prober).Start.func1.1({0x50b1300, 0xc009328e40})
35602026-03-11T20:45:50.805Z /work/oxidecomputer/cockroach/pkg/kv/kvprober/kvprober.go:293 +0x314
35612026-03-11T20:45:50.805Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
35622026-03-11T20:45:50.805Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
35632026-03-11T20:45:50.806Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
35642026-03-11T20:45:50.806Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
35652026-03-11T20:45:50.806Z
35662026-03-11T20:45:50.806Zgoroutine 10089 [sync.Cond.Wait]:
35672026-03-11T20:45:50.806Zsync.runtime_notifyListWait(0xc001e22250, 0x378c)
35682026-03-11T20:45:50.806Z /work/go/src/runtime/sema.go:569 +0x159
35692026-03-11T20:45:50.806Zsync.(*Cond).Wait(0x3aac600?)
35702026-03-11T20:45:50.806Z /work/go/src/sync/cond.go:70 +0x85
35712026-03-11T20:45:50.806Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583b90})
35722026-03-11T20:45:50.806Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
35732026-03-11T20:45:50.806Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
35742026-03-11T20:45:50.806Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
35752026-03-11T20:45:50.806Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
35762026-03-11T20:45:50.806Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
35772026-03-11T20:45:50.806Z
35782026-03-11T20:45:50.806Zgoroutine 10153 [select]:
35792026-03-11T20:45:50.806Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaScanner).waitAndProcess(0xc001ef68c0, {0x50b1300, 0xc009329c80}, {0x17f85c0?, 0x5091220?, 0x0?}, 0xc001d48008)
35802026-03-11T20:45:50.806Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scanner.go:224 +0x1f1
35812026-03-11T20:45:50.806Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaScanner).scanLoop.func1.1(0x5091220?)
35822026-03-11T20:45:50.806Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scanner.go:286 +0x45
35832026-03-11T20:45:50.806Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*storeReplicaVisitor).Visit(0xc007b26bd0, 0xc01c08fe40)
35842026-03-11T20:45:50.806Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:436 +0x202
35852026-03-11T20:45:50.806Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaScanner).scanLoop.func1({0x50b1300, 0xc009329c80})
35862026-03-11T20:45:50.806Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scanner.go:284 +0x1a3
35872026-03-11T20:45:50.806Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
35882026-03-11T20:45:50.806Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
35892026-03-11T20:45:50.806Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10124
35902026-03-11T20:45:50.806Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
35912026-03-11T20:45:50.806Z
35922026-03-11T20:45:50.806Zgoroutine 10046 [sync.Cond.Wait]:
35932026-03-11T20:45:50.806Zsync.runtime_notifyListWait(0xc001e22250, 0x3755)
35942026-03-11T20:45:50.806Z /work/go/src/runtime/sema.go:569 +0x159
35952026-03-11T20:45:50.806Zsync.(*Cond).Wait(0x3aac600?)
35962026-03-11T20:45:50.806Z /work/go/src/sync/cond.go:70 +0x85
35972026-03-11T20:45:50.806Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583380})
35982026-03-11T20:45:50.806Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
35992026-03-11T20:45:50.806Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
36002026-03-11T20:45:50.806Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
36012026-03-11T20:45:50.807Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
36022026-03-11T20:45:50.807Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
36032026-03-11T20:45:50.812Z
36042026-03-11T20:45:50.812Zgoroutine 9961 [chan receive, 59 minutes]:
36052026-03-11T20:45:50.812Zgithub.com/cockroachdb/cockroach/pkg/server.(*httpServer).start.func1({0x50b1300, 0xc0034a91d0})
36062026-03-11T20:45:50.812Z /work/oxidecomputer/cockroach/pkg/server/server_http.go:226 +0x45
36072026-03-11T20:45:50.812Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
36082026-03-11T20:45:50.812Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
36092026-03-11T20:45:50.812Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
36102026-03-11T20:45:50.812Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
36112026-03-11T20:45:50.812Z
36122026-03-11T20:45:50.812Zgoroutine 9962 [IO wait, 59 minutes]:
36132026-03-11T20:45:50.812Zinternal/poll.runtime_pollWait(0x7fc12eba3e10, 0x72)
36142026-03-11T20:45:50.812Z /work/go/src/runtime/netpoll.go:345 +0x85
36152026-03-11T20:45:50.812Zinternal/poll.(*pollDesc).wait(0xb?, 0xc0018bdd38?, 0x0)
36162026-03-11T20:45:50.812Z /work/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
36172026-03-11T20:45:50.812Zinternal/poll.(*pollDesc).waitRead(...)
36182026-03-11T20:45:50.812Z /work/go/src/internal/poll/fd_poll_runtime.go:89
36192026-03-11T20:45:50.812Zinternal/poll.(*FD).Accept(0xc001cc7b00)
36202026-03-11T20:45:50.812Z /work/go/src/internal/poll/fd_unix.go:611 +0x2ac
36212026-03-11T20:45:50.812Znet.(*netFD).accept(0xc001cc7b00)
36222026-03-11T20:45:50.812Z /work/go/src/net/fd_unix.go:172 +0x29
36232026-03-11T20:45:50.812Znet.(*TCPListener).accept(0xc0062f6320)
36242026-03-11T20:45:50.812Z /work/go/src/net/tcpsock_posix.go:159 +0x1e
36252026-03-11T20:45:50.812Znet.(*TCPListener).Accept(0xc0062f6320)
36262026-03-11T20:45:50.812Z /work/go/src/net/tcpsock.go:327 +0x30
36272026-03-11T20:45:50.812Zgithub.com/cockroachdb/cmux.(*cMux).Serve(0xc0034cb700)
36282026-03-11T20:45:50.813Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:124 +0x97
36292026-03-11T20:45:50.813Zgithub.com/cockroachdb/cockroach/pkg/server.(*httpServer).start.func2({0xc0013e6608?, 0x0?})
36302026-03-11T20:45:50.813Z /work/oxidecomputer/cockroach/pkg/server/server_http.go:243 +0x27
36312026-03-11T20:45:50.813Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
36322026-03-11T20:45:50.813Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
36332026-03-11T20:45:50.813Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
36342026-03-11T20:45:50.813Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
36352026-03-11T20:45:50.813Z
36362026-03-11T20:45:50.813Zgoroutine 10665 [select]:
36372026-03-11T20:45:50.813Znet.(*pipe).read(0xc006756880, {0xc002f5ec00, 0x600, 0x5?})
36382026-03-11T20:45:50.813Z /work/go/src/net/pipe.go:159 +0x155
36392026-03-11T20:45:50.813Znet.(*pipe).Read(0xc0066d9a40?, {0xc002f5ec00?, 0x53e0b0?, 0x8587f80?})
36402026-03-11T20:45:50.813Z /work/go/src/net/pipe.go:142 +0x1c
36412026-03-11T20:45:50.813Zcrypto/tls.(*atLeastReader).Read(0xc02928da40, {0xc002f5ec00?, 0x0?, 0xc02928da40?})
36422026-03-11T20:45:50.813Z /work/go/src/crypto/tls/conn.go:806 +0x3b
36432026-03-11T20:45:50.813Zbytes.(*Buffer).ReadFrom(0xc0066d9b30, {0x5081fe0, 0xc02928da40})
36442026-03-11T20:45:50.813Z /work/go/src/bytes/buffer.go:211 +0x98
36452026-03-11T20:45:50.813Zcrypto/tls.(*Conn).readFromUntil(0xc0066d9888, {0x7fc12c3c80b0, 0xc006756880}, 0xc00482e990?)
36462026-03-11T20:45:50.813Z /work/go/src/crypto/tls/conn.go:828 +0xde
36472026-03-11T20:45:50.813Zcrypto/tls.(*Conn).readRecordOrCCS(0xc0066d9888, 0x0)
36482026-03-11T20:45:50.813Z /work/go/src/crypto/tls/conn.go:626 +0x3cf
36492026-03-11T20:45:50.813Zcrypto/tls.(*Conn).readRecord(...)
36502026-03-11T20:45:50.813Z /work/go/src/crypto/tls/conn.go:588
36512026-03-11T20:45:50.813Zcrypto/tls.(*Conn).Read(0xc0066d9888, {0xc00674c000, 0x8000, 0x7fc12c3d6bd8?})
36522026-03-11T20:45:50.813Z /work/go/src/crypto/tls/conn.go:1370 +0x156
36532026-03-11T20:45:50.813Zbufio.(*Reader).Read(0xc004bb34a0, {0xc0036d8c80, 0x9, 0xc0047f2808?})
36542026-03-11T20:45:50.813Z /work/go/src/bufio/bufio.go:241 +0x197
36552026-03-11T20:45:50.813Zio.ReadAtLeast({0x507c320, 0xc004bb34a0}, {0xc0036d8c80, 0x9, 0x9}, 0x9)
36562026-03-11T20:45:50.813Z /work/go/src/io/io.go:335 +0x90
36572026-03-11T20:45:50.813Zio.ReadFull(...)
36582026-03-11T20:45:50.813Z /work/go/src/io/io.go:354
36592026-03-11T20:45:50.813Zgolang.org/x/net/http2.readFrameHeader({0xc0036d8c80, 0x9, 0xc028c572d8?}, {0x507c320?, 0xc004bb34a0?})
36602026-03-11T20:45:50.813Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x65
36612026-03-11T20:45:50.813Zgolang.org/x/net/http2.(*Framer).ReadFrame(0xc0036d8c40)
36622026-03-11T20:45:50.813Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:501 +0x85
36632026-03-11T20:45:50.813Zgoogle.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc0067249c0, 0xc006b55cb0, 0x42e6fc8)
36642026-03-11T20:45:50.813Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:611 +0x14e
36652026-03-11T20:45:50.813Zgoogle.golang.org/grpc.(*Server).serveStreams(0xc0076b8000, {0x50f1d80, 0xc0067249c0})
36662026-03-11T20:45:50.813Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:906 +0x114
36672026-03-11T20:45:50.814Zgoogle.golang.org/grpc.(*Server).handleRawConn.func1()
36682026-03-11T20:45:50.814Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:848 +0x45
36692026-03-11T20:45:50.814Zcreated by google.golang.org/grpc.(*Server).handleRawConn in goroutine 10600
36702026-03-11T20:45:50.814Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:847 +0x15b
36712026-03-11T20:45:50.814Z
36722026-03-11T20:45:50.814Zgoroutine 10463 [select]:
36732026-03-11T20:45:50.814Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents(0xc003283340, {0x50b1338, 0xc0076842d0}, 0xc007494b10, 0xc002070f00)
36742026-03-11T20:45:50.814Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:331 +0xc5
36752026-03-11T20:45:50.814Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func2({0x50b1338?, 0xc0076842d0?})
36762026-03-11T20:45:50.814Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:288 +0x32
36772026-03-11T20:45:50.814Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
36782026-03-11T20:45:50.814Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
36792026-03-11T20:45:50.814Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
36802026-03-11T20:45:50.814Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
36812026-03-11T20:45:50.814Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10456
36822026-03-11T20:45:50.814Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
36832026-03-11T20:45:50.814Z
36842026-03-11T20:45:50.814Zgoroutine 10461 [select]:
36852026-03-11T20:45:50.814Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents(0xc0032831e0, {0x50b1338, 0xc007684280}, 0xc0074949c0, 0xc002070de0)
36862026-03-11T20:45:50.814Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:331 +0xc5
36872026-03-11T20:45:50.814Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func2({0x50b1338?, 0xc007684280?})
36882026-03-11T20:45:50.814Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:288 +0x32
36892026-03-11T20:45:50.814Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
36902026-03-11T20:45:50.814Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
36912026-03-11T20:45:50.814Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
36922026-03-11T20:45:50.814Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
36932026-03-11T20:45:50.814Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10455
36942026-03-11T20:45:50.814Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
36952026-03-11T20:45:50.814Z
36962026-03-11T20:45:50.814Zgoroutine 11141 [chan receive, 59 minutes]:
36972026-03-11T20:45:50.814Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).rangeFeedWithRangeID(0xc001d7a808, 0x3d87860?, 0xc000a14120, {0x7fc12ce5a548, 0xc003c48050})
36982026-03-11T20:45:50.814Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:249 +0x76b
36992026-03-11T20:45:50.814Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed(...)
37002026-03-11T20:45:50.814Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:149
37012026-03-11T20:45:50.814Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).RangeFeed(0xc000f39208, 0xc000a14120, {0x50e5040, 0xc003c48050})
37022026-03-11T20:45:50.814Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:3082 +0x16f
37032026-03-11T20:45:50.819Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).RangeFeed(0xc001eb4380, 0xc000a14120, {0x50e5040, 0xc003c48050})
37042026-03-11T20:45:50.819Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/stores.go:216 +0x19a
37052026-03-11T20:45:50.819Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed(0x732ffb0?, 0x40055e0?, {0x50e5040, 0xc003c48050})
37062026-03-11T20:45:50.819Z /work/oxidecomputer/cockroach/pkg/server/node.go:1232 +0x28
37072026-03-11T20:45:50.819Zgithub.com/cockroachdb/cockroach/pkg/roachpb._Internal_RangeFeed_Handler({0x40055e0, 0xc004911b08}, {0x50df230, 0xc005b65880})
37082026-03-11T20:45:50.819Z /work/oxidecomputer/cockroach/pkg/roachpb/api.pb.go:9523 +0x107
37092026-03-11T20:45:50.819Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.StreamServerInterceptor.func14({0x40055e0, 0xc004911b08}, {0x50df9e0, 0xc00416e680}, 0xc002290858, 0x42e42f8)
37102026-03-11T20:45:50.819Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:205 +0x35b
37112026-03-11T20:45:50.819Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x40055e0?, 0xc004911b08?}, {0x50df9e0?, 0xc00416e680?})
37122026-03-11T20:45:50.819Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1408 +0x59
37132026-03-11T20:45:50.819Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func4({0x40055e0, 0xc004911b08}, {0x50df9e0, 0xc00416e680}, 0xc003bd2918?, 0xc004b39600)
37142026-03-11T20:45:50.819Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:247 +0x6c
37152026-03-11T20:45:50.819Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x40055e0?, 0xc004911b08?}, {0x50df9e0?, 0xc00416e680?})
37162026-03-11T20:45:50.819Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
37172026-03-11T20:45:50.819Zgithub.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({{{0x2a?}}}, {0x40055e0, 0xc004911b08}, {0x50df9e0, 0xc00416e680}, 0xc002290858, 0xc004b39600)
37182026-03-11T20:45:50.819Z /work/oxidecomputer/cockroach/pkg/rpc/auth.go:98 +0x19b
37192026-03-11T20:45:50.819Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x40055e0?, 0xc004911b08?}, {0x50df9e0?, 0xc00416e680?})
37202026-03-11T20:45:50.819Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
37212026-03-11T20:45:50.819Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func2.1({0x66?, 0x40?})
37222026-03-11T20:45:50.819Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:216 +0x27
37232026-03-11T20:45:50.819Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc003f40400, {0x50b1300, 0xc00435de90}, {0xc003bd2b28?, 0x415dbb?}, 0xc003bd2ac0)
37242026-03-11T20:45:50.819Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:344 +0xcb
37252026-03-11T20:45:50.819Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func2({0x40055e0, 0xc004911b08}, {0x50df9e0, 0xc00416e680}, 0xc002290858, 0xc004b39600)
37262026-03-11T20:45:50.819Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:215 +0xdf
37272026-03-11T20:45:50.819Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x40055e0?, 0xc004911b08?}, {0x50df9e0?, 0xc00416e680?})
37282026-03-11T20:45:50.819Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
37292026-03-11T20:45:50.819Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x40055e0, 0xc004911b08}, {0x50df9e0, 0xc00416e680}, 0xc002290858, 0x42e42f8)
37302026-03-11T20:45:50.819Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1413 +0x108
37312026-03-11T20:45:50.819Zgoogle.golang.org/grpc.(*Server).processStreamingRPC(0xc004feefc0, {0x50f1d80, 0xc001aef6c0}, 0xc00350d8c0, 0xc007661ad0, 0x73abba0, 0x0)
37322026-03-11T20:45:50.819Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1549 +0xe51
37332026-03-11T20:45:50.820Zgoogle.golang.org/grpc.(*Server).handleStream(0xc004feefc0, {0x50f1d80, 0xc001aef6c0}, 0xc00350d8c0, 0x0)
37342026-03-11T20:45:50.820Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1624 +0x999
37352026-03-11T20:45:50.820Zgoogle.golang.org/grpc.(*Server).serveStreams.func1.2()
37362026-03-11T20:45:50.820Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:922 +0x8d
37372026-03-11T20:45:50.820Zcreated by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 10956
37382026-03-11T20:45:50.820Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:920 +0x226
37392026-03-11T20:45:50.820Z
37402026-03-11T20:45:50.820Zgoroutine 9851 [semacquire, 59 minutes]:
37412026-03-11T20:45:50.820Zsync.runtime_Semacquire(0xc00212a090?)
37422026-03-11T20:45:50.820Z /work/go/src/runtime/sema.go:62 +0x25
37432026-03-11T20:45:50.820Zsync.(*WaitGroup).Wait(0x44e7a0?)
37442026-03-11T20:45:50.820Z /work/go/src/sync/waitgroup.go:116 +0x48
37452026-03-11T20:45:50.820Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc001f30000)
37462026-03-11T20:45:50.820Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
37472026-03-11T20:45:50.820Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc001f30000?, {0x50b1338?, 0xc005d14230?}})
37482026-03-11T20:45:50.820Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
37492026-03-11T20:45:50.820Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait({0x50b1300?, 0xc007582330?}, {0xc004343d78, 0x2, 0x4000000000000000?})
37502026-03-11T20:45:50.820Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:193 +0xff
37512026-03-11T20:45:50.820Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0xc003283ce0, {0x50b1300, 0xc007582330}, 0xc00485e720)
37522026-03-11T20:45:50.820Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:291 +0x59c
37532026-03-11T20:45:50.820Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).Start.func2({0x50b1300, 0xc00485e930})
37542026-03-11T20:45:50.821Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:215 +0x2f3
37552026-03-11T20:45:50.821Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
37562026-03-11T20:45:50.821Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
37572026-03-11T20:45:50.821Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
37582026-03-11T20:45:50.821Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
37592026-03-11T20:45:50.821Z
37602026-03-11T20:45:50.821Zgoroutine 10876 [IO wait, 59 minutes]:
37612026-03-11T20:45:50.821Zinternal/poll.runtime_pollWait(0x7fc12ea8f6a8, 0x72)
37622026-03-11T20:45:50.821Z /work/go/src/runtime/netpoll.go:345 +0x85
37632026-03-11T20:45:50.821Zinternal/poll.(*pollDesc).wait(0xf?, 0x359d6e5?, 0x0)
37642026-03-11T20:45:50.821Z /work/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
37652026-03-11T20:45:50.821Zinternal/poll.(*pollDesc).waitRead(...)
37662026-03-11T20:45:50.821Z /work/go/src/internal/poll/fd_poll_runtime.go:89
37672026-03-11T20:45:50.821Zinternal/poll.(*FD).Accept(0xc00667eb00)
37682026-03-11T20:45:50.821Z /work/go/src/internal/poll/fd_unix.go:611 +0x2ac
37692026-03-11T20:45:50.821Znet.(*netFD).accept(0xc00667eb00)
37702026-03-11T20:45:50.821Z /work/go/src/net/fd_unix.go:172 +0x29
37712026-03-11T20:45:50.821Znet.(*TCPListener).accept(0xc006635420)
37722026-03-11T20:45:50.821Z /work/go/src/net/tcpsock_posix.go:159 +0x1e
37732026-03-11T20:45:50.821Znet.(*TCPListener).Accept(0xc006635420)
37742026-03-11T20:45:50.821Z /work/go/src/net/tcpsock.go:327 +0x30
37752026-03-11T20:45:50.821Zgithub.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith(0xc000b26be8, {0x50b1300, 0xc002ff1980}, 0xc003dd7580, {0x50a4470, 0xc006635420}, 0xc003fac210)
37762026-03-11T20:45:50.821Z /work/oxidecomputer/cockroach/pkg/util/netutil/net.go:138 +0x77
37772026-03-11T20:45:50.821Zgithub.com/cockroachdb/cockroach/pkg/server.(*SQLServer).startServeSQL.func1({0x50b1300, 0xc002ff1980})
37782026-03-11T20:45:50.821Z /work/oxidecomputer/cockroach/pkg/server/server_sql.go:1374 +0xcb
37792026-03-11T20:45:50.821Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
37802026-03-11T20:45:50.821Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
37812026-03-11T20:45:50.821Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
37822026-03-11T20:45:50.821Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
37832026-03-11T20:45:50.821Z
37842026-03-11T20:45:50.821Zgoroutine 10114 [sync.Cond.Wait]:
37852026-03-11T20:45:50.821Zsync.runtime_notifyListWait(0xc001e22250, 0x3760)
37862026-03-11T20:45:50.821Z /work/go/src/runtime/sema.go:569 +0x159
37872026-03-11T20:45:50.821Zsync.(*Cond).Wait(0x3aac600?)
37882026-03-11T20:45:50.822Z /work/go/src/sync/cond.go:70 +0x85
37892026-03-11T20:45:50.822Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc003ebc150})
37902026-03-11T20:45:50.822Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
37912026-03-11T20:45:50.822Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
37922026-03-11T20:45:50.822Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
37932026-03-11T20:45:50.822Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
37942026-03-11T20:45:50.822Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
37952026-03-11T20:45:50.822Z
37962026-03-11T20:45:50.822Zgoroutine 10956 [IO wait]:
37972026-03-11T20:45:50.822Zinternal/poll.runtime_pollWait(0x7fc12eba3c20, 0x72)
37982026-03-11T20:45:50.822Z /work/go/src/runtime/netpoll.go:345 +0x85
37992026-03-11T20:45:50.822Zinternal/poll.(*pollDesc).wait(0xc001c2c000?, 0xc001694000?, 0x0)
38002026-03-11T20:45:50.822Z /work/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
38012026-03-11T20:45:50.822Zinternal/poll.(*pollDesc).waitRead(...)
38022026-03-11T20:45:50.822Z /work/go/src/internal/poll/fd_poll_runtime.go:89
38032026-03-11T20:45:50.828Zinternal/poll.(*FD).Read(0xc001c2c000, {0xc001694000, 0x600, 0x600})
38042026-03-11T20:45:50.828Z /work/go/src/internal/poll/fd_unix.go:164 +0x27a
38052026-03-11T20:45:50.828Znet.(*netFD).Read(0xc001c2c000, {0xc001694000?, 0xc00409c868?, 0x41e1e5?})
38062026-03-11T20:45:50.828Z /work/go/src/net/fd_posix.go:55 +0x25
38072026-03-11T20:45:50.828Znet.(*conn).Read(0xc00652a000, {0xc001694000?, 0x7fc12af111b0?, 0xc0304c2de0?})
38082026-03-11T20:45:50.828Z /work/go/src/net/net.go:185 +0x45
38092026-03-11T20:45:50.828Zgithub.com/cockroachdb/cmux.(*MuxConn).Read(0xc00119e120, {0xc001694000, 0x600, 0x600})
38102026-03-11T20:45:50.828Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x14f
38112026-03-11T20:45:50.828Zcrypto/tls.(*atLeastReader).Read(0xc0304c2de0, {0xc001694000?, 0x0?, 0xc0304c2de0?})
38122026-03-11T20:45:50.828Z /work/go/src/crypto/tls/conn.go:806 +0x3b
38132026-03-11T20:45:50.828Zbytes.(*Buffer).ReadFrom(0xc007b34630, {0x5081fe0, 0xc0304c2de0})
38142026-03-11T20:45:50.828Z /work/go/src/bytes/buffer.go:211 +0x98
38152026-03-11T20:45:50.828Zcrypto/tls.(*Conn).readFromUntil(0xc007b34388, {0x7fc12ce5a2d8, 0xc00119e120}, 0xc00409c990?)
38162026-03-11T20:45:50.828Z /work/go/src/crypto/tls/conn.go:828 +0xde
38172026-03-11T20:45:50.828Zcrypto/tls.(*Conn).readRecordOrCCS(0xc007b34388, 0x0)
38182026-03-11T20:45:50.828Z /work/go/src/crypto/tls/conn.go:626 +0x3cf
38192026-03-11T20:45:50.828Zcrypto/tls.(*Conn).readRecord(...)
38202026-03-11T20:45:50.828Z /work/go/src/crypto/tls/conn.go:588
38212026-03-11T20:45:50.828Zcrypto/tls.(*Conn).Read(0xc007b34388, {0xc004118000, 0x8000, 0x7fc12c3bbc90?})
38222026-03-11T20:45:50.828Z /work/go/src/crypto/tls/conn.go:1370 +0x156
38232026-03-11T20:45:50.828Zbufio.(*Reader).Read(0xc005534360, {0xc00386aac0, 0x9, 0xc0304c2dc8?})
38242026-03-11T20:45:50.828Z /work/go/src/bufio/bufio.go:241 +0x197
38252026-03-11T20:45:50.828Zio.ReadAtLeast({0x507c320, 0xc005534360}, {0xc00386aac0, 0x9, 0x9}, 0x9)
38262026-03-11T20:45:50.828Z /work/go/src/io/io.go:335 +0x90
38272026-03-11T20:45:50.828Zio.ReadFull(...)
38282026-03-11T20:45:50.828Z /work/go/src/io/io.go:354
38292026-03-11T20:45:50.828Zgolang.org/x/net/http2.readFrameHeader({0xc00386aac0, 0x9, 0xc00409ce10?}, {0x507c320?, 0xc005534360?})
38302026-03-11T20:45:50.828Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x65
38312026-03-11T20:45:50.829Zgolang.org/x/net/http2.(*Framer).ReadFrame(0xc00386aa80)
38322026-03-11T20:45:50.829Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:501 +0x85
38332026-03-11T20:45:50.829Zgoogle.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc001aef6c0, 0xc00279b080, 0x42e6fc8)
38342026-03-11T20:45:50.829Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:611 +0x14e
38352026-03-11T20:45:50.829Zgoogle.golang.org/grpc.(*Server).serveStreams(0xc004feefc0, {0x50f1d80, 0xc001aef6c0})
38362026-03-11T20:45:50.829Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:906 +0x114
38372026-03-11T20:45:50.829Zgoogle.golang.org/grpc.(*Server).handleRawConn.func1()
38382026-03-11T20:45:50.829Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:848 +0x45
38392026-03-11T20:45:50.829Zcreated by google.golang.org/grpc.(*Server).handleRawConn in goroutine 10868
38402026-03-11T20:45:50.829Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:847 +0x15b
38412026-03-11T20:45:50.829Z
38422026-03-11T20:45:50.829Zgoroutine 10971 [select, 59 minutes]:
38432026-03-11T20:45:50.829Zgoogle.golang.org/grpc.newClientStreamWithParams.func4()
38442026-03-11T20:45:50.829Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:341 +0x8c
38452026-03-11T20:45:50.829Zcreated by google.golang.org/grpc.newClientStreamWithParams in goroutine 10928
38462026-03-11T20:45:50.829Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:340 +0xb2a
38472026-03-11T20:45:50.829Z
38482026-03-11T20:45:50.829Zgoroutine 9850 [select, 59 minutes]:
38492026-03-11T20:45:50.829Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initCancelingWorkers.func1({0x50b1300, 0xc00485e6c0})
38502026-03-11T20:45:50.829Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:139 +0xc5
38512026-03-11T20:45:50.830Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
38522026-03-11T20:45:50.830Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
38532026-03-11T20:45:50.830Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
38542026-03-11T20:45:50.830Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
38552026-03-11T20:45:50.830Z
38562026-03-11T20:45:50.830Zgoroutine 10586 [select, 59 minutes]:
38572026-03-11T20:45:50.830Zgithub.com/cockroachdb/cockroach/pkg/util/hlc.(*Clock).StartMonitoringForwardClockJumps.func1()
38582026-03-11T20:45:50.830Z /work/oxidecomputer/cockroach/pkg/util/hlc/hlc.go:263 +0x148
38592026-03-11T20:45:50.830Zcreated by github.com/cockroachdb/cockroach/pkg/util/hlc.(*Clock).StartMonitoringForwardClockJumps in goroutine 9708
38602026-03-11T20:45:50.830Z /work/oxidecomputer/cockroach/pkg/util/hlc/hlc.go:255 +0x106
38612026-03-11T20:45:50.830Z
38622026-03-11T20:45:50.830Zgoroutine 10801 [sync.Cond.Wait, 59 minutes]:
38632026-03-11T20:45:50.830Zsync.runtime_notifyListWait(0xc0064c5050, 0x45)
38642026-03-11T20:45:50.830Z /work/go/src/runtime/sema.go:569 +0x159
38652026-03-11T20:45:50.830Zsync.(*Cond).Wait(0x0?)
38662026-03-11T20:45:50.830Z /work/go/src/sync/cond.go:70 +0x85
38672026-03-11T20:45:50.830Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3ad0})
38682026-03-11T20:45:50.830Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
38692026-03-11T20:45:50.830Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
38702026-03-11T20:45:50.830Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
38712026-03-11T20:45:50.830Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
38722026-03-11T20:45:50.830Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
38732026-03-11T20:45:50.830Z
38742026-03-11T20:45:50.830Zgoroutine 10059 [sync.Cond.Wait]:
38752026-03-11T20:45:50.830Zsync.runtime_notifyListWait(0xc001e22250, 0x3775)
38762026-03-11T20:45:50.830Z /work/go/src/runtime/sema.go:569 +0x159
38772026-03-11T20:45:50.830Zsync.(*Cond).Wait(0x3aac600?)
38782026-03-11T20:45:50.830Z /work/go/src/sync/cond.go:70 +0x85
38792026-03-11T20:45:50.830Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc0075835f0})
38802026-03-11T20:45:50.830Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
38812026-03-11T20:45:50.830Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
38822026-03-11T20:45:50.830Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
38832026-03-11T20:45:50.830Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
38842026-03-11T20:45:50.831Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
38852026-03-11T20:45:50.831Z
38862026-03-11T20:45:50.831Zgoroutine 10138 [select]:
38872026-03-11T20:45:50.831Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/liveness.(*NodeLiveness).Start.func1({0xc002344f08?, 0x0?})
38882026-03-11T20:45:50.831Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/liveness/liveness.go:789 +0x37f
38892026-03-11T20:45:50.831Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
38902026-03-11T20:45:50.831Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
38912026-03-11T20:45:50.831Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
38922026-03-11T20:45:50.831Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
38932026-03-11T20:45:50.831Z
38942026-03-11T20:45:50.831Zgoroutine 10934 [semacquire, 59 minutes]:
38952026-03-11T20:45:50.831Zsync.runtime_Semacquire(0xc005dbfce0?)
38962026-03-11T20:45:50.831Z /work/go/src/runtime/sema.go:62 +0x25
38972026-03-11T20:45:50.831Zsync.(*WaitGroup).Wait(0x44e7a0?)
38982026-03-11T20:45:50.831Z /work/go/src/sync/waitgroup.go:116 +0x48
38992026-03-11T20:45:50.831Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc000e48780)
39002026-03-11T20:45:50.831Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
39012026-03-11T20:45:50.831Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc000e48780?, {0x50b1338?, 0xc0010bcc30?}})
39022026-03-11T20:45:50.831Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
39032026-03-11T20:45:50.837Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait({0x50b1300?, 0xc005573b30?}, {0xc005a29d78, 0x2, 0xc005a29c70?})
39042026-03-11T20:45:50.837Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:193 +0xff
39052026-03-11T20:45:50.837Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0xc002e87a20, {0x50b1300, 0xc005573b30}, 0xc0055738c0)
39062026-03-11T20:45:50.837Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:291 +0x59c
39072026-03-11T20:45:50.837Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).Start.func2({0x50b1300, 0xc005573aa0})
39082026-03-11T20:45:50.837Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:215 +0x2f3
39092026-03-11T20:45:50.837Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
39102026-03-11T20:45:50.837Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
39112026-03-11T20:45:50.837Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10933
39122026-03-11T20:45:50.837Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
39132026-03-11T20:45:50.837Z
39142026-03-11T20:45:50.837Zgoroutine 9840 [select, 59 minutes]:
39152026-03-11T20:45:50.837Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc006b64908?, 0xc003621860?})
39162026-03-11T20:45:50.837Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
39172026-03-11T20:45:50.837Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
39182026-03-11T20:45:50.837Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
39192026-03-11T20:45:50.837Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
39202026-03-11T20:45:50.837Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
39212026-03-11T20:45:50.837Z
39222026-03-11T20:45:50.837Zgoroutine 9838 [select, 59 minutes]:
39232026-03-11T20:45:50.837Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc006b64308?, 0x0?})
39242026-03-11T20:45:50.837Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
39252026-03-11T20:45:50.837Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
39262026-03-11T20:45:50.837Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
39272026-03-11T20:45:50.837Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
39282026-03-11T20:45:50.837Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
39292026-03-11T20:45:50.837Z
39302026-03-11T20:45:50.837Zgoroutine 10074 [sync.Cond.Wait]:
39312026-03-11T20:45:50.837Zsync.runtime_notifyListWait(0xc001e22250, 0x3766)
39322026-03-11T20:45:50.837Z /work/go/src/runtime/sema.go:569 +0x159
39332026-03-11T20:45:50.837Zsync.(*Cond).Wait(0x3aac600?)
39342026-03-11T20:45:50.837Z /work/go/src/sync/cond.go:70 +0x85
39352026-03-11T20:45:50.837Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc0075838c0})
39362026-03-11T20:45:50.837Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
39372026-03-11T20:45:50.837Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
39382026-03-11T20:45:50.838Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
39392026-03-11T20:45:50.838Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
39402026-03-11T20:45:50.838Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
39412026-03-11T20:45:50.838Z
39422026-03-11T20:45:50.838Zgoroutine 10065 [sync.Cond.Wait]:
39432026-03-11T20:45:50.838Zsync.runtime_notifyListWait(0xc001e22250, 0x3778)
39442026-03-11T20:45:50.838Z /work/go/src/runtime/sema.go:569 +0x159
39452026-03-11T20:45:50.838Zsync.(*Cond).Wait(0x3aac600?)
39462026-03-11T20:45:50.838Z /work/go/src/sync/cond.go:70 +0x85
39472026-03-11T20:45:50.838Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583710})
39482026-03-11T20:45:50.838Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
39492026-03-11T20:45:50.838Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
39502026-03-11T20:45:50.838Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
39512026-03-11T20:45:50.838Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
39522026-03-11T20:45:50.838Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
39532026-03-11T20:45:50.838Z
39542026-03-11T20:45:50.838Zgoroutine 10071 [sync.Cond.Wait]:
39552026-03-11T20:45:50.838Zsync.runtime_notifyListWait(0xc001e22250, 0x378a)
39562026-03-11T20:45:50.838Z /work/go/src/runtime/sema.go:569 +0x159
39572026-03-11T20:45:50.838Zsync.(*Cond).Wait(0x3aac600?)
39582026-03-11T20:45:50.838Z /work/go/src/sync/cond.go:70 +0x85
39592026-03-11T20:45:50.838Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583830})
39602026-03-11T20:45:50.838Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
39612026-03-11T20:45:50.838Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
39622026-03-11T20:45:50.838Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
39632026-03-11T20:45:50.838Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
39642026-03-11T20:45:50.838Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
39652026-03-11T20:45:50.838Z
39662026-03-11T20:45:50.838Zgoroutine 10063 [sync.Cond.Wait]:
39672026-03-11T20:45:50.838Zsync.runtime_notifyListWait(0xc001e22250, 0x375a)
39682026-03-11T20:45:50.838Z /work/go/src/runtime/sema.go:569 +0x159
39692026-03-11T20:45:50.838Zsync.(*Cond).Wait(0x3aac600?)
39702026-03-11T20:45:50.838Z /work/go/src/sync/cond.go:70 +0x85
39712026-03-11T20:45:50.838Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc0075836b0})
39722026-03-11T20:45:50.838Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
39732026-03-11T20:45:50.838Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
39742026-03-11T20:45:50.838Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
39752026-03-11T20:45:50.839Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
39762026-03-11T20:45:50.839Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
39772026-03-11T20:45:50.839Z
39782026-03-11T20:45:50.839Zgoroutine 11100 [semacquire, 59 minutes]:
39792026-03-11T20:45:50.839Zsync.runtime_Semacquire(0xc003453ce0?)
39802026-03-11T20:45:50.839Z /work/go/src/runtime/sema.go:62 +0x25
39812026-03-11T20:45:50.839Zsync.(*WaitGroup).Wait(0x44e7a0?)
39822026-03-11T20:45:50.839Z /work/go/src/sync/waitgroup.go:116 +0x48
39832026-03-11T20:45:50.839Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc0036e00c0)
39842026-03-11T20:45:50.839Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
39852026-03-11T20:45:50.839Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc0036e00c0?, {0x50b1338?, 0xc001435400?}})
39862026-03-11T20:45:50.839Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
39872026-03-11T20:45:50.839Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed(0xc003d48008, {0x50b1338?, 0xc001435310?}, {0xc003dad080, 0x1, 0xc002fe1248?}, {0xc007661ad0?, 0x55b21e1?, 0xc0?}, 0x0, ...)
39882026-03-11T20:45:50.839Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:139 +0x6c5
39892026-03-11T20:45:50.840Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*dbAdapter).RangeFeed(0x115c72a197c241?, {0x50b1338?, 0xc001435310?}, {0xc003dad080?, 0x0?, 0xc004e0a008?}, {0x50b1300?, 0x402e270?, 0xc0?}, 0x0, ...)
39902026-03-11T20:45:50.840Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/db_adapter.go:78 +0x3e
39912026-03-11T20:45:50.840Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func1({0x50b1338?, 0xc001435310?})
39922026-03-11T20:45:50.840Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:285 +0x6c
39932026-03-11T20:45:50.840Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
39942026-03-11T20:45:50.840Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
39952026-03-11T20:45:50.840Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
39962026-03-11T20:45:50.840Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
39972026-03-11T20:45:50.840Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 11001
39982026-03-11T20:45:50.840Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
39992026-03-11T20:45:50.840Z
40002026-03-11T20:45:50.840Zgoroutine 10061 [sync.Cond.Wait]:
40012026-03-11T20:45:50.840Zsync.runtime_notifyListWait(0xc001e22250, 0x376b)
40022026-03-11T20:45:50.840Z /work/go/src/runtime/sema.go:569 +0x159
40032026-03-11T20:45:50.845Zsync.(*Cond).Wait(0x3aac600?)
40042026-03-11T20:45:50.845Z /work/go/src/sync/cond.go:70 +0x85
40052026-03-11T20:45:50.845Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583650})
40062026-03-11T20:45:50.845Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
40072026-03-11T20:45:50.845Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
40082026-03-11T20:45:50.845Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
40092026-03-11T20:45:50.845Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
40102026-03-11T20:45:50.845Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
40112026-03-11T20:45:50.845Z
40122026-03-11T20:45:50.845Zgoroutine 10460 [semacquire, 59 minutes]:
40132026-03-11T20:45:50.845Zsync.runtime_Semacquire(0xc004f350e0?)
40142026-03-11T20:45:50.845Z /work/go/src/runtime/sema.go:62 +0x25
40152026-03-11T20:45:50.845Zsync.(*WaitGroup).Wait(0x44e7a0?)
40162026-03-11T20:45:50.845Z /work/go/src/sync/waitgroup.go:116 +0x48
40172026-03-11T20:45:50.845Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc005d7acc0)
40182026-03-11T20:45:50.845Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
40192026-03-11T20:45:50.846Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc005d7acc0?, {0x50b1338?, 0xc007611a40?}})
40202026-03-11T20:45:50.846Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
40212026-03-11T20:45:50.846Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed(0xc002d6f408, {0x50b1338?, 0xc007684280?}, {0xc007494a50, 0x1, 0xc0074949c0?}, {0xc0032831e0?, 0xd8ae20?, 0xc0?}, 0x1, ...)
40222026-03-11T20:45:50.846Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:139 +0x6c5
40232026-03-11T20:45:50.846Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*dbAdapter).RangeFeed(0xc007494f60?, {0x50b1338?, 0xc007684280?}, {0xc007494a50?, 0x50b1300?, 0xc007494ab0?}, {0x7070000050b1300?, 0x50b1300?, 0x0?}, 0x1, ...)
40242026-03-11T20:45:50.846Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/db_adapter.go:78 +0x3e
40252026-03-11T20:45:50.846Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func1({0x50b1338?, 0xc007684280?})
40262026-03-11T20:45:50.846Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:285 +0x6c
40272026-03-11T20:45:50.846Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
40282026-03-11T20:45:50.846Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
40292026-03-11T20:45:50.846Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
40302026-03-11T20:45:50.846Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
40312026-03-11T20:45:50.846Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10455
40322026-03-11T20:45:50.846Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
40332026-03-11T20:45:50.846Z
40342026-03-11T20:45:50.846Zgoroutine 11060 [IO wait]:
40352026-03-11T20:45:50.846Zinternal/poll.runtime_pollWait(0x7fc12ea8f2c8, 0x72)
40362026-03-11T20:45:50.846Z /work/go/src/runtime/netpoll.go:345 +0x85
40372026-03-11T20:45:50.846Zinternal/poll.(*pollDesc).wait(0xc0008dd700?, 0xc00421a000?, 0x0)
40382026-03-11T20:45:50.846Z /work/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
40392026-03-11T20:45:50.847Zinternal/poll.(*pollDesc).waitRead(...)
40402026-03-11T20:45:50.847Z /work/go/src/internal/poll/fd_poll_runtime.go:89
40412026-03-11T20:45:50.847Zinternal/poll.(*FD).Read(0xc0008dd700, {0xc00421a000, 0x4000, 0x4000})
40422026-03-11T20:45:50.847Z /work/go/src/internal/poll/fd_unix.go:164 +0x27a
40432026-03-11T20:45:50.847Znet.(*netFD).Read(0xc0008dd700, {0xc00421a000?, 0x7fc128ac18b8?, 0xc0238f5d10?})
40442026-03-11T20:45:50.847Z /work/go/src/net/fd_posix.go:55 +0x25
40452026-03-11T20:45:50.847Znet.(*conn).Read(0xc004b0a9c0, {0xc00421a000?, 0xc005aae8e0?, 0x415dbb?})
40462026-03-11T20:45:50.847Z /work/go/src/net/net.go:185 +0x45
40472026-03-11T20:45:50.847Zcrypto/tls.(*atLeastReader).Read(0xc0238f5d10, {0xc00421a000?, 0x0?, 0xc0238f5d10?})
40482026-03-11T20:45:50.847Z /work/go/src/crypto/tls/conn.go:806 +0x3b
40492026-03-11T20:45:50.847Zbytes.(*Buffer).ReadFrom(0xc004ca7430, {0x5081fe0, 0xc0238f5d10})
40502026-03-11T20:45:50.847Z /work/go/src/bytes/buffer.go:211 +0x98
40512026-03-11T20:45:50.847Zcrypto/tls.(*Conn).readFromUntil(0xc004ca7188, {0x50821c0, 0xc004b0a9c0}, 0xc005aae928?)
40522026-03-11T20:45:50.847Z /work/go/src/crypto/tls/conn.go:828 +0xde
40532026-03-11T20:45:50.847Zcrypto/tls.(*Conn).readRecordOrCCS(0xc004ca7188, 0x0)
40542026-03-11T20:45:50.847Z /work/go/src/crypto/tls/conn.go:626 +0x3cf
40552026-03-11T20:45:50.847Zcrypto/tls.(*Conn).readRecord(...)
40562026-03-11T20:45:50.847Z /work/go/src/crypto/tls/conn.go:588
40572026-03-11T20:45:50.848Zcrypto/tls.(*Conn).Read(0xc004ca7188, {0xc005af4000, 0x8000, 0xc02f3df7a0?})
40582026-03-11T20:45:50.848Z /work/go/src/crypto/tls/conn.go:1370 +0x156
40592026-03-11T20:45:50.848Zbufio.(*Reader).Read(0xc00612f8c0, {0xc004b48ac0, 0x9, 0x0?})
40602026-03-11T20:45:50.848Z /work/go/src/bufio/bufio.go:241 +0x197
40612026-03-11T20:45:50.848Zio.ReadAtLeast({0x507c320, 0xc00612f8c0}, {0xc004b48ac0, 0x9, 0x9}, 0x9)
40622026-03-11T20:45:50.848Z /work/go/src/io/io.go:335 +0x90
40632026-03-11T20:45:50.848Zio.ReadFull(...)
40642026-03-11T20:45:50.848Z /work/go/src/io/io.go:354
40652026-03-11T20:45:50.848Zgolang.org/x/net/http2.readFrameHeader({0xc004b48ac0, 0x9, 0x0?}, {0x507c320?, 0xc00612f8c0?})
40662026-03-11T20:45:50.848Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x65
40672026-03-11T20:45:50.848Zgolang.org/x/net/http2.(*Framer).ReadFrame(0xc004b48a80)
40682026-03-11T20:45:50.848Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:501 +0x85
40692026-03-11T20:45:50.848Zgoogle.golang.org/grpc/internal/transport.(*http2Client).reader(0xc007515c20)
40702026-03-11T20:45:50.848Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1498 +0x41d
40712026-03-11T20:45:50.848Zcreated by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 11046
40722026-03-11T20:45:50.848Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:365 +0x184d
40732026-03-11T20:45:50.848Z
40742026-03-11T20:45:50.848Zgoroutine 10378 [select]:
40752026-03-11T20:45:50.848Zgithub.com/cockroachdb/cockroach/pkg/kv/kvprober.(*Prober).Start.func1.1({0x50b1300, 0xc009328d80})
40762026-03-11T20:45:50.848Z /work/oxidecomputer/cockroach/pkg/kv/kvprober/kvprober.go:293 +0x314
40772026-03-11T20:45:50.849Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
40782026-03-11T20:45:50.849Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
40792026-03-11T20:45:50.849Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
40802026-03-11T20:45:50.849Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
40812026-03-11T20:45:50.849Z
40822026-03-11T20:45:50.849Zgoroutine 11011 [IO wait]:
40832026-03-11T20:45:50.849Zinternal/poll.runtime_pollWait(0x7fc12c3bdef8, 0x72)
40842026-03-11T20:45:50.849Z /work/go/src/runtime/netpoll.go:345 +0x85
40852026-03-11T20:45:50.849Zinternal/poll.(*pollDesc).wait(0xc003422980?, 0xc002dfd200?, 0x0)
40862026-03-11T20:45:50.849Z /work/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
40872026-03-11T20:45:50.849Zinternal/poll.(*pollDesc).waitRead(...)
40882026-03-11T20:45:50.849Z /work/go/src/internal/poll/fd_poll_runtime.go:89
40892026-03-11T20:45:50.849Zinternal/poll.(*FD).Read(0xc003422980, {0xc002dfd200, 0x600, 0x600})
40902026-03-11T20:45:50.849Z /work/go/src/internal/poll/fd_unix.go:164 +0x27a
40912026-03-11T20:45:50.849Znet.(*netFD).Read(0xc003422980, {0xc002dfd200?, 0x7fc128b7ee00?, 0xc01e2f6d50?})
40922026-03-11T20:45:50.849Z /work/go/src/net/fd_posix.go:55 +0x25
40932026-03-11T20:45:50.849Znet.(*conn).Read(0xc004e7cde0, {0xc002dfd200?, 0xc0066dce08?, 0x415dbb?})
40942026-03-11T20:45:50.849Z /work/go/src/net/net.go:185 +0x45
40952026-03-11T20:45:50.849Zcrypto/tls.(*atLeastReader).Read(0xc01e2f6d50, {0xc002dfd200?, 0x0?, 0xc01e2f6d50?})
40962026-03-11T20:45:50.849Z /work/go/src/crypto/tls/conn.go:806 +0x3b
40972026-03-11T20:45:50.849Zbytes.(*Buffer).ReadFrom(0xc003afbeb0, {0x5081fe0, 0xc01e2f6d50})
40982026-03-11T20:45:50.849Z /work/go/src/bytes/buffer.go:211 +0x98
40992026-03-11T20:45:50.849Zcrypto/tls.(*Conn).readFromUntil(0xc003afbc08, {0x50821c0, 0xc004e7cde0}, 0xc0066dce50?)
41002026-03-11T20:45:50.849Z /work/go/src/crypto/tls/conn.go:828 +0xde
41012026-03-11T20:45:50.849Zcrypto/tls.(*Conn).readRecordOrCCS(0xc003afbc08, 0x0)
41022026-03-11T20:45:50.849Z /work/go/src/crypto/tls/conn.go:626 +0x3cf
41032026-03-11T20:45:50.858Zcrypto/tls.(*Conn).readRecord(...)
41042026-03-11T20:45:50.858Z /work/go/src/crypto/tls/conn.go:588
41052026-03-11T20:45:50.858Zcrypto/tls.(*Conn).Read(0xc003afbc08, {0xc003387000, 0x1000, 0x0?})
41062026-03-11T20:45:50.859Z /work/go/src/crypto/tls/conn.go:1370 +0x156
41072026-03-11T20:45:50.859Zgithub.com/cockroachdb/cockroach/pkg/sql/pgwire.(*readTimeoutConn).Read(0xc00583d830, {0xc003387000, 0x1000, 0x1000})
41082026-03-11T20:45:50.859Z /work/oxidecomputer/cockroach/pkg/sql/pgwire/conn.go:1840 +0xfe
41092026-03-11T20:45:50.859Zbufio.(*Reader).fill(0xc004b3ab10)
41102026-03-11T20:45:50.859Z /work/go/src/bufio/bufio.go:110 +0x103
41112026-03-11T20:45:50.859Zbufio.(*Reader).ReadByte(0xc004b3ab10)
41122026-03-11T20:45:50.859Z /work/go/src/bufio/bufio.go:269 +0x27
41132026-03-11T20:45:50.859Zgithub.com/cockroachdb/cockroach/pkg/sql/pgwire/pgwirebase.(*ReadBuffer).ReadTypedMsg(0xc004b3b2e0, {0x50aaee0, 0xc004b3ab10})
41142026-03-11T20:45:50.859Z /work/oxidecomputer/cockroach/pkg/sql/pgwire/pgwirebase/encoding.go:206 +0x2a
41152026-03-11T20:45:50.859Zgithub.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).serveImpl.func5(0xc004b3aa88, 0xc006486d60, 0xc0066dd48b, 0xc0066dd48d, {0x50aaeb0, 0xc0012095f0}, 0xc00394e308, 0xc00017ffb0, 0xc0066dd489)
41162026-03-11T20:45:50.859Z /work/oxidecomputer/cockroach/pkg/sql/pgwire/conn.go:400 +0x97
41172026-03-11T20:45:50.859Zgithub.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).serveImpl(0xc004b3aa88, {0x50b1300, 0xc003ed9a40}, 0xc006486d50, 0xc00394e308, {0x5400, 0x15000, 0xc00226d680, 0x0, 0x0}, ...)
41182026-03-11T20:45:50.859Z /work/oxidecomputer/cockroach/pkg/sql/pgwire/conn.go:540 +0xb2a
41192026-03-11T20:45:50.859Zgithub.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).serveConn(0xc001eae000, {0x50b1300, _}, {_, _}, {{{0xc00314301e, 0x4}}, 0x1, 0xc003ed99e0, 0xc003ed9a10, ...}, ...)
41202026-03-11T20:45:50.859Z /work/oxidecomputer/cockroach/pkg/sql/pgwire/conn.go:173 +0x2b6
41212026-03-11T20:45:50.859Zgithub.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).ServeConn(0xc001eae000, {0x50b1300, 0xc003ed9a40}, {0x50eb4b8, 0xc004e7cde0}, 0x1)
41222026-03-11T20:45:50.859Z /work/oxidecomputer/cockroach/pkg/sql/pgwire/server.go:874 +0xd49
41232026-03-11T20:45:50.859Zgithub.com/cockroachdb/cockroach/pkg/server.(*SQLServer).startServeSQL.func1.1({0x50b1300?, 0xc003cb7800?}, {0x50eb4b8, 0xc004e7cde0})
41242026-03-11T20:45:50.859Z /work/oxidecomputer/cockroach/pkg/server/server_sql.go:1378 +0xbd
41252026-03-11T20:45:50.859Zgithub.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith.func1({0x50b1300, 0xc003cb7800})
41262026-03-11T20:45:50.859Z /work/oxidecomputer/cockroach/pkg/util/netutil/net.go:162 +0x11a
41272026-03-11T20:45:50.859Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
41282026-03-11T20:45:50.859Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
41292026-03-11T20:45:50.859Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10584
41302026-03-11T20:45:50.859Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
41312026-03-11T20:45:50.859Z
41322026-03-11T20:45:50.859Zgoroutine 11155 [chan receive, 59 minutes]:
41332026-03-11T20:45:50.859Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).rangeFeedWithRangeID(0xc001d7b408, 0x3d87860?, 0xc002762b40, {0x7fc12ce5a548, 0xc000901b60})
41342026-03-11T20:45:50.859Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:249 +0x76b
41352026-03-11T20:45:50.859Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed(...)
41362026-03-11T20:45:50.859Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:149
41372026-03-11T20:45:50.859Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).RangeFeed(0xc000f39208, 0xc002762b40, {0x50e5040, 0xc000901b60})
41382026-03-11T20:45:50.859Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:3082 +0x16f
41392026-03-11T20:45:50.859Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).RangeFeed(0xc001eb4380, 0xc002762b40, {0x50e5040, 0xc000901b60})
41402026-03-11T20:45:50.859Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/stores.go:216 +0x19a
41412026-03-11T20:45:50.859Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed(0x732ffb0?, 0x40055e0?, {0x50e5040, 0xc000901b60})
41422026-03-11T20:45:50.860Z /work/oxidecomputer/cockroach/pkg/server/node.go:1232 +0x28
41432026-03-11T20:45:50.860Zgithub.com/cockroachdb/cockroach/pkg/roachpb._Internal_RangeFeed_Handler({0x40055e0, 0xc004911b08}, {0x50df230, 0xc004fa3d20})
41442026-03-11T20:45:50.860Z /work/oxidecomputer/cockroach/pkg/roachpb/api.pb.go:9523 +0x107
41452026-03-11T20:45:50.860Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.StreamServerInterceptor.func14({0x40055e0, 0xc004911b08}, {0x50df9e0, 0xc00300f930}, 0xc00349a558, 0x42e42f8)
41462026-03-11T20:45:50.860Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:205 +0x35b
41472026-03-11T20:45:50.860Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x40055e0?, 0xc004911b08?}, {0x50df9e0?, 0xc00300f930?})
41482026-03-11T20:45:50.860Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1408 +0x59
41492026-03-11T20:45:50.860Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func4({0x40055e0, 0xc004911b08}, {0x50df9e0, 0xc00300f930}, 0xc003bd9918?, 0xc007534f00)
41502026-03-11T20:45:50.860Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:247 +0x6c
41512026-03-11T20:45:50.860Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x40055e0?, 0xc004911b08?}, {0x50df9e0?, 0xc00300f930?})
41522026-03-11T20:45:50.860Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
41532026-03-11T20:45:50.860Zgithub.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({{{0x2b?}}}, {0x40055e0, 0xc004911b08}, {0x50df9e0, 0xc00300f930}, 0xc00349a558, 0xc007534f00)
41542026-03-11T20:45:50.860Z /work/oxidecomputer/cockroach/pkg/rpc/auth.go:98 +0x19b
41552026-03-11T20:45:50.860Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x40055e0?, 0xc004911b08?}, {0x50df9e0?, 0xc00300f930?})
41562026-03-11T20:45:50.860Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
41572026-03-11T20:45:50.861Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func2.1({0x66?, 0x40?})
41582026-03-11T20:45:50.861Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:216 +0x27
41592026-03-11T20:45:50.861Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc003f40400, {0x50b1300, 0xc0063c7620}, {0xc003bd9b28?, 0x415dbb?}, 0xc003bd9ac0)
41602026-03-11T20:45:50.861Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:344 +0xcb
41612026-03-11T20:45:50.861Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func2({0x40055e0, 0xc004911b08}, {0x50df9e0, 0xc00300f930}, 0xc00349a558, 0xc007534f00)
41622026-03-11T20:45:50.861Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:215 +0xdf
41632026-03-11T20:45:50.861Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x40055e0?, 0xc004911b08?}, {0x50df9e0?, 0xc00300f930?})
41642026-03-11T20:45:50.861Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
41652026-03-11T20:45:50.861Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x40055e0, 0xc004911b08}, {0x50df9e0, 0xc00300f930}, 0xc00349a558, 0x42e42f8)
41662026-03-11T20:45:50.861Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1413 +0x108
41672026-03-11T20:45:50.861Zgoogle.golang.org/grpc.(*Server).processStreamingRPC(0xc004feefc0, {0x50f1d80, 0xc001aef6c0}, 0xc002762a20, 0xc007661ad0, 0x73abba0, 0x0)
41682026-03-11T20:45:50.861Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1549 +0xe51
41692026-03-11T20:45:50.861Zgoogle.golang.org/grpc.(*Server).handleStream(0xc004feefc0, {0x50f1d80, 0xc001aef6c0}, 0xc002762a20, 0x0)
41702026-03-11T20:45:50.861Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1624 +0x999
41712026-03-11T20:45:50.861Zgoogle.golang.org/grpc.(*Server).serveStreams.func1.2()
41722026-03-11T20:45:50.861Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:922 +0x8d
41732026-03-11T20:45:50.861Zcreated by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 10956
41742026-03-11T20:45:50.861Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:920 +0x226
41752026-03-11T20:45:50.861Z
41762026-03-11T20:45:50.861Zgoroutine 10853 [select]:
41772026-03-11T20:45:50.861Zgoogle.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc006b56190, 0x1)
41782026-03-11T20:45:50.861Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:407 +0x113
41792026-03-11T20:45:50.861Zgoogle.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc005ff9b60)
41802026-03-11T20:45:50.862Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:534 +0x7b
41812026-03-11T20:45:50.862Zgoogle.golang.org/grpc/internal/transport.newHTTP2Client.func3()
41822026-03-11T20:45:50.862Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:415 +0x67
41832026-03-11T20:45:50.862Zcreated by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 9758
41842026-03-11T20:45:50.862Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:413 +0x1dcb
41852026-03-11T20:45:50.862Z
41862026-03-11T20:45:50.862Zgoroutine 10257 [chan receive, 59 minutes]:
41872026-03-11T20:45:50.862Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).rangeFeedWithRangeID(0xc001ded408, 0x3f3de40?, 0xc000f47e60, {0x7fc12cefd140, 0xc006498ec0})
41882026-03-11T20:45:50.862Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:249 +0x76b
41892026-03-11T20:45:50.862Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed(...)
41902026-03-11T20:45:50.862Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:149
41912026-03-11T20:45:50.863Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).RangeFeed(0xc000f39208, 0xc000f47e60, {0x50e5b30, 0xc006498ec0})
41922026-03-11T20:45:50.863Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:3082 +0x16f
41932026-03-11T20:45:50.863Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).RangeFeed(0xc001eb4380, 0xc000f47e60, {0x50e5b30, 0xc006498ec0})
41942026-03-11T20:45:50.863Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/stores.go:216 +0x19a
41952026-03-11T20:45:50.863Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed(0x3f3de40?, 0xc002228800?, {0x50e5b30, 0xc006498ec0})
41962026-03-11T20:45:50.863Z /work/oxidecomputer/cockroach/pkg/server/node.go:1232 +0x28
41972026-03-11T20:45:50.863Zgithub.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed.func1()
41982026-03-11T20:45:50.863Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:753 +0xa7
41992026-03-11T20:45:50.863Zcreated by github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed in goroutine 10256
42002026-03-11T20:45:50.863Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:750 +0x17c
42012026-03-11T20:45:50.863Z
42022026-03-11T20:45:50.863Zgoroutine 10072 [sync.Cond.Wait]:
42032026-03-11T20:45:50.869Zsync.runtime_notifyListWait(0xc001e22250, 0x377a)
42042026-03-11T20:45:50.869Z /work/go/src/runtime/sema.go:569 +0x159
42052026-03-11T20:45:50.869Zsync.(*Cond).Wait(0x3aac600?)
42062026-03-11T20:45:50.869Z /work/go/src/sync/cond.go:70 +0x85
42072026-03-11T20:45:50.869Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583860})
42082026-03-11T20:45:50.869Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
42092026-03-11T20:45:50.869Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
42102026-03-11T20:45:50.869Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
42112026-03-11T20:45:50.869Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
42122026-03-11T20:45:50.869Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
42132026-03-11T20:45:50.869Z
42142026-03-11T20:45:50.869Zgoroutine 11310 [chan receive, 59 minutes]:
42152026-03-11T20:45:50.869Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).rangeFeedWithRangeID(0xc001b49008, 0x3d87860?, 0xc0080a0240, {0x7fc12ce5a548, 0xc0060ba2f0})
42162026-03-11T20:45:50.869Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:249 +0x76b
42172026-03-11T20:45:50.870Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed(...)
42182026-03-11T20:45:50.870Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:149
42192026-03-11T20:45:50.870Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).RangeFeed(0xc000f39208, 0xc0080a0240, {0x50e5040, 0xc0060ba2f0})
42202026-03-11T20:45:50.870Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:3082 +0x16f
42212026-03-11T20:45:50.870Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).RangeFeed(0xc001eb4380, 0xc0080a0240, {0x50e5040, 0xc0060ba2f0})
42222026-03-11T20:45:50.870Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/stores.go:216 +0x19a
42232026-03-11T20:45:50.870Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed(0x732ffb0?, 0x40055e0?, {0x50e5040, 0xc0060ba2f0})
42242026-03-11T20:45:50.870Z /work/oxidecomputer/cockroach/pkg/server/node.go:1232 +0x28
42252026-03-11T20:45:50.870Zgithub.com/cockroachdb/cockroach/pkg/roachpb._Internal_RangeFeed_Handler({0x40055e0, 0xc004911b08}, {0x50df230, 0xc0062e5b80})
42262026-03-11T20:45:50.870Z /work/oxidecomputer/cockroach/pkg/roachpb/api.pb.go:9523 +0x107
42272026-03-11T20:45:50.870Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.StreamServerInterceptor.func14({0x40055e0, 0xc004911b08}, {0x50df9e0, 0xc007be1ba0}, 0xc005bc1cb0, 0x42e42f8)
42282026-03-11T20:45:50.870Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:205 +0x35b
42292026-03-11T20:45:50.870Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x40055e0?, 0xc004911b08?}, {0x50df9e0?, 0xc007be1ba0?})
42302026-03-11T20:45:50.870Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1408 +0x59
42312026-03-11T20:45:50.870Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func4({0x40055e0, 0xc004911b08}, {0x50df9e0, 0xc007be1ba0}, 0xc004099918?, 0xc008065580)
42322026-03-11T20:45:50.870Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:247 +0x6c
42332026-03-11T20:45:50.870Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x40055e0?, 0xc004911b08?}, {0x50df9e0?, 0xc007be1ba0?})
42342026-03-11T20:45:50.870Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
42352026-03-11T20:45:50.870Zgithub.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({{{0x7fc12eba6908?}}}, {0x40055e0, 0xc004911b08}, {0x50df9e0, 0xc007be1ba0}, 0xc005bc1cb0, 0xc008065580)
42362026-03-11T20:45:50.870Z /work/oxidecomputer/cockroach/pkg/rpc/auth.go:98 +0x19b
42372026-03-11T20:45:50.870Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x40055e0?, 0xc004911b08?}, {0x50df9e0?, 0xc007be1ba0?})
42382026-03-11T20:45:50.872Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
42392026-03-11T20:45:50.872Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func2.1({0x66?, 0x40?})
42402026-03-11T20:45:50.872Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:216 +0x27
42412026-03-11T20:45:50.872Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc003f40400, {0x50b1300, 0xc007e53470}, {0xc004099b28?, 0x415dbb?}, 0xc004099ac0)
42422026-03-11T20:45:50.872Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:344 +0xcb
42432026-03-11T20:45:50.872Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func2({0x40055e0, 0xc004911b08}, {0x50df9e0, 0xc007be1ba0}, 0xc005bc1cb0, 0xc008065580)
42442026-03-11T20:45:50.872Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:215 +0xdf
42452026-03-11T20:45:50.872Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x40055e0?, 0xc004911b08?}, {0x50df9e0?, 0xc007be1ba0?})
42462026-03-11T20:45:50.872Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
42472026-03-11T20:45:50.872Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x40055e0, 0xc004911b08}, {0x50df9e0, 0xc007be1ba0}, 0xc005bc1cb0, 0x42e42f8)
42482026-03-11T20:45:50.872Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1413 +0x108
42492026-03-11T20:45:50.872Zgoogle.golang.org/grpc.(*Server).processStreamingRPC(0xc004feefc0, {0x50f1d80, 0xc001aef6c0}, 0xc0080a0120, 0xc007661ad0, 0x73abba0, 0x0)
42502026-03-11T20:45:50.872Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1549 +0xe51
42512026-03-11T20:45:50.872Zgoogle.golang.org/grpc.(*Server).handleStream(0xc004feefc0, {0x50f1d80, 0xc001aef6c0}, 0xc0080a0120, 0x0)
42522026-03-11T20:45:50.872Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1624 +0x999
42532026-03-11T20:45:50.872Zgoogle.golang.org/grpc.(*Server).serveStreams.func1.2()
42542026-03-11T20:45:50.872Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:922 +0x8d
42552026-03-11T20:45:50.872Zcreated by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 10956
42562026-03-11T20:45:50.872Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:920 +0x226
42572026-03-11T20:45:50.872Z
42582026-03-11T20:45:50.872Zgoroutine 9848 [select, 59 minutes]:
42592026-03-11T20:45:50.872Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initCancelingWorkers.func1({0x50b1300, 0xc00485e5a0})
42602026-03-11T20:45:50.872Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:139 +0xc5
42612026-03-11T20:45:50.872Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
42622026-03-11T20:45:50.872Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
42632026-03-11T20:45:50.872Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
42642026-03-11T20:45:50.872Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
42652026-03-11T20:45:50.872Z
42662026-03-11T20:45:50.872Zgoroutine 9809 [chan receive, 59 minutes]:
42672026-03-11T20:45:50.872Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc0036f82d0?})
42682026-03-11T20:45:50.872Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
42692026-03-11T20:45:50.872Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x76cce00?, 0xc0023bec00?}}, 0xc00405bfa8)
42702026-03-11T20:45:50.872Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
42712026-03-11T20:45:50.872Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc00405bfb8?)
42722026-03-11T20:45:50.872Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
42732026-03-11T20:45:50.872Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
42742026-03-11T20:45:50.872Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
42752026-03-11T20:45:50.872Z
42762026-03-11T20:45:50.873Zgoroutine 10062 [sync.Cond.Wait]:
42772026-03-11T20:45:50.873Zsync.runtime_notifyListWait(0xc001e22250, 0x3774)
42782026-03-11T20:45:50.873Z /work/go/src/runtime/sema.go:569 +0x159
42792026-03-11T20:45:50.873Zsync.(*Cond).Wait(0x3aac600?)
42802026-03-11T20:45:50.873Z /work/go/src/sync/cond.go:70 +0x85
42812026-03-11T20:45:50.873Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583680})
42822026-03-11T20:45:50.873Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
42832026-03-11T20:45:50.873Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
42842026-03-11T20:45:50.873Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
42852026-03-11T20:45:50.873Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
42862026-03-11T20:45:50.873Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
42872026-03-11T20:45:50.873Z
42882026-03-11T20:45:50.873Zgoroutine 10954 [select]:
42892026-03-11T20:45:50.873Zgoogle.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc004be1d60, 0x1)
42902026-03-11T20:45:50.873Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:407 +0x113
42912026-03-11T20:45:50.873Zgoogle.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0055343c0)
42922026-03-11T20:45:50.873Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:534 +0x7b
42932026-03-11T20:45:50.873Zgoogle.golang.org/grpc/internal/transport.NewServerTransport.func2()
42942026-03-11T20:45:50.873Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:332 +0xc6
42952026-03-11T20:45:50.873Zcreated by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 10868
42962026-03-11T20:45:50.873Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:329 +0x184c
42972026-03-11T20:45:50.873Z
42982026-03-11T20:45:50.873Zgoroutine 10005 [sync.Cond.Wait]:
42992026-03-11T20:45:50.873Zsync.runtime_notifyListWait(0xc001e22250, 0x3788)
43002026-03-11T20:45:50.873Z /work/go/src/runtime/sema.go:569 +0x159
43012026-03-11T20:45:50.873Zsync.(*Cond).Wait(0x3aac600?)
43022026-03-11T20:45:50.873Z /work/go/src/sync/cond.go:70 +0x85
43032026-03-11T20:45:50.877Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007582ed0})
43042026-03-11T20:45:50.877Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
43052026-03-11T20:45:50.877Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
43062026-03-11T20:45:50.877Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
43072026-03-11T20:45:50.877Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
43082026-03-11T20:45:50.877Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
43092026-03-11T20:45:50.877Z
43102026-03-11T20:45:50.878Zgoroutine 10663 [select]:
43112026-03-11T20:45:50.878Zgoogle.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00677eb90, 0x1)
43122026-03-11T20:45:50.878Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:407 +0x113
43132026-03-11T20:45:50.878Zgoogle.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0064588a0)
43142026-03-11T20:45:50.878Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:534 +0x7b
43152026-03-11T20:45:50.878Zgoogle.golang.org/grpc/internal/transport.NewServerTransport.func2()
43162026-03-11T20:45:50.878Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:332 +0xc6
43172026-03-11T20:45:50.878Zcreated by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 10600
43182026-03-11T20:45:50.878Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:329 +0x184c
43192026-03-11T20:45:50.878Z
43202026-03-11T20:45:50.878Zgoroutine 10557 [semacquire, 59 minutes]:
43212026-03-11T20:45:50.878Zsync.runtime_Semacquire(0xc008c3eeb8?)
43222026-03-11T20:45:50.879Z /work/go/src/runtime/sema.go:62 +0x25
43232026-03-11T20:45:50.879Zsync.(*WaitGroup).Wait(0x44e7a0?)
43242026-03-11T20:45:50.879Z /work/go/src/sync/waitgroup.go:116 +0x48
43252026-03-11T20:45:50.879Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc0044a7940)
43262026-03-11T20:45:50.879Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
43272026-03-11T20:45:50.879Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc0044a7940?, {0x50b1338?, 0xc003234370?}})
43282026-03-11T20:45:50.879Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
43292026-03-11T20:45:50.879Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait({0x50b1300?, 0xc0066f4bd0?}, {0xc00636dd78, 0x2, 0xc00691fc70?})
43302026-03-11T20:45:50.879Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:193 +0xff
43312026-03-11T20:45:50.879Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0xc008050580, {0x50b1300, 0xc0066f4bd0}, 0xc0066f4a20)
43322026-03-11T20:45:50.879Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:291 +0x59c
43332026-03-11T20:45:50.879Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).Start.func2({0x50b1300, 0xc0066f4b40})
43342026-03-11T20:45:50.879Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:215 +0x2f3
43352026-03-11T20:45:50.879Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
43362026-03-11T20:45:50.880Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
43372026-03-11T20:45:50.880Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10630
43382026-03-11T20:45:50.880Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
43392026-03-11T20:45:50.880Z
43402026-03-11T20:45:50.880Zgoroutine 10697 [select]:
43412026-03-11T20:45:50.880Zgithub.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).manage.func1({0x50b1300, 0xc006e63860})
43422026-03-11T20:45:50.880Z /work/oxidecomputer/cockroach/pkg/gossip/gossip.go:1420 +0x2d2
43432026-03-11T20:45:50.880Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
43442026-03-11T20:45:50.880Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
43452026-03-11T20:45:50.880Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
43462026-03-11T20:45:50.880Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
43472026-03-11T20:45:50.880Z
43482026-03-11T20:45:50.880Zgoroutine 10703 [sync.Cond.Wait, 59 minutes]:
43492026-03-11T20:45:50.880Zsync.runtime_notifyListWait(0xc0064c5050, 0x1c)
43502026-03-11T20:45:50.880Z /work/go/src/runtime/sema.go:569 +0x159
43512026-03-11T20:45:50.880Zsync.(*Cond).Wait(0xb99e5b?)
43522026-03-11T20:45:50.880Z /work/go/src/sync/cond.go:70 +0x85
43532026-03-11T20:45:50.880Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd2b70})
43542026-03-11T20:45:50.880Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
43552026-03-11T20:45:50.880Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
43562026-03-11T20:45:50.880Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
43572026-03-11T20:45:50.880Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
43582026-03-11T20:45:50.880Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
43592026-03-11T20:45:50.880Z
43602026-03-11T20:45:50.880Zgoroutine 10702 [sync.Cond.Wait, 59 minutes]:
43612026-03-11T20:45:50.880Zsync.runtime_notifyListWait(0xc0064c5050, 0x1b)
43622026-03-11T20:45:50.880Z /work/go/src/runtime/sema.go:569 +0x159
43632026-03-11T20:45:50.880Zsync.(*Cond).Wait(0xc000f39208?)
43642026-03-11T20:45:50.880Z /work/go/src/sync/cond.go:70 +0x85
43652026-03-11T20:45:50.880Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd2b40})
43662026-03-11T20:45:50.880Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
43672026-03-11T20:45:50.880Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
43682026-03-11T20:45:50.880Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
43692026-03-11T20:45:50.880Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
43702026-03-11T20:45:50.880Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
43712026-03-11T20:45:50.880Z
43722026-03-11T20:45:50.880Zgoroutine 11082 [select]:
43732026-03-11T20:45:50.880Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).outputLoop(0xc004b15960, {0x50b1338, 0xc001379220})
43742026-03-11T20:45:50.881Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:310 +0x247
43752026-03-11T20:45:50.881Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).runOutputLoop(0xc004b15960, {0x50b1300, 0xc003c933e0}, 0xc000580008?)
43762026-03-11T20:45:50.881Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:335 +0xba
43772026-03-11T20:45:50.881Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run.func1({0x50b1300?, 0xc003c933e0?})
43782026-03-11T20:45:50.881Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:297 +0x46
43792026-03-11T20:45:50.881Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
43802026-03-11T20:45:50.881Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
43812026-03-11T20:45:50.881Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 11080
43822026-03-11T20:45:50.881Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
43832026-03-11T20:45:50.881Z
43842026-03-11T20:45:50.881Zgoroutine 10782 [sync.Cond.Wait, 59 minutes]:
43852026-03-11T20:45:50.881Zsync.runtime_notifyListWait(0xc0064c5050, 0x42)
43862026-03-11T20:45:50.881Z /work/go/src/runtime/sema.go:569 +0x159
43872026-03-11T20:45:50.881Zsync.(*Cond).Wait(0x50f7930?)
43882026-03-11T20:45:50.881Z /work/go/src/sync/cond.go:70 +0x85
43892026-03-11T20:45:50.881Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3740})
43902026-03-11T20:45:50.881Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
43912026-03-11T20:45:50.881Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
43922026-03-11T20:45:50.882Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
43932026-03-11T20:45:50.882Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
43942026-03-11T20:45:50.882Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
43952026-03-11T20:45:50.882Z
43962026-03-11T20:45:50.882Zgoroutine 11170 [select, 59 minutes]:
43972026-03-11T20:45:50.882Zgoogle.golang.org/grpc.newClientStreamWithParams.func4()
43982026-03-11T20:45:50.882Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:341 +0x8c
43992026-03-11T20:45:50.882Zcreated by google.golang.org/grpc.newClientStreamWithParams in goroutine 11137
44002026-03-11T20:45:50.882Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:340 +0xb2a
44012026-03-11T20:45:50.882Z
44022026-03-11T20:45:50.882Zgoroutine 9811 [chan receive, 59 minutes]:
44032026-03-11T20:45:50.887Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc0036f8480?})
44042026-03-11T20:45:50.887Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
44052026-03-11T20:45:50.887Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x50b1258?, 0xc003f084e0?}}, 0xc0040597a8)
44062026-03-11T20:45:50.887Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
44072026-03-11T20:45:50.887Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc0040597b8?)
44082026-03-11T20:45:50.887Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
44092026-03-11T20:45:50.887Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
44102026-03-11T20:45:50.887Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
44112026-03-11T20:45:50.887Z
44122026-03-11T20:45:50.887Zgoroutine 10745 [sync.Cond.Wait, 59 minutes]:
44132026-03-11T20:45:50.887Zsync.runtime_notifyListWait(0xc0064c5050, 0xd)
44142026-03-11T20:45:50.887Z /work/go/src/runtime/sema.go:569 +0x159
44152026-03-11T20:45:50.887Zsync.(*Cond).Wait(0xc006e3e878?)
44162026-03-11T20:45:50.887Z /work/go/src/sync/cond.go:70 +0x85
44172026-03-11T20:45:50.887Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3050})
44182026-03-11T20:45:50.887Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
44192026-03-11T20:45:50.887Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
44202026-03-11T20:45:50.887Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
44212026-03-11T20:45:50.887Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
44222026-03-11T20:45:50.887Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
44232026-03-11T20:45:50.888Z
44242026-03-11T20:45:50.888Zgoroutine 9945 [select, 59 minutes]:
44252026-03-11T20:45:50.888Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc004cb0308?, 0x0?})
44262026-03-11T20:45:50.888Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
44272026-03-11T20:45:50.888Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
44282026-03-11T20:45:50.888Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
44292026-03-11T20:45:50.888Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
44302026-03-11T20:45:50.888Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
44312026-03-11T20:45:50.888Z
44322026-03-11T20:45:50.888Zgoroutine 10754 [sync.Cond.Wait, 59 minutes]:
44332026-03-11T20:45:50.888Zsync.runtime_notifyListWait(0xc0064c5050, 0x15)
44342026-03-11T20:45:50.888Z /work/go/src/runtime/sema.go:569 +0x159
44352026-03-11T20:45:50.888Zsync.(*Cond).Wait(0x1eda2b2?)
44362026-03-11T20:45:50.888Z /work/go/src/sync/cond.go:70 +0x85
44372026-03-11T20:45:50.888Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3200})
44382026-03-11T20:45:50.888Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
44392026-03-11T20:45:50.888Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
44402026-03-11T20:45:50.888Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
44412026-03-11T20:45:50.888Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
44422026-03-11T20:45:50.888Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
44432026-03-11T20:45:50.888Z
44442026-03-11T20:45:50.888Zgoroutine 10655 [select]:
44452026-03-11T20:45:50.888Zgoogle.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc006581ae0, 0x1)
44462026-03-11T20:45:50.888Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:407 +0x113
44472026-03-11T20:45:50.888Zgoogle.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc006233b00)
44482026-03-11T20:45:50.888Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:534 +0x7b
44492026-03-11T20:45:50.888Zgoogle.golang.org/grpc/internal/transport.newHTTP2Client.func3()
44502026-03-11T20:45:50.888Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:415 +0x67
44512026-03-11T20:45:50.888Zcreated by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 10685
44522026-03-11T20:45:50.888Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:413 +0x1dcb
44532026-03-11T20:45:50.888Z
44542026-03-11T20:45:50.888Zgoroutine 9815 [chan receive, 59 minutes]:
44552026-03-11T20:45:50.888Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc005572510?})
44562026-03-11T20:45:50.888Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
44572026-03-11T20:45:50.888Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x0?, 0x0?}}, 0xc00061bfa8)
44582026-03-11T20:45:50.888Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
44592026-03-11T20:45:50.888Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
44602026-03-11T20:45:50.889Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
44612026-03-11T20:45:50.889Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
44622026-03-11T20:45:50.889Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
44632026-03-11T20:45:50.889Z
44642026-03-11T20:45:50.889Zgoroutine 9942 [select, 59 minutes]:
44652026-03-11T20:45:50.889Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc00167f808?, 0x0?})
44662026-03-11T20:45:50.889Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
44672026-03-11T20:45:50.889Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
44682026-03-11T20:45:50.889Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
44692026-03-11T20:45:50.889Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
44702026-03-11T20:45:50.889Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
44712026-03-11T20:45:50.889Z
44722026-03-11T20:45:50.889Zgoroutine 10129 [select, 9 minutes]:
44732026-03-11T20:45:50.889Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.KVAdmissionControllerImpl.SetTenantWeightProvider.func1()
44742026-03-11T20:45:50.889Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:3809 +0xd5
44752026-03-11T20:45:50.889Zcreated by github.com/cockroachdb/cockroach/pkg/kv/kvserver.KVAdmissionControllerImpl.SetTenantWeightProvider in goroutine 9708
44762026-03-11T20:45:50.889Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:3802 +0xcc
44772026-03-11T20:45:50.889Z
44782026-03-11T20:45:50.889Zgoroutine 9976 [select]:
44792026-03-11T20:45:50.889Zgoogle.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc001fb3900, 0x1)
44802026-03-11T20:45:50.889Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:407 +0x113
44812026-03-11T20:45:50.889Zgoogle.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc001be4a20)
44822026-03-11T20:45:50.889Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:534 +0x7b
44832026-03-11T20:45:50.889Zgoogle.golang.org/grpc/internal/transport.newHTTP2Client.func3()
44842026-03-11T20:45:50.889Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:415 +0x67
44852026-03-11T20:45:50.889Zcreated by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 9994
44862026-03-11T20:45:50.889Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:413 +0x1dcb
44872026-03-11T20:45:50.889Z
44882026-03-11T20:45:50.889Zgoroutine 10712 [select]:
44892026-03-11T20:45:50.889Zgoogle.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0021e54a0, 0x1)
44902026-03-11T20:45:50.889Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:407 +0x113
44912026-03-11T20:45:50.889Zgoogle.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0033c9f20)
44922026-03-11T20:45:50.889Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:534 +0x7b
44932026-03-11T20:45:50.889Zgoogle.golang.org/grpc/internal/transport.NewServerTransport.func2()
44942026-03-11T20:45:50.889Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:332 +0xc6
44952026-03-11T20:45:50.889Zcreated by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 10518
44962026-03-11T20:45:50.889Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:329 +0x184c
44972026-03-11T20:45:50.889Z
44982026-03-11T20:45:50.889Zgoroutine 10119 [select]:
44992026-03-11T20:45:50.890Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).coalescedHeartbeatsLoop(0xc000f39208, {0x50b1300, 0xc003ebc2a0})
45002026-03-11T20:45:50.890Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store_raft.go:678 +0xec
45012026-03-11T20:45:50.890Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
45022026-03-11T20:45:50.890Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
45032026-03-11T20:45:50.895Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
45042026-03-11T20:45:50.895Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
45052026-03-11T20:45:50.895Z
45062026-03-11T20:45:50.895Zgoroutine 10747 [sync.Cond.Wait, 59 minutes]:
45072026-03-11T20:45:50.895Zsync.runtime_notifyListWait(0xc0064c5050, 0x12)
45082026-03-11T20:45:50.895Z /work/go/src/runtime/sema.go:569 +0x159
45092026-03-11T20:45:50.895Zsync.(*Cond).Wait(0xc0064aca08?)
45102026-03-11T20:45:50.895Z /work/go/src/sync/cond.go:70 +0x85
45112026-03-11T20:45:50.895Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd30b0})
45122026-03-11T20:45:50.895Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
45132026-03-11T20:45:50.895Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
45142026-03-11T20:45:50.895Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
45152026-03-11T20:45:50.895Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
45162026-03-11T20:45:50.895Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
45172026-03-11T20:45:50.895Z
45182026-03-11T20:45:50.895Zgoroutine 9856 [chan receive, 59 minutes]:
45192026-03-11T20:45:50.895Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc007582bd0?})
45202026-03-11T20:45:50.895Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
45212026-03-11T20:45:50.895Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x1000000010000?, 0xc000d0a230?}}, 0xc000e077a8)
45222026-03-11T20:45:50.895Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
45232026-03-11T20:45:50.895Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
45242026-03-11T20:45:50.895Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
45252026-03-11T20:45:50.895Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
45262026-03-11T20:45:50.895Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
45272026-03-11T20:45:50.895Z
45282026-03-11T20:45:50.895Zgoroutine 10918 [select]:
45292026-03-11T20:45:50.895Zgithub.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func8({0x50b1300?, 0xc00067dbf0?})
45302026-03-11T20:45:50.896Z /work/oxidecomputer/cockroach/pkg/jobs/registry.go:811 +0x192
45312026-03-11T20:45:50.896Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
45322026-03-11T20:45:50.896Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
45332026-03-11T20:45:50.896Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
45342026-03-11T20:45:50.896Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
45352026-03-11T20:45:50.896Z
45362026-03-11T20:45:50.896Zgoroutine 10643 [select]:
45372026-03-11T20:45:50.896Zgithub.com/cockroachdb/cockroach/pkg/kv/kvprober.(*Prober).Start.func1.1({0x50b1300, 0xc0066f5770})
45382026-03-11T20:45:50.896Z /work/oxidecomputer/cockroach/pkg/kv/kvprober/kvprober.go:293 +0x314
45392026-03-11T20:45:50.896Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
45402026-03-11T20:45:50.896Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
45412026-03-11T20:45:50.896Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
45422026-03-11T20:45:50.896Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
45432026-03-11T20:45:50.896Z
45442026-03-11T20:45:50.897Zgoroutine 9949 [select, 59 minutes]:
45452026-03-11T20:45:50.897Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc004cb1208?, 0x0?})
45462026-03-11T20:45:50.897Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
45472026-03-11T20:45:50.897Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
45482026-03-11T20:45:50.897Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
45492026-03-11T20:45:50.897Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
45502026-03-11T20:45:50.897Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
45512026-03-11T20:45:50.897Z
45522026-03-11T20:45:50.897Zgoroutine 10642 [select]:
45532026-03-11T20:45:50.897Zgithub.com/cockroachdb/cockroach/pkg/kv/kvprober.(*Prober).Start.func1.1({0x50b1300, 0xc0066f5740})
45542026-03-11T20:45:50.897Z /work/oxidecomputer/cockroach/pkg/kv/kvprober/kvprober.go:293 +0x314
45552026-03-11T20:45:50.897Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
45562026-03-11T20:45:50.897Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
45572026-03-11T20:45:50.897Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
45582026-03-11T20:45:50.897Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
45592026-03-11T20:45:50.897Z
45602026-03-11T20:45:50.897Zgoroutine 10429 [select]:
45612026-03-11T20:45:50.897Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents(0xc00356a580, {0x50b1338, 0xc007611c70}, 0xc003457800, 0xc001c9c180)
45622026-03-11T20:45:50.897Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:331 +0xc5
45632026-03-11T20:45:50.898Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func2({0x50b1338?, 0xc007611c70?})
45642026-03-11T20:45:50.898Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:288 +0x32
45652026-03-11T20:45:50.898Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
45662026-03-11T20:45:50.898Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
45672026-03-11T20:45:50.898Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
45682026-03-11T20:45:50.898Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
45692026-03-11T20:45:50.898Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10331
45702026-03-11T20:45:50.898Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
45712026-03-11T20:45:50.898Z
45722026-03-11T20:45:50.898Zgoroutine 10076 [sync.Cond.Wait]:
45732026-03-11T20:45:50.898Zsync.runtime_notifyListWait(0xc001e22250, 0x3785)
45742026-03-11T20:45:50.898Z /work/go/src/runtime/sema.go:569 +0x159
45752026-03-11T20:45:50.898Zsync.(*Cond).Wait(0x3aac600?)
45762026-03-11T20:45:50.898Z /work/go/src/sync/cond.go:70 +0x85
45772026-03-11T20:45:50.898Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583920})
45782026-03-11T20:45:50.898Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
45792026-03-11T20:45:50.898Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
45802026-03-11T20:45:50.898Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
45812026-03-11T20:45:50.898Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
45822026-03-11T20:45:50.898Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
45832026-03-11T20:45:50.898Z
45842026-03-11T20:45:50.898Zgoroutine 10811 [sync.Cond.Wait, 59 minutes]:
45852026-03-11T20:45:50.898Zsync.runtime_notifyListWait(0xc0064c5050, 0x59)
45862026-03-11T20:45:50.898Z /work/go/src/runtime/sema.go:569 +0x159
45872026-03-11T20:45:50.898Zsync.(*Cond).Wait(0x0?)
45882026-03-11T20:45:50.898Z /work/go/src/sync/cond.go:70 +0x85
45892026-03-11T20:45:50.898Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3cb0})
45902026-03-11T20:45:50.898Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
45912026-03-11T20:45:50.898Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
45922026-03-11T20:45:50.898Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
45932026-03-11T20:45:50.898Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
45942026-03-11T20:45:50.898Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
45952026-03-11T20:45:50.898Z
45962026-03-11T20:45:50.898Zgoroutine 10914 [select]:
45972026-03-11T20:45:50.898Zgithub.com/cockroachdb/cockroach/pkg/sql/stmtdiagnostics.(*Registry).poll(0xc006067960, {0x50b1300, 0xc00067d0e0})
45982026-03-11T20:45:50.899Z /work/oxidecomputer/cockroach/pkg/sql/stmtdiagnostics/statement_diagnostics.go:181 +0x2b5
45992026-03-11T20:45:50.899Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
46002026-03-11T20:45:50.899Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
46012026-03-11T20:45:50.899Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
46022026-03-11T20:45:50.899Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
46032026-03-11T20:45:50.904Z
46042026-03-11T20:45:50.904Zgoroutine 10312 [select, 59 minutes]:
46052026-03-11T20:45:50.904Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1({0x50b1338, 0xc002340910})
46062026-03-11T20:45:50.904Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:116 +0x157
46072026-03-11T20:45:50.904Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.Group.GoCtx.func3()
46082026-03-11T20:45:50.904Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
46092026-03-11T20:45:50.904Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
46102026-03-11T20:45:50.904Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
46112026-03-11T20:45:50.904Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10310
46122026-03-11T20:45:50.904Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
46132026-03-11T20:45:50.904Z
46142026-03-11T20:45:50.904Zgoroutine 10372 [select, 59 minutes]:
46152026-03-11T20:45:50.904Zgoogle.golang.org/grpc.newClientStreamWithParams.func4()
46162026-03-11T20:45:50.904Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:341 +0x8c
46172026-03-11T20:45:50.904Zcreated by google.golang.org/grpc.newClientStreamWithParams in goroutine 10657
46182026-03-11T20:45:50.904Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:340 +0xb2a
46192026-03-11T20:45:50.904Z
46202026-03-11T20:45:50.904Zgoroutine 10974 [select]:
46212026-03-11T20:45:50.904Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).outputLoop(0xc000d39880, {0x50b1338, 0xc004ee87d0})
46222026-03-11T20:45:50.904Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:310 +0x247
46232026-03-11T20:45:50.905Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).runOutputLoop(0xc000d39880, {0x50b1300, 0xc001b002d0}, 0x1b?)
46242026-03-11T20:45:50.905Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:335 +0xba
46252026-03-11T20:45:50.905Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run.func1({0x50b1300?, 0xc001b002d0?})
46262026-03-11T20:45:50.905Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:297 +0x46
46272026-03-11T20:45:50.905Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
46282026-03-11T20:45:50.905Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
46292026-03-11T20:45:50.905Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10334
46302026-03-11T20:45:50.905Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
46312026-03-11T20:45:50.905Z
46322026-03-11T20:45:50.905Zgoroutine 9868 [select]:
46332026-03-11T20:45:50.905Zgithub.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
46342026-03-11T20:45:50.905Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:439 +0x1c9
46352026-03-11T20:45:50.906Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 9708
46362026-03-11T20:45:50.906Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:412 +0x4f
46372026-03-11T20:45:50.906Z
46382026-03-11T20:45:50.906Zgoroutine 10047 [sync.Cond.Wait]:
46392026-03-11T20:45:50.906Zsync.runtime_notifyListWait(0xc001e22250, 0x375e)
46402026-03-11T20:45:50.906Z /work/go/src/runtime/sema.go:569 +0x159
46412026-03-11T20:45:50.906Zsync.(*Cond).Wait(0x3aac600?)
46422026-03-11T20:45:50.906Z /work/go/src/sync/cond.go:70 +0x85
46432026-03-11T20:45:50.906Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc0075833b0})
46442026-03-11T20:45:50.906Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
46452026-03-11T20:45:50.906Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
46462026-03-11T20:45:50.906Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
46472026-03-11T20:45:50.906Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
46482026-03-11T20:45:50.906Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
46492026-03-11T20:45:50.906Z
46502026-03-11T20:45:50.906Zgoroutine 10917 [select]:
46512026-03-11T20:45:50.906Zgithub.com/cockroachdb/cockroach/pkg/sql/catalog/lease.(*Manager).PeriodicallyRefreshSomeLeases.func1({0x50b1300, 0xc00067d890})
46522026-03-11T20:45:50.906Z /work/oxidecomputer/cockroach/pkg/sql/catalog/lease/lease.go:1151 +0x2df
46532026-03-11T20:45:50.906Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
46542026-03-11T20:45:50.906Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
46552026-03-11T20:45:50.906Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
46562026-03-11T20:45:50.906Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
46572026-03-11T20:45:50.906Z
46582026-03-11T20:45:50.906Zgoroutine 10755 [sync.Cond.Wait, 59 minutes]:
46592026-03-11T20:45:50.906Zsync.runtime_notifyListWait(0xc0064c5050, 0x10)
46602026-03-11T20:45:50.906Z /work/go/src/runtime/sema.go:569 +0x159
46612026-03-11T20:45:50.906Zsync.(*Cond).Wait(0xc000f39208?)
46622026-03-11T20:45:50.906Z /work/go/src/sync/cond.go:70 +0x85
46632026-03-11T20:45:50.906Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3230})
46642026-03-11T20:45:50.906Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
46652026-03-11T20:45:50.906Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
46662026-03-11T20:45:50.906Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
46672026-03-11T20:45:50.906Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
46682026-03-11T20:45:50.906Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
46692026-03-11T20:45:50.906Z
46702026-03-11T20:45:50.906Zgoroutine 9776 [chan receive, 59 minutes]:
46712026-03-11T20:45:50.906Zgithub.com/cockroachdb/cockroach/pkg/util/netutil.MakeServer.func2({0xc000a9a308?, 0x0?})
46722026-03-11T20:45:50.906Z /work/oxidecomputer/cockroach/pkg/util/netutil/net.go:113 +0x49
46732026-03-11T20:45:50.906Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
46742026-03-11T20:45:50.907Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
46752026-03-11T20:45:50.907Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9963
46762026-03-11T20:45:50.907Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
46772026-03-11T20:45:50.907Z
46782026-03-11T20:45:50.907Zgoroutine 9964 [chan receive, 59 minutes]:
46792026-03-11T20:45:50.907Zgithub.com/cockroachdb/cmux.muxListener.Accept(...)
46802026-03-11T20:45:50.907Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184
46812026-03-11T20:45:50.907Zcrypto/tls.(*listener).Accept(0xc0059e3638)
46822026-03-11T20:45:50.907Z /work/go/src/crypto/tls/tls.go:66 +0x27
46832026-03-11T20:45:50.907Znet/http.(*Server).Serve(0xc00143a000, {0x509c4f0, 0xc0059e3638})
46842026-03-11T20:45:50.907Z /work/go/src/net/http/server.go:3260 +0x33e
46852026-03-11T20:45:50.907Zgithub.com/cockroachdb/cockroach/pkg/server.(*httpServer).start.func4({0xc0013e6c08?, 0x0?})
46862026-03-11T20:45:50.907Z /work/oxidecomputer/cockroach/pkg/server/server_http.go:277 +0x25
46872026-03-11T20:45:50.907Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
46882026-03-11T20:45:50.907Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
46892026-03-11T20:45:50.907Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
46902026-03-11T20:45:50.907Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
46912026-03-11T20:45:50.907Z
46922026-03-11T20:45:50.907Zgoroutine 9865 [select]:
46932026-03-11T20:45:50.907Zgithub.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue.func2()
46942026-03-11T20:45:50.907Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:366 +0x85
46952026-03-11T20:45:50.907Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue in goroutine 9708
46962026-03-11T20:45:50.907Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:363 +0x3ed
46972026-03-11T20:45:50.907Z
46982026-03-11T20:45:50.907Zgoroutine 10748 [sync.Cond.Wait, 59 minutes]:
46992026-03-11T20:45:50.907Zsync.runtime_notifyListWait(0xc0064c5050, 0x14)
47002026-03-11T20:45:50.907Z /work/go/src/runtime/sema.go:569 +0x159
47012026-03-11T20:45:50.907Zsync.(*Cond).Wait(0xc006e3e878?)
47022026-03-11T20:45:50.907Z /work/go/src/sync/cond.go:70 +0x85
47032026-03-11T20:45:50.912Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd30e0})
47042026-03-11T20:45:50.912Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
47052026-03-11T20:45:50.912Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
47062026-03-11T20:45:50.912Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
47072026-03-11T20:45:50.912Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
47082026-03-11T20:45:50.912Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
47092026-03-11T20:45:50.912Z
47102026-03-11T20:45:50.912Zgoroutine 9869 [select]:
47112026-03-11T20:45:50.912Zgithub.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue.func2()
47122026-03-11T20:45:50.912Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:366 +0x85
47132026-03-11T20:45:50.912Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue in goroutine 9708
47142026-03-11T20:45:50.913Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:363 +0x3ed
47152026-03-11T20:45:50.913Z
47162026-03-11T20:45:50.913Zgoroutine 10332 [select]:
47172026-03-11T20:45:50.913Zgithub.com/cockroachdb/cockroach/pkg/rpc.respStreamClientAdapter.recvInternal(...)
47182026-03-11T20:45:50.913Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:694
47192026-03-11T20:45:50.913Zgithub.com/cockroachdb/cockroach/pkg/rpc.rangeFeedClientAdapter.Recv(...)
47202026-03-11T20:45:50.913Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:723
47212026-03-11T20:45:50.913Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed.func3()
47222026-03-11T20:45:50.913Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:516 +0x25
47232026-03-11T20:45:50.913Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*stuckRangeFeedCanceler).do(0xc0022ba410, 0xc003519a50)
47242026-03-11T20:45:50.913Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed_canceler.go:100 +0x1b3
47252026-03-11T20:45:50.913Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed(0xc002d6f408, {0x50b1338?, 0xc002340910?}, {{0xc008f7c248, 0x1, 0x8}, {0xc008f7c250, 0x1, 0x8}}, {0x0, ...}, ...)
47262026-03-11T20:45:50.913Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:515 +0xc98
47272026-03-11T20:45:50.913Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).partialRangeFeed(0xc002d6f408, {0x50b1338, 0xc002340910}, 0xc0023408c0, {{0xc008f7c248, 0x1, 0x8}, {0xc008f7c250, 0x1, 0x8}}, ...)
47282026-03-11T20:45:50.913Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:336 +0x745
47292026-03-11T20:45:50.913Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.1({0x50b1338?, 0xc002340910?})
47302026-03-11T20:45:50.914Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:120 +0xa9
47312026-03-11T20:45:50.914Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.Group.GoCtx.2()
47322026-03-11T20:45:50.914Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
47332026-03-11T20:45:50.914Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
47342026-03-11T20:45:50.914Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
47352026-03-11T20:45:50.914Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10312
47362026-03-11T20:45:50.914Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
47372026-03-11T20:45:50.914Z
47382026-03-11T20:45:50.914Zgoroutine 11001 [semacquire, 59 minutes]:
47392026-03-11T20:45:50.914Zsync.runtime_Semacquire(0xc003453c38?)
47402026-03-11T20:45:50.914Z /work/go/src/runtime/sema.go:62 +0x25
47412026-03-11T20:45:50.914Zsync.(*WaitGroup).Wait(0x44e7a0?)
47422026-03-11T20:45:50.914Z /work/go/src/sync/waitgroup.go:116 +0x48
47432026-03-11T20:45:50.914Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc0036e0080)
47442026-03-11T20:45:50.914Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
47452026-03-11T20:45:50.914Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc0036e0080?, {0x50b1338?, 0xc001435310?}})
47462026-03-11T20:45:50.914Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
47472026-03-11T20:45:50.914Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait({0x50b1300?, 0xc003dad470?}, {0xc0069e9d78, 0x2, 0xc0069e9c70?})
47482026-03-11T20:45:50.914Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:193 +0xff
47492026-03-11T20:45:50.915Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0xc003b48c60, {0x50b1300, 0xc003dad470}, 0xc003dacff0)
47502026-03-11T20:45:50.915Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:291 +0x59c
47512026-03-11T20:45:50.915Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).Start.func2({0x50b1300, 0xc003dad3b0})
47522026-03-11T20:45:50.915Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:215 +0x2f3
47532026-03-11T20:45:50.915Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
47542026-03-11T20:45:50.915Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
47552026-03-11T20:45:50.915Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 11000
47562026-03-11T20:45:50.915Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
47572026-03-11T20:45:50.915Z
47582026-03-11T20:45:50.915Zgoroutine 11041 [select]:
47592026-03-11T20:45:50.915Zgoogle.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc004c69770, {0xc002290628, 0x5, 0x5})
47602026-03-11T20:45:50.915Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:191 +0x95
47612026-03-11T20:45:50.915Zgoogle.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc004c69770, {0xc002290628?, 0xc0009d93f8?, 0xc0072cf048?})
47622026-03-11T20:45:50.915Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:171 +0x12d
47632026-03-11T20:45:50.915Zgoogle.golang.org/grpc/internal/transport.(*transportReader).Read(0xc00435d8f0, {0xc002290628?, 0xc0072cf0c0?, 0xa40865?})
47642026-03-11T20:45:50.915Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:485 +0x2c
47652026-03-11T20:45:50.915Zio.ReadAtLeast({0x5086380, 0xc00435d8f0}, {0xc002290628, 0x5, 0x5}, 0x5)
47662026-03-11T20:45:50.916Z /work/go/src/io/io.go:335 +0x90
47672026-03-11T20:45:50.916Zio.ReadFull(...)
47682026-03-11T20:45:50.916Z /work/go/src/io/io.go:354
47692026-03-11T20:45:50.916Zgoogle.golang.org/grpc/internal/transport.(*Stream).Read(0xc00350d200, {0xc002290628, 0x5, 0x5})
47702026-03-11T20:45:50.916Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:469 +0x96
47712026-03-11T20:45:50.916Zgoogle.golang.org/grpc.(*parser).recvMsg(0xc002290618, 0x7fffffff)
47722026-03-11T20:45:50.916Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:559 +0x42
47732026-03-11T20:45:50.916Zgoogle.golang.org/grpc.recvAndDecompress(0x60?, 0xc00350d200, {0x0, 0x0}, 0x7fffffff, 0xc0072cf378, {0x50a8e40, 0x77577e0})
47742026-03-11T20:45:50.916Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:690 +0x5a
47752026-03-11T20:45:50.916Zgoogle.golang.org/grpc.recv(0x50900a0?, {0x7fc12cff4640, 0xc0004ce590}, 0x2b?, {0x0?, 0x0?}, {0x3f6ba40, 0xc01a6c4700}, 0x10?, 0xc0072cf378, ...)
47762026-03-11T20:45:50.916Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:756 +0x68
47772026-03-11T20:45:50.916Zgoogle.golang.org/grpc.(*csAttempt).recvMsg(0xc0041cc840, {0x3f6ba40, 0xc01a6c4700}, 0x10?)
47782026-03-11T20:45:50.916Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:983 +0x27e
47792026-03-11T20:45:50.916Zgoogle.golang.org/grpc.(*clientStream).RecvMsg.func1(0x41e1e5?)
47802026-03-11T20:45:50.916Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:834 +0x1f
47812026-03-11T20:45:50.916Zgoogle.golang.org/grpc.(*clientStream).withRetry(0xc00350cd80, 0xc0072cf4a8, 0xc0072cf498)
47822026-03-11T20:45:50.916Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:688 +0x2ee
47832026-03-11T20:45:50.916Zgoogle.golang.org/grpc.(*clientStream).RecvMsg(0xc00350cd80, {0x3f6ba40?, 0xc01a6c4700?})
47842026-03-11T20:45:50.916Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:833 +0x107
47852026-03-11T20:45:50.916Zgithub.com/cockroachdb/cockroach/pkg/util/tracing.(*tracingClientStream).RecvMsg(0xc005b657a0, {0x3f6ba40?, 0xc01a6c4700?})
47862026-03-11T20:45:50.916Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:441 +0x31
47872026-03-11T20:45:50.916Zgithub.com/cockroachdb/cockroach/pkg/roachpb.(*internalRangeFeedClient).Recv(0xc007079920)
47882026-03-11T20:45:50.916Z /work/oxidecomputer/cockroach/pkg/roachpb/api.pb.go:9289 +0x46
47892026-03-11T20:45:50.916Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed.func3()
47902026-03-11T20:45:50.916Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:516 +0x25
47912026-03-11T20:45:50.916Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*stuckRangeFeedCanceler).do(0xc004c69630, 0xc0072cfa50)
47922026-03-11T20:45:50.916Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed_canceler.go:100 +0x1b3
47932026-03-11T20:45:50.916Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed(0xc003d48008, {0x50b1338?, 0xc004c69540?}, {{0xc00712d468, 0x1, 0x8}, {0xc00712d470, 0x1, 0x8}}, {0x189be113f3bf8c0c, ...}, ...)
47942026-03-11T20:45:50.916Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:515 +0xc98
47952026-03-11T20:45:50.916Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).partialRangeFeed(0xc003d48008, {0x50b1338, 0xc004c69540}, 0xc004c694f0, {{0xc00712d468, 0x1, 0x8}, {0xc00712d470, 0x1, 0x8}}, ...)
47962026-03-11T20:45:50.916Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:336 +0x745
47972026-03-11T20:45:50.916Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.1({0x50b1338?, 0xc004c69540?})
47982026-03-11T20:45:50.916Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:120 +0xa9
47992026-03-11T20:45:50.916Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.Group.GoCtx.2()
48002026-03-11T20:45:50.916Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
48012026-03-11T20:45:50.916Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
48022026-03-11T20:45:50.916Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
48032026-03-11T20:45:50.921Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 11039
48042026-03-11T20:45:50.921Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
48052026-03-11T20:45:50.921Z
48062026-03-11T20:45:50.921Zgoroutine 9870 [select]:
48072026-03-11T20:45:50.921Zgithub.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
48082026-03-11T20:45:50.921Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:439 +0x1c9
48092026-03-11T20:45:50.921Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 9708
48102026-03-11T20:45:50.922Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:412 +0x4f
48112026-03-11T20:45:50.922Z
48122026-03-11T20:45:50.922Zgoroutine 10926 [select, 59 minutes]:
48132026-03-11T20:45:50.922Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1({0x50b1338, 0xc0078ad220})
48142026-03-11T20:45:50.922Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:116 +0x157
48152026-03-11T20:45:50.922Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.Group.GoCtx.func3()
48162026-03-11T20:45:50.922Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
48172026-03-11T20:45:50.922Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
48182026-03-11T20:45:50.922Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
48192026-03-11T20:45:50.922Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10922
48202026-03-11T20:45:50.922Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
48212026-03-11T20:45:50.922Z
48222026-03-11T20:45:50.922Zgoroutine 11080 [select]:
48232026-03-11T20:45:50.922Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run(0xc00629f200, {0x50b1300?, 0xc003c93320?}, 0x2c?, 0xc005d362a0, 0xc003f40400)
48242026-03-11T20:45:50.922Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:269 +0x4dc
48252026-03-11T20:45:50.922Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).Start.func1({0x50b1300?, 0xc003c93320?})
48262026-03-11T20:45:50.922Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:222 +0x36
48272026-03-11T20:45:50.922Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
48282026-03-11T20:45:50.922Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
48292026-03-11T20:45:50.922Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 11079
48302026-03-11T20:45:50.922Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
48312026-03-11T20:45:50.922Z
48322026-03-11T20:45:50.922Zgoroutine 9863 [select]:
48332026-03-11T20:45:50.922Zgithub.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue.func2()
48342026-03-11T20:45:50.922Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:366 +0x85
48352026-03-11T20:45:50.922Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue in goroutine 9708
48362026-03-11T20:45:50.922Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:363 +0x3ed
48372026-03-11T20:45:50.922Z
48382026-03-11T20:45:50.922Zgoroutine 10928 [select]:
48392026-03-11T20:45:50.922Zgoogle.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc004ee85a0, {0xc00583c160, 0x5, 0x5})
48402026-03-11T20:45:50.922Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:191 +0x95
48412026-03-11T20:45:50.922Zgoogle.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc004ee85a0, {0xc00583c160?, 0xc0009d8558?, 0xc0041f1048?})
48422026-03-11T20:45:50.922Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:171 +0x12d
48432026-03-11T20:45:50.922Zgoogle.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0032c9710, {0xc00583c160?, 0xc0041f10c0?, 0xa40865?})
48442026-03-11T20:45:50.922Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:485 +0x2c
48452026-03-11T20:45:50.923Zio.ReadAtLeast({0x5086380, 0xc0032c9710}, {0xc00583c160, 0x5, 0x5}, 0x5)
48462026-03-11T20:45:50.923Z /work/go/src/io/io.go:335 +0x90
48472026-03-11T20:45:50.923Zio.ReadFull(...)
48482026-03-11T20:45:50.923Z /work/go/src/io/io.go:354
48492026-03-11T20:45:50.923Zgoogle.golang.org/grpc/internal/transport.(*Stream).Read(0xc004e62a20, {0xc00583c160, 0x5, 0x5})
48502026-03-11T20:45:50.923Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:469 +0x96
48512026-03-11T20:45:50.923Zgoogle.golang.org/grpc.(*parser).recvMsg(0xc00583c150, 0x7fffffff)
48522026-03-11T20:45:50.923Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:559 +0x42
48532026-03-11T20:45:50.923Zgoogle.golang.org/grpc.recvAndDecompress(0x60?, 0xc004e62a20, {0x0, 0x0}, 0x7fffffff, 0xc0041f1378, {0x50a8e40, 0x77577e0})
48542026-03-11T20:45:50.923Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:690 +0x5a
48552026-03-11T20:45:50.923Zgoogle.golang.org/grpc.recv(0x50900a0?, {0x7fc12cff4640, 0xc0004ce590}, 0xc0041f1300?, {0x0?, 0x0?}, {0x3f6ba40, 0xc0021d99c0}, 0x10?, 0xc0041f1378, ...)
48562026-03-11T20:45:50.923Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:756 +0x68
48572026-03-11T20:45:50.923Zgoogle.golang.org/grpc.(*csAttempt).recvMsg(0xc006372b00, {0x3f6ba40, 0xc0021d99c0}, 0x10?)
48582026-03-11T20:45:50.923Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:983 +0x27e
48592026-03-11T20:45:50.923Zgoogle.golang.org/grpc.(*clientStream).RecvMsg.func1(0x41e1e5?)
48602026-03-11T20:45:50.923Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:834 +0x1f
48612026-03-11T20:45:50.924Zgoogle.golang.org/grpc.(*clientStream).withRetry(0xc004e626c0, 0xc0041f14a8, 0xc0041f1498)
48622026-03-11T20:45:50.924Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:688 +0x2ee
48632026-03-11T20:45:50.924Zgoogle.golang.org/grpc.(*clientStream).RecvMsg(0xc004e626c0, {0x3f6ba40?, 0xc0021d99c0?})
48642026-03-11T20:45:50.924Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:833 +0x107
48652026-03-11T20:45:50.924Zgithub.com/cockroachdb/cockroach/pkg/util/tracing.(*tracingClientStream).RecvMsg(0xc00827ba40, {0x3f6ba40?, 0xc0021d99c0?})
48662026-03-11T20:45:50.924Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:441 +0x31
48672026-03-11T20:45:50.924Zgithub.com/cockroachdb/cockroach/pkg/roachpb.(*internalRangeFeedClient).Recv(0xc008587d90)
48682026-03-11T20:45:50.924Z /work/oxidecomputer/cockroach/pkg/roachpb/api.pb.go:9289 +0x46
48692026-03-11T20:45:50.924Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed.func3()
48702026-03-11T20:45:50.924Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:516 +0x25
48712026-03-11T20:45:50.924Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*stuckRangeFeedCanceler).do(0xc0078ad360, 0xc0041f1a50)
48722026-03-11T20:45:50.924Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed_canceler.go:100 +0x1b3
48732026-03-11T20:45:50.924Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed(0xc003d48008, {0x50b1338?, 0xc0078ad220?}, {{0xc0062b2a38, 0x1, 0x8}, {0xc0062b2a40, 0x1, 0x8}}, {0x0, ...}, ...)
48742026-03-11T20:45:50.924Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:515 +0xc98
48752026-03-11T20:45:50.924Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).partialRangeFeed(0xc003d48008, {0x50b1338, 0xc0078ad220}, 0xc0078ad1d0, {{0xc0062b2a38, 0x1, 0x8}, {0xc0062b2a40, 0x1, 0x8}}, ...)
48762026-03-11T20:45:50.924Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:336 +0x745
48772026-03-11T20:45:50.924Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.1({0x50b1338?, 0xc0078ad220?})
48782026-03-11T20:45:50.924Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:120 +0xa9
48792026-03-11T20:45:50.924Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.Group.GoCtx.2()
48802026-03-11T20:45:50.924Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
48812026-03-11T20:45:50.924Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
48822026-03-11T20:45:50.924Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
48832026-03-11T20:45:50.924Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10926
48842026-03-11T20:45:50.924Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
48852026-03-11T20:45:50.924Z
48862026-03-11T20:45:50.924Zgoroutine 10466 [select]:
48872026-03-11T20:45:50.924Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).outputLoop(0xc000750fc0, {0x50b1338, 0xc002228820})
48882026-03-11T20:45:50.924Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:310 +0x247
48892026-03-11T20:45:50.924Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).runOutputLoop(0xc000750fc0, {0x50b1300, 0xc0054ce900}, 0x1?)
48902026-03-11T20:45:50.924Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:335 +0xba
48912026-03-11T20:45:50.924Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run.func1({0x50b1300?, 0xc0054ce900?})
48922026-03-11T20:45:50.924Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:297 +0x46
48932026-03-11T20:45:50.924Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
48942026-03-11T20:45:50.924Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
48952026-03-11T20:45:50.924Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10334
48962026-03-11T20:45:50.924Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
48972026-03-11T20:45:50.924Z
48982026-03-11T20:45:50.924Zgoroutine 10045 [sync.Cond.Wait]:
48992026-03-11T20:45:50.924Zsync.runtime_notifyListWait(0xc001e22250, 0x3758)
49002026-03-11T20:45:50.925Z /work/go/src/runtime/sema.go:569 +0x159
49012026-03-11T20:45:50.925Zsync.(*Cond).Wait(0x3aac600?)
49022026-03-11T20:45:50.925Z /work/go/src/sync/cond.go:70 +0x85
49032026-03-11T20:45:50.934Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583350})
49042026-03-11T20:45:50.934Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
49052026-03-11T20:45:50.934Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
49062026-03-11T20:45:50.934Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
49072026-03-11T20:45:50.934Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
49082026-03-11T20:45:50.934Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
49092026-03-11T20:45:50.934Z
49102026-03-11T20:45:50.934Zgoroutine 9990 [chan receive, 59 minutes]:
49112026-03-11T20:45:50.934Zgithub.com/cockroachdb/cockroach/pkg/server.startListenRPCAndSQL.func3({0xc0013e7208?, 0x0?})
49122026-03-11T20:45:50.934Z /work/oxidecomputer/cockroach/pkg/server/start_listen.go:108 +0x3a
49132026-03-11T20:45:50.934Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
49142026-03-11T20:45:50.934Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
49152026-03-11T20:45:50.934Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
49162026-03-11T20:45:50.934Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
49172026-03-11T20:45:50.935Z
49182026-03-11T20:45:50.935Zgoroutine 10835 [select]:
49192026-03-11T20:45:50.935Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/liveness.(*NodeLiveness).Start.func1({0xc007166c08?, 0x0?})
49202026-03-11T20:45:50.935Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/liveness/liveness.go:789 +0x37f
49212026-03-11T20:45:50.935Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
49222026-03-11T20:45:50.935Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
49232026-03-11T20:45:50.935Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
49242026-03-11T20:45:50.935Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
49252026-03-11T20:45:50.935Z
49262026-03-11T20:45:50.935Zgoroutine 10882 [select]:
49272026-03-11T20:45:50.935Zgoogle.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc00677e5a0, 0x1)
49282026-03-11T20:45:50.935Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:407 +0x113
49292026-03-11T20:45:50.935Zgoogle.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0062aab40)
49302026-03-11T20:45:50.935Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:534 +0x7b
49312026-03-11T20:45:50.935Zgoogle.golang.org/grpc/internal/transport.newHTTP2Client.func3()
49322026-03-11T20:45:50.935Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:415 +0x67
49332026-03-11T20:45:50.935Zcreated by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 10302
49342026-03-11T20:45:50.935Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:413 +0x1dcb
49352026-03-11T20:45:50.935Z
49362026-03-11T20:45:50.935Zgoroutine 9943 [select, 59 minutes]:
49372026-03-11T20:45:50.935Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc00167fb08?, 0x0?})
49382026-03-11T20:45:50.935Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
49392026-03-11T20:45:50.935Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
49402026-03-11T20:45:50.936Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
49412026-03-11T20:45:50.936Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
49422026-03-11T20:45:50.936Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
49432026-03-11T20:45:50.936Z
49442026-03-11T20:45:50.936Zgoroutine 10006 [sync.Cond.Wait]:
49452026-03-11T20:45:50.936Zsync.runtime_notifyListWait(0xc001e22250, 0x3789)
49462026-03-11T20:45:50.936Z /work/go/src/runtime/sema.go:569 +0x159
49472026-03-11T20:45:50.936Zsync.(*Cond).Wait(0x3aac600?)
49482026-03-11T20:45:50.936Z /work/go/src/sync/cond.go:70 +0x85
49492026-03-11T20:45:50.936Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007582f00})
49502026-03-11T20:45:50.936Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
49512026-03-11T20:45:50.936Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
49522026-03-11T20:45:50.936Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
49532026-03-11T20:45:50.936Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
49542026-03-11T20:45:50.936Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
49552026-03-11T20:45:50.936Z
49562026-03-11T20:45:50.936Zgoroutine 9939 [select, 59 minutes]:
49572026-03-11T20:45:50.936Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc00167ef08?, 0x0?})
49582026-03-11T20:45:50.936Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
49592026-03-11T20:45:50.936Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
49602026-03-11T20:45:50.936Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
49612026-03-11T20:45:50.936Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
49622026-03-11T20:45:50.936Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
49632026-03-11T20:45:50.936Z
49642026-03-11T20:45:50.936Zgoroutine 10007 [sync.Cond.Wait]:
49652026-03-11T20:45:50.936Zsync.runtime_notifyListWait(0xc001e22250, 0x3787)
49662026-03-11T20:45:50.936Z /work/go/src/runtime/sema.go:569 +0x159
49672026-03-11T20:45:50.936Zsync.(*Cond).Wait(0x3aac600?)
49682026-03-11T20:45:50.936Z /work/go/src/sync/cond.go:70 +0x85
49692026-03-11T20:45:50.936Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007582f30})
49702026-03-11T20:45:50.937Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
49712026-03-11T20:45:50.937Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
49722026-03-11T20:45:50.937Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
49732026-03-11T20:45:50.937Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
49742026-03-11T20:45:50.937Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
49752026-03-11T20:45:50.937Z
49762026-03-11T20:45:50.937Zgoroutine 10011 [sync.Cond.Wait]:
49772026-03-11T20:45:50.937Zsync.runtime_notifyListWait(0xc001e22250, 0x379a)
49782026-03-11T20:45:50.937Z /work/go/src/runtime/sema.go:569 +0x159
49792026-03-11T20:45:50.937Zsync.(*Cond).Wait(0x3aac600?)
49802026-03-11T20:45:50.937Z /work/go/src/sync/cond.go:70 +0x85
49812026-03-11T20:45:50.937Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007582ff0})
49822026-03-11T20:45:50.937Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
49832026-03-11T20:45:50.937Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
49842026-03-11T20:45:50.937Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
49852026-03-11T20:45:50.937Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
49862026-03-11T20:45:50.937Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
49872026-03-11T20:45:50.937Z
49882026-03-11T20:45:50.937Zgoroutine 10012 [sync.Cond.Wait]:
49892026-03-11T20:45:50.937Zsync.runtime_notifyListWait(0xc001e22250, 0x3743)
49902026-03-11T20:45:50.937Z /work/go/src/runtime/sema.go:569 +0x159
49912026-03-11T20:45:50.937Zsync.(*Cond).Wait(0x3aac600?)
49922026-03-11T20:45:50.937Z /work/go/src/sync/cond.go:70 +0x85
49932026-03-11T20:45:50.937Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583020})
49942026-03-11T20:45:50.937Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
49952026-03-11T20:45:50.937Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
49962026-03-11T20:45:50.937Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
49972026-03-11T20:45:50.937Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
49982026-03-11T20:45:50.937Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
49992026-03-11T20:45:50.937Z
50002026-03-11T20:45:50.937Zgoroutine 10008 [sync.Cond.Wait]:
50012026-03-11T20:45:50.937Zsync.runtime_notifyListWait(0xc001e22250, 0x3790)
50022026-03-11T20:45:50.937Z /work/go/src/runtime/sema.go:569 +0x159
50032026-03-11T20:45:50.944Zsync.(*Cond).Wait(0x3aac600?)
50042026-03-11T20:45:50.944Z /work/go/src/sync/cond.go:70 +0x85
50052026-03-11T20:45:50.944Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007582f60})
50062026-03-11T20:45:50.944Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
50072026-03-11T20:45:50.944Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
50082026-03-11T20:45:50.944Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
50092026-03-11T20:45:50.944Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
50102026-03-11T20:45:50.945Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
50112026-03-11T20:45:50.945Z
50122026-03-11T20:45:50.945Zgoroutine 10630 [select]:
50132026-03-11T20:45:50.945Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.(*Watcher).Run(0xc0011795f0, {0x50b1338, 0xc006581360})
50142026-03-11T20:45:50.945Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:319 +0x955
50152026-03-11T20:45:50.945Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.Start.func1({0x50b1300?, 0xc006858c30?})
50162026-03-11T20:45:50.945Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:189 +0x2d1
50172026-03-11T20:45:50.945Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
50182026-03-11T20:45:50.945Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
50192026-03-11T20:45:50.945Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
50202026-03-11T20:45:50.945Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
50212026-03-11T20:45:50.945Z
50222026-03-11T20:45:50.945Zgoroutine 10305 [IO wait]:
50232026-03-11T20:45:50.945Zinternal/poll.runtime_pollWait(0x7fc12ea8f4b8, 0x72)
50242026-03-11T20:45:50.945Z /work/go/src/runtime/netpoll.go:345 +0x85
50252026-03-11T20:45:50.945Zinternal/poll.(*pollDesc).wait(0xc00667e100?, 0xc006958000?, 0x0)
50262026-03-11T20:45:50.945Z /work/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
50272026-03-11T20:45:50.945Zinternal/poll.(*pollDesc).waitRead(...)
50282026-03-11T20:45:50.945Z /work/go/src/internal/poll/fd_poll_runtime.go:89
50292026-03-11T20:45:50.945Zinternal/poll.(*FD).Read(0xc00667e100, {0xc006958000, 0x5500, 0x5500})
50302026-03-11T20:45:50.945Z /work/go/src/internal/poll/fd_unix.go:164 +0x27a
50312026-03-11T20:45:50.945Znet.(*netFD).Read(0xc00667e100, {0xc006958000?, 0x7fc127b457a8?, 0xc0284e7668?})
50322026-03-11T20:45:50.945Z /work/go/src/net/fd_posix.go:55 +0x25
50332026-03-11T20:45:50.945Znet.(*conn).Read(0xc003c82670, {0xc006958000?, 0xc00409e8e0?, 0x415dbb?})
50342026-03-11T20:45:50.945Z /work/go/src/net/net.go:185 +0x45
50352026-03-11T20:45:50.945Zcrypto/tls.(*atLeastReader).Read(0xc0284e7668, {0xc006958000?, 0x0?, 0xc0284e7668?})
50362026-03-11T20:45:50.945Z /work/go/src/crypto/tls/conn.go:806 +0x3b
50372026-03-11T20:45:50.945Zbytes.(*Buffer).ReadFrom(0xc0068122b0, {0x5081fe0, 0xc0284e7668})
50382026-03-11T20:45:50.945Z /work/go/src/bytes/buffer.go:211 +0x98
50392026-03-11T20:45:50.945Zcrypto/tls.(*Conn).readFromUntil(0xc006812008, {0x50821c0, 0xc003c82670}, 0xc00409e928?)
50402026-03-11T20:45:50.945Z /work/go/src/crypto/tls/conn.go:828 +0xde
50412026-03-11T20:45:50.945Zcrypto/tls.(*Conn).readRecordOrCCS(0xc006812008, 0x0)
50422026-03-11T20:45:50.945Z /work/go/src/crypto/tls/conn.go:626 +0x3cf
50432026-03-11T20:45:50.946Zcrypto/tls.(*Conn).readRecord(...)
50442026-03-11T20:45:50.946Z /work/go/src/crypto/tls/conn.go:588
50452026-03-11T20:45:50.946Zcrypto/tls.(*Conn).Read(0xc006812008, {0xc00230c000, 0x8000, 0xc00bca70e0?})
50462026-03-11T20:45:50.946Z /work/go/src/crypto/tls/conn.go:1370 +0x156
50472026-03-11T20:45:50.946Zbufio.(*Reader).Read(0xc0062aaae0, {0xc0034ce200, 0x9, 0x0?})
50482026-03-11T20:45:50.946Z /work/go/src/bufio/bufio.go:241 +0x197
50492026-03-11T20:45:50.946Zio.ReadAtLeast({0x507c320, 0xc0062aaae0}, {0xc0034ce200, 0x9, 0x9}, 0x9)
50502026-03-11T20:45:50.946Z /work/go/src/io/io.go:335 +0x90
50512026-03-11T20:45:50.946Zio.ReadFull(...)
50522026-03-11T20:45:50.946Z /work/go/src/io/io.go:354
50532026-03-11T20:45:50.946Zgolang.org/x/net/http2.readFrameHeader({0xc0034ce200, 0x9, 0x0?}, {0x507c320?, 0xc0062aaae0?})
50542026-03-11T20:45:50.946Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x65
50552026-03-11T20:45:50.946Zgolang.org/x/net/http2.(*Framer).ReadFrame(0xc0034ce1c0)
50562026-03-11T20:45:50.946Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:501 +0x85
50572026-03-11T20:45:50.946Zgoogle.golang.org/grpc/internal/transport.(*http2Client).reader(0xc006302000)
50582026-03-11T20:45:50.946Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1498 +0x41d
50592026-03-11T20:45:50.947Zcreated by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 10302
50602026-03-11T20:45:50.947Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:365 +0x184d
50612026-03-11T20:45:50.947Z
50622026-03-11T20:45:50.947Zgoroutine 9847 [select, 59 minutes]:
50632026-03-11T20:45:50.947Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initCancelingWorkers.func1({0x50b1300, 0xc00485e570})
50642026-03-11T20:45:50.947Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:139 +0xc5
50652026-03-11T20:45:50.947Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
50662026-03-11T20:45:50.947Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
50672026-03-11T20:45:50.947Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
50682026-03-11T20:45:50.947Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
50692026-03-11T20:45:50.947Z
50702026-03-11T20:45:50.947Zgoroutine 11075 [select]:
50712026-03-11T20:45:50.947Zgoogle.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc002297d60, 0x1)
50722026-03-11T20:45:50.947Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:407 +0x113
50732026-03-11T20:45:50.947Zgoogle.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc005789800)
50742026-03-11T20:45:50.947Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:534 +0x7b
50752026-03-11T20:45:50.947Zgoogle.golang.org/grpc/internal/transport.NewServerTransport.func2()
50762026-03-11T20:45:50.947Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:332 +0xc6
50772026-03-11T20:45:50.948Zcreated by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 11048
50782026-03-11T20:45:50.948Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:329 +0x184c
50792026-03-11T20:45:50.948Z
50802026-03-11T20:45:50.948Zgoroutine 10740 [sync.Cond.Wait, 59 minutes]:
50812026-03-11T20:45:50.948Zsync.runtime_notifyListWait(0xc0064c5050, 0x4)
50822026-03-11T20:45:50.948Z /work/go/src/runtime/sema.go:569 +0x159
50832026-03-11T20:45:50.948Zsync.(*Cond).Wait(0x1eda2b2?)
50842026-03-11T20:45:50.948Z /work/go/src/sync/cond.go:70 +0x85
50852026-03-11T20:45:50.948Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd2f60})
50862026-03-11T20:45:50.948Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
50872026-03-11T20:45:50.948Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
50882026-03-11T20:45:50.948Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
50892026-03-11T20:45:50.948Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
50902026-03-11T20:45:50.948Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
50912026-03-11T20:45:50.948Z
50922026-03-11T20:45:50.948Zgoroutine 9938 [select, 59 minutes]:
50932026-03-11T20:45:50.948Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc00167ec08?, 0xc00485e090?})
50942026-03-11T20:45:50.948Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
50952026-03-11T20:45:50.948Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
50962026-03-11T20:45:50.948Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
50972026-03-11T20:45:50.948Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
50982026-03-11T20:45:50.948Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
50992026-03-11T20:45:50.948Z
51002026-03-11T20:45:50.948Zgoroutine 11308 [select, 59 minutes]:
51012026-03-11T20:45:50.948Zgoogle.golang.org/grpc.newClientStreamWithParams.func4()
51022026-03-11T20:45:50.948Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:341 +0x8c
51032026-03-11T20:45:50.955Zcreated by google.golang.org/grpc.newClientStreamWithParams in goroutine 11307
51042026-03-11T20:45:50.955Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:340 +0xb2a
51052026-03-11T20:45:50.955Z
51062026-03-11T20:45:50.955Zgoroutine 10118 [select]:
51072026-03-11T20:45:50.955Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).raftTickLoop(0xc000f39208, {0x2786?, 0x0?})
51082026-03-11T20:45:50.955Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store_raft.go:618 +0x145
51092026-03-11T20:45:50.955Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
51102026-03-11T20:45:50.955Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
51112026-03-11T20:45:50.955Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
51122026-03-11T20:45:50.955Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
51132026-03-11T20:45:50.955Z
51142026-03-11T20:45:50.955Zgoroutine 9819 [chan receive, 59 minutes]:
51152026-03-11T20:45:50.955Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc0055724b0?})
51162026-03-11T20:45:50.955Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
51172026-03-11T20:45:50.955Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x50b1338?, 0xc002fa46e0?}}, 0xc004056fa8)
51182026-03-11T20:45:50.955Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
51192026-03-11T20:45:50.955Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc002d2f680?)
51202026-03-11T20:45:50.955Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
51212026-03-11T20:45:50.955Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
51222026-03-11T20:45:50.955Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
51232026-03-11T20:45:50.955Z
51242026-03-11T20:45:50.955Zgoroutine 10497 [chan receive, 59 minutes]:
51252026-03-11T20:45:50.955Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).rangeFeedWithRangeID(0xc001d7a808, 0x3f3de40?, 0xc006096120, {0x7fc12cefd140, 0xc00734f220})
51262026-03-11T20:45:50.955Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:249 +0x76b
51272026-03-11T20:45:50.955Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed(...)
51282026-03-11T20:45:50.955Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:149
51292026-03-11T20:45:50.955Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).RangeFeed(0xc000f39208, 0xc006096120, {0x50e5b30, 0xc00734f220})
51302026-03-11T20:45:50.955Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:3082 +0x16f
51312026-03-11T20:45:50.955Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).RangeFeed(0xc001eb4380, 0xc006096120, {0x50e5b30, 0xc00734f220})
51322026-03-11T20:45:50.955Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/stores.go:216 +0x19a
51332026-03-11T20:45:50.955Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed(0x3f3de40?, 0xc003c52080?, {0x50e5b30, 0xc00734f220})
51342026-03-11T20:45:50.955Z /work/oxidecomputer/cockroach/pkg/server/node.go:1232 +0x28
51352026-03-11T20:45:50.955Zgithub.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed.func1()
51362026-03-11T20:45:50.955Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:753 +0xa7
51372026-03-11T20:45:50.955Zcreated by github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed in goroutine 10496
51382026-03-11T20:45:50.955Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:750 +0x17c
51392026-03-11T20:45:50.955Z
51402026-03-11T20:45:50.955Zgoroutine 10496 [select]:
51412026-03-11T20:45:50.956Zgithub.com/cockroachdb/cockroach/pkg/rpc.respStreamClientAdapter.recvInternal(...)
51422026-03-11T20:45:50.956Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:694
51432026-03-11T20:45:50.956Zgithub.com/cockroachdb/cockroach/pkg/rpc.rangeFeedClientAdapter.Recv(...)
51442026-03-11T20:45:50.956Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:723
51452026-03-11T20:45:50.956Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed.func3()
51462026-03-11T20:45:50.956Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:516 +0x25
51472026-03-11T20:45:50.956Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*stuckRangeFeedCanceler).do(0xc005f39ef0, 0xc0060a1a50)
51482026-03-11T20:45:50.956Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed_canceler.go:100 +0x1b3
51492026-03-11T20:45:50.956Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed(0xc002d6f408, {0x50b1338?, 0xc005f39e00?}, {{0xc00922efd0, 0x1, 0x8}, {0xc00922efd8, 0x1, 0x8}}, {0x189be113ecd14117, ...}, ...)
51502026-03-11T20:45:50.956Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:515 +0xc98
51512026-03-11T20:45:50.956Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).partialRangeFeed(0xc002d6f408, {0x50b1338, 0xc005f39e00}, 0xc005f39db0, {{0xc00922efd0, 0x1, 0x8}, {0xc00922efd8, 0x1, 0x8}}, ...)
51522026-03-11T20:45:50.956Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:336 +0x745
51532026-03-11T20:45:50.956Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.1({0x50b1338?, 0xc005f39e00?})
51542026-03-11T20:45:50.956Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:120 +0xa9
51552026-03-11T20:45:50.956Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.Group.GoCtx.2()
51562026-03-11T20:45:50.956Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
51572026-03-11T20:45:50.956Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
51582026-03-11T20:45:50.956Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
51592026-03-11T20:45:50.956Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10494
51602026-03-11T20:45:50.956Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
51612026-03-11T20:45:50.956Z
51622026-03-11T20:45:50.956Zgoroutine 10714 [IO wait]:
51632026-03-11T20:45:50.956Zinternal/poll.runtime_pollWait(0x7fc12ea8f0d8, 0x72)
51642026-03-11T20:45:50.956Z /work/go/src/runtime/netpoll.go:345 +0x85
51652026-03-11T20:45:50.956Zinternal/poll.(*pollDesc).wait(0xc00692c000?, 0xc0031f2600?, 0x0)
51662026-03-11T20:45:50.956Z /work/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
51672026-03-11T20:45:50.956Zinternal/poll.(*pollDesc).waitRead(...)
51682026-03-11T20:45:50.956Z /work/go/src/internal/poll/fd_poll_runtime.go:89
51692026-03-11T20:45:50.956Zinternal/poll.(*FD).Read(0xc00692c000, {0xc0031f2600, 0x600, 0x600})
51702026-03-11T20:45:50.956Z /work/go/src/internal/poll/fd_unix.go:164 +0x27a
51712026-03-11T20:45:50.956Znet.(*netFD).Read(0xc00692c000, {0xc0031f2600?, 0xc004345868?, 0x41e1e5?})
51722026-03-11T20:45:50.956Z /work/go/src/net/fd_posix.go:55 +0x25
51732026-03-11T20:45:50.956Znet.(*conn).Read(0xc004b0a048, {0xc0031f2600?, 0x7fc12afd43f0?, 0xc00d4cddd0?})
51742026-03-11T20:45:50.956Z /work/go/src/net/net.go:185 +0x45
51752026-03-11T20:45:50.956Zgithub.com/cockroachdb/cmux.(*MuxConn).Read(0xc002f4a000, {0xc0031f2600, 0x600, 0x600})
51762026-03-11T20:45:50.956Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x14f
51772026-03-11T20:45:50.956Zcrypto/tls.(*atLeastReader).Read(0xc00d4cddd0, {0xc0031f2600?, 0x0?, 0xc00d4cddd0?})
51782026-03-11T20:45:50.956Z /work/go/src/crypto/tls/conn.go:806 +0x3b
51792026-03-11T20:45:50.957Zbytes.(*Buffer).ReadFrom(0xc0065762b0, {0x5081fe0, 0xc00d4cddd0})
51802026-03-11T20:45:50.957Z /work/go/src/bytes/buffer.go:211 +0x98
51812026-03-11T20:45:50.957Zcrypto/tls.(*Conn).readFromUntil(0xc006576008, {0x7fc12ce5a2d8, 0xc002f4a000}, 0xc004345990?)
51822026-03-11T20:45:50.957Z /work/go/src/crypto/tls/conn.go:828 +0xde
51832026-03-11T20:45:50.957Zcrypto/tls.(*Conn).readRecordOrCCS(0xc006576008, 0x0)
51842026-03-11T20:45:50.957Z /work/go/src/crypto/tls/conn.go:626 +0x3cf
51852026-03-11T20:45:50.957Zcrypto/tls.(*Conn).readRecord(...)
51862026-03-11T20:45:50.957Z /work/go/src/crypto/tls/conn.go:588
51872026-03-11T20:45:50.957Zcrypto/tls.(*Conn).Read(0xc006576008, {0xc000dc8000, 0x8000, 0xc004345ce8?})
51882026-03-11T20:45:50.957Z /work/go/src/crypto/tls/conn.go:1370 +0x156
51892026-03-11T20:45:50.957Zbufio.(*Reader).Read(0xc0033c9ec0, {0xc002bf22e0, 0x9, 0xc00d4cddb8?})
51902026-03-11T20:45:50.957Z /work/go/src/bufio/bufio.go:241 +0x197
51912026-03-11T20:45:50.957Zio.ReadAtLeast({0x507c320, 0xc0033c9ec0}, {0xc002bf22e0, 0x9, 0x9}, 0x9)
51922026-03-11T20:45:50.957Z /work/go/src/io/io.go:335 +0x90
51932026-03-11T20:45:50.957Zio.ReadFull(...)
51942026-03-11T20:45:50.958Z /work/go/src/io/io.go:354
51952026-03-11T20:45:50.958Zgolang.org/x/net/http2.readFrameHeader({0xc002bf22e0, 0x9, 0xc004345e10?}, {0x507c320?, 0xc0033c9ec0?})
51962026-03-11T20:45:50.958Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x65
51972026-03-11T20:45:50.958Zgolang.org/x/net/http2.(*Framer).ReadFrame(0xc002bf22a0)
51982026-03-11T20:45:50.958Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:501 +0x85
51992026-03-11T20:45:50.958Zgoogle.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc00785b040, 0xc007080ab0, 0x42e6fc8)
52002026-03-11T20:45:50.958Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:611 +0x14e
52012026-03-11T20:45:50.958Zgoogle.golang.org/grpc.(*Server).serveStreams(0xc004feefc0, {0x50f1d80, 0xc00785b040})
52022026-03-11T20:45:50.958Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:906 +0x114
52032026-03-11T20:45:50.962Zgoogle.golang.org/grpc.(*Server).handleRawConn.func1()
52042026-03-11T20:45:50.962Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:848 +0x45
52052026-03-11T20:45:50.962Zcreated by google.golang.org/grpc.(*Server).handleRawConn in goroutine 10518
52062026-03-11T20:45:50.962Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:847 +0x15b
52072026-03-11T20:45:50.962Z
52082026-03-11T20:45:50.962Zgoroutine 9867 [select]:
52092026-03-11T20:45:50.962Zgithub.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue.func2()
52102026-03-11T20:45:50.962Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:366 +0x85
52112026-03-11T20:45:50.962Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue in goroutine 9708
52122026-03-11T20:45:50.962Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:363 +0x3ed
52132026-03-11T20:45:50.962Z
52142026-03-11T20:45:50.962Zgoroutine 10775 [sync.Cond.Wait, 59 minutes]:
52152026-03-11T20:45:50.962Zsync.runtime_notifyListWait(0xc0064c5050, 0x39)
52162026-03-11T20:45:50.962Z /work/go/src/runtime/sema.go:569 +0x159
52172026-03-11T20:45:50.962Zsync.(*Cond).Wait(0x50f7930?)
52182026-03-11T20:45:50.962Z /work/go/src/sync/cond.go:70 +0x85
52192026-03-11T20:45:50.962Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd35f0})
52202026-03-11T20:45:50.962Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
52212026-03-11T20:45:50.962Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
52222026-03-11T20:45:50.962Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
52232026-03-11T20:45:50.962Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
52242026-03-11T20:45:50.962Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
52252026-03-11T20:45:50.962Z
52262026-03-11T20:45:50.962Zgoroutine 9907 [chan receive, 59 minutes]:
52272026-03-11T20:45:50.962Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc002e5a360?})
52282026-03-11T20:45:50.962Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
52292026-03-11T20:45:50.963Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x0?, 0x0?}}, 0xc004243fa8)
52302026-03-11T20:45:50.963Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
52312026-03-11T20:45:50.963Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
52322026-03-11T20:45:50.963Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
52332026-03-11T20:45:50.963Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
52342026-03-11T20:45:50.963Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
52352026-03-11T20:45:50.963Z
52362026-03-11T20:45:50.963Zgoroutine 9849 [select, 59 minutes]:
52372026-03-11T20:45:50.963Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initCancelingWorkers.func1({0x50b1300, 0xc00485e600})
52382026-03-11T20:45:50.963Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:139 +0xc5
52392026-03-11T20:45:50.963Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
52402026-03-11T20:45:50.963Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
52412026-03-11T20:45:50.963Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
52422026-03-11T20:45:50.963Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
52432026-03-11T20:45:50.963Z
52442026-03-11T20:45:50.963Zgoroutine 9072 [select]:
52452026-03-11T20:45:50.963Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaScanner).waitAndProcess(0xc00388e1c0, {0x50b1300, 0xc0076b67e0}, {0x0?, 0x0?, 0x0?}, 0x0)
52462026-03-11T20:45:50.963Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scanner.go:224 +0x1f1
52472026-03-11T20:45:50.963Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaScanner).scanLoop.func1({0x50b1300, 0xc0076b67e0})
52482026-03-11T20:45:50.963Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scanner.go:291 +0x1e5
52492026-03-11T20:45:50.963Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
52502026-03-11T20:45:50.963Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
52512026-03-11T20:45:50.963Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10821
52522026-03-11T20:45:50.963Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
52532026-03-11T20:45:50.963Z
52542026-03-11T20:45:50.963Zgoroutine 10488 [chan receive, 59 minutes]:
52552026-03-11T20:45:50.963Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).rangeFeedWithRangeID(0xc001d7b408, 0x3f3de40?, 0xc00601dd40, {0x7fc12cefd140, 0xc00734e240})
52562026-03-11T20:45:50.963Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:249 +0x76b
52572026-03-11T20:45:50.963Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed(...)
52582026-03-11T20:45:50.963Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:149
52592026-03-11T20:45:50.963Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).RangeFeed(0xc000f39208, 0xc00601dd40, {0x50e5b30, 0xc00734e240})
52602026-03-11T20:45:50.963Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:3082 +0x16f
52612026-03-11T20:45:50.963Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).RangeFeed(0xc001eb4380, 0xc00601dd40, {0x50e5b30, 0xc00734e240})
52622026-03-11T20:45:50.963Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/stores.go:216 +0x19a
52632026-03-11T20:45:50.963Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed(0x3f3de40?, 0xc005f39bb0?, {0x50e5b30, 0xc00734e240})
52642026-03-11T20:45:50.963Z /work/oxidecomputer/cockroach/pkg/server/node.go:1232 +0x28
52652026-03-11T20:45:50.963Zgithub.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed.func1()
52662026-03-11T20:45:50.963Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:753 +0xa7
52672026-03-11T20:45:50.963Zcreated by github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed in goroutine 10487
52682026-03-11T20:45:50.964Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:750 +0x17c
52692026-03-11T20:45:50.964Z
52702026-03-11T20:45:50.964Zgoroutine 9814 [chan receive, 59 minutes]:
52712026-03-11T20:45:50.964Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc0036f80f0?})
52722026-03-11T20:45:50.964Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
52732026-03-11T20:45:50.964Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0xc00879c840?, 0x1028cc0?}}, 0xc004058fa8)
52742026-03-11T20:45:50.964Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
52752026-03-11T20:45:50.964Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc004058f98?)
52762026-03-11T20:45:50.964Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
52772026-03-11T20:45:50.964Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
52782026-03-11T20:45:50.964Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
52792026-03-11T20:45:50.964Z
52802026-03-11T20:45:50.964Zgoroutine 10688 [IO wait, 59 minutes]:
52812026-03-11T20:45:50.964Zinternal/poll.runtime_pollWait(0x7fc12ea8ecf8, 0x72)
52822026-03-11T20:45:50.964Z /work/go/src/runtime/netpoll.go:345 +0x85
52832026-03-11T20:45:50.965Zinternal/poll.(*pollDesc).wait(0xe?, 0x0?, 0x0)
52842026-03-11T20:45:50.965Z /work/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
52852026-03-11T20:45:50.965Zinternal/poll.(*pollDesc).waitRead(...)
52862026-03-11T20:45:50.965Z /work/go/src/internal/poll/fd_poll_runtime.go:89
52872026-03-11T20:45:50.965Zinternal/poll.(*FD).Accept(0xc00667ea80)
52882026-03-11T20:45:50.965Z /work/go/src/internal/poll/fd_unix.go:611 +0x2ac
52892026-03-11T20:45:50.965Znet.(*netFD).accept(0xc00667ea80)
52902026-03-11T20:45:50.965Z /work/go/src/net/fd_unix.go:172 +0x29
52912026-03-11T20:45:50.965Znet.(*TCPListener).accept(0xc006635400)
52922026-03-11T20:45:50.965Z /work/go/src/net/tcpsock_posix.go:159 +0x1e
52932026-03-11T20:45:50.965Znet.(*TCPListener).Accept(0xc006635400)
52942026-03-11T20:45:50.965Z /work/go/src/net/tcpsock.go:327 +0x30
52952026-03-11T20:45:50.965Zgithub.com/cockroachdb/cmux.(*cMux).Serve(0xc0064c45c0)
52962026-03-11T20:45:50.965Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:124 +0x97
52972026-03-11T20:45:50.965Zgithub.com/cockroachdb/cockroach/pkg/server.startListenRPCAndSQL.func5.2.1()
52982026-03-11T20:45:50.965Z /work/oxidecomputer/cockroach/pkg/server/start_listen.go:139 +0x27
52992026-03-11T20:45:50.965Zsync.(*Once).doSlow(0xc0020beed8?, 0xc0020bef18?)
53002026-03-11T20:45:50.965Z /work/go/src/sync/once.go:74 +0xc2
53012026-03-11T20:45:50.965Zsync.(*Once).Do(...)
53022026-03-11T20:45:50.965Z /work/go/src/sync/once.go:65
53032026-03-11T20:45:50.970Zgithub.com/cockroachdb/cockroach/pkg/server.startListenRPCAndSQL.func5.2({0xc006e9c308?, 0xc007583710?})
53042026-03-11T20:45:50.970Z /work/oxidecomputer/cockroach/pkg/server/start_listen.go:138 +0x4f
53052026-03-11T20:45:50.970Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
53062026-03-11T20:45:50.970Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
53072026-03-11T20:45:50.970Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
53082026-03-11T20:45:50.970Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
53092026-03-11T20:45:50.970Z
53102026-03-11T20:45:50.970Zgoroutine 10487 [select]:
53112026-03-11T20:45:50.970Zgithub.com/cockroachdb/cockroach/pkg/rpc.respStreamClientAdapter.recvInternal(...)
53122026-03-11T20:45:50.970Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:694
53132026-03-11T20:45:50.970Zgithub.com/cockroachdb/cockroach/pkg/rpc.rangeFeedClientAdapter.Recv(...)
53142026-03-11T20:45:50.970Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:723
53152026-03-11T20:45:50.970Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed.func3()
53162026-03-11T20:45:50.970Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:516 +0x25
53172026-03-11T20:45:50.970Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*stuckRangeFeedCanceler).do(0xc005f39ae0, 0xc00605fa50)
53182026-03-11T20:45:50.971Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed_canceler.go:100 +0x1b3
53192026-03-11T20:45:50.971Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed(0xc002d6f408, {0x50b1338?, 0xc005f399f0?}, {{0xc007003498, 0x2, 0x8}, {0xc0070034a0, 0x2, 0x8}}, {0x189be113eccc04ba, ...}, ...)
53202026-03-11T20:45:50.971Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:515 +0xc98
53212026-03-11T20:45:50.971Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).partialRangeFeed(0xc002d6f408, {0x50b1338, 0xc005f399f0}, 0xc005f399a0, {{0xc007003498, 0x2, 0x8}, {0xc0070034a0, 0x2, 0x8}}, ...)
53222026-03-11T20:45:50.971Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:336 +0x745
53232026-03-11T20:45:50.971Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.1({0x50b1338?, 0xc005f399f0?})
53242026-03-11T20:45:50.971Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:120 +0xa9
53252026-03-11T20:45:50.971Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.Group.GoCtx.2()
53262026-03-11T20:45:50.971Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
53272026-03-11T20:45:50.971Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
53282026-03-11T20:45:50.971Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
53292026-03-11T20:45:50.971Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10485
53302026-03-11T20:45:50.971Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
53312026-03-11T20:45:50.971Z
53322026-03-11T20:45:50.971Zgoroutine 10903 [select]:
53332026-03-11T20:45:50.971Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.(*Watcher).Run(0xc003a03320, {0x50b1338, 0xc00677e780})
53342026-03-11T20:45:50.971Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:319 +0x955
53352026-03-11T20:45:50.971Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.Start.func1({0x50b1300?, 0xc0076b68d0?})
53362026-03-11T20:45:50.971Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:189 +0x2d1
53372026-03-11T20:45:50.971Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
53382026-03-11T20:45:50.971Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
53392026-03-11T20:45:50.971Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
53402026-03-11T20:45:50.971Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
53412026-03-11T20:45:50.971Z
53422026-03-11T20:45:50.971Zgoroutine 10700 [select, 59 minutes]:
53432026-03-11T20:45:50.971Zgithub.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run(0xc002b7dce0, {0x50b1300, 0xc006fd2990})
53442026-03-11T20:45:50.971Z /work/oxidecomputer/cockroach/pkg/internal/client/requestbatcher/batcher.go:475 +0x326
53452026-03-11T20:45:50.971Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
53462026-03-11T20:45:50.971Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
53472026-03-11T20:45:50.971Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
53482026-03-11T20:45:50.971Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
53492026-03-11T20:45:50.971Z
53502026-03-11T20:45:50.971Zgoroutine 11029 [sync.Cond.Wait, 59 minutes]:
53512026-03-11T20:45:50.971Zsync.runtime_notifyListWait(0xc004a57250, 0x3)
53522026-03-11T20:45:50.971Z /work/go/src/runtime/sema.go:569 +0x159
53532026-03-11T20:45:50.971Zsync.(*Cond).Wait(0xc006e177d8?)
53542026-03-11T20:45:50.971Z /work/go/src/sync/cond.go:70 +0x85
53552026-03-11T20:45:50.972Zgithub.com/cockroachdb/cockroach/pkg/sql.(*StmtBuf).CurCmd(0xc004b3b0e8)
53562026-03-11T20:45:50.972Z /work/oxidecomputer/cockroach/pkg/sql/conn_io.go:453 +0xad
53572026-03-11T20:45:50.972Zgithub.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd(0xc004c74008)
53582026-03-11T20:45:50.972Z /work/oxidecomputer/cockroach/pkg/sql/conn_executor.go:1984 +0x66
53592026-03-11T20:45:50.972Zgithub.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0xc004c74008, {0x50b1338, 0xc006cfce60}, 0xd88ce36e?, {0x5400, 0x15000, 0xc00226d680, 0x0, 0x0}, 0xc006486d90)
53602026-03-11T20:45:50.972Z /work/oxidecomputer/cockroach/pkg/sql/conn_executor.go:1958 +0x265
53612026-03-11T20:45:50.972Zgithub.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn(0x222f68?, {0x50b1338?, 0xc006cfce60?}, {0x3?}, {0x5400, 0x15000, 0xc00226d680, 0x0, 0x0}, 0xc006486d90)
53622026-03-11T20:45:50.972Z /work/oxidecomputer/cockroach/pkg/sql/conn_executor.go:763 +0xe6
53632026-03-11T20:45:50.972Zgithub.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1()
53642026-03-11T20:45:50.972Z /work/oxidecomputer/cockroach/pkg/sql/pgwire/conn.go:730 +0x3ee
53652026-03-11T20:45:50.972Zcreated by github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync in goroutine 11011
53662026-03-11T20:45:50.972Z /work/oxidecomputer/cockroach/pkg/sql/pgwire/conn.go:641 +0x247
53672026-03-11T20:45:50.972Z
53682026-03-11T20:45:50.972Zgoroutine 10695 [chan receive, 59 minutes]:
53692026-03-11T20:45:50.972Zgithub.com/cockroachdb/cockroach/pkg/gossip.(*server).start.func3({0xc006e9cc08?, 0xc00684bad0?})
53702026-03-11T20:45:50.972Z /work/oxidecomputer/cockroach/pkg/gossip/server.go:380 +0x48
53712026-03-11T20:45:50.972Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
53722026-03-11T20:45:50.972Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
53732026-03-11T20:45:50.972Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
53742026-03-11T20:45:50.972Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
53752026-03-11T20:45:50.972Z
53762026-03-11T20:45:50.972Zgoroutine 9835 [select, 59 minutes]:
53772026-03-11T20:45:50.973Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc005957208?, 0x0?})
53782026-03-11T20:45:50.973Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
53792026-03-11T20:45:50.974Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
53802026-03-11T20:45:50.974Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
53812026-03-11T20:45:50.974Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
53822026-03-11T20:45:50.974Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
53832026-03-11T20:45:50.974Z
53842026-03-11T20:45:50.974Zgoroutine 10094 [sync.Cond.Wait]:
53852026-03-11T20:45:50.974Zsync.runtime_notifyListWait(0xc001e22250, 0x3797)
53862026-03-11T20:45:50.974Z /work/go/src/runtime/sema.go:569 +0x159
53872026-03-11T20:45:50.974Zsync.(*Cond).Wait(0x3aac600?)
53882026-03-11T20:45:50.974Z /work/go/src/sync/cond.go:70 +0x85
53892026-03-11T20:45:50.974Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583c80})
53902026-03-11T20:45:50.974Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
53912026-03-11T20:45:50.974Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
53922026-03-11T20:45:50.974Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
53932026-03-11T20:45:50.974Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
53942026-03-11T20:45:50.974Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
53952026-03-11T20:45:50.974Z
53962026-03-11T20:45:50.974Zgoroutine 9873 [select]:
53972026-03-11T20:45:50.974Zgithub.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).startTickerLocked.func1()
53982026-03-11T20:45:50.974Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:365 +0xd2
53992026-03-11T20:45:50.974Zcreated by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).startTickerLocked in goroutine 9708
54002026-03-11T20:45:50.975Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:360 +0x6e
54012026-03-11T20:45:50.975Z
54022026-03-11T20:45:50.975Zgoroutine 10128 [select]:
54032026-03-11T20:45:50.980Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).startComputePeriodicMetrics.func1({0x50b1300, 0xc003ebc840})
54042026-03-11T20:45:50.980Z /work/oxidecomputer/cockroach/pkg/server/node.go:748 +0xff
54052026-03-11T20:45:50.980Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
54062026-03-11T20:45:50.980Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
54072026-03-11T20:45:50.980Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
54082026-03-11T20:45:50.980Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
54092026-03-11T20:45:50.980Z
54102026-03-11T20:45:50.980Zgoroutine 10902 [select]:
54112026-03-11T20:45:50.980Zgithub.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigkvsubscriber.(*KVSubscriber).Start.func1({0x50b1300, 0xc0076b68a0})
54122026-03-11T20:45:50.980Z /work/oxidecomputer/cockroach/pkg/spanconfig/spanconfigkvsubscriber/kvsubscriber.go:268 +0x227
54132026-03-11T20:45:50.980Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
54142026-03-11T20:45:50.981Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
54152026-03-11T20:45:50.981Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
54162026-03-11T20:45:50.981Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
54172026-03-11T20:45:50.981Z
54182026-03-11T20:45:50.981Zgoroutine 11059 [select]:
54192026-03-11T20:45:50.981Zgoogle.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc007515c20)
54202026-03-11T20:45:50.981Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1572 +0x137
54212026-03-11T20:45:50.981Zcreated by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 11046
54222026-03-11T20:45:50.981Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:360 +0x180b
54232026-03-11T20:45:50.981Z
54242026-03-11T20:45:50.981Zgoroutine 9963 [chan receive, 59 minutes]:
54252026-03-11T20:45:50.981Zgithub.com/cockroachdb/cmux.muxListener.Accept(...)
54262026-03-11T20:45:50.981Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184
54272026-03-11T20:45:50.981Znet/http.(*Server).Serve(0xc000f4e000, {0x509c4c0, 0xc0059e35f0})
54282026-03-11T20:45:50.981Z /work/go/src/net/http/server.go:3260 +0x33e
54292026-03-11T20:45:50.981Zgithub.com/cockroachdb/cockroach/pkg/server.(*httpServer).start.func3({0xc0013e6908?, 0x0?})
54302026-03-11T20:45:50.981Z /work/oxidecomputer/cockroach/pkg/server/server_http.go:264 +0x126
54312026-03-11T20:45:50.981Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
54322026-03-11T20:45:50.981Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
54332026-03-11T20:45:50.981Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
54342026-03-11T20:45:50.981Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
54352026-03-11T20:45:50.981Z
54362026-03-11T20:45:50.981Zgoroutine 11000 [select]:
54372026-03-11T20:45:50.981Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.(*Watcher).Run(0xc001c245a0, {0x50b1338, 0xc006d1ae60})
54382026-03-11T20:45:50.981Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:319 +0x955
54392026-03-11T20:45:50.981Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.Start.func1({0x50b1300?, 0xc003dacfc0?})
54402026-03-11T20:45:50.981Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:189 +0x2d1
54412026-03-11T20:45:50.981Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
54422026-03-11T20:45:50.981Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
54432026-03-11T20:45:50.981Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
54442026-03-11T20:45:50.981Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
54452026-03-11T20:45:50.981Z
54462026-03-11T20:45:50.981Zgoroutine 11311 [select]:
54472026-03-11T20:45:50.981Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).outputLoop(0xc004f10620, {0x50b1338, 0xc0080c20a0})
54482026-03-11T20:45:50.981Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:310 +0x247
54492026-03-11T20:45:50.981Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).runOutputLoop(0xc004f10620, {0x50b1300, 0xc007e53710}, 0x1b?)
54502026-03-11T20:45:50.981Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:335 +0xba
54512026-03-11T20:45:50.981Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run.func1({0x50b1300?, 0xc007e53710?})
54522026-03-11T20:45:50.982Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:297 +0x46
54532026-03-11T20:45:50.982Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
54542026-03-11T20:45:50.982Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
54552026-03-11T20:45:50.982Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10144
54562026-03-11T20:45:50.982Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
54572026-03-11T20:45:50.982Z
54582026-03-11T20:45:50.982Zgoroutine 10706 [select, 59 minutes]:
54592026-03-11T20:45:50.982Zgoogle.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc006c9ca80)
54602026-03-11T20:45:50.982Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:112 +0x6d
54612026-03-11T20:45:50.982Zcreated by google.golang.org/grpc.newCCBalancerWrapper in goroutine 10657
54622026-03-11T20:45:50.982Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:73 +0x22b
54632026-03-11T20:45:50.982Z
54642026-03-11T20:45:50.982Zgoroutine 10742 [sync.Cond.Wait, 59 minutes]:
54652026-03-11T20:45:50.982Zsync.runtime_notifyListWait(0xc0064c5050, 0x6)
54662026-03-11T20:45:50.982Z /work/go/src/runtime/sema.go:569 +0x159
54672026-03-11T20:45:50.982Zsync.(*Cond).Wait(0x1eda2b2?)
54682026-03-11T20:45:50.982Z /work/go/src/sync/cond.go:70 +0x85
54692026-03-11T20:45:50.982Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd2fc0})
54702026-03-11T20:45:50.982Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
54712026-03-11T20:45:50.982Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
54722026-03-11T20:45:50.982Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
54732026-03-11T20:45:50.982Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
54742026-03-11T20:45:50.982Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
54752026-03-11T20:45:50.982Z
54762026-03-11T20:45:50.982Zgoroutine 9957 [semacquire, 59 minutes]:
54772026-03-11T20:45:50.982Zsync.runtime_Semacquire(0xc004320018?)
54782026-03-11T20:45:50.982Z /work/go/src/runtime/sema.go:62 +0x25
54792026-03-11T20:45:50.982Zsync.(*WaitGroup).Wait(0x44e7a0?)
54802026-03-11T20:45:50.982Z /work/go/src/sync/waitgroup.go:116 +0x48
54812026-03-11T20:45:50.982Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc0006ca000)
54822026-03-11T20:45:50.982Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
54832026-03-11T20:45:50.982Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc0006ca000?, {0x50b1338?, 0xc000afc0f0?}})
54842026-03-11T20:45:50.982Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
54852026-03-11T20:45:50.982Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait({0x50b1300?, 0xc00432c060?}, {0xc004344d78, 0x2, 0x4000000000000000?})
54862026-03-11T20:45:50.982Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:193 +0xff
54872026-03-11T20:45:50.982Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0xc003cd4dc0, {0x50b1300, 0xc00432c060}, 0xc0076599b0)
54882026-03-11T20:45:50.982Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:291 +0x59c
54892026-03-11T20:45:50.983Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).Start.func2({0x50b1300, 0xc007659b00})
54902026-03-11T20:45:50.983Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:215 +0x2f3
54912026-03-11T20:45:50.983Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
54922026-03-11T20:45:50.983Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
54932026-03-11T20:45:50.983Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
54942026-03-11T20:45:50.983Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
54952026-03-11T20:45:50.983Z
54962026-03-11T20:45:50.983Zgoroutine 10311 [select]:
54972026-03-11T20:45:50.983Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents(0xc00356a420, {0x50b1338, 0xc0023407d0}, 0xc003456e10, 0xc007125620)
54982026-03-11T20:45:50.983Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:331 +0xc5
54992026-03-11T20:45:50.983Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func2({0x50b1338?, 0xc0023407d0?})
55002026-03-11T20:45:50.983Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:288 +0x32
55012026-03-11T20:45:50.983Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
55022026-03-11T20:45:50.983Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
55032026-03-11T20:45:50.990Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
55042026-03-11T20:45:50.990Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
55052026-03-11T20:45:50.990Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10323
55062026-03-11T20:45:50.990Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
55072026-03-11T20:45:50.990Z
55082026-03-11T20:45:50.990Zgoroutine 10485 [select, 59 minutes]:
55092026-03-11T20:45:50.990Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1({0x50b1338, 0xc005f399f0})
55102026-03-11T20:45:50.990Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:116 +0x157
55112026-03-11T20:45:50.990Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.Group.GoCtx.func3()
55122026-03-11T20:45:50.990Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
55132026-03-11T20:45:50.990Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
55142026-03-11T20:45:50.990Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
55152026-03-11T20:45:50.990Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10444
55162026-03-11T20:45:50.990Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
55172026-03-11T20:45:50.990Z
55182026-03-11T20:45:50.990Zgoroutine 10018 [select]:
55192026-03-11T20:45:50.990Znet.(*pipe).read(0xc00184c000, {0xc00628a700, 0x700, 0x5?})
55202026-03-11T20:45:50.990Z /work/go/src/net/pipe.go:159 +0x155
55212026-03-11T20:45:50.990Znet.(*pipe).Read(0xc0042b4540?, {0xc00628a700?, 0x90?, 0x780008?})
55222026-03-11T20:45:50.990Z /work/go/src/net/pipe.go:142 +0x1c
55232026-03-11T20:45:50.990Zcrypto/tls.(*atLeastReader).Read(0xc0283c2a20, {0xc00628a700?, 0x0?, 0xc0283c2a20?})
55242026-03-11T20:45:50.990Z /work/go/src/crypto/tls/conn.go:806 +0x3b
55252026-03-11T20:45:50.990Zbytes.(*Buffer).ReadFrom(0xc0042b4630, {0x5081fe0, 0xc0283c2a20})
55262026-03-11T20:45:50.990Z /work/go/src/bytes/buffer.go:211 +0x98
55272026-03-11T20:45:50.990Zcrypto/tls.(*Conn).readFromUntil(0xc0042b4388, {0x7fc12c3c80b0, 0xc00184c000}, 0xc0046e7958?)
55282026-03-11T20:45:50.990Z /work/go/src/crypto/tls/conn.go:828 +0xde
55292026-03-11T20:45:50.990Zcrypto/tls.(*Conn).readRecordOrCCS(0xc0042b4388, 0x0)
55302026-03-11T20:45:50.991Z /work/go/src/crypto/tls/conn.go:626 +0x3cf
55312026-03-11T20:45:50.991Zcrypto/tls.(*Conn).readRecord(...)
55322026-03-11T20:45:50.991Z /work/go/src/crypto/tls/conn.go:588
55332026-03-11T20:45:50.991Zcrypto/tls.(*Conn).Read(0xc0042b4388, {0xc001c40000, 0x8000, 0xc0046e7ca8?})
55342026-03-11T20:45:50.991Z /work/go/src/crypto/tls/conn.go:1370 +0x156
55352026-03-11T20:45:50.991Zbufio.(*Reader).Read(0xc0090ef2c0, {0xc000c858c0, 0x9, 0xc0046e7d30?})
55362026-03-11T20:45:50.991Z /work/go/src/bufio/bufio.go:241 +0x197
55372026-03-11T20:45:50.991Zio.ReadAtLeast({0x507c320, 0xc0090ef2c0}, {0xc000c858c0, 0x9, 0x9}, 0x9)
55382026-03-11T20:45:50.991Z /work/go/src/io/io.go:335 +0x90
55392026-03-11T20:45:50.991Zio.ReadFull(...)
55402026-03-11T20:45:50.991Z /work/go/src/io/io.go:354
55412026-03-11T20:45:50.991Zgolang.org/x/net/http2.readFrameHeader({0xc000c858c0, 0x9, 0x3c27d00?}, {0x507c320?, 0xc0090ef2c0?})
55422026-03-11T20:45:50.991Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x65
55432026-03-11T20:45:50.991Zgolang.org/x/net/http2.(*Framer).ReadFrame(0xc000c85880)
55442026-03-11T20:45:50.991Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:501 +0x85
55452026-03-11T20:45:50.991Zgoogle.golang.org/grpc/internal/transport.(*http2Client).reader(0xc005cbba40)
55462026-03-11T20:45:50.991Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1498 +0x41d
55472026-03-11T20:45:50.991Zcreated by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 9994
55482026-03-11T20:45:50.991Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:365 +0x184d
55492026-03-11T20:45:50.991Z
55502026-03-11T20:45:50.991Zgoroutine 9912 [chan receive, 59 minutes]:
55512026-03-11T20:45:50.991Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc002e5a870?})
55522026-03-11T20:45:50.991Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
55532026-03-11T20:45:50.991Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x0?, 0x0?}}, 0xc0042467a8)
55542026-03-11T20:45:50.991Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
55552026-03-11T20:45:50.991Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
55562026-03-11T20:45:50.991Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
55572026-03-11T20:45:50.991Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
55582026-03-11T20:45:50.992Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
55592026-03-11T20:45:50.992Z
55602026-03-11T20:45:50.992Zgoroutine 10739 [sync.Cond.Wait, 59 minutes]:
55612026-03-11T20:45:50.992Zsync.runtime_notifyListWait(0xc0064c5050, 0x3)
55622026-03-11T20:45:50.992Z /work/go/src/runtime/sema.go:569 +0x159
55632026-03-11T20:45:50.992Zsync.(*Cond).Wait(0xc000f39208?)
55642026-03-11T20:45:50.992Z /work/go/src/sync/cond.go:70 +0x85
55652026-03-11T20:45:50.992Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd2f30})
55662026-03-11T20:45:50.992Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
55672026-03-11T20:45:50.992Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
55682026-03-11T20:45:50.992Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
55692026-03-11T20:45:50.992Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
55702026-03-11T20:45:50.992Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
55712026-03-11T20:45:50.992Z
55722026-03-11T20:45:50.992Zgoroutine 10360 [select, 4 minutes]:
55732026-03-11T20:45:50.992Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/protectedts/ptreconcile.(*Reconciler).run(0xc000c7cf50, {0x50b1300, 0xc00417a7b0}, 0xc003f40400)
55742026-03-11T20:45:50.992Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/protectedts/ptreconcile/reconciler.go:104 +0x265
55752026-03-11T20:45:50.992Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/protectedts/ptreconcile.(*Reconciler).StartReconciler.func1({0x50b1300?, 0xc00417a7b0?})
55762026-03-11T20:45:50.992Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/protectedts/ptreconcile/reconciler.go:77 +0x2e
55772026-03-11T20:45:50.992Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
55782026-03-11T20:45:50.992Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
55792026-03-11T20:45:50.992Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10391
55802026-03-11T20:45:50.992Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
55812026-03-11T20:45:50.993Z
55822026-03-11T20:45:50.993Zgoroutine 10146 [select, 59 minutes]:
55832026-03-11T20:45:50.993Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2({0x50b1300, 0xc009329b00})
55842026-03-11T20:45:50.993Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/queue.go:823 +0xff
55852026-03-11T20:45:50.993Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
55862026-03-11T20:45:50.993Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
55872026-03-11T20:45:50.993Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10124
55882026-03-11T20:45:50.993Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
55892026-03-11T20:45:50.993Z
55902026-03-11T20:45:50.993Zgoroutine 9782 [chan receive, 59 minutes]:
55912026-03-11T20:45:50.993Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc0055722a0?})
55922026-03-11T20:45:50.993Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
55932026-03-11T20:45:50.993Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x100000001?, 0xc00225adc0?}}, 0xc002bebfa8)
55942026-03-11T20:45:50.993Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
55952026-03-11T20:45:50.993Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
55962026-03-11T20:45:50.993Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
55972026-03-11T20:45:50.993Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
55982026-03-11T20:45:50.993Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
55992026-03-11T20:45:50.993Z
56002026-03-11T20:45:50.993Zgoroutine 10462 [semacquire, 59 minutes]:
56012026-03-11T20:45:50.993Zsync.runtime_Semacquire(0xc0034d6510?)
56022026-03-11T20:45:50.993Z /work/go/src/runtime/sema.go:62 +0x25
56032026-03-11T20:45:50.998Zsync.(*WaitGroup).Wait(0x44e7a0?)
56042026-03-11T20:45:50.998Z /work/go/src/sync/waitgroup.go:116 +0x48
56052026-03-11T20:45:50.998Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc0048260c0)
56062026-03-11T20:45:50.998Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
56072026-03-11T20:45:50.998Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc0048260c0?, {0x50b1338?, 0xc006bc6b40?}})
56082026-03-11T20:45:50.998Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
56092026-03-11T20:45:50.998Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed(0xc002d6f408, {0x50b1338?, 0xc0076842d0?}, {0xc007494ba0, 0x1, 0xc007494b10?}, {0xc003283340?, 0xd8aea0?, 0xc0?}, 0x0, ...)
56102026-03-11T20:45:50.998Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:139 +0x6c5
56112026-03-11T20:45:50.998Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*dbAdapter).RangeFeed(0xc0074950b0?, {0x50b1338?, 0xc0076842d0?}, {0xc007494ba0?, 0x50b1300?, 0xc007494c00?}, {0x7070000050b1300?, 0x50b1300?, 0x0?}, 0x0, ...)
56122026-03-11T20:45:50.998Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/db_adapter.go:78 +0x3e
56132026-03-11T20:45:50.998Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func1({0x50b1338?, 0xc0076842d0?})
56142026-03-11T20:45:50.998Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:285 +0x6c
56152026-03-11T20:45:50.999Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
56162026-03-11T20:45:50.999Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
56172026-03-11T20:45:50.999Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
56182026-03-11T20:45:50.999Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
56192026-03-11T20:45:50.999Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10456
56202026-03-11T20:45:50.999Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
56212026-03-11T20:45:50.999Z
56222026-03-11T20:45:50.999Zgoroutine 9060 [select]:
56232026-03-11T20:45:50.999Zgoogle.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0077020f0, 0x1)
56242026-03-11T20:45:50.999Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:407 +0x113
56252026-03-11T20:45:50.999Zgoogle.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00625e660)
56262026-03-11T20:45:50.999Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:534 +0x7b
56272026-03-11T20:45:50.999Zgoogle.golang.org/grpc/internal/transport.NewServerTransport.func2()
56282026-03-11T20:45:50.999Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:332 +0xc6
56292026-03-11T20:45:50.999Zcreated by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 10710
56302026-03-11T20:45:50.999Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:329 +0x184c
56312026-03-11T20:45:50.999Z
56322026-03-11T20:45:50.999Zgoroutine 9862 [select]:
56332026-03-11T20:45:50.999Zgithub.com/cockroachdb/cockroach/pkg/gossip.newInfoStore.func1({0xc001bb4308?, 0x17eee95?})
56342026-03-11T20:45:50.999Z /work/oxidecomputer/cockroach/pkg/gossip/infostore.go:192 +0x74
56352026-03-11T20:45:50.999Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
56362026-03-11T20:45:50.999Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
56372026-03-11T20:45:50.999Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
56382026-03-11T20:45:50.999Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
56392026-03-11T20:45:50.999Z
56402026-03-11T20:45:50.999Zgoroutine 9861 [select, 59 minutes]:
56412026-03-11T20:45:50.999Zgithub.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler.func1()
56422026-03-11T20:45:50.999Z /work/oxidecomputer/cockroach/pkg/security/certificate_manager.go:241 +0x113
56432026-03-11T20:45:50.999Zcreated by github.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler in goroutine 9708
56442026-03-11T20:45:50.999Z /work/oxidecomputer/cockroach/pkg/security/certificate_manager.go:238 +0x7d
56452026-03-11T20:45:50.999Z
56462026-03-11T20:45:50.999Zgoroutine 9978 [select]:
56472026-03-11T20:45:50.999Zgoogle.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc001d0e1a0)
56482026-03-11T20:45:50.999Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1131 +0x205
56492026-03-11T20:45:50.999Zcreated by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 9974
56502026-03-11T20:45:50.999Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:341 +0x188e
56512026-03-11T20:45:50.999Z
56522026-03-11T20:45:51.000Zgoroutine 9886 [select]:
56532026-03-11T20:45:51.000Zgithub.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1()
56542026-03-11T20:45:51.000Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:71 +0xcb
56552026-03-11T20:45:51.000Zcreated by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 9708
56562026-03-11T20:45:51.000Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:66 +0x58
56572026-03-11T20:45:51.000Z
56582026-03-11T20:45:51.000Zgoroutine 9885 [chan receive, 59 minutes]:
56592026-03-11T20:45:51.000Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc00315f8f0?})
56602026-03-11T20:45:51.000Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
56612026-03-11T20:45:51.000Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x41244010041244?, 0x4124401004124401?}}, 0xc003b29fa8)
56622026-03-11T20:45:51.000Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
56632026-03-11T20:45:51.000Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xfb6db6c41e29078a?)
56642026-03-11T20:45:51.000Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
56652026-03-11T20:45:51.000Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
56662026-03-11T20:45:51.000Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
56672026-03-11T20:45:51.000Z
56682026-03-11T20:45:51.000Zgoroutine 10116 [sync.Cond.Wait]:
56692026-03-11T20:45:51.000Zsync.runtime_notifyListWait(0xc001e22250, 0x375d)
56702026-03-11T20:45:51.001Z /work/go/src/runtime/sema.go:569 +0x159
56712026-03-11T20:45:51.001Zsync.(*Cond).Wait(0x3aac600?)
56722026-03-11T20:45:51.001Z /work/go/src/sync/cond.go:70 +0x85
56732026-03-11T20:45:51.001Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc003ebc1b0})
56742026-03-11T20:45:51.001Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
56752026-03-11T20:45:51.001Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
56762026-03-11T20:45:51.001Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
56772026-03-11T20:45:51.001Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
56782026-03-11T20:45:51.001Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
56792026-03-11T20:45:51.001Z
56802026-03-11T20:45:51.001Zgoroutine 11134 [select]:
56812026-03-11T20:45:51.001Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents(0xc002e87a20, {0x50b1338, 0xc0010bcc30}, 0xc0055738c0, 0xc000f5a900)
56822026-03-11T20:45:51.001Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:331 +0xc5
56832026-03-11T20:45:51.001Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func2({0x50b1338?, 0xc0010bcc30?})
56842026-03-11T20:45:51.001Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:288 +0x32
56852026-03-11T20:45:51.001Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
56862026-03-11T20:45:51.001Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
56872026-03-11T20:45:51.001Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
56882026-03-11T20:45:51.002Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
56892026-03-11T20:45:51.002Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10934
56902026-03-11T20:45:51.002Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
56912026-03-11T20:45:51.002Z
56922026-03-11T20:45:51.002Zgoroutine 10042 [sync.Cond.Wait]:
56932026-03-11T20:45:51.002Zsync.runtime_notifyListWait(0xc001e22250, 0x3751)
56942026-03-11T20:45:51.002Z /work/go/src/runtime/sema.go:569 +0x159
56952026-03-11T20:45:51.002Zsync.(*Cond).Wait(0x3aac600?)
56962026-03-11T20:45:51.002Z /work/go/src/sync/cond.go:70 +0x85
56972026-03-11T20:45:51.002Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc0075832c0})
56982026-03-11T20:45:51.002Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
56992026-03-11T20:45:51.002Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
57002026-03-11T20:45:51.002Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
57012026-03-11T20:45:51.002Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
57022026-03-11T20:45:51.002Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
57032026-03-11T20:45:51.009Z
57042026-03-11T20:45:51.009Zgoroutine 11104 [select]:
57052026-03-11T20:45:51.009Zgoogle.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc001435630, {0xc003453e28, 0x5, 0x5})
57062026-03-11T20:45:51.009Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:191 +0x95
57072026-03-11T20:45:51.009Zgoogle.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc001435630, {0xc003453e28?, 0xc0083d6ee8?, 0xc005ebb048?})
57082026-03-11T20:45:51.009Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:171 +0x12d
57092026-03-11T20:45:51.009Zgoogle.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0070e39e0, {0xc003453e28?, 0xc005ebb0c0?, 0xa40865?})
57102026-03-11T20:45:51.009Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:485 +0x2c
57112026-03-11T20:45:51.009Zio.ReadAtLeast({0x5086380, 0xc0070e39e0}, {0xc003453e28, 0x5, 0x5}, 0x5)
57122026-03-11T20:45:51.009Z /work/go/src/io/io.go:335 +0x90
57132026-03-11T20:45:51.009Zio.ReadFull(...)
57142026-03-11T20:45:51.009Z /work/go/src/io/io.go:354
57152026-03-11T20:45:51.009Zgoogle.golang.org/grpc/internal/transport.(*Stream).Read(0xc000a9c360, {0xc003453e28, 0x5, 0x5})
57162026-03-11T20:45:51.009Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:469 +0x96
57172026-03-11T20:45:51.009Zgoogle.golang.org/grpc.(*parser).recvMsg(0xc003453e18, 0x7fffffff)
57182026-03-11T20:45:51.009Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:559 +0x42
57192026-03-11T20:45:51.009Zgoogle.golang.org/grpc.recvAndDecompress(0x60?, 0xc000a9c360, {0x0, 0x0}, 0x7fffffff, 0xc005ebb378, {0x50a8e40, 0x77577e0})
57202026-03-11T20:45:51.009Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:690 +0x5a
57212026-03-11T20:45:51.009Zgoogle.golang.org/grpc.recv(0x50900a0?, {0x7fc12cff4640, 0xc0004ce590}, 0xc005ebb300?, {0x0?, 0x0?}, {0x3f6ba40, 0xc01a6c46c0}, 0x10?, 0xc005ebb378, ...)
57222026-03-11T20:45:51.009Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:756 +0x68
57232026-03-11T20:45:51.009Zgoogle.golang.org/grpc.(*csAttempt).recvMsg(0xc006728e70, {0x3f6ba40, 0xc01a6c46c0}, 0x10?)
57242026-03-11T20:45:51.009Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:983 +0x27e
57252026-03-11T20:45:51.009Zgoogle.golang.org/grpc.(*clientStream).RecvMsg.func1(0x41e1e5?)
57262026-03-11T20:45:51.009Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:834 +0x1f
57272026-03-11T20:45:51.009Zgoogle.golang.org/grpc.(*clientStream).withRetry(0xc002277c20, 0xc005ebb4a8, 0xc005ebb498)
57282026-03-11T20:45:51.009Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:688 +0x2ee
57292026-03-11T20:45:51.010Zgoogle.golang.org/grpc.(*clientStream).RecvMsg(0xc002277c20, {0x3f6ba40?, 0xc01a6c46c0?})
57302026-03-11T20:45:51.010Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:833 +0x107
57312026-03-11T20:45:51.010Zgithub.com/cockroachdb/cockroach/pkg/util/tracing.(*tracingClientStream).RecvMsg(0xc004052260, {0x3f6ba40?, 0xc01a6c46c0?})
57322026-03-11T20:45:51.010Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:441 +0x31
57332026-03-11T20:45:51.010Zgithub.com/cockroachdb/cockroach/pkg/roachpb.(*internalRangeFeedClient).Recv(0xc005553120)
57342026-03-11T20:45:51.010Z /work/oxidecomputer/cockroach/pkg/roachpb/api.pb.go:9289 +0x46
57352026-03-11T20:45:51.010Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed.func3()
57362026-03-11T20:45:51.010Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:516 +0x25
57372026-03-11T20:45:51.010Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*stuckRangeFeedCanceler).do(0xc0014354f0, 0xc005ebba50)
57382026-03-11T20:45:51.010Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed_canceler.go:100 +0x1b3
57392026-03-11T20:45:51.010Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed(0xc003d48008, {0x50b1338?, 0xc001435400?}, {{0xc005f3a710, 0x1, 0x8}, {0xc005f3a718, 0x1, 0x8}}, {0x189be113f3dfcd31, ...}, ...)
57402026-03-11T20:45:51.010Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:515 +0xc98
57412026-03-11T20:45:51.010Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).partialRangeFeed(0xc003d48008, {0x50b1338, 0xc001435400}, 0xc001435360, {{0xc005f3a710, 0x1, 0x8}, {0xc005f3a718, 0x1, 0x8}}, ...)
57422026-03-11T20:45:51.010Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:336 +0x745
57432026-03-11T20:45:51.010Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.1({0x50b1338?, 0xc001435400?})
57442026-03-11T20:45:51.010Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:120 +0xa9
57452026-03-11T20:45:51.010Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.Group.GoCtx.2()
57462026-03-11T20:45:51.010Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
57472026-03-11T20:45:51.011Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
57482026-03-11T20:45:51.011Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
57492026-03-11T20:45:51.011Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 11102
57502026-03-11T20:45:51.011Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
57512026-03-11T20:45:51.011Z
57522026-03-11T20:45:51.011Zgoroutine 9884 [chan receive, 59 minutes]:
57532026-03-11T20:45:51.011Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc007582cf0?})
57542026-03-11T20:45:51.011Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
57552026-03-11T20:45:51.011Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x0?, 0x0?}}, 0xc003b297a8)
57562026-03-11T20:45:51.011Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
57572026-03-11T20:45:51.011Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
57582026-03-11T20:45:51.011Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
57592026-03-11T20:45:51.011Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
57602026-03-11T20:45:51.011Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
57612026-03-11T20:45:51.011Z
57622026-03-11T20:45:51.011Zgoroutine 10998 [select, 59 minutes]:
57632026-03-11T20:45:51.012Zgithub.com/cockroachdb/cockroach/pkg/server.(*Server).startAttemptUpgrade.func1({0x50b1300?, 0xc0032947b0?})
57642026-03-11T20:45:51.012Z /work/oxidecomputer/cockroach/pkg/server/auto_upgrade.go:45 +0x1cf
57652026-03-11T20:45:51.012Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
57662026-03-11T20:45:51.012Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
57672026-03-11T20:45:51.012Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
57682026-03-11T20:45:51.012Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
57692026-03-11T20:45:51.012Z
57702026-03-11T20:45:51.012Zgoroutine 10904 [select, 9 minutes]:
57712026-03-11T20:45:51.012Zgithub.com/cockroachdb/cockroach/pkg/server.(*Server).startSystemLogsGC.func1({0x50b1300, 0xc0076b6930})
57722026-03-11T20:45:51.012Z /work/oxidecomputer/cockroach/pkg/server/server_systemlog_gc.go:179 +0x153
57732026-03-11T20:45:51.012Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
57742026-03-11T20:45:51.012Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
57752026-03-11T20:45:51.012Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
57762026-03-11T20:45:51.012Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
57772026-03-11T20:45:51.012Z
57782026-03-11T20:45:51.012Zgoroutine 10112 [sync.Cond.Wait]:
57792026-03-11T20:45:51.012Zsync.runtime_notifyListWait(0xc001e22250, 0x376e)
57802026-03-11T20:45:51.013Z /work/go/src/runtime/sema.go:569 +0x159
57812026-03-11T20:45:51.013Zsync.(*Cond).Wait(0x3aac600?)
57822026-03-11T20:45:51.013Z /work/go/src/sync/cond.go:70 +0x85
57832026-03-11T20:45:51.013Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc003ebc090})
57842026-03-11T20:45:51.013Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
57852026-03-11T20:45:51.013Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
57862026-03-11T20:45:51.013Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
57872026-03-11T20:45:51.013Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
57882026-03-11T20:45:51.013Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
57892026-03-11T20:45:51.013Z
57902026-03-11T20:45:51.013Zgoroutine 9871 [select]:
57912026-03-11T20:45:51.013Zgithub.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue.func2()
57922026-03-11T20:45:51.013Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:366 +0x85
57932026-03-11T20:45:51.013Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue in goroutine 9708
57942026-03-11T20:45:51.013Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:363 +0x3ed
57952026-03-11T20:45:51.013Z
57962026-03-11T20:45:51.013Zgoroutine 10756 [sync.Cond.Wait, 59 minutes]:
57972026-03-11T20:45:51.013Zsync.runtime_notifyListWait(0xc0064c5050, 0x1d)
57982026-03-11T20:45:51.013Z /work/go/src/runtime/sema.go:569 +0x159
57992026-03-11T20:45:51.013Zsync.(*Cond).Wait(0xc006e3e878?)
58002026-03-11T20:45:51.013Z /work/go/src/sync/cond.go:70 +0x85
58012026-03-11T20:45:51.013Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3260})
58022026-03-11T20:45:51.013Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
58032026-03-11T20:45:51.020Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
58042026-03-11T20:45:51.020Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
58052026-03-11T20:45:51.020Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
58062026-03-11T20:45:51.020Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
58072026-03-11T20:45:51.020Z
58082026-03-11T20:45:51.020Zgoroutine 10757 [sync.Cond.Wait, 59 minutes]:
58092026-03-11T20:45:51.020Zsync.runtime_notifyListWait(0xc0064c5050, 0x22)
58102026-03-11T20:45:51.020Z /work/go/src/runtime/sema.go:569 +0x159
58112026-03-11T20:45:51.020Zsync.(*Cond).Wait(0xc006e3e878?)
58122026-03-11T20:45:51.020Z /work/go/src/sync/cond.go:70 +0x85
58132026-03-11T20:45:51.020Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3290})
58142026-03-11T20:45:51.020Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
58152026-03-11T20:45:51.020Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
58162026-03-11T20:45:51.020Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
58172026-03-11T20:45:51.020Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
58182026-03-11T20:45:51.020Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
58192026-03-11T20:45:51.020Z
58202026-03-11T20:45:51.020Zgoroutine 11076 [select]:
58212026-03-11T20:45:51.020Zgoogle.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc00169cb60)
58222026-03-11T20:45:51.020Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1131 +0x205
58232026-03-11T20:45:51.020Zcreated by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 11048
58242026-03-11T20:45:51.020Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:341 +0x188e
58252026-03-11T20:45:51.020Z
58262026-03-11T20:45:51.020Zgoroutine 9960 [chan receive, 59 minutes]:
58272026-03-11T20:45:51.020Zgithub.com/cockroachdb/cockroach/pkg/util/netutil.MakeServer.func2({0xc0013e6008?, 0x0?})
58282026-03-11T20:45:51.020Z /work/oxidecomputer/cockroach/pkg/util/netutil/net.go:113 +0x49
58292026-03-11T20:45:51.020Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
58302026-03-11T20:45:51.020Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
58312026-03-11T20:45:51.020Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
58322026-03-11T20:45:51.020Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
58332026-03-11T20:45:51.020Z
58342026-03-11T20:45:51.020Zgoroutine 10069 [sync.Cond.Wait]:
58352026-03-11T20:45:51.021Zsync.runtime_notifyListWait(0xc001e22250, 0x377b)
58362026-03-11T20:45:51.021Z /work/go/src/runtime/sema.go:569 +0x159
58372026-03-11T20:45:51.021Zsync.(*Cond).Wait(0x3aac600?)
58382026-03-11T20:45:51.021Z /work/go/src/sync/cond.go:70 +0x85
58392026-03-11T20:45:51.021Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc0075837d0})
58402026-03-11T20:45:51.021Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
58412026-03-11T20:45:51.021Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
58422026-03-11T20:45:51.021Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
58432026-03-11T20:45:51.021Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
58442026-03-11T20:45:51.021Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
58452026-03-11T20:45:51.021Z
58462026-03-11T20:45:51.021Zgoroutine 11149 [select]:
58472026-03-11T20:45:51.021Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).outputLoop(0xc004d52ee0, {0x50b1338, 0xc0023c7040})
58482026-03-11T20:45:51.021Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:310 +0x247
58492026-03-11T20:45:51.021Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).runOutputLoop(0xc004d52ee0, {0x50b1300, 0xc006eea780}, 0x1b?)
58502026-03-11T20:45:51.021Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:335 +0xba
58512026-03-11T20:45:51.021Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run.func1({0x50b1300?, 0xc006eea780?})
58522026-03-11T20:45:51.021Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:297 +0x46
58532026-03-11T20:45:51.021Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
58542026-03-11T20:45:51.021Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
58552026-03-11T20:45:51.021Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10334
58562026-03-11T20:45:51.021Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
58572026-03-11T20:45:51.021Z
58582026-03-11T20:45:51.021Zgoroutine 9883 [chan receive, 59 minutes]:
58592026-03-11T20:45:51.021Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc007582c30?})
58602026-03-11T20:45:51.021Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
58612026-03-11T20:45:51.021Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0xe?, 0xc2648f7c7368d12d?}}, 0xc003b28fa8)
58622026-03-11T20:45:51.021Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
58632026-03-11T20:45:51.021Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
58642026-03-11T20:45:51.021Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
58652026-03-11T20:45:51.021Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
58662026-03-11T20:45:51.021Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
58672026-03-11T20:45:51.021Z
58682026-03-11T20:45:51.021Zgoroutine 10084 [sync.Cond.Wait]:
58692026-03-11T20:45:51.021Zsync.runtime_notifyListWait(0xc001e22250, 0x3783)
58702026-03-11T20:45:51.022Z /work/go/src/runtime/sema.go:569 +0x159
58712026-03-11T20:45:51.022Zsync.(*Cond).Wait(0x3aac600?)
58722026-03-11T20:45:51.022Z /work/go/src/sync/cond.go:70 +0x85
58732026-03-11T20:45:51.022Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583aa0})
58742026-03-11T20:45:51.022Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
58752026-03-11T20:45:51.022Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
58762026-03-11T20:45:51.022Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
58772026-03-11T20:45:51.022Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
58782026-03-11T20:45:51.022Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
58792026-03-11T20:45:51.022Z
58802026-03-11T20:45:51.022Zgoroutine 10783 [sync.Cond.Wait, 59 minutes]:
58812026-03-11T20:45:51.022Zsync.runtime_notifyListWait(0xc0064c5050, 0x46)
58822026-03-11T20:45:51.022Z /work/go/src/runtime/sema.go:569 +0x159
58832026-03-11T20:45:51.022Zsync.(*Cond).Wait(0xc000e2c960?)
58842026-03-11T20:45:51.022Z /work/go/src/sync/cond.go:70 +0x85
58852026-03-11T20:45:51.022Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3770})
58862026-03-11T20:45:51.022Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
58872026-03-11T20:45:51.022Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
58882026-03-11T20:45:51.022Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
58892026-03-11T20:45:51.022Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
58902026-03-11T20:45:51.022Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
58912026-03-11T20:45:51.022Z
58922026-03-11T20:45:51.022Zgoroutine 11102 [select, 59 minutes]:
58932026-03-11T20:45:51.022Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1({0x50b1338, 0xc001435400})
58942026-03-11T20:45:51.022Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:116 +0x157
58952026-03-11T20:45:51.022Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.Group.GoCtx.func3()
58962026-03-11T20:45:51.022Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
58972026-03-11T20:45:51.022Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
58982026-03-11T20:45:51.022Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
58992026-03-11T20:45:51.022Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 11100
59002026-03-11T20:45:51.023Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
59012026-03-11T20:45:51.023Z
59022026-03-11T20:45:51.023Zgoroutine 10148 [select, 59 minutes]:
59032026-03-11T20:45:51.031Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2({0x50b1300, 0xc009329b60})
59042026-03-11T20:45:51.031Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/queue.go:823 +0xff
59052026-03-11T20:45:51.031Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
59062026-03-11T20:45:51.032Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
59072026-03-11T20:45:51.032Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10124
59082026-03-11T20:45:51.032Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
59092026-03-11T20:45:51.032Z
59102026-03-11T20:45:51.032Zgoroutine 9779 [chan receive, 59 minutes]:
59112026-03-11T20:45:51.032Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc005572150?})
59122026-03-11T20:45:51.032Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
59132026-03-11T20:45:51.032Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0xc00350d9e0?, 0xc00350da20?}}, 0xc002beafa8)
59142026-03-11T20:45:51.032Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
59152026-03-11T20:45:51.032Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
59162026-03-11T20:45:51.032Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
59172026-03-11T20:45:51.032Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
59182026-03-11T20:45:51.032Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
59192026-03-11T20:45:51.032Z
59202026-03-11T20:45:51.032Zgoroutine 10430 [select, 59 minutes]:
59212026-03-11T20:45:51.032Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1({0x50b1338, 0xc007611e00})
59222026-03-11T20:45:51.032Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:116 +0x157
59232026-03-11T20:45:51.032Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.Group.GoCtx.func3()
59242026-03-11T20:45:51.032Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
59252026-03-11T20:45:51.032Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
59262026-03-11T20:45:51.032Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
59272026-03-11T20:45:51.033Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10428
59282026-03-11T20:45:51.033Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
59292026-03-11T20:45:51.033Z
59302026-03-11T20:45:51.033Zgoroutine 10735 [sync.Cond.Wait, 59 minutes]:
59312026-03-11T20:45:51.033Zsync.runtime_notifyListWait(0xc0064c5050, 0x8)
59322026-03-11T20:45:51.033Z /work/go/src/runtime/sema.go:569 +0x159
59332026-03-11T20:45:51.033Zsync.(*Cond).Wait(0x10000c006b42008?)
59342026-03-11T20:45:51.033Z /work/go/src/sync/cond.go:70 +0x85
59352026-03-11T20:45:51.033Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd2e70})
59362026-03-11T20:45:51.033Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
59372026-03-11T20:45:51.033Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
59382026-03-11T20:45:51.033Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
59392026-03-11T20:45:51.033Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
59402026-03-11T20:45:51.033Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
59412026-03-11T20:45:51.033Z
59422026-03-11T20:45:51.033Zgoroutine 9803 [select]:
59432026-03-11T20:45:51.033Zgithub.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
59442026-03-11T20:45:51.033Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:439 +0x1c9
59452026-03-11T20:45:51.033Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 9708
59462026-03-11T20:45:51.033Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:412 +0x4f
59472026-03-11T20:45:51.033Z
59482026-03-11T20:45:51.033Zgoroutine 9946 [select, 59 minutes]:
59492026-03-11T20:45:51.033Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc004cb0608?, 0x0?})
59502026-03-11T20:45:51.033Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
59512026-03-11T20:45:51.033Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
59522026-03-11T20:45:51.033Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
59532026-03-11T20:45:51.033Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
59542026-03-11T20:45:51.033Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
59552026-03-11T20:45:51.033Z
59562026-03-11T20:45:51.033Zgoroutine 10433 [chan receive, 59 minutes]:
59572026-03-11T20:45:51.033Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).rangeFeedWithRangeID(0xc001ded408, 0x3f3de40?, 0xc005dffc20, {0x7fc12cefd140, 0xc005dbb5e0})
59582026-03-11T20:45:51.033Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:249 +0x76b
59592026-03-11T20:45:51.033Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed(...)
59602026-03-11T20:45:51.033Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:149
59612026-03-11T20:45:51.033Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).RangeFeed(0xc000f39208, 0xc005dffc20, {0x50e5b30, 0xc005dbb5e0})
59622026-03-11T20:45:51.033Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:3082 +0x16f
59632026-03-11T20:45:51.033Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).RangeFeed(0xc001eb4380, 0xc005dffc20, {0x50e5b30, 0xc005dbb5e0})
59642026-03-11T20:45:51.033Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/stores.go:216 +0x19a
59652026-03-11T20:45:51.034Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed(0x3f3de40?, 0xc00619a080?, {0x50e5b30, 0xc005dbb5e0})
59662026-03-11T20:45:51.034Z /work/oxidecomputer/cockroach/pkg/server/node.go:1232 +0x28
59672026-03-11T20:45:51.034Zgithub.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed.func1()
59682026-03-11T20:45:51.034Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:753 +0xa7
59692026-03-11T20:45:51.034Zcreated by github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed in goroutine 10432
59702026-03-11T20:45:51.034Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:750 +0x17c
59712026-03-11T20:45:51.034Z
59722026-03-11T20:45:51.034Zgoroutine 10003 [select]:
59732026-03-11T20:45:51.034Zgithub.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run(0xc002c1a580, {0x50b1300, 0xc007b27620})
59742026-03-11T20:45:51.034Z /work/oxidecomputer/cockroach/pkg/internal/client/requestbatcher/batcher.go:475 +0x326
59752026-03-11T20:45:51.034Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
59762026-03-11T20:45:51.034Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
59772026-03-11T20:45:51.034Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
59782026-03-11T20:45:51.034Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
59792026-03-11T20:45:51.034Z
59802026-03-11T20:45:51.034Zgoroutine 10041 [sync.Cond.Wait]:
59812026-03-11T20:45:51.035Zsync.runtime_notifyListWait(0xc001e22250, 0x3756)
59822026-03-11T20:45:51.035Z /work/go/src/runtime/sema.go:569 +0x159
59832026-03-11T20:45:51.035Zsync.(*Cond).Wait(0x3aac600?)
59842026-03-11T20:45:51.035Z /work/go/src/sync/cond.go:70 +0x85
59852026-03-11T20:45:51.035Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583290})
59862026-03-11T20:45:51.035Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
59872026-03-11T20:45:51.035Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
59882026-03-11T20:45:51.035Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
59892026-03-11T20:45:51.035Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
59902026-03-11T20:45:51.035Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
59912026-03-11T20:45:51.035Z
59922026-03-11T20:45:51.035Zgoroutine 10472 [select]:
59932026-03-11T20:45:51.035Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents(0xc007576000, {0x50b1338, 0xc003a62af0}, 0xc006c0a870, 0xc009224120)
59942026-03-11T20:45:51.035Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:331 +0xc5
59952026-03-11T20:45:51.035Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func2({0x50b1338?, 0xc003a62af0?})
59962026-03-11T20:45:51.035Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:288 +0x32
59972026-03-11T20:45:51.035Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
59982026-03-11T20:45:51.035Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
59992026-03-11T20:45:51.035Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
60002026-03-11T20:45:51.035Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
60012026-03-11T20:45:51.035Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 8998
60022026-03-11T20:45:51.035Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
60032026-03-11T20:45:51.043Z
60042026-03-11T20:45:51.043Zgoroutine 9940 [select, 59 minutes]:
60052026-03-11T20:45:51.043Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc00167f208?, 0x0?})
60062026-03-11T20:45:51.043Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
60072026-03-11T20:45:51.043Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
60082026-03-11T20:45:51.043Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
60092026-03-11T20:45:51.043Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
60102026-03-11T20:45:51.043Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
60112026-03-11T20:45:51.043Z
60122026-03-11T20:45:51.043Zgoroutine 11142 [select]:
60132026-03-11T20:45:51.044Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).outputLoop(0xc004d52540, {0x50b1338, 0xc004c69bd0})
60142026-03-11T20:45:51.044Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:310 +0x247
60152026-03-11T20:45:51.044Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).runOutputLoop(0xc004d52540, {0x50b1300, 0xc0020161e0}, 0x1b?)
60162026-03-11T20:45:51.044Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:335 +0xba
60172026-03-11T20:45:51.044Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run.func1({0x50b1300?, 0xc0020161e0?})
60182026-03-11T20:45:51.044Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:297 +0x46
60192026-03-11T20:45:51.044Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
60202026-03-11T20:45:51.044Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
60212026-03-11T20:45:51.044Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10498
60222026-03-11T20:45:51.044Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
60232026-03-11T20:45:51.044Z
60242026-03-11T20:45:51.044Zgoroutine 11101 [select]:
60252026-03-11T20:45:51.044Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents(0xc003b48c60, {0x50b1338, 0xc001435310}, 0xc003dacff0, 0xc000eca8a0)
60262026-03-11T20:45:51.044Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:331 +0xc5
60272026-03-11T20:45:51.044Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func2({0x50b1338?, 0xc001435310?})
60282026-03-11T20:45:51.044Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:288 +0x32
60292026-03-11T20:45:51.044Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
60302026-03-11T20:45:51.044Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
60312026-03-11T20:45:51.044Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
60322026-03-11T20:45:51.044Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
60332026-03-11T20:45:51.044Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 11001
60342026-03-11T20:45:51.044Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
60352026-03-11T20:45:51.044Z
60362026-03-11T20:45:51.044Zgoroutine 9833 [select, 59 minutes]:
60372026-03-11T20:45:51.044Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc005956c08?, 0x0?})
60382026-03-11T20:45:51.045Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
60392026-03-11T20:45:51.045Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
60402026-03-11T20:45:51.045Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
60412026-03-11T20:45:51.045Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
60422026-03-11T20:45:51.045Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
60432026-03-11T20:45:51.045Z
60442026-03-11T20:45:51.045Zgoroutine 10777 [sync.Cond.Wait, 59 minutes]:
60452026-03-11T20:45:51.045Zsync.runtime_notifyListWait(0xc0064c5050, 0x3b)
60462026-03-11T20:45:51.045Z /work/go/src/runtime/sema.go:569 +0x159
60472026-03-11T20:45:51.045Zsync.(*Cond).Wait(0xc0031205d0?)
60482026-03-11T20:45:51.045Z /work/go/src/sync/cond.go:70 +0x85
60492026-03-11T20:45:51.045Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3650})
60502026-03-11T20:45:51.045Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
60512026-03-11T20:45:51.045Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
60522026-03-11T20:45:51.045Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
60532026-03-11T20:45:51.045Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
60542026-03-11T20:45:51.045Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
60552026-03-11T20:45:51.045Z
60562026-03-11T20:45:51.045Zgoroutine 9875 [chan receive, 59 minutes]:
60572026-03-11T20:45:51.045Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc005572b40?})
60582026-03-11T20:45:51.045Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
60592026-03-11T20:45:51.045Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x0?, 0x0?}}, 0xc000e02fa8)
60602026-03-11T20:45:51.045Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
60612026-03-11T20:45:51.045Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
60622026-03-11T20:45:51.045Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
60632026-03-11T20:45:51.045Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
60642026-03-11T20:45:51.045Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
60652026-03-11T20:45:51.045Z
60662026-03-11T20:45:51.045Zgoroutine 10651 [chan receive, 59 minutes]:
60672026-03-11T20:45:51.045Zgithub.com/cockroachdb/cockroach/pkg/util/netutil.MakeServer.func2({0xc006c90c08?, 0x0?})
60682026-03-11T20:45:51.045Z /work/oxidecomputer/cockroach/pkg/util/netutil/net.go:113 +0x49
60692026-03-11T20:45:51.045Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
60702026-03-11T20:45:51.045Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
60712026-03-11T20:45:51.045Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10590
60722026-03-11T20:45:51.045Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
60732026-03-11T20:45:51.046Z
60742026-03-11T20:45:51.046Zgoroutine 10120 [select, 59 minutes]:
60752026-03-11T20:45:51.046Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Start.func3({0xc0011ccf08?, 0x0?})
60762026-03-11T20:45:51.046Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:1978 +0xb2
60772026-03-11T20:45:51.046Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
60782026-03-11T20:45:51.046Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
60792026-03-11T20:45:51.046Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
60802026-03-11T20:45:51.046Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
60812026-03-11T20:45:51.046Z
60822026-03-11T20:45:51.046Zgoroutine 10121 [select]:
60832026-03-11T20:45:51.046Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).startGossip.func3({0x50b1300, 0xc003ebc420})
60842026-03-11T20:45:51.046Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:2152 +0x225
60852026-03-11T20:45:51.046Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
60862026-03-11T20:45:51.046Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
60872026-03-11T20:45:51.046Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
60882026-03-11T20:45:51.046Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
60892026-03-11T20:45:51.046Z
60902026-03-11T20:45:51.046Zgoroutine 10122 [select, 1 minutes]:
60912026-03-11T20:45:51.046Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).startGossip.func3({0x50b1300, 0xc003ebc450})
60922026-03-11T20:45:51.047Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:2152 +0x225
60932026-03-11T20:45:51.047Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
60942026-03-11T20:45:51.047Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
60952026-03-11T20:45:51.047Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
60962026-03-11T20:45:51.047Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
60972026-03-11T20:45:51.047Z
60982026-03-11T20:45:51.047Zgoroutine 9880 [chan receive, 59 minutes]:
60992026-03-11T20:45:51.047Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc005572d50?})
61002026-03-11T20:45:51.047Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
61012026-03-11T20:45:51.047Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x1?, 0x1?}}, 0xc003b277a8)
61022026-03-11T20:45:51.047Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
61032026-03-11T20:45:51.052Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
61042026-03-11T20:45:51.052Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
61052026-03-11T20:45:51.052Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
61062026-03-11T20:45:51.052Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
61072026-03-11T20:45:51.052Z
61082026-03-11T20:45:51.052Zgoroutine 9844 [select, 59 minutes]:
61092026-03-11T20:45:51.052Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc006b65808?, 0xc001af6ac0?})
61102026-03-11T20:45:51.052Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
61112026-03-11T20:45:51.052Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
61122026-03-11T20:45:51.052Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
61132026-03-11T20:45:51.052Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
61142026-03-11T20:45:51.052Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
61152026-03-11T20:45:51.052Z
61162026-03-11T20:45:51.052Zgoroutine 10803 [sync.Cond.Wait, 59 minutes]:
61172026-03-11T20:45:51.052Zsync.runtime_notifyListWait(0xc0064c5050, 0x48)
61182026-03-11T20:45:51.052Z /work/go/src/runtime/sema.go:569 +0x159
61192026-03-11T20:45:51.052Zsync.(*Cond).Wait(0x0?)
61202026-03-11T20:45:51.052Z /work/go/src/sync/cond.go:70 +0x85
61212026-03-11T20:45:51.052Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3b30})
61222026-03-11T20:45:51.052Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
61232026-03-11T20:45:51.052Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
61242026-03-11T20:45:51.052Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
61252026-03-11T20:45:51.052Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
61262026-03-11T20:45:51.052Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
61272026-03-11T20:45:51.052Z
61282026-03-11T20:45:51.052Zgoroutine 10058 [sync.Cond.Wait]:
61292026-03-11T20:45:51.052Zsync.runtime_notifyListWait(0xc001e22250, 0x3777)
61302026-03-11T20:45:51.052Z /work/go/src/runtime/sema.go:569 +0x159
61312026-03-11T20:45:51.052Zsync.(*Cond).Wait(0x3aac600?)
61322026-03-11T20:45:51.052Z /work/go/src/sync/cond.go:70 +0x85
61332026-03-11T20:45:51.052Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc0075835c0})
61342026-03-11T20:45:51.053Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
61352026-03-11T20:45:51.053Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
61362026-03-11T20:45:51.053Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
61372026-03-11T20:45:51.053Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
61382026-03-11T20:45:51.053Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
61392026-03-11T20:45:51.053Z
61402026-03-11T20:45:51.053Zgoroutine 10057 [sync.Cond.Wait]:
61412026-03-11T20:45:51.053Zsync.runtime_notifyListWait(0xc001e22250, 0x3771)
61422026-03-11T20:45:51.053Z /work/go/src/runtime/sema.go:569 +0x159
61432026-03-11T20:45:51.053Zsync.(*Cond).Wait(0x3aac600?)
61442026-03-11T20:45:51.053Z /work/go/src/sync/cond.go:70 +0x85
61452026-03-11T20:45:51.053Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583590})
61462026-03-11T20:45:51.053Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
61472026-03-11T20:45:51.053Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
61482026-03-11T20:45:51.053Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
61492026-03-11T20:45:51.053Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
61502026-03-11T20:45:51.053Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
61512026-03-11T20:45:51.053Z
61522026-03-11T20:45:51.053Zgoroutine 10132 [select, 1 minutes]:
61532026-03-11T20:45:51.053Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/reports.(*Reporter).Start.func2({0x50b1300?, 0xc003ebca50?})
61542026-03-11T20:45:51.053Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/reports/reporter.go:165 +0x305
61552026-03-11T20:45:51.053Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
61562026-03-11T20:45:51.053Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
61572026-03-11T20:45:51.053Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
61582026-03-11T20:45:51.053Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
61592026-03-11T20:45:51.053Z
61602026-03-11T20:45:51.053Zgoroutine 11105 [select, 59 minutes]:
61612026-03-11T20:45:51.053Zgoogle.golang.org/grpc.newClientStreamWithParams.func4()
61622026-03-11T20:45:51.053Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:341 +0x8c
61632026-03-11T20:45:51.053Zcreated by google.golang.org/grpc.newClientStreamWithParams in goroutine 11104
61642026-03-11T20:45:51.053Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:340 +0xb2a
61652026-03-11T20:45:51.054Z
61662026-03-11T20:45:51.054Zgoroutine 9857 [chan receive, 59 minutes]:
61672026-03-11T20:45:51.054Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc007582ab0?})
61682026-03-11T20:45:51.054Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
61692026-03-11T20:45:51.054Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0xc00350dc20?, 0xc00350dc60?}}, 0xc000e047a8)
61702026-03-11T20:45:51.054Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
61712026-03-11T20:45:51.054Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
61722026-03-11T20:45:51.054Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
61732026-03-11T20:45:51.054Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
61742026-03-11T20:45:51.054Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
61752026-03-11T20:45:51.054Z
61762026-03-11T20:45:51.054Zgoroutine 10698 [select, 59 minutes]:
61772026-03-11T20:45:51.054Zgithub.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run(0xc002b7da20, {0x50b1300, 0xc006fd28d0})
61782026-03-11T20:45:51.054Z /work/oxidecomputer/cockroach/pkg/internal/client/requestbatcher/batcher.go:475 +0x326
61792026-03-11T20:45:51.054Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
61802026-03-11T20:45:51.054Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
61812026-03-11T20:45:51.054Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
61822026-03-11T20:45:51.054Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
61832026-03-11T20:45:51.054Z
61842026-03-11T20:45:51.054Zgoroutine 10916 [select, 59 minutes]:
61852026-03-11T20:45:51.055Zgithub.com/cockroachdb/cockroach/pkg/sql/catalog/lease.(*Manager).RefreshLeases.func1({0x50b1300, 0xc00067d860})
61862026-03-11T20:45:51.055Z /work/oxidecomputer/cockroach/pkg/sql/catalog/lease/lease.go:1041 +0x105
61872026-03-11T20:45:51.055Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
61882026-03-11T20:45:51.055Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
61892026-03-11T20:45:51.055Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
61902026-03-11T20:45:51.055Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
61912026-03-11T20:45:51.055Z
61922026-03-11T20:45:51.055Zgoroutine 9950 [select, 59 minutes]:
61932026-03-11T20:45:51.055Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc004cb1508?, 0x0?})
61942026-03-11T20:45:51.055Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
61952026-03-11T20:45:51.055Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
61962026-03-11T20:45:51.055Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
61972026-03-11T20:45:51.055Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
61982026-03-11T20:45:51.055Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
61992026-03-11T20:45:51.055Z
62002026-03-11T20:45:51.055Zgoroutine 10933 [select]:
62012026-03-11T20:45:51.055Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.(*Watcher).Run(0xc0017ea6c0, {0x50b1338, 0xc007703c70})
62022026-03-11T20:45:51.055Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:319 +0x955
62032026-03-11T20:45:51.062Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.Start.func1({0x50b1300?, 0xc005573890?})
62042026-03-11T20:45:51.062Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:189 +0x2d1
62052026-03-11T20:45:51.062Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
62062026-03-11T20:45:51.062Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
62072026-03-11T20:45:51.062Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
62082026-03-11T20:45:51.062Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
62092026-03-11T20:45:51.062Z
62102026-03-11T20:45:51.062Zgoroutine 10037 [sync.Cond.Wait]:
62112026-03-11T20:45:51.062Zsync.runtime_notifyListWait(0xc001e22250, 0x374c)
62122026-03-11T20:45:51.062Z /work/go/src/runtime/sema.go:569 +0x159
62132026-03-11T20:45:51.062Zsync.(*Cond).Wait(0x3aac600?)
62142026-03-11T20:45:51.062Z /work/go/src/sync/cond.go:70 +0x85
62152026-03-11T20:45:51.062Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc0075831d0})
62162026-03-11T20:45:51.062Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
62172026-03-11T20:45:51.062Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
62182026-03-11T20:45:51.062Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
62192026-03-11T20:45:51.062Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
62202026-03-11T20:45:51.062Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
62212026-03-11T20:45:51.063Z
62222026-03-11T20:45:51.063Zgoroutine 9070 [select, 59 minutes]:
62232026-03-11T20:45:51.063Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2({0x50b1300, 0xc0076b6780})
62242026-03-11T20:45:51.063Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/queue.go:823 +0xff
62252026-03-11T20:45:51.063Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
62262026-03-11T20:45:51.063Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
62272026-03-11T20:45:51.063Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10821
62282026-03-11T20:45:51.063Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
62292026-03-11T20:45:51.063Z
62302026-03-11T20:45:51.063Zgoroutine 10793 [sync.Cond.Wait, 59 minutes]:
62312026-03-11T20:45:51.063Zsync.runtime_notifyListWait(0xc0064c5050, 0x5e)
62322026-03-11T20:45:51.063Z /work/go/src/runtime/sema.go:569 +0x159
62332026-03-11T20:45:51.063Zsync.(*Cond).Wait(0x50b1300?)
62342026-03-11T20:45:51.063Z /work/go/src/sync/cond.go:70 +0x85
62352026-03-11T20:45:51.063Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3950})
62362026-03-11T20:45:51.063Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
62372026-03-11T20:45:51.063Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
62382026-03-11T20:45:51.063Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
62392026-03-11T20:45:51.063Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
62402026-03-11T20:45:51.063Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
62412026-03-11T20:45:51.063Z
62422026-03-11T20:45:51.063Zgoroutine 10816 [select]:
62432026-03-11T20:45:51.063Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).coalescedHeartbeatsLoop(0xc006fce008, {0x50b1300, 0xc006fd3da0})
62442026-03-11T20:45:51.063Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store_raft.go:678 +0xec
62452026-03-11T20:45:51.063Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
62462026-03-11T20:45:51.063Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
62472026-03-11T20:45:51.063Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
62482026-03-11T20:45:51.063Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
62492026-03-11T20:45:51.063Z
62502026-03-11T20:45:51.063Zgoroutine 10815 [select]:
62512026-03-11T20:45:51.063Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).raftTickLoop(0xc006fce008, {0x2a3f?, 0x0?})
62522026-03-11T20:45:51.063Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store_raft.go:618 +0x145
62532026-03-11T20:45:51.063Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
62542026-03-11T20:45:51.063Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
62552026-03-11T20:45:51.063Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
62562026-03-11T20:45:51.063Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
62572026-03-11T20:45:51.063Z
62582026-03-11T20:45:51.063Zgoroutine 10147 [select, 59 minutes]:
62592026-03-11T20:45:51.063Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2({0x50b1300, 0xc009329b30})
62602026-03-11T20:45:51.064Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/queue.go:823 +0xff
62612026-03-11T20:45:51.064Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
62622026-03-11T20:45:51.064Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
62632026-03-11T20:45:51.064Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10124
62642026-03-11T20:45:51.064Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
62652026-03-11T20:45:51.064Z
62662026-03-11T20:45:51.064Zgoroutine 9881 [chan receive, 59 minutes]:
62672026-03-11T20:45:51.064Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc005572c30?})
62682026-03-11T20:45:51.064Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
62692026-03-11T20:45:51.064Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0xaaaa2492492aaaa2?, 0x2493c4f262492492?}}, 0xc003b27fa8)
62702026-03-11T20:45:51.064Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
62712026-03-11T20:45:51.064Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x92aaaa2492492aaa?)
62722026-03-11T20:45:51.064Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
62732026-03-11T20:45:51.064Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
62742026-03-11T20:45:51.064Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
62752026-03-11T20:45:51.064Z
62762026-03-11T20:45:51.064Zgoroutine 9877 [chan receive, 59 minutes]:
62772026-03-11T20:45:51.064Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc005572a20?})
62782026-03-11T20:45:51.064Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
62792026-03-11T20:45:51.064Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x100000001024180?, 0x50f1d80?}}, 0xc000e057a8)
62802026-03-11T20:45:51.064Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
62812026-03-11T20:45:51.064Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
62822026-03-11T20:45:51.064Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
62832026-03-11T20:45:51.064Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
62842026-03-11T20:45:51.064Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
62852026-03-11T20:45:51.064Z
62862026-03-11T20:45:51.064Zgoroutine 9882 [chan receive, 59 minutes]:
62872026-03-11T20:45:51.064Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc007582c90?})
62882026-03-11T20:45:51.064Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
62892026-03-11T20:45:51.064Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x0?, 0x0?}}, 0xc003b287a8)
62902026-03-11T20:45:51.064Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
62912026-03-11T20:45:51.064Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
62922026-03-11T20:45:51.064Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
62932026-03-11T20:45:51.064Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
62942026-03-11T20:45:51.065Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
62952026-03-11T20:45:51.065Z
62962026-03-11T20:45:51.065Zgoroutine 10813 [sync.Cond.Wait, 59 minutes]:
62972026-03-11T20:45:51.065Zsync.runtime_notifyListWait(0xc0064c5050, 0x57)
62982026-03-11T20:45:51.065Z /work/go/src/runtime/sema.go:569 +0x159
62992026-03-11T20:45:51.065Zsync.(*Cond).Wait(0x0?)
63002026-03-11T20:45:51.065Z /work/go/src/sync/cond.go:70 +0x85
63012026-03-11T20:45:51.065Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3d10})
63022026-03-11T20:45:51.065Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
63032026-03-11T20:45:51.069Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
63042026-03-11T20:45:51.069Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
63052026-03-11T20:45:51.069Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
63062026-03-11T20:45:51.069Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
63072026-03-11T20:45:51.069Z
63082026-03-11T20:45:51.069Zgoroutine 10111 [sync.Cond.Wait]:
63092026-03-11T20:45:51.069Zsync.runtime_notifyListWait(0xc001e22250, 0x3767)
63102026-03-11T20:45:51.069Z /work/go/src/runtime/sema.go:569 +0x159
63112026-03-11T20:45:51.069Zsync.(*Cond).Wait(0x3aac600?)
63122026-03-11T20:45:51.070Z /work/go/src/sync/cond.go:70 +0x85
63132026-03-11T20:45:51.070Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc003ebc060})
63142026-03-11T20:45:51.070Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
63152026-03-11T20:45:51.070Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
63162026-03-11T20:45:51.070Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
63172026-03-11T20:45:51.070Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
63182026-03-11T20:45:51.070Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
63192026-03-11T20:45:51.070Z
63202026-03-11T20:45:51.070Zgoroutine 11047 [select]:
63212026-03-11T20:45:51.070Zgithub.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc003efa200, {0x50b1300, 0xc0043e0a80}, 0xc003f0c820, {0xc007bda030, 0xf}, 0xc002cfb560)
63222026-03-11T20:45:51.070Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:1462 +0x27b
63232026-03-11T20:45:51.070Zgithub.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1({0x50b1300, 0xc0043e0a80})
63242026-03-11T20:45:51.070Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:1380 +0x85
63252026-03-11T20:45:51.070Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
63262026-03-11T20:45:51.070Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
63272026-03-11T20:45:51.070Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
63282026-03-11T20:45:51.070Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
63292026-03-11T20:45:51.070Z
63302026-03-11T20:45:51.070Zgoroutine 10060 [sync.Cond.Wait]:
63312026-03-11T20:45:51.070Zsync.runtime_notifyListWait(0xc001e22250, 0x376f)
63322026-03-11T20:45:51.070Z /work/go/src/runtime/sema.go:569 +0x159
63332026-03-11T20:45:51.070Zsync.(*Cond).Wait(0x3aac600?)
63342026-03-11T20:45:51.070Z /work/go/src/sync/cond.go:70 +0x85
63352026-03-11T20:45:51.070Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583620})
63362026-03-11T20:45:51.070Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
63372026-03-11T20:45:51.070Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
63382026-03-11T20:45:51.070Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
63392026-03-11T20:45:51.070Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
63402026-03-11T20:45:51.070Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
63412026-03-11T20:45:51.070Z
63422026-03-11T20:45:51.070Zgoroutine 10814 [semacquire, 59 minutes]:
63432026-03-11T20:45:51.070Zsync.runtime_Semacquire(0x0?)
63442026-03-11T20:45:51.070Z /work/go/src/runtime/sema.go:62 +0x25
63452026-03-11T20:45:51.070Zsync.(*WaitGroup).Wait(0x0?)
63462026-03-11T20:45:51.070Z /work/go/src/sync/waitgroup.go:116 +0x48
63472026-03-11T20:45:51.070Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Wait(...)
63482026-03-11T20:45:51.070Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:236
63492026-03-11T20:45:51.071Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
63502026-03-11T20:45:51.071Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
63512026-03-11T20:45:51.071Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
63522026-03-11T20:45:51.071Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
63532026-03-11T20:45:51.071Z
63542026-03-11T20:45:51.071Zgoroutine 9959 [select, 59 minutes]:
63552026-03-11T20:45:51.071Zgithub.com/cockroachdb/cockroach/pkg/util/hlc.(*Clock).StartMonitoringForwardClockJumps.func1()
63562026-03-11T20:45:51.071Z /work/oxidecomputer/cockroach/pkg/util/hlc/hlc.go:263 +0x148
63572026-03-11T20:45:51.071Zcreated by github.com/cockroachdb/cockroach/pkg/util/hlc.(*Clock).StartMonitoringForwardClockJumps in goroutine 9708
63582026-03-11T20:45:51.071Z /work/oxidecomputer/cockroach/pkg/util/hlc/hlc.go:255 +0x106
63592026-03-11T20:45:51.071Z
63602026-03-11T20:45:51.071Zgoroutine 10722 [sync.Cond.Wait, 59 minutes]:
63612026-03-11T20:45:51.071Zsync.runtime_notifyListWait(0xc0064c5050, 0x26)
63622026-03-11T20:45:51.071Z /work/go/src/runtime/sema.go:569 +0x159
63632026-03-11T20:45:51.071Zsync.(*Cond).Wait(0x40?)
63642026-03-11T20:45:51.071Z /work/go/src/sync/cond.go:70 +0x85
63652026-03-11T20:45:51.071Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd2c00})
63662026-03-11T20:45:51.071Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
63672026-03-11T20:45:51.071Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
63682026-03-11T20:45:51.071Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
63692026-03-11T20:45:51.071Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
63702026-03-11T20:45:51.071Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
63712026-03-11T20:45:51.071Z
63722026-03-11T20:45:51.071Zgoroutine 9878 [chan receive, 59 minutes]:
63732026-03-11T20:45:51.071Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc005572c90?})
63742026-03-11T20:45:51.071Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
63752026-03-11T20:45:51.071Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x1024180?, 0xc00433cd10?}}, 0xc000e04fa8)
63762026-03-11T20:45:51.071Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
63772026-03-11T20:45:51.071Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
63782026-03-11T20:45:51.071Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
63792026-03-11T20:45:51.071Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
63802026-03-11T20:45:51.071Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
63812026-03-11T20:45:51.071Z
63822026-03-11T20:45:51.071Zgoroutine 10081 [sync.Cond.Wait]:
63832026-03-11T20:45:51.071Zsync.runtime_notifyListWait(0xc001e22250, 0x377c)
63842026-03-11T20:45:51.071Z /work/go/src/runtime/sema.go:569 +0x159
63852026-03-11T20:45:51.071Zsync.(*Cond).Wait(0x3aac600?)
63862026-03-11T20:45:51.071Z /work/go/src/sync/cond.go:70 +0x85
63872026-03-11T20:45:51.071Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583a10})
63882026-03-11T20:45:51.072Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
63892026-03-11T20:45:51.072Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
63902026-03-11T20:45:51.072Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
63912026-03-11T20:45:51.072Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
63922026-03-11T20:45:51.072Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
63932026-03-11T20:45:51.072Z
63942026-03-11T20:45:51.072Zgoroutine 9866 [select]:
63952026-03-11T20:45:51.072Zgithub.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
63962026-03-11T20:45:51.072Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:439 +0x1c9
63972026-03-11T20:45:51.072Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 9708
63982026-03-11T20:45:51.072Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:412 +0x4f
63992026-03-11T20:45:51.072Z
64002026-03-11T20:45:51.072Zgoroutine 10921 [select, 9 minutes]:
64012026-03-11T20:45:51.072Zgithub.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigmanager.(*Manager).run(0xc001fb3310, {0x50b1300, 0xc00067dd40})
64022026-03-11T20:45:51.072Z /work/oxidecomputer/cockroach/pkg/spanconfig/spanconfigmanager/manager.go:159 +0x3ba
64032026-03-11T20:45:51.076Zgithub.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigmanager.(*Manager).Start.func1({0x50b1300?, 0xc00067dd40?})
64042026-03-11T20:45:51.076Z /work/oxidecomputer/cockroach/pkg/spanconfig/spanconfigmanager/manager.go:101 +0x2a
64052026-03-11T20:45:51.076Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
64062026-03-11T20:45:51.076Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
64072026-03-11T20:45:51.076Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
64082026-03-11T20:45:51.076Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
64092026-03-11T20:45:51.076Z
64102026-03-11T20:45:51.076Zgoroutine 9952 [select, 59 minutes]:
64112026-03-11T20:45:51.076Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc004cb1b08?, 0x0?})
64122026-03-11T20:45:51.076Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
64132026-03-11T20:45:51.076Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
64142026-03-11T20:45:51.076Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
64152026-03-11T20:45:51.076Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
64162026-03-11T20:45:51.076Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
64172026-03-11T20:45:51.076Z
64182026-03-11T20:45:51.076Zgoroutine 10083 [sync.Cond.Wait]:
64192026-03-11T20:45:51.076Zsync.runtime_notifyListWait(0xc001e22250, 0x377f)
64202026-03-11T20:45:51.076Z /work/go/src/runtime/sema.go:569 +0x159
64212026-03-11T20:45:51.076Zsync.(*Cond).Wait(0x3aac600?)
64222026-03-11T20:45:51.077Z /work/go/src/sync/cond.go:70 +0x85
64232026-03-11T20:45:51.077Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583a70})
64242026-03-11T20:45:51.077Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
64252026-03-11T20:45:51.077Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
64262026-03-11T20:45:51.077Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
64272026-03-11T20:45:51.077Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
64282026-03-11T20:45:51.077Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
64292026-03-11T20:45:51.077Z
64302026-03-11T20:45:51.077Zgoroutine 9056 [select, 9 minutes]:
64312026-03-11T20:45:51.077Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2({0x50b1300, 0xc009329aa0})
64322026-03-11T20:45:51.077Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/queue.go:823 +0xff
64332026-03-11T20:45:51.077Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
64342026-03-11T20:45:51.077Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
64352026-03-11T20:45:51.077Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10124
64362026-03-11T20:45:51.077Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
64372026-03-11T20:45:51.077Z
64382026-03-11T20:45:51.077Zgoroutine 9837 [select, 59 minutes]:
64392026-03-11T20:45:51.077Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc006b64008?, 0xc003ce5270?})
64402026-03-11T20:45:51.077Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
64412026-03-11T20:45:51.077Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
64422026-03-11T20:45:51.078Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
64432026-03-11T20:45:51.078Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
64442026-03-11T20:45:51.078Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
64452026-03-11T20:45:51.078Z
64462026-03-11T20:45:51.078Zgoroutine 10723 [sync.Cond.Wait, 59 minutes]:
64472026-03-11T20:45:51.078Zsync.runtime_notifyListWait(0xc0064c5050, 0x29)
64482026-03-11T20:45:51.078Z /work/go/src/runtime/sema.go:569 +0x159
64492026-03-11T20:45:51.078Zsync.(*Cond).Wait(0xc006e3e878?)
64502026-03-11T20:45:51.078Z /work/go/src/sync/cond.go:70 +0x85
64512026-03-11T20:45:51.078Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd2c30})
64522026-03-11T20:45:51.078Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
64532026-03-11T20:45:51.078Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
64542026-03-11T20:45:51.078Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
64552026-03-11T20:45:51.078Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
64562026-03-11T20:45:51.078Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
64572026-03-11T20:45:51.078Z
64582026-03-11T20:45:51.078Zgoroutine 9068 [select, 59 minutes]:
64592026-03-11T20:45:51.078Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2({0x50b1300, 0xc0076b6720})
64602026-03-11T20:45:51.078Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/queue.go:823 +0xff
64612026-03-11T20:45:51.078Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
64622026-03-11T20:45:51.078Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
64632026-03-11T20:45:51.078Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10821
64642026-03-11T20:45:51.078Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
64652026-03-11T20:45:51.078Z
64662026-03-11T20:45:51.078Zgoroutine 9061 [select]:
64672026-03-11T20:45:51.079Zgoogle.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc007738000)
64682026-03-11T20:45:51.079Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1131 +0x205
64692026-03-11T20:45:51.079Zcreated by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 10710
64702026-03-11T20:45:51.079Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:341 +0x188e
64712026-03-11T20:45:51.079Z
64722026-03-11T20:45:51.079Zgoroutine 11171 [select, 59 minutes]:
64732026-03-11T20:45:51.079Zgithub.com/cockroachdb/cockroach/pkg/util/tracing.newTracingClientStream.func2()
64742026-03-11T20:45:51.079Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:386 +0x67
64752026-03-11T20:45:51.079Zcreated by github.com/cockroachdb/cockroach/pkg/util/tracing.newTracingClientStream in goroutine 11137
64762026-03-11T20:45:51.079Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:385 +0x125
64772026-03-11T20:45:51.079Z
64782026-03-11T20:45:51.079Zgoroutine 8977 [chan receive, 59 minutes]:
64792026-03-11T20:45:51.079Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc007582030?})
64802026-03-11T20:45:51.079Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
64812026-03-11T20:45:51.079Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0xc003514bc0?, 0x1028cc0?}}, 0xc002be9fa8)
64822026-03-11T20:45:51.079Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
64832026-03-11T20:45:51.079Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc002be9f98?)
64842026-03-11T20:45:51.079Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
64852026-03-11T20:45:51.079Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
64862026-03-11T20:45:51.079Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
64872026-03-11T20:45:51.079Z
64882026-03-11T20:45:51.079Zgoroutine 10784 [sync.Cond.Wait, 59 minutes]:
64892026-03-11T20:45:51.079Zsync.runtime_notifyListWait(0xc0064c5050, 0x4b)
64902026-03-11T20:45:51.079Z /work/go/src/runtime/sema.go:569 +0x159
64912026-03-11T20:45:51.079Zsync.(*Cond).Wait(0x50f7930?)
64922026-03-11T20:45:51.079Z /work/go/src/sync/cond.go:70 +0x85
64932026-03-11T20:45:51.079Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd37a0})
64942026-03-11T20:45:51.079Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
64952026-03-11T20:45:51.080Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
64962026-03-11T20:45:51.080Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
64972026-03-11T20:45:51.080Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
64982026-03-11T20:45:51.080Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
64992026-03-11T20:45:51.080Z
65002026-03-11T20:45:51.080Zgoroutine 9876 [chan receive, 59 minutes]:
65012026-03-11T20:45:51.080Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc005572bd0?})
65022026-03-11T20:45:51.080Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
65032026-03-11T20:45:51.091Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x0?, 0x0?}}, 0xc000e027a8)
65042026-03-11T20:45:51.091Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
65052026-03-11T20:45:51.091Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
65062026-03-11T20:45:51.091Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
65072026-03-11T20:45:51.091Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
65082026-03-11T20:45:51.092Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
65092026-03-11T20:45:51.092Z
65102026-03-11T20:45:51.092Zgoroutine 9057 [select, 59 minutes]:
65112026-03-11T20:45:51.092Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2({0x50b1300, 0xc009329ad0})
65122026-03-11T20:45:51.092Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/queue.go:823 +0xff
65132026-03-11T20:45:51.092Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
65142026-03-11T20:45:51.092Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
65152026-03-11T20:45:51.092Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10124
65162026-03-11T20:45:51.092Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
65172026-03-11T20:45:51.092Z
65182026-03-11T20:45:51.092Zgoroutine 10423 [select, 59 minutes]:
65192026-03-11T20:45:51.092Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1({0x50b1338, 0xc007611a40})
65202026-03-11T20:45:51.092Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:116 +0x157
65212026-03-11T20:45:51.092Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.Group.GoCtx.func3()
65222026-03-11T20:45:51.092Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
65232026-03-11T20:45:51.092Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
65242026-03-11T20:45:51.092Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
65252026-03-11T20:45:51.092Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10460
65262026-03-11T20:45:51.092Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
65272026-03-11T20:45:51.092Z
65282026-03-11T20:45:51.092Zgoroutine 9944 [select, 59 minutes]:
65292026-03-11T20:45:51.092Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc004cb0008?, 0x0?})
65302026-03-11T20:45:51.093Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
65312026-03-11T20:45:51.093Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
65322026-03-11T20:45:51.093Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
65332026-03-11T20:45:51.093Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
65342026-03-11T20:45:51.093Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
65352026-03-11T20:45:51.093Z
65362026-03-11T20:45:51.093Zgoroutine 10514 [select]:
65372026-03-11T20:45:51.093Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).outputLoop(0xc0022110a0, {0x50b1338, 0xc00619a0a0})
65382026-03-11T20:45:51.093Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:310 +0x247
65392026-03-11T20:45:51.093Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).runOutputLoop(0xc0022110a0, {0x50b1300, 0xc007182d20}, 0x1?)
65402026-03-11T20:45:51.093Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:335 +0xba
65412026-03-11T20:45:51.093Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run.func1({0x50b1300?, 0xc007182d20?})
65422026-03-11T20:45:51.093Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:297 +0x46
65432026-03-11T20:45:51.093Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
65442026-03-11T20:45:51.093Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
65452026-03-11T20:45:51.093Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10334
65462026-03-11T20:45:51.093Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
65472026-03-11T20:45:51.093Z
65482026-03-11T20:45:51.093Zgoroutine 10336 [select]:
65492026-03-11T20:45:51.093Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).outputLoop(0xc0021ee000, {0x50b1338, 0xc0022ba820})
65502026-03-11T20:45:51.093Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:310 +0x247
65512026-03-11T20:45:51.093Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).runOutputLoop(0xc0021ee000, {0x50b1300, 0xc0006a98c0}, 0x0?)
65522026-03-11T20:45:51.093Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:335 +0xba
65532026-03-11T20:45:51.093Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run.func1({0x50b1300?, 0xc0006a98c0?})
65542026-03-11T20:45:51.093Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:297 +0x46
65552026-03-11T20:45:51.093Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
65562026-03-11T20:45:51.093Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
65572026-03-11T20:45:51.093Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10334
65582026-03-11T20:45:51.093Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
65592026-03-11T20:45:51.093Z
65602026-03-11T20:45:51.093Zgoroutine 10812 [sync.Cond.Wait, 59 minutes]:
65612026-03-11T20:45:51.093Zsync.runtime_notifyListWait(0xc0064c5050, 0x55)
65622026-03-11T20:45:51.094Z /work/go/src/runtime/sema.go:569 +0x159
65632026-03-11T20:45:51.094Zsync.(*Cond).Wait(0xc006e3e878?)
65642026-03-11T20:45:51.094Z /work/go/src/sync/cond.go:70 +0x85
65652026-03-11T20:45:51.094Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3ce0})
65662026-03-11T20:45:51.094Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
65672026-03-11T20:45:51.094Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
65682026-03-11T20:45:51.094Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
65692026-03-11T20:45:51.094Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
65702026-03-11T20:45:51.094Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
65712026-03-11T20:45:51.094Z
65722026-03-11T20:45:51.094Zgoroutine 10016 [sync.Cond.Wait]:
65732026-03-11T20:45:51.094Zsync.runtime_notifyListWait(0xc001e22250, 0x37a1)
65742026-03-11T20:45:51.094Z /work/go/src/runtime/sema.go:569 +0x159
65752026-03-11T20:45:51.094Zsync.(*Cond).Wait(0x3aac600?)
65762026-03-11T20:45:51.094Z /work/go/src/sync/cond.go:70 +0x85
65772026-03-11T20:45:51.094Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc0075830e0})
65782026-03-11T20:45:51.094Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
65792026-03-11T20:45:51.094Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
65802026-03-11T20:45:51.094Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
65812026-03-11T20:45:51.094Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
65822026-03-11T20:45:51.094Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
65832026-03-11T20:45:51.094Z
65842026-03-11T20:45:51.094Zgoroutine 10834 [select]:
65852026-03-11T20:45:51.094Zgithub.com/cockroachdb/cockroach/pkg/util/admission.(*StoreGrantCoordinators).SetPebbleMetricsProvider.func1()
65862026-03-11T20:45:51.094Z /work/oxidecomputer/cockroach/pkg/util/admission/granter.go:1261 +0xd5
65872026-03-11T20:45:51.094Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.(*StoreGrantCoordinators).SetPebbleMetricsProvider in goroutine 9708
65882026-03-11T20:45:51.094Z /work/oxidecomputer/cockroach/pkg/util/admission/granter.go:1256 +0x28d
65892026-03-11T20:45:51.094Z
65902026-03-11T20:45:51.094Zgoroutine 10898 [IO wait]:
65912026-03-11T20:45:51.094Zinternal/poll.runtime_pollWait(0x7fc12c3bd928, 0x72)
65922026-03-11T20:45:51.094Z /work/go/src/runtime/netpoll.go:345 +0x85
65932026-03-11T20:45:51.094Zinternal/poll.(*pollDesc).wait(0xc006b5c080?, 0xc0051b6000?, 0x0)
65942026-03-11T20:45:51.094Z /work/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
65952026-03-11T20:45:51.094Zinternal/poll.(*pollDesc).waitRead(...)
65962026-03-11T20:45:51.094Z /work/go/src/internal/poll/fd_poll_runtime.go:89
65972026-03-11T20:45:51.094Zinternal/poll.(*FD).Read(0xc006b5c080, {0xc0051b6000, 0xa000, 0xa000})
65982026-03-11T20:45:51.094Z /work/go/src/internal/poll/fd_unix.go:164 +0x27a
65992026-03-11T20:45:51.094Znet.(*netFD).Read(0xc006b5c080, {0xc0051b6000?, 0xc004828868?, 0x41e1e5?})
66002026-03-11T20:45:51.094Z /work/go/src/net/fd_posix.go:55 +0x25
66012026-03-11T20:45:51.095Znet.(*conn).Read(0xc004e7c038, {0xc0051b6000?, 0x7fc127b457a8?, 0xc0284e7500?})
66022026-03-11T20:45:51.095Z /work/go/src/net/net.go:185 +0x45
66032026-03-11T20:45:51.100Zgithub.com/cockroachdb/cmux.(*MuxConn).Read(0xc00726dc80, {0xc0051b6000, 0xa000, 0xa000})
66042026-03-11T20:45:51.100Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x14f
66052026-03-11T20:45:51.100Zcrypto/tls.(*atLeastReader).Read(0xc0284e7500, {0xc0051b6000?, 0x0?, 0xc0284e7500?})
66062026-03-11T20:45:51.100Z /work/go/src/crypto/tls/conn.go:806 +0x3b
66072026-03-11T20:45:51.100Zbytes.(*Buffer).ReadFrom(0xc002ce29b0, {0x5081fe0, 0xc0284e7500})
66082026-03-11T20:45:51.100Z /work/go/src/bytes/buffer.go:211 +0x98
66092026-03-11T20:45:51.100Zcrypto/tls.(*Conn).readFromUntil(0xc002ce2708, {0x7fc12ce5a2d8, 0xc00726dc80}, 0xc004828990?)
66102026-03-11T20:45:51.100Z /work/go/src/crypto/tls/conn.go:828 +0xde
66112026-03-11T20:45:51.100Zcrypto/tls.(*Conn).readRecordOrCCS(0xc002ce2708, 0x0)
66122026-03-11T20:45:51.100Z /work/go/src/crypto/tls/conn.go:626 +0x3cf
66132026-03-11T20:45:51.100Zcrypto/tls.(*Conn).readRecord(...)
66142026-03-11T20:45:51.100Z /work/go/src/crypto/tls/conn.go:588
66152026-03-11T20:45:51.100Zcrypto/tls.(*Conn).Read(0xc002ce2708, {0xc007af8000, 0x8000, 0xc004828ce8?})
66162026-03-11T20:45:51.100Z /work/go/src/crypto/tls/conn.go:1370 +0x156
66172026-03-11T20:45:51.100Zbufio.(*Reader).Read(0xc00625f1a0, {0xc0022102e0, 0x9, 0xc0284e74e8?})
66182026-03-11T20:45:51.100Z /work/go/src/bufio/bufio.go:241 +0x197
66192026-03-11T20:45:51.100Zio.ReadAtLeast({0x507c320, 0xc00625f1a0}, {0xc0022102e0, 0x9, 0x9}, 0x9)
66202026-03-11T20:45:51.100Z /work/go/src/io/io.go:335 +0x90
66212026-03-11T20:45:51.100Zio.ReadFull(...)
66222026-03-11T20:45:51.100Z /work/go/src/io/io.go:354
66232026-03-11T20:45:51.100Zgolang.org/x/net/http2.readFrameHeader({0xc0022102e0, 0x9, 0xc004828e10?}, {0x507c320?, 0xc00625f1a0?})
66242026-03-11T20:45:51.100Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x65
66252026-03-11T20:45:51.100Zgolang.org/x/net/http2.(*Framer).ReadFrame(0xc0022102a0)
66262026-03-11T20:45:51.100Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:501 +0x85
66272026-03-11T20:45:51.100Zgoogle.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc003d1a1a0, 0xc006f09620, 0x42e6fc8)
66282026-03-11T20:45:51.100Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:611 +0x14e
66292026-03-11T20:45:51.100Zgoogle.golang.org/grpc.(*Server).serveStreams(0xc004feefc0, {0x50f1d80, 0xc003d1a1a0})
66302026-03-11T20:45:51.100Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:906 +0x114
66312026-03-11T20:45:51.101Zgoogle.golang.org/grpc.(*Server).handleRawConn.func1()
66322026-03-11T20:45:51.101Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:848 +0x45
66332026-03-11T20:45:51.101Zcreated by google.golang.org/grpc.(*Server).handleRawConn in goroutine 10669
66342026-03-11T20:45:51.101Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:847 +0x15b
66352026-03-11T20:45:51.101Z
66362026-03-11T20:45:51.101Zgoroutine 10198 [select, 9 minutes]:
66372026-03-11T20:45:51.101Zgithub.com/cockroachdb/cockroach/pkg/server.(*Server).startSystemLogsGC.func1({0x50b1300, 0xc0006e2150})
66382026-03-11T20:45:51.101Z /work/oxidecomputer/cockroach/pkg/server/server_systemlog_gc.go:179 +0x153
66392026-03-11T20:45:51.101Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
66402026-03-11T20:45:51.101Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
66412026-03-11T20:45:51.101Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
66422026-03-11T20:45:51.101Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
66432026-03-11T20:45:51.101Z
66442026-03-11T20:45:51.101Zgoroutine 10093 [sync.Cond.Wait]:
66452026-03-11T20:45:51.101Zsync.runtime_notifyListWait(0xc001e22250, 0x37a0)
66462026-03-11T20:45:51.101Z /work/go/src/runtime/sema.go:569 +0x159
66472026-03-11T20:45:51.101Zsync.(*Cond).Wait(0x3aac600?)
66482026-03-11T20:45:51.101Z /work/go/src/sync/cond.go:70 +0x85
66492026-03-11T20:45:51.101Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583c50})
66502026-03-11T20:45:51.101Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
66512026-03-11T20:45:51.101Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
66522026-03-11T20:45:51.101Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
66532026-03-11T20:45:51.101Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
66542026-03-11T20:45:51.101Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
66552026-03-11T20:45:51.101Z
66562026-03-11T20:45:51.101Zgoroutine 9839 [select, 59 minutes]:
66572026-03-11T20:45:51.102Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc006b64608?, 0xc004a3a2a0?})
66582026-03-11T20:45:51.102Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
66592026-03-11T20:45:51.102Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
66602026-03-11T20:45:51.102Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
66612026-03-11T20:45:51.102Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
66622026-03-11T20:45:51.102Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
66632026-03-11T20:45:51.102Z
66642026-03-11T20:45:51.102Zgoroutine 10108 [sync.Cond.Wait]:
66652026-03-11T20:45:51.102Zsync.runtime_notifyListWait(0xc001e22250, 0x3770)
66662026-03-11T20:45:51.102Z /work/go/src/runtime/sema.go:569 +0x159
66672026-03-11T20:45:51.102Zsync.(*Cond).Wait(0x3aac600?)
66682026-03-11T20:45:51.102Z /work/go/src/sync/cond.go:70 +0x85
66692026-03-11T20:45:51.102Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583f20})
66702026-03-11T20:45:51.102Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
66712026-03-11T20:45:51.102Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
66722026-03-11T20:45:51.102Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
66732026-03-11T20:45:51.102Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
66742026-03-11T20:45:51.102Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
66752026-03-11T20:45:51.102Z
66762026-03-11T20:45:51.102Zgoroutine 9953 [select, 59 minutes]:
66772026-03-11T20:45:51.102Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initCancelingWorkers.func1({0x50b1300, 0xc007659890})
66782026-03-11T20:45:51.102Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:139 +0xc5
66792026-03-11T20:45:51.102Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
66802026-03-11T20:45:51.102Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
66812026-03-11T20:45:51.102Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
66822026-03-11T20:45:51.102Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
66832026-03-11T20:45:51.102Z
66842026-03-11T20:45:51.102Zgoroutine 9954 [select, 59 minutes]:
66852026-03-11T20:45:51.103Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initCancelingWorkers.func1({0x50b1300, 0xc0076598f0})
66862026-03-11T20:45:51.103Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:139 +0xc5
66872026-03-11T20:45:51.103Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
66882026-03-11T20:45:51.103Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
66892026-03-11T20:45:51.103Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
66902026-03-11T20:45:51.103Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
66912026-03-11T20:45:51.103Z
66922026-03-11T20:45:51.103Zgoroutine 10098 [sync.Cond.Wait]:
66932026-03-11T20:45:51.103Zsync.runtime_notifyListWait(0xc001e22250, 0x3794)
66942026-03-11T20:45:51.103Z /work/go/src/runtime/sema.go:569 +0x159
66952026-03-11T20:45:51.103Zsync.(*Cond).Wait(0x3aac600?)
66962026-03-11T20:45:51.103Z /work/go/src/sync/cond.go:70 +0x85
66972026-03-11T20:45:51.103Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583d40})
66982026-03-11T20:45:51.103Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
66992026-03-11T20:45:51.103Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
67002026-03-11T20:45:51.103Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
67012026-03-11T20:45:51.103Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
67022026-03-11T20:45:51.103Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
67032026-03-11T20:45:51.111Z
67042026-03-11T20:45:51.111Zgoroutine 9921 [chan receive, 59 minutes]:
67052026-03-11T20:45:51.111Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc0036f88a0?})
67062026-03-11T20:45:51.111Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
67072026-03-11T20:45:51.111Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x0?, 0x0?}}, 0xc004654fa8)
67082026-03-11T20:45:51.111Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
67092026-03-11T20:45:51.111Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
67102026-03-11T20:45:51.111Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
67112026-03-11T20:45:51.111Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
67122026-03-11T20:45:51.111Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
67132026-03-11T20:45:51.111Z
67142026-03-11T20:45:51.111Zgoroutine 9834 [select, 59 minutes]:
67152026-03-11T20:45:51.112Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc005956f08?, 0xc003634bec?})
67162026-03-11T20:45:51.112Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
67172026-03-11T20:45:51.112Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
67182026-03-11T20:45:51.112Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
67192026-03-11T20:45:51.112Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
67202026-03-11T20:45:51.112Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
67212026-03-11T20:45:51.112Z
67222026-03-11T20:45:51.112Zgoroutine 10086 [sync.Cond.Wait]:
67232026-03-11T20:45:51.112Zsync.runtime_notifyListWait(0xc001e22250, 0x3780)
67242026-03-11T20:45:51.112Z /work/go/src/runtime/sema.go:569 +0x159
67252026-03-11T20:45:51.112Zsync.(*Cond).Wait(0x3aac600?)
67262026-03-11T20:45:51.112Z /work/go/src/sync/cond.go:70 +0x85
67272026-03-11T20:45:51.112Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583b00})
67282026-03-11T20:45:51.112Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
67292026-03-11T20:45:51.112Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
67302026-03-11T20:45:51.112Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
67312026-03-11T20:45:51.112Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
67322026-03-11T20:45:51.112Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
67332026-03-11T20:45:51.112Z
67342026-03-11T20:45:51.112Zgoroutine 9064 [select, 59 minutes]:
67352026-03-11T20:45:51.112Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2({0x50b1300, 0xc0076b6660})
67362026-03-11T20:45:51.112Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/queue.go:823 +0xff
67372026-03-11T20:45:51.112Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
67382026-03-11T20:45:51.112Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
67392026-03-11T20:45:51.112Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10821
67402026-03-11T20:45:51.112Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
67412026-03-11T20:45:51.112Z
67422026-03-11T20:45:51.112Zgoroutine 9922 [select]:
67432026-03-11T20:45:51.112Zgithub.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1()
67442026-03-11T20:45:51.112Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:71 +0xcb
67452026-03-11T20:45:51.112Zcreated by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 9708
67462026-03-11T20:45:51.112Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:66 +0x58
67472026-03-11T20:45:51.112Z
67482026-03-11T20:45:51.112Zgoroutine 10995 [select]:
67492026-03-11T20:45:51.112Zgithub.com/cockroachdb/cockroach/pkg/jobs.(*jobScheduler).runDaemon.func1({0x50b1300?, 0xc0032942a0?})
67502026-03-11T20:45:51.112Z /work/oxidecomputer/cockroach/pkg/jobs/job_scheduler.go:443 +0x293
67512026-03-11T20:45:51.112Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
67522026-03-11T20:45:51.112Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
67532026-03-11T20:45:51.112Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
67542026-03-11T20:45:51.113Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
67552026-03-11T20:45:51.113Z
67562026-03-11T20:45:51.113Zgoroutine 10050 [sync.Cond.Wait]:
67572026-03-11T20:45:51.113Zsync.runtime_notifyListWait(0xc001e22250, 0x3761)
67582026-03-11T20:45:51.113Z /work/go/src/runtime/sema.go:569 +0x159
67592026-03-11T20:45:51.113Zsync.(*Cond).Wait(0x3aac600?)
67602026-03-11T20:45:51.113Z /work/go/src/sync/cond.go:70 +0x85
67612026-03-11T20:45:51.113Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583440})
67622026-03-11T20:45:51.113Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
67632026-03-11T20:45:51.113Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
67642026-03-11T20:45:51.113Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
67652026-03-11T20:45:51.113Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
67662026-03-11T20:45:51.113Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
67672026-03-11T20:45:51.113Z
67682026-03-11T20:45:51.113Zgoroutine 10036 [sync.Cond.Wait]:
67692026-03-11T20:45:51.113Zsync.runtime_notifyListWait(0xc001e22250, 0x374a)
67702026-03-11T20:45:51.113Z /work/go/src/runtime/sema.go:569 +0x159
67712026-03-11T20:45:51.113Zsync.(*Cond).Wait(0x3aac600?)
67722026-03-11T20:45:51.113Z /work/go/src/sync/cond.go:70 +0x85
67732026-03-11T20:45:51.113Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc0075831a0})
67742026-03-11T20:45:51.113Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
67752026-03-11T20:45:51.113Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
67762026-03-11T20:45:51.113Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
67772026-03-11T20:45:51.113Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
67782026-03-11T20:45:51.113Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
67792026-03-11T20:45:51.113Z
67802026-03-11T20:45:51.113Zgoroutine 9804 [select]:
67812026-03-11T20:45:51.113Zgithub.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue.func2()
67822026-03-11T20:45:51.113Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:366 +0x85
67832026-03-11T20:45:51.113Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue in goroutine 9708
67842026-03-11T20:45:51.113Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:363 +0x3ed
67852026-03-11T20:45:51.113Z
67862026-03-11T20:45:51.113Zgoroutine 11140 [chan receive, 59 minutes]:
67872026-03-11T20:45:51.113Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).rangeFeedWithRangeID(0xc001ded408, 0x3d87860?, 0xc001c5f7a0, {0x7fc12ce5a548, 0xc000829800})
67882026-03-11T20:45:51.114Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:249 +0x76b
67892026-03-11T20:45:51.114Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed(...)
67902026-03-11T20:45:51.114Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:149
67912026-03-11T20:45:51.114Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).RangeFeed(0xc000f39208, 0xc001c5f7a0, {0x50e5040, 0xc000829800})
67922026-03-11T20:45:51.114Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:3082 +0x16f
67932026-03-11T20:45:51.114Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).RangeFeed(0xc001eb4380, 0xc001c5f7a0, {0x50e5040, 0xc000829800})
67942026-03-11T20:45:51.114Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/stores.go:216 +0x19a
67952026-03-11T20:45:51.114Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed(0x732ffb0?, 0x40055e0?, {0x50e5040, 0xc000829800})
67962026-03-11T20:45:51.114Z /work/oxidecomputer/cockroach/pkg/server/node.go:1232 +0x28
67972026-03-11T20:45:51.114Zgithub.com/cockroachdb/cockroach/pkg/roachpb._Internal_RangeFeed_Handler({0x40055e0, 0xc004911b08}, {0x50df230, 0xc0062f7d60})
67982026-03-11T20:45:51.114Z /work/oxidecomputer/cockroach/pkg/roachpb/api.pb.go:9523 +0x107
67992026-03-11T20:45:51.114Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.StreamServerInterceptor.func14({0x40055e0, 0xc004911b08}, {0x50df9e0, 0xc0079b9930}, 0xc005dbe5e8, 0x42e42f8)
68002026-03-11T20:45:51.114Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:205 +0x35b
68012026-03-11T20:45:51.114Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x40055e0?, 0xc004911b08?}, {0x50df9e0?, 0xc0079b9930?})
68022026-03-11T20:45:51.114Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1408 +0x59
68032026-03-11T20:45:51.118Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func4({0x40055e0, 0xc004911b08}, {0x50df9e0, 0xc0079b9930}, 0xc004098918?, 0xc000ee73c0)
68042026-03-11T20:45:51.118Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:247 +0x6c
68052026-03-11T20:45:51.118Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x40055e0?, 0xc004911b08?}, {0x50df9e0?, 0xc0079b9930?})
68062026-03-11T20:45:51.118Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
68072026-03-11T20:45:51.118Zgithub.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({{{0x6?}}}, {0x40055e0, 0xc004911b08}, {0x50df9e0, 0xc0079b9930}, 0xc005dbe5e8, 0xc000ee73c0)
68082026-03-11T20:45:51.118Z /work/oxidecomputer/cockroach/pkg/rpc/auth.go:98 +0x19b
68092026-03-11T20:45:51.119Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x40055e0?, 0xc004911b08?}, {0x50df9e0?, 0xc0079b9930?})
68102026-03-11T20:45:51.119Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
68112026-03-11T20:45:51.119Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func2.1({0x66?, 0x40?})
68122026-03-11T20:45:51.119Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:216 +0x27
68132026-03-11T20:45:51.119Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc003f40400, {0x50b1300, 0xc006f649c0}, {0x41556f?, 0x7fc12ce972f8?}, 0xc004098ac0)
68142026-03-11T20:45:51.119Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:344 +0xcb
68152026-03-11T20:45:51.119Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func2({0x40055e0, 0xc004911b08}, {0x50df9e0, 0xc0079b9930}, 0xc005dbe5e8, 0xc000ee73c0)
68162026-03-11T20:45:51.119Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:215 +0xdf
68172026-03-11T20:45:51.119Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x40055e0?, 0xc004911b08?}, {0x50df9e0?, 0xc0079b9930?})
68182026-03-11T20:45:51.119Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
68192026-03-11T20:45:51.119Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x40055e0, 0xc004911b08}, {0x50df9e0, 0xc0079b9930}, 0xc005dbe5e8, 0x42e42f8)
68202026-03-11T20:45:51.119Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1413 +0x108
68212026-03-11T20:45:51.119Zgoogle.golang.org/grpc.(*Server).processStreamingRPC(0xc004feefc0, {0x50f1d80, 0xc001aef6c0}, 0xc00350d320, 0xc007661ad0, 0x73abba0, 0x0)
68222026-03-11T20:45:51.119Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1549 +0xe51
68232026-03-11T20:45:51.119Zgoogle.golang.org/grpc.(*Server).handleStream(0xc004feefc0, {0x50f1d80, 0xc001aef6c0}, 0xc00350d320, 0x0)
68242026-03-11T20:45:51.119Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1624 +0x999
68252026-03-11T20:45:51.119Zgoogle.golang.org/grpc.(*Server).serveStreams.func1.2()
68262026-03-11T20:45:51.119Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:922 +0x8d
68272026-03-11T20:45:51.119Zcreated by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 10956
68282026-03-11T20:45:51.119Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:920 +0x226
68292026-03-11T20:45:51.119Z
68302026-03-11T20:45:51.119Zgoroutine 11079 [chan receive, 59 minutes]:
68312026-03-11T20:45:51.119Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).rangeFeedWithRangeID(0xc001e56008, 0x3f3de40?, 0xc003925200, {0x7fc12cefd140, 0xc001f88b40})
68322026-03-11T20:45:51.119Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:249 +0x76b
68332026-03-11T20:45:51.119Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed(...)
68342026-03-11T20:45:51.119Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:149
68352026-03-11T20:45:51.119Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).RangeFeed(0xc000f39208, 0xc003925200, {0x50e5b30, 0xc001f88b40})
68362026-03-11T20:45:51.119Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:3082 +0x16f
68372026-03-11T20:45:51.119Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).RangeFeed(0xc001eb4380, 0xc003925200, {0x50e5b30, 0xc001f88b40})
68382026-03-11T20:45:51.119Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/stores.go:216 +0x19a
68392026-03-11T20:45:51.119Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed(0x3f3de40?, 0xc001379070?, {0x50e5b30, 0xc001f88b40})
68402026-03-11T20:45:51.119Z /work/oxidecomputer/cockroach/pkg/server/node.go:1232 +0x28
68412026-03-11T20:45:51.119Zgithub.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed.func1()
68422026-03-11T20:45:51.119Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:753 +0xa7
68432026-03-11T20:45:51.119Zcreated by github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed in goroutine 11078
68442026-03-11T20:45:51.119Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:750 +0x17c
68452026-03-11T20:45:51.119Z
68462026-03-11T20:45:51.120Zgoroutine 10068 [sync.Cond.Wait]:
68472026-03-11T20:45:51.120Zsync.runtime_notifyListWait(0xc001e22250, 0x377d)
68482026-03-11T20:45:51.120Z /work/go/src/runtime/sema.go:569 +0x159
68492026-03-11T20:45:51.120Zsync.(*Cond).Wait(0x3aac600?)
68502026-03-11T20:45:51.120Z /work/go/src/sync/cond.go:70 +0x85
68512026-03-11T20:45:51.120Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc0075837a0})
68522026-03-11T20:45:51.120Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
68532026-03-11T20:45:51.120Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
68542026-03-11T20:45:51.120Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
68552026-03-11T20:45:51.120Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
68562026-03-11T20:45:51.120Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
68572026-03-11T20:45:51.120Z
68582026-03-11T20:45:51.120Zgoroutine 10713 [select]:
68592026-03-11T20:45:51.120Zgoogle.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc00785b040)
68602026-03-11T20:45:51.120Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1131 +0x205
68612026-03-11T20:45:51.120Zcreated by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 10518
68622026-03-11T20:45:51.120Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:341 +0x188e
68632026-03-11T20:45:51.120Z
68642026-03-11T20:45:51.120Zgoroutine 10591 [chan receive, 59 minutes]:
68652026-03-11T20:45:51.120Zgithub.com/cockroachdb/cmux.muxListener.Accept(...)
68662026-03-11T20:45:51.120Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184
68672026-03-11T20:45:51.120Zcrypto/tls.(*listener).Accept(0xc006e57cf8)
68682026-03-11T20:45:51.120Z /work/go/src/crypto/tls/tls.go:66 +0x27
68692026-03-11T20:45:51.120Znet/http.(*Server).Serve(0xc00205db30, {0x509c4f0, 0xc006e57cf8})
68702026-03-11T20:45:51.120Z /work/go/src/net/http/server.go:3260 +0x33e
68712026-03-11T20:45:51.120Zgithub.com/cockroachdb/cockroach/pkg/server.(*httpServer).start.func4({0xc006e54c08?, 0x0?})
68722026-03-11T20:45:51.120Z /work/oxidecomputer/cockroach/pkg/server/server_http.go:277 +0x25
68732026-03-11T20:45:51.120Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
68742026-03-11T20:45:51.120Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
68752026-03-11T20:45:51.120Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
68762026-03-11T20:45:51.120Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
68772026-03-11T20:45:51.120Z
68782026-03-11T20:45:51.120Zgoroutine 10064 [sync.Cond.Wait]:
68792026-03-11T20:45:51.120Zsync.runtime_notifyListWait(0xc001e22250, 0x376a)
68802026-03-11T20:45:51.120Z /work/go/src/runtime/sema.go:569 +0x159
68812026-03-11T20:45:51.120Zsync.(*Cond).Wait(0x3aac600?)
68822026-03-11T20:45:51.120Z /work/go/src/sync/cond.go:70 +0x85
68832026-03-11T20:45:51.121Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc0075836e0})
68842026-03-11T20:45:51.121Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
68852026-03-11T20:45:51.121Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
68862026-03-11T20:45:51.121Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
68872026-03-11T20:45:51.121Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
68882026-03-11T20:45:51.121Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
68892026-03-11T20:45:51.121Z
68902026-03-11T20:45:51.121Zgoroutine 10113 [sync.Cond.Wait]:
68912026-03-11T20:45:51.121Zsync.runtime_notifyListWait(0xc001e22250, 0x3776)
68922026-03-11T20:45:51.121Z /work/go/src/runtime/sema.go:569 +0x159
68932026-03-11T20:45:51.121Zsync.(*Cond).Wait(0x3aac600?)
68942026-03-11T20:45:51.121Z /work/go/src/sync/cond.go:70 +0x85
68952026-03-11T20:45:51.121Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc003ebc0f0})
68962026-03-11T20:45:51.121Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
68972026-03-11T20:45:51.122Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
68982026-03-11T20:45:51.122Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
68992026-03-11T20:45:51.122Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
69002026-03-11T20:45:51.122Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
69012026-03-11T20:45:51.122Z
69022026-03-11T20:45:51.122Zgoroutine 9879 [chan receive, 59 minutes]:
69032026-03-11T20:45:51.128Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc005572cf0?})
69042026-03-11T20:45:51.128Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
69052026-03-11T20:45:51.128Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x3?, 0x0?}}, 0xc003b26fa8)
69062026-03-11T20:45:51.128Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
69072026-03-11T20:45:51.128Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x1?)
69082026-03-11T20:45:51.129Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
69092026-03-11T20:45:51.129Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
69102026-03-11T20:45:51.129Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
69112026-03-11T20:45:51.129Z
69122026-03-11T20:45:51.129Zgoroutine 10085 [sync.Cond.Wait]:
69132026-03-11T20:45:51.129Zsync.runtime_notifyListWait(0xc001e22250, 0x3781)
69142026-03-11T20:45:51.129Z /work/go/src/runtime/sema.go:569 +0x159
69152026-03-11T20:45:51.129Zsync.(*Cond).Wait(0x3aac600?)
69162026-03-11T20:45:51.129Z /work/go/src/sync/cond.go:70 +0x85
69172026-03-11T20:45:51.129Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583ad0})
69182026-03-11T20:45:51.129Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
69192026-03-11T20:45:51.129Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
69202026-03-11T20:45:51.129Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
69212026-03-11T20:45:51.129Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
69222026-03-11T20:45:51.129Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
69232026-03-11T20:45:51.129Z
69242026-03-11T20:45:51.129Zgoroutine 10000 [select, 59 minutes]:
69252026-03-11T20:45:51.129Zgithub.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).bootstrap.func1({0x50b1300?, 0xc006b5b350?})
69262026-03-11T20:45:51.129Z /work/oxidecomputer/cockroach/pkg/gossip/gossip.go:1385 +0x253
69272026-03-11T20:45:51.129Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
69282026-03-11T20:45:51.129Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
69292026-03-11T20:45:51.130Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
69302026-03-11T20:45:51.130Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
69312026-03-11T20:45:51.130Z
69322026-03-11T20:45:51.130Zgoroutine 10733 [sync.Cond.Wait, 59 minutes]:
69332026-03-11T20:45:51.130Zsync.runtime_notifyListWait(0xc0064c5050, 0x2)
69342026-03-11T20:45:51.130Z /work/go/src/runtime/sema.go:569 +0x159
69352026-03-11T20:45:51.130Zsync.(*Cond).Wait(0xc007e413e0?)
69362026-03-11T20:45:51.130Z /work/go/src/sync/cond.go:70 +0x85
69372026-03-11T20:45:51.130Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd2e10})
69382026-03-11T20:45:51.130Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
69392026-03-11T20:45:51.130Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
69402026-03-11T20:45:51.130Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
69412026-03-11T20:45:51.130Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
69422026-03-11T20:45:51.130Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
69432026-03-11T20:45:51.130Z
69442026-03-11T20:45:51.130Zgoroutine 10010 [sync.Cond.Wait]:
69452026-03-11T20:45:51.130Zsync.runtime_notifyListWait(0xc001e22250, 0x3793)
69462026-03-11T20:45:51.130Z /work/go/src/runtime/sema.go:569 +0x159
69472026-03-11T20:45:51.130Zsync.(*Cond).Wait(0x3aac600?)
69482026-03-11T20:45:51.130Z /work/go/src/sync/cond.go:70 +0x85
69492026-03-11T20:45:51.130Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007582fc0})
69502026-03-11T20:45:51.130Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
69512026-03-11T20:45:51.130Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
69522026-03-11T20:45:51.130Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
69532026-03-11T20:45:51.130Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
69542026-03-11T20:45:51.130Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
69552026-03-11T20:45:51.131Z
69562026-03-11T20:45:51.131Zgoroutine 3501264 [select]:
69572026-03-11T20:45:51.131Zgithub.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1()
69582026-03-11T20:45:51.131Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:71 +0xcb
69592026-03-11T20:45:51.131Zcreated by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 10013
69602026-03-11T20:45:51.131Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:66 +0x58
69612026-03-11T20:45:51.131Z
69622026-03-11T20:45:51.131Zgoroutine 8968 [select, 59 minutes]:
69632026-03-11T20:45:51.131Zgithub.com/cockroachdb/cockroach/pkg/util/log.(*fileSink).gcDaemon(0xc006938410, {0x50b1338, 0xc006cfc460})
69642026-03-11T20:45:51.131Z /work/oxidecomputer/cockroach/pkg/util/log/file_log_gc.go:25 +0x79
69652026-03-11T20:45:51.131Zcreated by github.com/cockroachdb/cockroach/pkg/util/log.ApplyConfig in goroutine 9708
69662026-03-11T20:45:51.131Z /work/oxidecomputer/cockroach/pkg/util/log/flags.go:293 +0x10f5
69672026-03-11T20:45:51.131Z
69682026-03-11T20:45:51.131Zgoroutine 10732 [sync.Cond.Wait, 59 minutes]:
69692026-03-11T20:45:51.131Zsync.runtime_notifyListWait(0xc0064c5050, 0x0)
69702026-03-11T20:45:51.131Z /work/go/src/runtime/sema.go:569 +0x159
69712026-03-11T20:45:51.131Zsync.(*Cond).Wait(0xc000f39208?)
69722026-03-11T20:45:51.131Z /work/go/src/sync/cond.go:70 +0x85
69732026-03-11T20:45:51.131Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd2de0})
69742026-03-11T20:45:51.131Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
69752026-03-11T20:45:51.131Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
69762026-03-11T20:45:51.131Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
69772026-03-11T20:45:51.131Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
69782026-03-11T20:45:51.131Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
69792026-03-11T20:45:51.131Z
69802026-03-11T20:45:51.131Zgoroutine 9991 [chan receive, 59 minutes]:
69812026-03-11T20:45:51.131Zgithub.com/cockroachdb/cockroach/pkg/server.configureGRPCGateway.func1({0xc0013e7b08?, 0x0?})
69822026-03-11T20:45:51.132Z /work/oxidecomputer/cockroach/pkg/server/grpc_gateway.go:86 +0x31
69832026-03-11T20:45:51.132Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
69842026-03-11T20:45:51.132Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
69852026-03-11T20:45:51.132Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
69862026-03-11T20:45:51.132Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
69872026-03-11T20:45:51.132Z
69882026-03-11T20:45:51.132Zgoroutine 9711 [select, 59 minutes]:
69892026-03-11T20:45:51.132Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1({0x50b1338, 0xc005d142d0})
69902026-03-11T20:45:51.132Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:116 +0x157
69912026-03-11T20:45:51.132Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.Group.GoCtx.func3()
69922026-03-11T20:45:51.132Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
69932026-03-11T20:45:51.132Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
69942026-03-11T20:45:51.132Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
69952026-03-11T20:45:51.132Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9709
69962026-03-11T20:45:51.132Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
69972026-03-11T20:45:51.132Z
69982026-03-11T20:45:51.132Zgoroutine 10004 [chan receive, 59 minutes]:
69992026-03-11T20:45:51.132Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func1({0xc000a9af08?, 0x0?})
70002026-03-11T20:45:51.132Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:203 +0x37
70012026-03-11T20:45:51.132Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
70022026-03-11T20:45:51.132Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
70032026-03-11T20:45:51.139Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
70042026-03-11T20:45:51.139Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
70052026-03-11T20:45:51.139Z
70062026-03-11T20:45:51.139Zgoroutine 9665 [select, 59 minutes]:
70072026-03-11T20:45:51.139Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1({0x50b1338, 0xc003aae050})
70082026-03-11T20:45:51.139Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:116 +0x157
70092026-03-11T20:45:51.139Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.Group.GoCtx.func3()
70102026-03-11T20:45:51.139Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
70112026-03-11T20:45:51.139Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
70122026-03-11T20:45:51.139Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
70132026-03-11T20:45:51.139Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9892
70142026-03-11T20:45:51.139Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
70152026-03-11T20:45:51.139Z
70162026-03-11T20:45:51.139Zgoroutine 9071 [select, 59 minutes]:
70172026-03-11T20:45:51.139Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2({0x50b1300, 0xc0076b67b0})
70182026-03-11T20:45:51.139Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/queue.go:823 +0xff
70192026-03-11T20:45:51.139Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
70202026-03-11T20:45:51.139Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
70212026-03-11T20:45:51.139Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10821
70222026-03-11T20:45:51.139Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
70232026-03-11T20:45:51.139Z
70242026-03-11T20:45:51.139Zgoroutine 8974 [chan receive, 59 minutes]:
70252026-03-11T20:45:51.139Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc0075821b0?})
70262026-03-11T20:45:51.139Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
70272026-03-11T20:45:51.139Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0xc002259560?, 0xc0022595a0?}}, 0xc0004e5fa8)
70282026-03-11T20:45:51.139Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
70292026-03-11T20:45:51.139Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
70302026-03-11T20:45:51.140Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
70312026-03-11T20:45:51.140Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
70322026-03-11T20:45:51.140Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
70332026-03-11T20:45:51.140Z
70342026-03-11T20:45:51.140Zgoroutine 10374 [select]:
70352026-03-11T20:45:51.140Zgoogle.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc002053ae0, {0xc00766eaf0, 0x5, 0x5})
70362026-03-11T20:45:51.140Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:191 +0x95
70372026-03-11T20:45:51.140Zgoogle.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc002053ae0, {0xc00766eaf0?, 0xc0068200c0?, 0xc0045c5958?})
70382026-03-11T20:45:51.140Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:171 +0x12d
70392026-03-11T20:45:51.140Zgoogle.golang.org/grpc/internal/transport.(*transportReader).Read(0xc001681a70, {0xc00766eaf0?, 0xc0045c59d0?, 0xa40865?})
70402026-03-11T20:45:51.140Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:485 +0x2c
70412026-03-11T20:45:51.140Zio.ReadAtLeast({0x5086380, 0xc001681a70}, {0xc00766eaf0, 0x5, 0x5}, 0x5)
70422026-03-11T20:45:51.140Z /work/go/src/io/io.go:335 +0x90
70432026-03-11T20:45:51.140Zio.ReadFull(...)
70442026-03-11T20:45:51.140Z /work/go/src/io/io.go:354
70452026-03-11T20:45:51.140Zgoogle.golang.org/grpc/internal/transport.(*Stream).Read(0xc002351320, {0xc00766eaf0, 0x5, 0x5})
70462026-03-11T20:45:51.140Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:469 +0x96
70472026-03-11T20:45:51.140Zgoogle.golang.org/grpc.(*parser).recvMsg(0xc00766eae0, 0x7fffffff)
70482026-03-11T20:45:51.140Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:559 +0x42
70492026-03-11T20:45:51.140Zgoogle.golang.org/grpc.recvAndDecompress(0x47b2f2?, 0xc002351320, {0x0, 0x0}, 0x7fffffff, 0xc0045c5c88, {0x50a8e40, 0x77577e0})
70502026-03-11T20:45:51.140Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:690 +0x5a
70512026-03-11T20:45:51.140Zgoogle.golang.org/grpc.recv(0xffffffffffffffff?, {0x7fc12cff4640, 0xc0004ce590}, 0x60000c0288e0bf8?, {0x0?, 0x0?}, {0x3efaf60, 0xc00846f310}, 0x3a8ed40?, 0xc0045c5c88, ...)
70522026-03-11T20:45:51.140Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:756 +0x68
70532026-03-11T20:45:51.140Zgoogle.golang.org/grpc.(*csAttempt).recvMsg(0xc006622dc0, {0x3efaf60, 0xc00846f310}, 0x10?)
70542026-03-11T20:45:51.140Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:983 +0x27e
70552026-03-11T20:45:51.140Zgoogle.golang.org/grpc.(*clientStream).RecvMsg.func1(0x41e1e5?)
70562026-03-11T20:45:51.140Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:834 +0x1f
70572026-03-11T20:45:51.140Zgoogle.golang.org/grpc.(*clientStream).withRetry(0xc002350fc0, 0xc0045c5db8, 0xc0045c5da8)
70582026-03-11T20:45:51.140Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:688 +0x2ee
70592026-03-11T20:45:51.140Zgoogle.golang.org/grpc.(*clientStream).RecvMsg(0xc002350fc0, {0x3efaf60?, 0xc00846f310?})
70602026-03-11T20:45:51.140Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:833 +0x107
70612026-03-11T20:45:51.140Zgithub.com/cockroachdb/cockroach/pkg/util/tracing.(*tracingClientStream).RecvMsg(0xc0043c4ba0, {0x3efaf60?, 0xc00846f310?})
70622026-03-11T20:45:51.140Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:441 +0x31
70632026-03-11T20:45:51.140Zgithub.com/cockroachdb/cockroach/pkg/gossip.(*gossipGossipClient).Recv(0xc00082acd0)
70642026-03-11T20:45:51.140Z /work/oxidecomputer/cockroach/pkg/gossip/gossip.pb.go:636 +0x46
70652026-03-11T20:45:51.140Zgithub.com/cockroachdb/cockroach/pkg/gossip.(*client).gossip.func2.1(0xc0065468a0, {0x50ebb00, 0xc00082acd0}, 0xc00740e410, {0x50b1300, 0xc001681d70}, 0xc003d1ab60)
70662026-03-11T20:45:51.140Z /work/oxidecomputer/cockroach/pkg/gossip/client.go:317 +0x62
70672026-03-11T20:45:51.140Zgithub.com/cockroachdb/cockroach/pkg/gossip.(*client).gossip.func2({0x50b1300?, 0xc001681d70?})
70682026-03-11T20:45:51.141Z /work/oxidecomputer/cockroach/pkg/gossip/client.go:332 +0x86
70692026-03-11T20:45:51.141Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
70702026-03-11T20:45:51.141Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
70712026-03-11T20:45:51.141Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10657
70722026-03-11T20:45:51.141Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
70732026-03-11T20:45:51.141Z
70742026-03-11T20:45:51.141Zgoroutine 10972 [select, 59 minutes]:
70752026-03-11T20:45:51.141Zgithub.com/cockroachdb/cockroach/pkg/util/tracing.newTracingClientStream.func2()
70762026-03-11T20:45:51.141Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:386 +0x67
70772026-03-11T20:45:51.141Zcreated by github.com/cockroachdb/cockroach/pkg/util/tracing.newTracingClientStream in goroutine 10928
70782026-03-11T20:45:51.141Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:385 +0x125
70792026-03-11T20:45:51.141Z
70802026-03-11T20:45:51.141Zgoroutine 10699 [select, 59 minutes]:
70812026-03-11T20:45:51.141Zgithub.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run(0xc002b7db80, {0x50b1300, 0xc006fd2930})
70822026-03-11T20:45:51.141Z /work/oxidecomputer/cockroach/pkg/internal/client/requestbatcher/batcher.go:475 +0x326
70832026-03-11T20:45:51.141Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
70842026-03-11T20:45:51.141Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
70852026-03-11T20:45:51.141Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
70862026-03-11T20:45:51.141Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
70872026-03-11T20:45:51.141Z
70882026-03-11T20:45:51.141Zgoroutine 11121 [select]:
70892026-03-11T20:45:51.141Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents(0xc001131600, {0x50b1338, 0xc00218ee60}, 0xc007b07f50, 0xc001384960)
70902026-03-11T20:45:51.141Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:331 +0xc5
70912026-03-11T20:45:51.141Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func2({0x50b1338?, 0xc00218ee60?})
70922026-03-11T20:45:51.141Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:288 +0x32
70932026-03-11T20:45:51.141Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
70942026-03-11T20:45:51.141Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
70952026-03-11T20:45:51.141Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
70962026-03-11T20:45:51.141Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
70972026-03-11T20:45:51.141Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10908
70982026-03-11T20:45:51.141Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
70992026-03-11T20:45:51.141Z
71002026-03-11T20:45:51.141Zgoroutine 9785 [chan receive, 59 minutes]:
71012026-03-11T20:45:51.141Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc005572360?})
71022026-03-11T20:45:51.141Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
71032026-03-11T20:45:51.148Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x0?, 0x0?}}, 0xc0000bb7a8)
71042026-03-11T20:45:51.148Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
71052026-03-11T20:45:51.148Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
71062026-03-11T20:45:51.148Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
71072026-03-11T20:45:51.148Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
71082026-03-11T20:45:51.148Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
71092026-03-11T20:45:51.148Z
71102026-03-11T20:45:51.148Zgoroutine 10696 [select, 59 minutes]:
71112026-03-11T20:45:51.148Zgithub.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).bootstrap.func1({0x50b1300?, 0xc006e63830?})
71122026-03-11T20:45:51.148Z /work/oxidecomputer/cockroach/pkg/gossip/gossip.go:1385 +0x253
71132026-03-11T20:45:51.148Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
71142026-03-11T20:45:51.148Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
71152026-03-11T20:45:51.148Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
71162026-03-11T20:45:51.148Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
71172026-03-11T20:45:51.148Z
71182026-03-11T20:45:51.148Zgoroutine 10734 [sync.Cond.Wait, 59 minutes]:
71192026-03-11T20:45:51.148Zsync.runtime_notifyListWait(0xc0064c5050, 0x9)
71202026-03-11T20:45:51.148Z /work/go/src/runtime/sema.go:569 +0x159
71212026-03-11T20:45:51.148Zsync.(*Cond).Wait(0x0?)
71222026-03-11T20:45:51.148Z /work/go/src/sync/cond.go:70 +0x85
71232026-03-11T20:45:51.148Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd2e40})
71242026-03-11T20:45:51.148Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
71252026-03-11T20:45:51.148Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
71262026-03-11T20:45:51.148Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
71272026-03-11T20:45:51.148Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
71282026-03-11T20:45:51.148Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
71292026-03-11T20:45:51.148Z
71302026-03-11T20:45:51.148Zgoroutine 10493 [select]:
71312026-03-11T20:45:51.148Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents(0xc002326160, {0x50b1338, 0xc005f39d60}, 0xc00081af90, 0xc0091d60c0)
71322026-03-11T20:45:51.148Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:331 +0xc5
71332026-03-11T20:45:51.148Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func2({0x50b1338?, 0xc005f39d60?})
71342026-03-11T20:45:51.149Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:288 +0x32
71352026-03-11T20:45:51.149Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
71362026-03-11T20:45:51.149Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
71372026-03-11T20:45:51.149Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
71382026-03-11T20:45:51.149Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
71392026-03-11T20:45:51.149Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10202
71402026-03-11T20:45:51.149Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
71412026-03-11T20:45:51.149Z
71422026-03-11T20:45:51.149Zgoroutine 10373 [select, 59 minutes]:
71432026-03-11T20:45:51.149Zgithub.com/cockroachdb/cockroach/pkg/util/tracing.newTracingClientStream.func2()
71442026-03-11T20:45:51.149Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:386 +0x67
71452026-03-11T20:45:51.149Zcreated by github.com/cockroachdb/cockroach/pkg/util/tracing.newTracingClientStream in goroutine 10657
71462026-03-11T20:45:51.149Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:385 +0x125
71472026-03-11T20:45:51.149Z
71482026-03-11T20:45:51.149Zgoroutine 10017 [sync.Cond.Wait]:
71492026-03-11T20:45:51.149Zsync.runtime_notifyListWait(0xc001e22250, 0x3748)
71502026-03-11T20:45:51.149Z /work/go/src/runtime/sema.go:569 +0x159
71512026-03-11T20:45:51.149Zsync.(*Cond).Wait(0x3aac600?)
71522026-03-11T20:45:51.149Z /work/go/src/sync/cond.go:70 +0x85
71532026-03-11T20:45:51.149Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583110})
71542026-03-11T20:45:51.149Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
71552026-03-11T20:45:51.149Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
71562026-03-11T20:45:51.149Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
71572026-03-11T20:45:51.149Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
71582026-03-11T20:45:51.149Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
71592026-03-11T20:45:51.149Z
71602026-03-11T20:45:51.149Zgoroutine 11078 [select]:
71612026-03-11T20:45:51.149Zgithub.com/cockroachdb/cockroach/pkg/rpc.respStreamClientAdapter.recvInternal(...)
71622026-03-11T20:45:51.149Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:694
71632026-03-11T20:45:51.149Zgithub.com/cockroachdb/cockroach/pkg/rpc.rangeFeedClientAdapter.Recv(...)
71642026-03-11T20:45:51.149Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:723
71652026-03-11T20:45:51.149Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed.func3()
71662026-03-11T20:45:51.149Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:516 +0x25
71672026-03-11T20:45:51.149Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*stuckRangeFeedCanceler).do(0xc001378eb0, 0xc00768da50)
71682026-03-11T20:45:51.149Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed_canceler.go:100 +0x1b3
71692026-03-11T20:45:51.149Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed(0xc002d6f408, {0x50b1338?, 0xc0032344b0?}, {{0xc003d99708, 0x1, 0x8}, {0xc003d99710, 0x1, 0x8}}, {0x189be113f26a211c, ...}, ...)
71702026-03-11T20:45:51.150Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:515 +0xc98
71712026-03-11T20:45:51.150Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).partialRangeFeed(0xc002d6f408, {0x50b1338, 0xc0032344b0}, 0xc003234460, {{0xc003d99708, 0x1, 0x8}, {0xc003d99710, 0x1, 0x8}}, ...)
71722026-03-11T20:45:51.150Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:336 +0x745
71732026-03-11T20:45:51.150Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.1({0x50b1338?, 0xc0032344b0?})
71742026-03-11T20:45:51.150Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:120 +0xa9
71752026-03-11T20:45:51.150Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.Group.GoCtx.2()
71762026-03-11T20:45:51.150Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
71772026-03-11T20:45:51.150Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
71782026-03-11T20:45:51.150Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
71792026-03-11T20:45:51.150Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 11055
71802026-03-11T20:45:51.150Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
71812026-03-11T20:45:51.150Z
71822026-03-11T20:45:51.150Zgoroutine 9798 [select]:
71832026-03-11T20:45:51.150Zgithub.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue.func2()
71842026-03-11T20:45:51.150Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:366 +0x85
71852026-03-11T20:45:51.150Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue in goroutine 9708
71862026-03-11T20:45:51.150Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:363 +0x3ed
71872026-03-11T20:45:51.150Z
71882026-03-11T20:45:51.150Zgoroutine 10034 [sync.Cond.Wait]:
71892026-03-11T20:45:51.150Zsync.runtime_notifyListWait(0xc001e22250, 0x379b)
71902026-03-11T20:45:51.150Z /work/go/src/runtime/sema.go:569 +0x159
71912026-03-11T20:45:51.150Zsync.(*Cond).Wait(0x3aac600?)
71922026-03-11T20:45:51.150Z /work/go/src/sync/cond.go:70 +0x85
71932026-03-11T20:45:51.150Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583140})
71942026-03-11T20:45:51.150Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
71952026-03-11T20:45:51.150Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
71962026-03-11T20:45:51.150Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
71972026-03-11T20:45:51.150Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
71982026-03-11T20:45:51.150Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
71992026-03-11T20:45:51.150Z
72002026-03-11T20:45:51.150Zgoroutine 10705 [sync.Cond.Wait, 59 minutes]:
72012026-03-11T20:45:51.150Zsync.runtime_notifyListWait(0xc0064c5050, 0x24)
72022026-03-11T20:45:51.150Z /work/go/src/runtime/sema.go:569 +0x159
72032026-03-11T20:45:51.155Zsync.(*Cond).Wait(0xc0021782d0?)
72042026-03-11T20:45:51.155Z /work/go/src/sync/cond.go:70 +0x85
72052026-03-11T20:45:51.155Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd2bd0})
72062026-03-11T20:45:51.155Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
72072026-03-11T20:45:51.155Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
72082026-03-11T20:45:51.155Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
72092026-03-11T20:45:51.155Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
72102026-03-11T20:45:51.155Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
72112026-03-11T20:45:51.155Z
72122026-03-11T20:45:51.155Zgoroutine 10741 [sync.Cond.Wait, 59 minutes]:
72132026-03-11T20:45:51.155Zsync.runtime_notifyListWait(0xc0064c5050, 0x5)
72142026-03-11T20:45:51.155Z /work/go/src/runtime/sema.go:569 +0x159
72152026-03-11T20:45:51.155Zsync.(*Cond).Wait(0x10000c0018a0008?)
72162026-03-11T20:45:51.155Z /work/go/src/sync/cond.go:70 +0x85
72172026-03-11T20:45:51.155Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd2f90})
72182026-03-11T20:45:51.155Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
72192026-03-11T20:45:51.155Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
72202026-03-11T20:45:51.155Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
72212026-03-11T20:45:51.155Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
72222026-03-11T20:45:51.155Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
72232026-03-11T20:45:51.155Z
72242026-03-11T20:45:51.156Zgoroutine 9066 [select, 59 minutes]:
72252026-03-11T20:45:51.156Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2({0x50b1300, 0xc0076b66c0})
72262026-03-11T20:45:51.156Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/queue.go:823 +0xff
72272026-03-11T20:45:51.156Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
72282026-03-11T20:45:51.156Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
72292026-03-11T20:45:51.156Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10821
72302026-03-11T20:45:51.156Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
72312026-03-11T20:45:51.156Z
72322026-03-11T20:45:51.156Zgoroutine 9864 [select]:
72332026-03-11T20:45:51.156Zgithub.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
72342026-03-11T20:45:51.156Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:439 +0x1c9
72352026-03-11T20:45:51.156Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 9708
72362026-03-11T20:45:51.156Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:412 +0x4f
72372026-03-11T20:45:51.156Z
72382026-03-11T20:45:51.156Zgoroutine 10743 [sync.Cond.Wait, 59 minutes]:
72392026-03-11T20:45:51.157Zsync.runtime_notifyListWait(0xc0064c5050, 0x7)
72402026-03-11T20:45:51.157Z /work/go/src/runtime/sema.go:569 +0x159
72412026-03-11T20:45:51.157Zsync.(*Cond).Wait(0x50f7930?)
72422026-03-11T20:45:51.157Z /work/go/src/sync/cond.go:70 +0x85
72432026-03-11T20:45:51.157Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd2ff0})
72442026-03-11T20:45:51.157Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
72452026-03-11T20:45:51.157Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
72462026-03-11T20:45:51.157Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
72472026-03-11T20:45:51.157Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
72482026-03-11T20:45:51.157Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
72492026-03-11T20:45:51.157Z
72502026-03-11T20:45:51.157Zgoroutine 10744 [sync.Cond.Wait, 59 minutes]:
72512026-03-11T20:45:51.157Zsync.runtime_notifyListWait(0xc0064c5050, 0x1)
72522026-03-11T20:45:51.157Z /work/go/src/runtime/sema.go:569 +0x159
72532026-03-11T20:45:51.157Zsync.(*Cond).Wait(0xc000f39208?)
72542026-03-11T20:45:51.157Z /work/go/src/sync/cond.go:70 +0x85
72552026-03-11T20:45:51.157Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3020})
72562026-03-11T20:45:51.157Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
72572026-03-11T20:45:51.157Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
72582026-03-11T20:45:51.157Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
72592026-03-11T20:45:51.157Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
72602026-03-11T20:45:51.157Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
72612026-03-11T20:45:51.157Z
72622026-03-11T20:45:51.157Zgoroutine 10955 [select]:
72632026-03-11T20:45:51.157Zgoogle.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc001aef6c0)
72642026-03-11T20:45:51.157Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1131 +0x205
72652026-03-11T20:45:51.157Zcreated by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 10868
72662026-03-11T20:45:51.157Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:341 +0x188e
72672026-03-11T20:45:51.157Z
72682026-03-11T20:45:51.157Zgoroutine 11072 [select]:
72692026-03-11T20:45:51.157Zgoogle.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc00540c6e0, {0xc00349a3b8, 0x5, 0x5})
72702026-03-11T20:45:51.158Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:191 +0x95
72712026-03-11T20:45:51.158Zgoogle.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc00540c6e0, {0xc00349a3b8?, 0xc0061410f8?, 0xc0041c5048?})
72722026-03-11T20:45:51.158Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:171 +0x12d
72732026-03-11T20:45:51.158Zgoogle.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0063c72c0, {0xc00349a3b8?, 0xc0041c50c0?, 0xa40865?})
72742026-03-11T20:45:51.158Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:485 +0x2c
72752026-03-11T20:45:51.158Zio.ReadAtLeast({0x5086380, 0xc0063c72c0}, {0xc00349a3b8, 0x5, 0x5}, 0x5)
72762026-03-11T20:45:51.158Z /work/go/src/io/io.go:335 +0x90
72772026-03-11T20:45:51.158Zio.ReadFull(...)
72782026-03-11T20:45:51.158Z /work/go/src/io/io.go:354
72792026-03-11T20:45:51.158Zgoogle.golang.org/grpc/internal/transport.(*Stream).Read(0xc002762900, {0xc00349a3b8, 0x5, 0x5})
72802026-03-11T20:45:51.158Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:469 +0x96
72812026-03-11T20:45:51.158Zgoogle.golang.org/grpc.(*parser).recvMsg(0xc00349a3a8, 0x7fffffff)
72822026-03-11T20:45:51.158Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:559 +0x42
72832026-03-11T20:45:51.158Zgoogle.golang.org/grpc.recvAndDecompress(0x60?, 0xc002762900, {0x0, 0x0}, 0x7fffffff, 0xc0041c5378, {0x50a8e40, 0x77577e0})
72842026-03-11T20:45:51.158Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:690 +0x5a
72852026-03-11T20:45:51.159Zgoogle.golang.org/grpc.recv(0x50900a0?, {0x7fc12cff4640, 0xc0004ce590}, 0x2f?, {0x0?, 0x0?}, {0x3f6ba40, 0xc012c341e0}, 0x10?, 0xc0041c5378, ...)
72862026-03-11T20:45:51.159Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:756 +0x68
72872026-03-11T20:45:51.159Zgoogle.golang.org/grpc.(*csAttempt).recvMsg(0xc001b25600, {0x3f6ba40, 0xc012c341e0}, 0x10?)
72882026-03-11T20:45:51.159Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:983 +0x27e
72892026-03-11T20:45:51.159Zgoogle.golang.org/grpc.(*clientStream).RecvMsg.func1(0x41e1e5?)
72902026-03-11T20:45:51.159Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:834 +0x1f
72912026-03-11T20:45:51.159Zgoogle.golang.org/grpc.(*clientStream).withRetry(0xc002762000, 0xc0041c54a8, 0xc0041c5498)
72922026-03-11T20:45:51.159Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:688 +0x2ee
72932026-03-11T20:45:51.159Zgoogle.golang.org/grpc.(*clientStream).RecvMsg(0xc002762000, {0x3f6ba40?, 0xc012c341e0?})
72942026-03-11T20:45:51.159Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:833 +0x107
72952026-03-11T20:45:51.159Zgithub.com/cockroachdb/cockroach/pkg/util/tracing.(*tracingClientStream).RecvMsg(0xc004fa3c40, {0x3f6ba40?, 0xc012c341e0?})
72962026-03-11T20:45:51.159Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:441 +0x31
72972026-03-11T20:45:51.159Zgithub.com/cockroachdb/cockroach/pkg/roachpb.(*internalRangeFeedClient).Recv(0xc000901760)
72982026-03-11T20:45:51.159Z /work/oxidecomputer/cockroach/pkg/roachpb/api.pb.go:9289 +0x46
72992026-03-11T20:45:51.159Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed.func3()
73002026-03-11T20:45:51.159Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:516 +0x25
73012026-03-11T20:45:51.159Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*stuckRangeFeedCanceler).do(0xc00540c3c0, 0xc0041c5a50)
73022026-03-11T20:45:51.159Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed_canceler.go:100 +0x1b3
73032026-03-11T20:45:51.165Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed(0xc003d48008, {0x50b1338?, 0xc00540c280?}, {{0xc003babac0, 0x2, 0x8}, {0xc003babac8, 0x2, 0x8}}, {0x189be113f3b9fb77, ...}, ...)
73042026-03-11T20:45:51.165Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:515 +0xc98
73052026-03-11T20:45:51.165Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).partialRangeFeed(0xc003d48008, {0x50b1338, 0xc00540c280}, 0xc00540c230, {{0xc003babac0, 0x2, 0x8}, {0xc003babac8, 0x2, 0x8}}, ...)
73062026-03-11T20:45:51.166Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:336 +0x745
73072026-03-11T20:45:51.166Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.1({0x50b1338?, 0xc00540c280?})
73082026-03-11T20:45:51.166Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:120 +0xa9
73092026-03-11T20:45:51.166Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.Group.GoCtx.2()
73102026-03-11T20:45:51.166Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
73112026-03-11T20:45:51.166Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
73122026-03-11T20:45:51.166Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
73132026-03-11T20:45:51.166Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 11070
73142026-03-11T20:45:51.166Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
73152026-03-11T20:45:51.166Z
73162026-03-11T20:45:51.166Zgoroutine 11127 [select, 59 minutes]:
73172026-03-11T20:45:51.166Zgoogle.golang.org/grpc.newClientStreamWithParams.func4()
73182026-03-11T20:45:51.166Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:341 +0x8c
73192026-03-11T20:45:51.167Zcreated by google.golang.org/grpc.newClientStreamWithParams in goroutine 11126
73202026-03-11T20:45:51.167Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:340 +0xb2a
73212026-03-11T20:45:51.167Z
73222026-03-11T20:45:51.167Zgoroutine 9947 [select, 59 minutes]:
73232026-03-11T20:45:51.167Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc004cb0c08?, 0x0?})
73242026-03-11T20:45:51.167Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
73252026-03-11T20:45:51.167Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
73262026-03-11T20:45:51.167Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
73272026-03-11T20:45:51.167Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
73282026-03-11T20:45:51.167Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
73292026-03-11T20:45:51.167Z
73302026-03-11T20:45:51.167Zgoroutine 9843 [select, 59 minutes]:
73312026-03-11T20:45:51.167Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc006b65208?, 0xc001e1c050?})
73322026-03-11T20:45:51.167Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
73332026-03-11T20:45:51.167Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
73342026-03-11T20:45:51.167Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
73352026-03-11T20:45:51.167Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
73362026-03-11T20:45:51.167Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
73372026-03-11T20:45:51.167Z
73382026-03-11T20:45:51.167Zgoroutine 10915 [semacquire, 59 minutes]:
73392026-03-11T20:45:51.167Zsync.runtime_Semacquire(0xc006b6b170?)
73402026-03-11T20:45:51.167Z /work/go/src/runtime/sema.go:62 +0x25
73412026-03-11T20:45:51.167Zsync.(*WaitGroup).Wait(0x44e7a0?)
73422026-03-11T20:45:51.167Z /work/go/src/sync/waitgroup.go:116 +0x48
73432026-03-11T20:45:51.167Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc006c9d980)
73442026-03-11T20:45:51.167Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
73452026-03-11T20:45:51.167Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc006c9d980?, {0x50b1338?, 0xc006b57bd0?}})
73462026-03-11T20:45:51.167Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
73472026-03-11T20:45:51.167Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait({0x50b1300?, 0xc003aac450?}, {0xc002c7fd78, 0x2, 0x4000000000000000?})
73482026-03-11T20:45:51.167Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:193 +0xff
73492026-03-11T20:45:51.167Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0xc00750cb00, {0x50b1300, 0xc003aac450}, 0xc00067d200)
73502026-03-11T20:45:51.167Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:291 +0x59c
73512026-03-11T20:45:51.167Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).Start.func2({0x50b1300, 0xc00067d6b0})
73522026-03-11T20:45:51.167Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:215 +0x2f3
73532026-03-11T20:45:51.167Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
73542026-03-11T20:45:51.167Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
73552026-03-11T20:45:51.167Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
73562026-03-11T20:45:51.167Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
73572026-03-11T20:45:51.168Z
73582026-03-11T20:45:51.168Zgoroutine 9069 [select, 59 minutes]:
73592026-03-11T20:45:51.168Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2({0x50b1300, 0xc0076b6750})
73602026-03-11T20:45:51.168Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/queue.go:823 +0xff
73612026-03-11T20:45:51.168Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
73622026-03-11T20:45:51.168Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
73632026-03-11T20:45:51.168Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10821
73642026-03-11T20:45:51.168Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
73652026-03-11T20:45:51.168Z
73662026-03-11T20:45:51.168Zgoroutine 10123 [select, 1 minutes]:
73672026-03-11T20:45:51.168Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).startGossip.func3({0x50b1300, 0xc003ebc480})
73682026-03-11T20:45:51.168Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:2152 +0x225
73692026-03-11T20:45:51.168Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
73702026-03-11T20:45:51.168Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
73712026-03-11T20:45:51.168Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
73722026-03-11T20:45:51.168Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
73732026-03-11T20:45:51.168Z
73742026-03-11T20:45:51.168Zgoroutine 11068 [semacquire, 59 minutes]:
73752026-03-11T20:45:51.168Zsync.runtime_Semacquire(0xc00349a210?)
73762026-03-11T20:45:51.168Z /work/go/src/runtime/sema.go:62 +0x25
73772026-03-11T20:45:51.168Zsync.(*WaitGroup).Wait(0x44e7a0?)
73782026-03-11T20:45:51.168Z /work/go/src/sync/waitgroup.go:116 +0x48
73792026-03-11T20:45:51.168Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc007534d40)
73802026-03-11T20:45:51.168Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
73812026-03-11T20:45:51.168Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc007534d40?, {0x50b1338?, 0xc00540c280?}})
73822026-03-11T20:45:51.168Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
73832026-03-11T20:45:51.168Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed(0xc003d48008, {0x50b1338?, 0xc00540c1e0?}, {0xc0033fb8c0, 0x1, 0xc0045d3d68?}, {0x1?, 0x8?, 0x0?}, 0x1, ...)
73842026-03-11T20:45:51.168Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:139 +0x6c5
73852026-03-11T20:45:51.168Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*dbAdapter).RangeFeed(0xc004fa3aa0?, {0x50b1338?, 0xc00540c1e0?}, {0xc0033fb8c0?, 0x50b1338?, 0xc0013d2820?}, {0xc003c31320?, 0x4a22420?, 0xc0?}, 0x1, ...)
73862026-03-11T20:45:51.168Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/db_adapter.go:78 +0x3e
73872026-03-11T20:45:51.168Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func1({0x50b1338?, 0xc00540c1e0?})
73882026-03-11T20:45:51.168Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:285 +0x6c
73892026-03-11T20:45:51.168Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
73902026-03-11T20:45:51.168Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
73912026-03-11T20:45:51.168Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
73922026-03-11T20:45:51.168Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
73932026-03-11T20:45:51.168Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10883
73942026-03-11T20:45:51.169Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
73952026-03-11T20:45:51.169Z
73962026-03-11T20:45:51.169Zgoroutine 9874 [chan receive, 59 minutes]:
73972026-03-11T20:45:51.169Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc005572ae0?})
73982026-03-11T20:45:51.169Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
73992026-03-11T20:45:51.169Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x1000000000001?, 0xc000d0a370?}}, 0xc000e067a8)
74002026-03-11T20:45:51.169Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
74012026-03-11T20:45:51.169Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
74022026-03-11T20:45:51.169Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
74032026-03-11T20:45:51.177Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
74042026-03-11T20:45:51.177Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
74052026-03-11T20:45:51.177Z
74062026-03-11T20:45:51.177Zgoroutine 10066 [sync.Cond.Wait]:
74072026-03-11T20:45:51.177Zsync.runtime_notifyListWait(0xc001e22250, 0x377e)
74082026-03-11T20:45:51.178Z /work/go/src/runtime/sema.go:569 +0x159
74092026-03-11T20:45:51.178Zsync.(*Cond).Wait(0x3aac600?)
74102026-03-11T20:45:51.178Z /work/go/src/sync/cond.go:70 +0x85
74112026-03-11T20:45:51.178Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583740})
74122026-03-11T20:45:51.178Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
74132026-03-11T20:45:51.178Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
74142026-03-11T20:45:51.178Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
74152026-03-11T20:45:51.178Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
74162026-03-11T20:45:51.178Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
74172026-03-11T20:45:51.178Z
74182026-03-11T20:45:51.178Zgoroutine 10070 [sync.Cond.Wait]:
74192026-03-11T20:45:51.178Zsync.runtime_notifyListWait(0xc001e22250, 0x3769)
74202026-03-11T20:45:51.178Z /work/go/src/runtime/sema.go:569 +0x159
74212026-03-11T20:45:51.178Zsync.(*Cond).Wait(0x3aac600?)
74222026-03-11T20:45:51.178Z /work/go/src/sync/cond.go:70 +0x85
74232026-03-11T20:45:51.178Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583800})
74242026-03-11T20:45:51.178Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
74252026-03-11T20:45:51.178Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
74262026-03-11T20:45:51.178Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
74272026-03-11T20:45:51.178Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
74282026-03-11T20:45:51.178Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
74292026-03-11T20:45:51.178Z
74302026-03-11T20:45:51.178Zgoroutine 10200 [select, 59 minutes]:
74312026-03-11T20:45:51.178Zgithub.com/cockroachdb/cockroach/pkg/sql/sqlliveness/slstorage.(*Storage).deleteSessionsLoop(0xc0071df0e0, {0x50b1300?, 0xc00081ae40?})
74322026-03-11T20:45:51.178Z /work/oxidecomputer/cockroach/pkg/sql/sqlliveness/slstorage/slstorage.go:354 +0x165
74332026-03-11T20:45:51.178Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
74342026-03-11T20:45:51.178Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
74352026-03-11T20:45:51.178Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
74362026-03-11T20:45:51.178Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
74372026-03-11T20:45:51.178Z
74382026-03-11T20:45:51.178Zgoroutine 10067 [sync.Cond.Wait]:
74392026-03-11T20:45:51.178Zsync.runtime_notifyListWait(0xc001e22250, 0x3784)
74402026-03-11T20:45:51.178Z /work/go/src/runtime/sema.go:569 +0x159
74412026-03-11T20:45:51.178Zsync.(*Cond).Wait(0x3aac600?)
74422026-03-11T20:45:51.178Z /work/go/src/sync/cond.go:70 +0x85
74432026-03-11T20:45:51.178Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583770})
74442026-03-11T20:45:51.178Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
74452026-03-11T20:45:51.178Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
74462026-03-11T20:45:51.179Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
74472026-03-11T20:45:51.179Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
74482026-03-11T20:45:51.179Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
74492026-03-11T20:45:51.179Z
74502026-03-11T20:45:51.179Zgoroutine 10073 [sync.Cond.Wait]:
74512026-03-11T20:45:51.179Zsync.runtime_notifyListWait(0xc001e22250, 0x3773)
74522026-03-11T20:45:51.179Z /work/go/src/runtime/sema.go:569 +0x159
74532026-03-11T20:45:51.179Zsync.(*Cond).Wait(0x3aac600?)
74542026-03-11T20:45:51.179Z /work/go/src/sync/cond.go:70 +0x85
74552026-03-11T20:45:51.179Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583890})
74562026-03-11T20:45:51.179Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
74572026-03-11T20:45:51.179Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
74582026-03-11T20:45:51.179Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
74592026-03-11T20:45:51.179Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
74602026-03-11T20:45:51.179Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
74612026-03-11T20:45:51.179Z
74622026-03-11T20:45:51.179Zgoroutine 11069 [select]:
74632026-03-11T20:45:51.179Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents(0xc00426f8c0, {0x50b1338, 0xc00540c1e0}, 0xc0033fb830, 0xc0021a0060)
74642026-03-11T20:45:51.179Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:331 +0xc5
74652026-03-11T20:45:51.179Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func2({0x50b1338?, 0xc00540c1e0?})
74662026-03-11T20:45:51.179Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:288 +0x32
74672026-03-11T20:45:51.179Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
74682026-03-11T20:45:51.179Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
74692026-03-11T20:45:51.179Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
74702026-03-11T20:45:51.179Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
74712026-03-11T20:45:51.179Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10883
74722026-03-11T20:45:51.179Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
74732026-03-11T20:45:51.179Z
74742026-03-11T20:45:51.180Zgoroutine 10781 [sync.Cond.Wait, 59 minutes]:
74752026-03-11T20:45:51.180Zsync.runtime_notifyListWait(0xc0064c5050, 0x40)
74762026-03-11T20:45:51.180Z /work/go/src/runtime/sema.go:569 +0x159
74772026-03-11T20:45:51.180Zsync.(*Cond).Wait(0xc002051650?)
74782026-03-11T20:45:51.180Z /work/go/src/sync/cond.go:70 +0x85
74792026-03-11T20:45:51.180Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3710})
74802026-03-11T20:45:51.180Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
74812026-03-11T20:45:51.180Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
74822026-03-11T20:45:51.180Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
74832026-03-11T20:45:51.180Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
74842026-03-11T20:45:51.180Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
74852026-03-11T20:45:51.180Z
74862026-03-11T20:45:51.180Zgoroutine 10865 [select]:
74872026-03-11T20:45:51.180Zgithub.com/cockroachdb/cockroach/pkg/sql/stats.(*Refresher).Start.func1({0x50b1300, 0xc00067d080})
74882026-03-11T20:45:51.180Z /work/oxidecomputer/cockroach/pkg/sql/stats/automatic_stats.go:374 +0x1b9
74892026-03-11T20:45:51.180Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
74902026-03-11T20:45:51.180Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
74912026-03-11T20:45:51.180Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
74922026-03-11T20:45:51.180Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
74932026-03-11T20:45:51.180Z
74942026-03-11T20:45:51.180Zgoroutine 10686 [chan receive, 59 minutes]:
74952026-03-11T20:45:51.180Zgithub.com/cockroachdb/cockroach/pkg/server.configureGRPCGateway.func5({0x50b1300, 0xc006e62a80})
74962026-03-11T20:45:51.180Z /work/oxidecomputer/cockroach/pkg/server/grpc_gateway.go:127 +0x45
74972026-03-11T20:45:51.181Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
74982026-03-11T20:45:51.181Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
74992026-03-11T20:45:51.181Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
75002026-03-11T20:45:51.181Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
75012026-03-11T20:45:51.181Z
75022026-03-11T20:45:51.181Zgoroutine 9852 [select, 4 minutes]:
75032026-03-11T20:45:51.186Zgithub.com/cockroachdb/cockroach/pkg/server.startPurgeOldSessions.func1({0xc0073bd808?, 0xc004f0bbc0?})
75042026-03-11T20:45:51.186Z /work/oxidecomputer/cockroach/pkg/server/purge_auth_session.go:69 +0x205
75052026-03-11T20:45:51.186Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
75062026-03-11T20:45:51.186Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
75072026-03-11T20:45:51.186Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
75082026-03-11T20:45:51.186Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
75092026-03-11T20:45:51.186Z
75102026-03-11T20:45:51.186Zgoroutine 9999 [chan receive, 59 minutes]:
75112026-03-11T20:45:51.186Zgithub.com/cockroachdb/cockroach/pkg/gossip.(*server).start.func3({0xc0022c3208?, 0x0?})
75122026-03-11T20:45:51.186Z /work/oxidecomputer/cockroach/pkg/gossip/server.go:380 +0x48
75132026-03-11T20:45:51.186Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
75142026-03-11T20:45:51.186Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
75152026-03-11T20:45:51.186Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
75162026-03-11T20:45:51.186Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
75172026-03-11T20:45:51.186Z
75182026-03-11T20:45:51.186Zgoroutine 10738 [sync.Cond.Wait, 59 minutes]:
75192026-03-11T20:45:51.186Zsync.runtime_notifyListWait(0xc0064c5050, 0xc)
75202026-03-11T20:45:51.186Z /work/go/src/runtime/sema.go:569 +0x159
75212026-03-11T20:45:51.187Zsync.(*Cond).Wait(0xc000f39208?)
75222026-03-11T20:45:51.187Z /work/go/src/sync/cond.go:70 +0x85
75232026-03-11T20:45:51.187Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd2f00})
75242026-03-11T20:45:51.187Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
75252026-03-11T20:45:51.187Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
75262026-03-11T20:45:51.187Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
75272026-03-11T20:45:51.187Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
75282026-03-11T20:45:51.187Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
75292026-03-11T20:45:51.187Z
75302026-03-11T20:45:51.187Zgoroutine 10477 [select]:
75312026-03-11T20:45:51.187Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).outputLoop(0xc00296a000, {0x50b1338, 0xc003a62d70})
75322026-03-11T20:45:51.187Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:310 +0x247
75332026-03-11T20:45:51.187Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).runOutputLoop(0xc00296a000, {0x50b1300, 0xc004bb02a0}, 0x1?)
75342026-03-11T20:45:51.187Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:335 +0xba
75352026-03-11T20:45:51.187Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run.func1({0x50b1300?, 0xc004bb02a0?})
75362026-03-11T20:45:51.187Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:297 +0x46
75372026-03-11T20:45:51.187Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
75382026-03-11T20:45:51.187Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
75392026-03-11T20:45:51.187Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10334
75402026-03-11T20:45:51.187Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
75412026-03-11T20:45:51.187Z
75422026-03-11T20:45:51.187Zgoroutine 9805 [select]:
75432026-03-11T20:45:51.187Zgithub.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
75442026-03-11T20:45:51.187Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:439 +0x1c9
75452026-03-11T20:45:51.187Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 9708
75462026-03-11T20:45:51.187Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:412 +0x4f
75472026-03-11T20:45:51.187Z
75482026-03-11T20:45:51.187Zgoroutine 10583 [select]:
75492026-03-11T20:45:51.187Zgithub.com/cockroachdb/cockroach/pkg/ts.(*poller).start.func1({0x50b1300, 0xc0064c1890})
75502026-03-11T20:45:51.187Z /work/oxidecomputer/cockroach/pkg/ts/db.go:176 +0xea
75512026-03-11T20:45:51.187Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
75522026-03-11T20:45:51.187Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
75532026-03-11T20:45:51.187Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
75542026-03-11T20:45:51.187Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
75552026-03-11T20:45:51.187Z
75562026-03-11T20:45:51.187Zgoroutine 10049 [sync.Cond.Wait]:
75572026-03-11T20:45:51.187Zsync.runtime_notifyListWait(0xc001e22250, 0x3765)
75582026-03-11T20:45:51.188Z /work/go/src/runtime/sema.go:569 +0x159
75592026-03-11T20:45:51.188Zsync.(*Cond).Wait(0x3aac600?)
75602026-03-11T20:45:51.188Z /work/go/src/sync/cond.go:70 +0x85
75612026-03-11T20:45:51.188Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583410})
75622026-03-11T20:45:51.188Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
75632026-03-11T20:45:51.188Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
75642026-03-11T20:45:51.188Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
75652026-03-11T20:45:51.188Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
75662026-03-11T20:45:51.188Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
75672026-03-11T20:45:51.188Z
75682026-03-11T20:45:51.188Zgoroutine 10817 [select, 59 minutes]:
75692026-03-11T20:45:51.188Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Start.func3({0xc007138008?, 0x0?})
75702026-03-11T20:45:51.188Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:1978 +0xb2
75712026-03-11T20:45:51.188Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
75722026-03-11T20:45:51.188Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
75732026-03-11T20:45:51.188Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
75742026-03-11T20:45:51.188Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
75752026-03-11T20:45:51.188Z
75762026-03-11T20:45:51.188Zgoroutine 10110 [sync.Cond.Wait]:
75772026-03-11T20:45:51.188Zsync.runtime_notifyListWait(0xc001e22250, 0x3768)
75782026-03-11T20:45:51.188Z /work/go/src/runtime/sema.go:569 +0x159
75792026-03-11T20:45:51.188Zsync.(*Cond).Wait(0x3aac600?)
75802026-03-11T20:45:51.188Z /work/go/src/sync/cond.go:70 +0x85
75812026-03-11T20:45:51.188Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc003ebc000})
75822026-03-11T20:45:51.188Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
75832026-03-11T20:45:51.188Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
75842026-03-11T20:45:51.188Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
75852026-03-11T20:45:51.189Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
75862026-03-11T20:45:51.189Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
75872026-03-11T20:45:51.189Z
75882026-03-11T20:45:51.189Zgoroutine 10561 [select, 1 minutes]:
75892026-03-11T20:45:51.189Zgithub.com/cockroachdb/cockroach/pkg/kv/kvprober.(*Prober).Start.func1.1({0x50b1300, 0xc0066f5710})
75902026-03-11T20:45:51.189Z /work/oxidecomputer/cockroach/pkg/kv/kvprober/kvprober.go:293 +0x314
75912026-03-11T20:45:51.189Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
75922026-03-11T20:45:51.189Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
75932026-03-11T20:45:51.189Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
75942026-03-11T20:45:51.189Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
75952026-03-11T20:45:51.189Z
75962026-03-11T20:45:51.189Zgoroutine 9796 [select]:
75972026-03-11T20:45:51.189Zgithub.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue.func2()
75982026-03-11T20:45:51.189Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:366 +0x85
75992026-03-11T20:45:51.189Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue in goroutine 9708
76002026-03-11T20:45:51.189Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:363 +0x3ed
76012026-03-11T20:45:51.189Z
76022026-03-11T20:45:51.189Zgoroutine 10001 [select]:
76032026-03-11T20:45:51.423Zgithub.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).manage.func1({0x50b1300, 0xc006b5b380})
76042026-03-11T20:45:51.423Z /work/oxidecomputer/cockroach/pkg/gossip/gossip.go:1420 +0x2d2
76052026-03-11T20:45:51.423Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
76062026-03-11T20:45:51.423Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
76072026-03-11T20:45:51.423Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
76082026-03-11T20:45:51.423Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
76092026-03-11T20:45:51.423Z
76102026-03-11T20:45:51.423Zgoroutine 11137 [select]:
76112026-03-11T20:45:51.423Zgoogle.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc0010bd540, {0xc005dbfe70, 0x5, 0x5})
76122026-03-11T20:45:51.423Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:191 +0x95
76132026-03-11T20:45:51.423Zgoogle.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc0010bd540, {0xc005dbfe70?, 0xc0044004e0?, 0xc00408d048?})
76142026-03-11T20:45:51.423Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:171 +0x12d
76152026-03-11T20:45:51.423Zgoogle.golang.org/grpc/internal/transport.(*transportReader).Read(0xc004bb05a0, {0xc005dbfe70?, 0xc00408d0c0?, 0xa40865?})
76162026-03-11T20:45:51.423Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:485 +0x2c
76172026-03-11T20:45:51.423Zio.ReadAtLeast({0x5086380, 0xc004bb05a0}, {0xc005dbfe70, 0x5, 0x5}, 0x5)
76182026-03-11T20:45:51.425Z /work/go/src/io/io.go:335 +0x90
76192026-03-11T20:45:51.425Zio.ReadFull(...)
76202026-03-11T20:45:51.425Z /work/go/src/io/io.go:354
76212026-03-11T20:45:51.425Zgoogle.golang.org/grpc/internal/transport.(*Stream).Read(0xc001a65b00, {0xc005dbfe70, 0x5, 0x5})
76222026-03-11T20:45:51.425Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:469 +0x96
76232026-03-11T20:45:51.425Zgoogle.golang.org/grpc.(*parser).recvMsg(0xc005dbfe60, 0x7fffffff)
76242026-03-11T20:45:51.425Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:559 +0x42
76252026-03-11T20:45:51.425Zgoogle.golang.org/grpc.recvAndDecompress(0x60?, 0xc001a65b00, {0x0, 0x0}, 0x7fffffff, 0xc00408d378, {0x50a8e40, 0x77577e0})
76262026-03-11T20:45:51.425Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:690 +0x5a
76272026-03-11T20:45:51.425Zgoogle.golang.org/grpc.recv(0x50900a0?, {0x7fc12cff4640, 0xc0004ce590}, 0xc00408d300?, {0x0?, 0x0?}, {0x3f6ba40, 0xc00b0e67a0}, 0x10?, 0xc00408d378, ...)
76282026-03-11T20:45:51.425Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:756 +0x68
76292026-03-11T20:45:51.425Zgoogle.golang.org/grpc.(*csAttempt).recvMsg(0xc0064333f0, {0x3f6ba40, 0xc00b0e67a0}, 0x10?)
76302026-03-11T20:45:51.425Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:983 +0x27e
76312026-03-11T20:45:51.425Zgoogle.golang.org/grpc.(*clientStream).RecvMsg.func1(0x41e1e5?)
76322026-03-11T20:45:51.425Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:834 +0x1f
76332026-03-11T20:45:51.425Zgoogle.golang.org/grpc.(*clientStream).withRetry(0xc001a65680, 0xc00408d4a8, 0xc00408d498)
76342026-03-11T20:45:51.425Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:688 +0x2ee
76352026-03-11T20:45:51.425Zgoogle.golang.org/grpc.(*clientStream).RecvMsg(0xc001a65680, {0x3f6ba40?, 0xc00b0e67a0?})
76362026-03-11T20:45:51.425Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:833 +0x107
76372026-03-11T20:45:51.425Zgithub.com/cockroachdb/cockroach/pkg/util/tracing.(*tracingClientStream).RecvMsg(0xc0003e23c0, {0x3f6ba40?, 0xc00b0e67a0?})
76382026-03-11T20:45:51.425Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:441 +0x31
76392026-03-11T20:45:51.425Zgithub.com/cockroachdb/cockroach/pkg/roachpb.(*internalRangeFeedClient).Recv(0xc006416e40)
76402026-03-11T20:45:51.425Z /work/oxidecomputer/cockroach/pkg/roachpb/api.pb.go:9289 +0x46
76412026-03-11T20:45:51.425Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed.func3()
76422026-03-11T20:45:51.425Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:516 +0x25
76432026-03-11T20:45:51.425Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*stuckRangeFeedCanceler).do(0xc0010bd400, 0xc00408da50)
76442026-03-11T20:45:51.425Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed_canceler.go:100 +0x1b3
76452026-03-11T20:45:51.425Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed(0xc003d48008, {0x50b1338?, 0xc0010bd310?}, {{0xc00642ebe8, 0x1, 0x8}, {0xc00642ebf0, 0x1, 0x8}}, {0x189be113f3ce56a5, ...}, ...)
76462026-03-11T20:45:51.425Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:515 +0xc98
76472026-03-11T20:45:51.425Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).partialRangeFeed(0xc003d48008, {0x50b1338, 0xc0010bd310}, 0xc0010bd2c0, {{0xc00642ebe8, 0x1, 0x8}, {0xc00642ebf0, 0x1, 0x8}}, ...)
76482026-03-11T20:45:51.425Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:336 +0x745
76492026-03-11T20:45:51.425Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.1({0x50b1338?, 0xc0010bd310?})
76502026-03-11T20:45:51.425Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:120 +0xa9
76512026-03-11T20:45:51.425Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.Group.GoCtx.2()
76522026-03-11T20:45:51.425Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
76532026-03-11T20:45:51.425Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
76542026-03-11T20:45:51.425Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
76552026-03-11T20:45:51.426Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 11135
76562026-03-11T20:45:51.426Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
76572026-03-11T20:45:51.426Z
76582026-03-11T20:45:51.426Zgoroutine 9955 [select, 59 minutes]:
76592026-03-11T20:45:51.426Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initCancelingWorkers.func1({0x50b1300, 0xc007659920})
76602026-03-11T20:45:51.426Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:139 +0xc5
76612026-03-11T20:45:51.426Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
76622026-03-11T20:45:51.426Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
76632026-03-11T20:45:51.426Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
76642026-03-11T20:45:51.426Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
76652026-03-11T20:45:51.426Z
76662026-03-11T20:45:51.426Zgoroutine 10500 [select]:
76672026-03-11T20:45:51.426Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).outputLoop(0xc002759340, {0x50b1338, 0xc003c52190})
76682026-03-11T20:45:51.426Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:310 +0x247
76692026-03-11T20:45:51.426Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).runOutputLoop(0xc002759340, {0x50b1300, 0xc004230960}, 0x3?)
76702026-03-11T20:45:51.426Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:335 +0xba
76712026-03-11T20:45:51.427Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run.func1({0x50b1300?, 0xc004230960?})
76722026-03-11T20:45:51.427Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:297 +0x46
76732026-03-11T20:45:51.427Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
76742026-03-11T20:45:51.427Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
76752026-03-11T20:45:51.427Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10498
76762026-03-11T20:45:51.427Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
76772026-03-11T20:45:51.427Z
76782026-03-11T20:45:51.427Zgoroutine 10303 [select]:
76792026-03-11T20:45:51.427Zgithub.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc000e28800, {0x50b1300, 0xc0024ffe90}, 0xc00785fdb0, {0xc0041a99e0, 0xf}, 0xc002f90960)
76802026-03-11T20:45:51.427Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:1462 +0x27b
76812026-03-11T20:45:51.427Zgithub.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1({0x50b1300, 0xc0024ffe90})
76822026-03-11T20:45:51.427Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:1380 +0x85
76832026-03-11T20:45:51.427Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
76842026-03-11T20:45:51.427Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
76852026-03-11T20:45:51.427Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
76862026-03-11T20:45:51.427Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
76872026-03-11T20:45:51.427Z
76882026-03-11T20:45:51.427Zgoroutine 10104 [sync.Cond.Wait]:
76892026-03-11T20:45:51.427Zsync.runtime_notifyListWait(0xc001e22250, 0x37a2)
76902026-03-11T20:45:51.427Z /work/go/src/runtime/sema.go:569 +0x159
76912026-03-11T20:45:51.427Zsync.(*Cond).Wait(0x3aac600?)
76922026-03-11T20:45:51.427Z /work/go/src/sync/cond.go:70 +0x85
76932026-03-11T20:45:51.427Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583e60})
76942026-03-11T20:45:51.427Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
76952026-03-11T20:45:51.427Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
76962026-03-11T20:45:51.427Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
76972026-03-11T20:45:51.427Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
76982026-03-11T20:45:51.427Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
76992026-03-11T20:45:51.427Z
77002026-03-11T20:45:51.427Zgoroutine 10091 [sync.Cond.Wait]:
77012026-03-11T20:45:51.427Zsync.runtime_notifyListWait(0xc001e22250, 0x3799)
77022026-03-11T20:45:51.427Z /work/go/src/runtime/sema.go:569 +0x159
77032026-03-11T20:45:51.437Zsync.(*Cond).Wait(0x3aac600?)
77042026-03-11T20:45:51.437Z /work/go/src/sync/cond.go:70 +0x85
77052026-03-11T20:45:51.437Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583bf0})
77062026-03-11T20:45:51.437Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
77072026-03-11T20:45:51.437Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
77082026-03-11T20:45:51.438Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
77092026-03-11T20:45:51.438Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
77102026-03-11T20:45:51.438Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
77112026-03-11T20:45:51.438Z
77122026-03-11T20:45:51.438Zgoroutine 9977 [select]:
77132026-03-11T20:45:51.438Zgoogle.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc003aafb80, 0x1)
77142026-03-11T20:45:51.438Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:407 +0x113
77152026-03-11T20:45:51.438Zgoogle.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc001081c80)
77162026-03-11T20:45:51.438Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:534 +0x7b
77172026-03-11T20:45:51.438Zgoogle.golang.org/grpc/internal/transport.NewServerTransport.func2()
77182026-03-11T20:45:51.438Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:332 +0xc6
77192026-03-11T20:45:51.438Zcreated by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 9974
77202026-03-11T20:45:51.438Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:329 +0x184c
77212026-03-11T20:45:51.438Z
77222026-03-11T20:45:51.438Zgoroutine 10444 [semacquire, 59 minutes]:
77232026-03-11T20:45:51.438Zsync.runtime_Semacquire(0xc007ee0120?)
77242026-03-11T20:45:51.438Z /work/go/src/runtime/sema.go:62 +0x25
77252026-03-11T20:45:51.438Zsync.(*WaitGroup).Wait(0x44e7a0?)
77262026-03-11T20:45:51.438Z /work/go/src/sync/waitgroup.go:116 +0x48
77272026-03-11T20:45:51.439Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc006042b40)
77282026-03-11T20:45:51.439Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
77292026-03-11T20:45:51.439Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc006042b40?, {0x50b1338?, 0xc005f399f0?}})
77302026-03-11T20:45:51.439Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
77312026-03-11T20:45:51.439Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed(0xc002d6f408, {0x50b1338?, 0xc003eef5e0?}, {0xc0040cb770, 0x1, 0xc00530a5c0?}, {0xc001e86748?, 0x180fe75?, 0x0?}, 0x1, ...)
77322026-03-11T20:45:51.439Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:139 +0x6c5
77332026-03-11T20:45:51.439Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*dbAdapter).RangeFeed(0x1f43f824a2b1bb?, {0x50b1338?, 0xc003eef5e0?}, {0xc0040cb770?, 0x0?, 0xc0046e2a08?}, {0x50b1300?, 0x407d3b0?, 0xc0?}, 0x1, ...)
77342026-03-11T20:45:51.439Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/db_adapter.go:78 +0x3e
77352026-03-11T20:45:51.439Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func1({0x50b1338?, 0xc003eef5e0?})
77362026-03-11T20:45:51.439Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:285 +0x6c
77372026-03-11T20:45:51.439Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
77382026-03-11T20:45:51.439Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
77392026-03-11T20:45:51.439Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
77402026-03-11T20:45:51.439Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
77412026-03-11T20:45:51.439Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10179
77422026-03-11T20:45:51.439Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
77432026-03-11T20:45:51.439Z
77442026-03-11T20:45:51.439Zgoroutine 10795 [sync.Cond.Wait, 59 minutes]:
77452026-03-11T20:45:51.439Zsync.runtime_notifyListWait(0xc0064c5050, 0x5a)
77462026-03-11T20:45:51.439Z /work/go/src/runtime/sema.go:569 +0x159
77472026-03-11T20:45:51.439Zsync.(*Cond).Wait(0x50f7930?)
77482026-03-11T20:45:51.439Z /work/go/src/sync/cond.go:70 +0x85
77492026-03-11T20:45:51.439Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd39b0})
77502026-03-11T20:45:51.439Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
77512026-03-11T20:45:51.439Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
77522026-03-11T20:45:51.439Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
77532026-03-11T20:45:51.439Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
77542026-03-11T20:45:51.439Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
77552026-03-11T20:45:51.439Z
77562026-03-11T20:45:51.439Zgoroutine 10056 [sync.Cond.Wait]:
77572026-03-11T20:45:51.440Zsync.runtime_notifyListWait(0xc001e22250, 0x376d)
77582026-03-11T20:45:51.440Z /work/go/src/runtime/sema.go:569 +0x159
77592026-03-11T20:45:51.440Zsync.(*Cond).Wait(0x3aac600?)
77602026-03-11T20:45:51.440Z /work/go/src/sync/cond.go:70 +0x85
77612026-03-11T20:45:51.440Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583560})
77622026-03-11T20:45:51.440Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
77632026-03-11T20:45:51.440Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
77642026-03-11T20:45:51.440Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
77652026-03-11T20:45:51.440Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
77662026-03-11T20:45:51.440Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
77672026-03-11T20:45:51.440Z
77682026-03-11T20:45:51.440Zgoroutine 10476 [chan receive, 59 minutes]:
77692026-03-11T20:45:51.440Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).rangeFeedWithRangeID(0xc001ded408, 0x3f3de40?, 0xc006052240, {0x7fc12cefd140, 0xc006c46c20})
77702026-03-11T20:45:51.440Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:249 +0x76b
77712026-03-11T20:45:51.440Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed(...)
77722026-03-11T20:45:51.440Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:149
77732026-03-11T20:45:51.440Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).RangeFeed(0xc000f39208, 0xc006052240, {0x50e5b30, 0xc006c46c20})
77742026-03-11T20:45:51.441Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:3082 +0x16f
77752026-03-11T20:45:51.441Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).RangeFeed(0xc001eb4380, 0xc006052240, {0x50e5b30, 0xc006c46c20})
77762026-03-11T20:45:51.441Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/stores.go:216 +0x19a
77772026-03-11T20:45:51.441Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed(0x3f3de40?, 0xc003a62d50?, {0x50e5b30, 0xc006c46c20})
77782026-03-11T20:45:51.441Z /work/oxidecomputer/cockroach/pkg/server/node.go:1232 +0x28
77792026-03-11T20:45:51.441Zgithub.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed.func1()
77802026-03-11T20:45:51.441Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:753 +0xa7
77812026-03-11T20:45:51.441Zcreated by github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed in goroutine 10475
77822026-03-11T20:45:51.441Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:750 +0x17c
77832026-03-11T20:45:51.441Z
77842026-03-11T20:45:51.441Zgoroutine 10577 [select]:
77852026-03-11T20:45:51.441Zgoogle.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc003d1a1a0)
77862026-03-11T20:45:51.441Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1131 +0x205
77872026-03-11T20:45:51.441Zcreated by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 10669
77882026-03-11T20:45:51.441Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:341 +0x188e
77892026-03-11T20:45:51.441Z
77902026-03-11T20:45:51.441Zgoroutine 11307 [select]:
77912026-03-11T20:45:51.441Zgoogle.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc008057db0, {0xc005bc1b88, 0x5, 0x5})
77922026-03-11T20:45:51.441Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:191 +0x95
77932026-03-11T20:45:51.441Zgoogle.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc008057db0, {0xc005bc1b88?, 0xc0033d7f80?, 0xc004009048?})
77942026-03-11T20:45:51.441Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:171 +0x12d
77952026-03-11T20:45:51.441Zgoogle.golang.org/grpc/internal/transport.(*transportReader).Read(0xc007e53110, {0xc005bc1b88?, 0xc0040090c0?, 0xa40865?})
77962026-03-11T20:45:51.441Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:485 +0x2c
77972026-03-11T20:45:51.441Zio.ReadAtLeast({0x5086380, 0xc007e53110}, {0xc005bc1b88, 0x5, 0x5}, 0x5)
77982026-03-11T20:45:51.441Z /work/go/src/io/io.go:335 +0x90
77992026-03-11T20:45:51.441Zio.ReadFull(...)
78002026-03-11T20:45:51.441Z /work/go/src/io/io.go:354
78012026-03-11T20:45:51.441Zgoogle.golang.org/grpc/internal/transport.(*Stream).Read(0xc0080a0000, {0xc005bc1b88, 0x5, 0x5})
78022026-03-11T20:45:51.441Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:469 +0x96
78032026-03-11T20:45:51.448Zgoogle.golang.org/grpc.(*parser).recvMsg(0xc005bc1b78, 0x7fffffff)
78042026-03-11T20:45:51.448Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:559 +0x42
78052026-03-11T20:45:51.448Zgoogle.golang.org/grpc.recvAndDecompress(0x60?, 0xc0080a0000, {0x0, 0x0}, 0x7fffffff, 0xc004009378, {0x50a8e40, 0x77577e0})
78062026-03-11T20:45:51.448Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:690 +0x5a
78072026-03-11T20:45:51.448Zgoogle.golang.org/grpc.recv(0x50900a0?, {0x7fc12cff4640, 0xc0004ce590}, 0x2b?, {0x0?, 0x0?}, {0x3f6ba40, 0xc006f9a3e0}, 0x10?, 0xc004009378, ...)
78082026-03-11T20:45:51.448Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:756 +0x68
78092026-03-11T20:45:51.448Zgoogle.golang.org/grpc.(*csAttempt).recvMsg(0xc00809a0b0, {0x3f6ba40, 0xc006f9a3e0}, 0x10?)
78102026-03-11T20:45:51.448Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:983 +0x27e
78112026-03-11T20:45:51.448Zgoogle.golang.org/grpc.(*clientStream).RecvMsg.func1(0x41e1e5?)
78122026-03-11T20:45:51.448Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:834 +0x1f
78132026-03-11T20:45:51.448Zgoogle.golang.org/grpc.(*clientStream).withRetry(0xc00806bd40, 0xc0040094a8, 0xc004009498)
78142026-03-11T20:45:51.448Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:688 +0x2ee
78152026-03-11T20:45:51.448Zgoogle.golang.org/grpc.(*clientStream).RecvMsg(0xc00806bd40, {0x3f6ba40?, 0xc006f9a3e0?})
78162026-03-11T20:45:51.448Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:833 +0x107
78172026-03-11T20:45:51.448Zgithub.com/cockroachdb/cockroach/pkg/util/tracing.(*tracingClientStream).RecvMsg(0xc0062e5ae0, {0x3f6ba40?, 0xc006f9a3e0?})
78182026-03-11T20:45:51.448Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:441 +0x31
78192026-03-11T20:45:51.448Zgithub.com/cockroachdb/cockroach/pkg/roachpb.(*internalRangeFeedClient).Recv(0xc0060ba0f0)
78202026-03-11T20:45:51.448Z /work/oxidecomputer/cockroach/pkg/roachpb/api.pb.go:9289 +0x46
78212026-03-11T20:45:51.448Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed.func3()
78222026-03-11T20:45:51.448Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:516 +0x25
78232026-03-11T20:45:51.448Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*stuckRangeFeedCanceler).do(0xc008057c70, 0xc004009a50)
78242026-03-11T20:45:51.448Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed_canceler.go:100 +0x1b3
78252026-03-11T20:45:51.448Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed(0xc003d48008, {0x50b1338?, 0xc003aae050?}, {{0xc008584780, 0x1, 0x8}, {0xc008584788, 0x1, 0x8}}, {0x189be113e956a777, ...}, ...)
78262026-03-11T20:45:51.448Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:515 +0xc98
78272026-03-11T20:45:51.449Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).partialRangeFeed(0xc003d48008, {0x50b1338, 0xc003aae050}, 0xc003aae000, {{0xc008584780, 0x1, 0x8}, {0xc008584788, 0x1, 0x8}}, ...)
78282026-03-11T20:45:51.449Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:336 +0x745
78292026-03-11T20:45:51.449Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.1({0x50b1338?, 0xc003aae050?})
78302026-03-11T20:45:51.449Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:120 +0xa9
78312026-03-11T20:45:51.449Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.Group.GoCtx.2()
78322026-03-11T20:45:51.449Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
78332026-03-11T20:45:51.449Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
78342026-03-11T20:45:51.449Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
78352026-03-11T20:45:51.449Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9665
78362026-03-11T20:45:51.449Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
78372026-03-11T20:45:51.449Z
78382026-03-11T20:45:51.449Zgoroutine 11122 [semacquire, 59 minutes]:
78392026-03-11T20:45:51.449Zsync.runtime_Semacquire(0xc005dbe378?)
78402026-03-11T20:45:51.449Z /work/go/src/runtime/sema.go:62 +0x25
78412026-03-11T20:45:51.449Zsync.(*WaitGroup).Wait(0x44e7a0?)
78422026-03-11T20:45:51.449Z /work/go/src/sync/waitgroup.go:116 +0x48
78432026-03-11T20:45:51.450Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc000ee6bc0)
78442026-03-11T20:45:51.450Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
78452026-03-11T20:45:51.450Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc000ee6bc0?, {0x50b1338?, 0xc0013d28c0?}})
78462026-03-11T20:45:51.450Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
78472026-03-11T20:45:51.450Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed(0xc003d48008, {0x50b1338?, 0xc0013d2820?}, {0xc003c31410, 0x1, 0x10?}, {0xc007661ad0?, 0x55b21e1?, 0xc0?}, 0x0, ...)
78482026-03-11T20:45:51.450Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:139 +0x6c5
78492026-03-11T20:45:51.450Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*dbAdapter).RangeFeed(0x1a8b24fc351132?, {0x50b1338?, 0xc0013d2820?}, {0xc003c31410?, 0x0?, 0xc004096a08?}, {0x50b1300?, 0x6341c50?, 0xc0?}, 0x0, ...)
78502026-03-11T20:45:51.450Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/db_adapter.go:78 +0x3e
78512026-03-11T20:45:51.450Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func1({0x50b1338?, 0xc0013d2820?})
78522026-03-11T20:45:51.450Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:285 +0x6c
78532026-03-11T20:45:51.450Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
78542026-03-11T20:45:51.450Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
78552026-03-11T20:45:51.450Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
78562026-03-11T20:45:51.450Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
78572026-03-11T20:45:51.450Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10838
78582026-03-11T20:45:51.450Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
78592026-03-11T20:45:51.451Z
78602026-03-11T20:45:51.451Zgoroutine 9941 [select, 59 minutes]:
78612026-03-11T20:45:51.451Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc00167f508?, 0xc004f18030?})
78622026-03-11T20:45:51.451Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
78632026-03-11T20:45:51.451Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
78642026-03-11T20:45:51.451Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
78652026-03-11T20:45:51.451Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
78662026-03-11T20:45:51.451Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
78672026-03-11T20:45:51.451Z
78682026-03-11T20:45:51.451Zgoroutine 9067 [select, 59 minutes]:
78692026-03-11T20:45:51.451Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2({0x50b1300, 0xc0076b66f0})
78702026-03-11T20:45:51.451Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/queue.go:823 +0xff
78712026-03-11T20:45:51.451Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
78722026-03-11T20:45:51.451Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
78732026-03-11T20:45:51.451Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10821
78742026-03-11T20:45:51.451Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
78752026-03-11T20:45:51.451Z
78762026-03-11T20:45:51.452Zgoroutine 10749 [sync.Cond.Wait, 59 minutes]:
78772026-03-11T20:45:51.452Zsync.runtime_notifyListWait(0xc0064c5050, 0x16)
78782026-03-11T20:45:51.452Z /work/go/src/runtime/sema.go:569 +0x159
78792026-03-11T20:45:51.452Zsync.(*Cond).Wait(0x50b1300?)
78802026-03-11T20:45:51.452Z /work/go/src/sync/cond.go:70 +0x85
78812026-03-11T20:45:51.452Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3110})
78822026-03-11T20:45:51.452Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
78832026-03-11T20:45:51.452Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
78842026-03-11T20:45:51.452Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
78852026-03-11T20:45:51.452Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
78862026-03-11T20:45:51.452Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
78872026-03-11T20:45:51.452Z
78882026-03-11T20:45:51.452Zgoroutine 10796 [sync.Cond.Wait, 59 minutes]:
78892026-03-11T20:45:51.452Zsync.runtime_notifyListWait(0xc0064c5050, 0x5c)
78902026-03-11T20:45:51.452Z /work/go/src/runtime/sema.go:569 +0x159
78912026-03-11T20:45:51.452Zsync.(*Cond).Wait(0x50f7930?)
78922026-03-11T20:45:51.452Z /work/go/src/sync/cond.go:70 +0x85
78932026-03-11T20:45:51.452Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd39e0})
78942026-03-11T20:45:51.452Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
78952026-03-11T20:45:51.452Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
78962026-03-11T20:45:51.452Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
78972026-03-11T20:45:51.452Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
78982026-03-11T20:45:51.452Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
78992026-03-11T20:45:51.452Z
79002026-03-11T20:45:51.452Zgoroutine 9778 [chan receive, 59 minutes]:
79012026-03-11T20:45:51.452Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc0055720f0?})
79022026-03-11T20:45:51.452Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
79032026-03-11T20:45:51.465Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x0?, 0xc0009b62c0?}}, 0xc002be6fa8)
79042026-03-11T20:45:51.465Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
79052026-03-11T20:45:51.465Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
79062026-03-11T20:45:51.465Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
79072026-03-11T20:45:51.465Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
79082026-03-11T20:45:51.465Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
79092026-03-11T20:45:51.465Z
79102026-03-11T20:45:51.465Zgoroutine 9025 [select, 59 minutes]:
79112026-03-11T20:45:51.465Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc00167e908?, 0xc00114906c?})
79122026-03-11T20:45:51.465Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
79132026-03-11T20:45:51.465Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
79142026-03-11T20:45:51.465Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
79152026-03-11T20:45:51.465Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
79162026-03-11T20:45:51.465Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
79172026-03-11T20:45:51.466Z
79182026-03-11T20:45:51.466Zgoroutine 10039 [sync.Cond.Wait]:
79192026-03-11T20:45:51.466Zsync.runtime_notifyListWait(0xc001e22250, 0x374d)
79202026-03-11T20:45:51.466Z /work/go/src/runtime/sema.go:569 +0x159
79212026-03-11T20:45:51.466Zsync.(*Cond).Wait(0x3aac600?)
79222026-03-11T20:45:51.466Z /work/go/src/sync/cond.go:70 +0x85
79232026-03-11T20:45:51.466Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583230})
79242026-03-11T20:45:51.466Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
79252026-03-11T20:45:51.466Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
79262026-03-11T20:45:51.466Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
79272026-03-11T20:45:51.466Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
79282026-03-11T20:45:51.466Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
79292026-03-11T20:45:51.466Z
79302026-03-11T20:45:51.466Zgoroutine 10966 [select]:
79312026-03-11T20:45:51.466Zgoogle.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc0066c23c0)
79322026-03-11T20:45:51.466Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1572 +0x137
79332026-03-11T20:45:51.466Zcreated by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 10962
79342026-03-11T20:45:51.466Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:360 +0x180b
79352026-03-11T20:45:51.466Z
79362026-03-11T20:45:51.466Zgoroutine 11045 [select, 59 minutes]:
79372026-03-11T20:45:51.466Zgoogle.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0070f86c0)
79382026-03-11T20:45:51.466Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:112 +0x6d
79392026-03-11T20:45:51.466Zcreated by google.golang.org/grpc.newCCBalancerWrapper in goroutine 9708
79402026-03-11T20:45:51.466Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:73 +0x22b
79412026-03-11T20:45:51.466Z
79422026-03-11T20:45:51.466Zgoroutine 9007 [select]:
79432026-03-11T20:45:51.466Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).outputLoop(0xc002758000, {0x50b1338, 0xc000ebdae0})
79442026-03-11T20:45:51.466Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:310 +0x247
79452026-03-11T20:45:51.466Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).runOutputLoop(0xc002758000, {0x50b1300, 0xc002017140}, 0xc0020cf9e0?)
79462026-03-11T20:45:51.466Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:335 +0xba
79472026-03-11T20:45:51.466Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run.func1({0x50b1300?, 0xc002017140?})
79482026-03-11T20:45:51.466Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:297 +0x46
79492026-03-11T20:45:51.466Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
79502026-03-11T20:45:51.466Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
79512026-03-11T20:45:51.466Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9005
79522026-03-11T20:45:51.466Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
79532026-03-11T20:45:51.466Z
79542026-03-11T20:45:51.467Zgoroutine 10701 [chan receive, 59 minutes]:
79552026-03-11T20:45:51.467Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func1({0xc00700a008?, 0xc006f2a570?})
79562026-03-11T20:45:51.467Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:203 +0x37
79572026-03-11T20:45:51.467Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
79582026-03-11T20:45:51.467Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
79592026-03-11T20:45:51.467Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
79602026-03-11T20:45:51.467Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
79612026-03-11T20:45:51.467Z
79622026-03-11T20:45:51.467Zgoroutine 9872 [select]:
79632026-03-11T20:45:51.467Zgithub.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
79642026-03-11T20:45:51.467Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:439 +0x1c9
79652026-03-11T20:45:51.467Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 9708
79662026-03-11T20:45:51.467Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:412 +0x4f
79672026-03-11T20:45:51.467Z
79682026-03-11T20:45:51.467Zgoroutine 9065 [select, 59 minutes]:
79692026-03-11T20:45:51.467Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2({0x50b1300, 0xc0076b6690})
79702026-03-11T20:45:51.467Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/queue.go:823 +0xff
79712026-03-11T20:45:51.467Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
79722026-03-11T20:45:51.467Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
79732026-03-11T20:45:51.467Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10821
79742026-03-11T20:45:51.467Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
79752026-03-11T20:45:51.467Z
79762026-03-11T20:45:51.467Zgoroutine 10704 [sync.Cond.Wait, 59 minutes]:
79772026-03-11T20:45:51.467Zsync.runtime_notifyListWait(0xc0064c5050, 0x1f)
79782026-03-11T20:45:51.467Z /work/go/src/runtime/sema.go:569 +0x159
79792026-03-11T20:45:51.467Zsync.(*Cond).Wait(0xc006e3e878?)
79802026-03-11T20:45:51.467Z /work/go/src/sync/cond.go:70 +0x85
79812026-03-11T20:45:51.467Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd2ba0})
79822026-03-11T20:45:51.467Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
79832026-03-11T20:45:51.467Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
79842026-03-11T20:45:51.467Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
79852026-03-11T20:45:51.467Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
79862026-03-11T20:45:51.467Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
79872026-03-11T20:45:51.467Z
79882026-03-11T20:45:51.467Zgoroutine 10475 [select]:
79892026-03-11T20:45:51.467Zgithub.com/cockroachdb/cockroach/pkg/rpc.respStreamClientAdapter.recvInternal(...)
79902026-03-11T20:45:51.467Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:694
79912026-03-11T20:45:51.467Zgithub.com/cockroachdb/cockroach/pkg/rpc.rangeFeedClientAdapter.Recv(...)
79922026-03-11T20:45:51.467Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:723
79932026-03-11T20:45:51.468Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed.func3()
79942026-03-11T20:45:51.468Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:516 +0x25
79952026-03-11T20:45:51.468Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*stuckRangeFeedCanceler).do(0xc003a62c80, 0xc005ef7a50)
79962026-03-11T20:45:51.468Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed_canceler.go:100 +0x1b3
79972026-03-11T20:45:51.468Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed(0xc002d6f408, {0x50b1338?, 0xc003a62b90?}, {{0xc00768a0d8, 0x1, 0x8}, {0xc00768a0e0, 0x1, 0x8}}, {0x189be113ecde6ad9, ...}, ...)
79982026-03-11T20:45:51.468Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:515 +0xc98
79992026-03-11T20:45:51.468Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).partialRangeFeed(0xc002d6f408, {0x50b1338, 0xc003a62b90}, 0xc003a62b40, {{0xc00768a0d8, 0x1, 0x8}, {0xc00768a0e0, 0x1, 0x8}}, ...)
80002026-03-11T20:45:51.468Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:336 +0x745
80012026-03-11T20:45:51.468Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.1({0x50b1338?, 0xc003a62b90?})
80022026-03-11T20:45:51.468Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:120 +0xa9
80032026-03-11T20:45:51.473Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.Group.GoCtx.2()
80042026-03-11T20:45:51.473Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
80052026-03-11T20:45:51.473Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
80062026-03-11T20:45:51.473Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
80072026-03-11T20:45:51.474Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10473
80082026-03-11T20:45:51.474Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
80092026-03-11T20:45:51.474Z
80102026-03-11T20:45:51.474Zgoroutine 10432 [select]:
80112026-03-11T20:45:51.474Zgithub.com/cockroachdb/cockroach/pkg/rpc.respStreamClientAdapter.recvInternal(...)
80122026-03-11T20:45:51.474Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:694
80132026-03-11T20:45:51.474Zgithub.com/cockroachdb/cockroach/pkg/rpc.rangeFeedClientAdapter.Recv(...)
80142026-03-11T20:45:51.474Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:723
80152026-03-11T20:45:51.474Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed.func3()
80162026-03-11T20:45:51.474Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:516 +0x25
80172026-03-11T20:45:51.474Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*stuckRangeFeedCanceler).do(0xc007611ef0, 0xc005e27a50)
80182026-03-11T20:45:51.474Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed_canceler.go:100 +0x1b3
80192026-03-11T20:45:51.474Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed(0xc002d6f408, {0x50b1338?, 0xc007611e00?}, {{0xc008f7c2f0, 0x1, 0x8}, {0xc008f7c2f8, 0x1, 0x8}}, {0x189be113ecd9c5b3, ...}, ...)
80202026-03-11T20:45:51.474Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:515 +0xc98
80212026-03-11T20:45:51.474Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).partialRangeFeed(0xc002d6f408, {0x50b1338, 0xc007611e00}, 0xc007611cc0, {{0xc008f7c2f0, 0x1, 0x8}, {0xc008f7c2f8, 0x1, 0x8}}, ...)
80222026-03-11T20:45:51.475Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:336 +0x745
80232026-03-11T20:45:51.475Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.1({0x50b1338?, 0xc007611e00?})
80242026-03-11T20:45:51.475Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:120 +0xa9
80252026-03-11T20:45:51.475Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.Group.GoCtx.2()
80262026-03-11T20:45:51.475Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
80272026-03-11T20:45:51.475Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
80282026-03-11T20:45:51.475Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
80292026-03-11T20:45:51.475Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10430
80302026-03-11T20:45:51.475Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
80312026-03-11T20:45:51.475Z
80322026-03-11T20:45:51.475Zgoroutine 11154 [select, 59 minutes]:
80332026-03-11T20:45:51.475Zgithub.com/cockroachdb/cockroach/pkg/util/tracing.newTracingClientStream.func2()
80342026-03-11T20:45:51.475Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:386 +0x67
80352026-03-11T20:45:51.475Zcreated by github.com/cockroachdb/cockroach/pkg/util/tracing.newTracingClientStream in goroutine 11072
80362026-03-11T20:45:51.475Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:385 +0x125
80372026-03-11T20:45:51.475Z
80382026-03-11T20:45:51.475Zgoroutine 10014 [sync.Cond.Wait]:
80392026-03-11T20:45:51.475Zsync.runtime_notifyListWait(0xc001e22250, 0x379d)
80402026-03-11T20:45:51.475Z /work/go/src/runtime/sema.go:569 +0x159
80412026-03-11T20:45:51.475Zsync.(*Cond).Wait(0x3aac600?)
80422026-03-11T20:45:51.475Z /work/go/src/sync/cond.go:70 +0x85
80432026-03-11T20:45:51.475Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583080})
80442026-03-11T20:45:51.475Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
80452026-03-11T20:45:51.475Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
80462026-03-11T20:45:51.475Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
80472026-03-11T20:45:51.475Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
80482026-03-11T20:45:51.475Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
80492026-03-11T20:45:51.475Z
80502026-03-11T20:45:51.475Zgoroutine 10861 [select]:
80512026-03-11T20:45:51.475Zgithub.com/cockroachdb/cockroach/pkg/sql/sqlstats/persistedsqlstats.(*PersistedSQLStats).startSQLStatsFlushLoop.func1({0x50b1300, 0xc00067cde0})
80522026-03-11T20:45:51.475Z /work/oxidecomputer/cockroach/pkg/sql/sqlstats/persistedsqlstats/provider.go:137 +0x236
80532026-03-11T20:45:51.475Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
80542026-03-11T20:45:51.475Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
80552026-03-11T20:45:51.475Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
80562026-03-11T20:45:51.475Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
80572026-03-11T20:45:51.475Z
80582026-03-11T20:45:51.475Zgoroutine 10150 [select, 59 minutes]:
80592026-03-11T20:45:51.476Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2({0x50b1300, 0xc009329bf0})
80602026-03-11T20:45:51.476Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/queue.go:823 +0xff
80612026-03-11T20:45:51.476Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
80622026-03-11T20:45:51.476Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
80632026-03-11T20:45:51.476Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10124
80642026-03-11T20:45:51.476Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
80652026-03-11T20:45:51.476Z
80662026-03-11T20:45:51.476Zgoroutine 10035 [sync.Cond.Wait]:
80672026-03-11T20:45:51.476Zsync.runtime_notifyListWait(0xc001e22250, 0x3746)
80682026-03-11T20:45:51.476Z /work/go/src/runtime/sema.go:569 +0x159
80692026-03-11T20:45:51.476Zsync.(*Cond).Wait(0x3aac600?)
80702026-03-11T20:45:51.476Z /work/go/src/sync/cond.go:70 +0x85
80712026-03-11T20:45:51.476Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583170})
80722026-03-11T20:45:51.476Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
80732026-03-11T20:45:51.476Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
80742026-03-11T20:45:51.477Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
80752026-03-11T20:45:51.477Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
80762026-03-11T20:45:51.477Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
80772026-03-11T20:45:51.477Z
80782026-03-11T20:45:51.477Zgoroutine 10015 [sync.Cond.Wait]:
80792026-03-11T20:45:51.477Zsync.runtime_notifyListWait(0xc001e22250, 0x379f)
80802026-03-11T20:45:51.477Z /work/go/src/runtime/sema.go:569 +0x159
80812026-03-11T20:45:51.477Zsync.(*Cond).Wait(0x3aac600?)
80822026-03-11T20:45:51.477Z /work/go/src/sync/cond.go:70 +0x85
80832026-03-11T20:45:51.477Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc0075830b0})
80842026-03-11T20:45:51.477Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
80852026-03-11T20:45:51.477Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
80862026-03-11T20:45:51.477Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
80872026-03-11T20:45:51.477Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
80882026-03-11T20:45:51.477Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
80892026-03-11T20:45:51.477Z
80902026-03-11T20:45:51.477Zgoroutine 10149 [select]:
80912026-03-11T20:45:51.477Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2({0x50b1300, 0xc009329b90})
80922026-03-11T20:45:51.477Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/queue.go:823 +0xff
80932026-03-11T20:45:51.477Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
80942026-03-11T20:45:51.477Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
80952026-03-11T20:45:51.477Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10124
80962026-03-11T20:45:51.477Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
80972026-03-11T20:45:51.477Z
80982026-03-11T20:45:51.477Zgoroutine 9075 [select]:
80992026-03-11T20:45:51.477Zgithub.com/cockroachdb/cockroach/pkg/gossip.(*server).Gossip(0xc006d3e300, {0x50ebb58, 0xc006e341d0})
81002026-03-11T20:45:51.477Z /work/oxidecomputer/cockroach/pkg/gossip/server.go:190 +0x87b
81012026-03-11T20:45:51.477Zgithub.com/cockroachdb/cockroach/pkg/gossip._Gossip_Gossip_Handler({0x3f222a0?, 0xc006d3e300}, {0x50df230, 0xc0078ea060})
81022026-03-11T20:45:51.477Z /work/oxidecomputer/cockroach/pkg/gossip/gossip.pb.go:660 +0xd8
81032026-03-11T20:45:51.484Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.StreamServerInterceptor.func14({0x3f222a0, 0xc006d3e300}, {0x50df9e0, 0xc0063ce000}, 0xc0078b00f0, 0x42e70b8)
81042026-03-11T20:45:51.484Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:205 +0x35b
81052026-03-11T20:45:51.484Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x3f222a0?, 0xc006d3e300?}, {0x50df9e0?, 0xc0063ce000?})
81062026-03-11T20:45:51.484Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1408 +0x59
81072026-03-11T20:45:51.484Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func4({0x3f222a0, 0xc006d3e300}, {0x50df9e0, 0xc0063ce000}, 0x40b826?, 0xc0078ae0c0)
81082026-03-11T20:45:51.484Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:247 +0x6c
81092026-03-11T20:45:51.484Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x3f222a0?, 0xc006d3e300?}, {0x50df9e0?, 0xc0063ce000?})
81102026-03-11T20:45:51.484Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
81112026-03-11T20:45:51.484Zgithub.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({{{0x3b9aca00?}}}, {0x3f222a0, 0xc006d3e300}, {0x50df9e0, 0xc0063ce000}, 0xc0078b00f0, 0xc0078ae0c0)
81122026-03-11T20:45:51.484Z /work/oxidecomputer/cockroach/pkg/rpc/auth.go:98 +0x19b
81132026-03-11T20:45:51.484Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x3f222a0?, 0xc006d3e300?}, {0x50df9e0?, 0xc0063ce000?})
81142026-03-11T20:45:51.484Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
81152026-03-11T20:45:51.484Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func2.1({0x66?, 0x40?})
81162026-03-11T20:45:51.484Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:216 +0x27
81172026-03-11T20:45:51.484Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc003f40400, {0x50b1300, 0xc0078b62a0}, {0xc005aafb28?, 0x415de5?}, 0xc005aafac0)
81182026-03-11T20:45:51.484Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:344 +0xcb
81192026-03-11T20:45:51.484Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func2({0x3f222a0, 0xc006d3e300}, {0x50df9e0, 0xc0063ce000}, 0xc0078b00f0, 0xc0078ae0c0)
81202026-03-11T20:45:51.484Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:215 +0xdf
81212026-03-11T20:45:51.484Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x3f222a0?, 0xc006d3e300?}, {0x50df9e0?, 0xc0063ce000?})
81222026-03-11T20:45:51.484Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
81232026-03-11T20:45:51.484Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x3f222a0, 0xc006d3e300}, {0x50df9e0, 0xc0063ce000}, 0xc0078b00f0, 0x42e70b8)
81242026-03-11T20:45:51.484Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1413 +0x108
81252026-03-11T20:45:51.484Zgoogle.golang.org/grpc.(*Server).processStreamingRPC(0xc004feefc0, {0x50f1d80, 0xc007738000}, 0xc0078e8000, 0xc007660630, 0x73879c0, 0x0)
81262026-03-11T20:45:51.484Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1549 +0xe51
81272026-03-11T20:45:51.484Zgoogle.golang.org/grpc.(*Server).handleStream(0xc004feefc0, {0x50f1d80, 0xc007738000}, 0xc0078e8000, 0x0)
81282026-03-11T20:45:51.484Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1624 +0x999
81292026-03-11T20:45:51.484Zgoogle.golang.org/grpc.(*Server).serveStreams.func1.2()
81302026-03-11T20:45:51.484Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:922 +0x8d
81312026-03-11T20:45:51.484Zcreated by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 9062
81322026-03-11T20:45:51.484Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:920 +0x226
81332026-03-11T20:45:51.485Z
81342026-03-11T20:45:51.485Zgoroutine 9233 [IO wait]:
81352026-03-11T20:45:51.485Zinternal/poll.runtime_pollWait(0x7fc12ea8fa88, 0x72)
81362026-03-11T20:45:51.485Z /work/go/src/runtime/netpoll.go:345 +0x85
81372026-03-11T20:45:51.485Zinternal/poll.(*pollDesc).wait(0xc006756a80?, 0xc00363c700?, 0x0)
81382026-03-11T20:45:51.485Z /work/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
81392026-03-11T20:45:51.485Zinternal/poll.(*pollDesc).waitRead(...)
81402026-03-11T20:45:51.485Z /work/go/src/internal/poll/fd_poll_runtime.go:89
81412026-03-11T20:45:51.485Zinternal/poll.(*FD).Read(0xc006756a80, {0xc00363c700, 0x700, 0x700})
81422026-03-11T20:45:51.485Z /work/go/src/internal/poll/fd_unix.go:164 +0x27a
81432026-03-11T20:45:51.485Znet.(*netFD).Read(0xc006756a80, {0xc00363c700?, 0x7fc12afd7da0?, 0xc009212570?})
81442026-03-11T20:45:51.485Z /work/go/src/net/fd_posix.go:55 +0x25
81452026-03-11T20:45:51.485Znet.(*conn).Read(0xc005406db0, {0xc00363c700?, 0xc006bfc8e0?, 0x415dbb?})
81462026-03-11T20:45:51.485Z /work/go/src/net/net.go:185 +0x45
81472026-03-11T20:45:51.486Zcrypto/tls.(*atLeastReader).Read(0xc009212570, {0xc00363c700?, 0x0?, 0xc009212570?})
81482026-03-11T20:45:51.486Z /work/go/src/crypto/tls/conn.go:806 +0x3b
81492026-03-11T20:45:51.486Zbytes.(*Buffer).ReadFrom(0xc006f142b0, {0x5081fe0, 0xc009212570})
81502026-03-11T20:45:51.486Z /work/go/src/bytes/buffer.go:211 +0x98
81512026-03-11T20:45:51.486Zcrypto/tls.(*Conn).readFromUntil(0xc006f14008, {0x50821c0, 0xc005406db0}, 0xc006bfc928?)
81522026-03-11T20:45:51.486Z /work/go/src/crypto/tls/conn.go:828 +0xde
81532026-03-11T20:45:51.486Zcrypto/tls.(*Conn).readRecordOrCCS(0xc006f14008, 0x0)
81542026-03-11T20:45:51.486Z /work/go/src/crypto/tls/conn.go:626 +0x3cf
81552026-03-11T20:45:51.486Zcrypto/tls.(*Conn).readRecord(...)
81562026-03-11T20:45:51.486Z /work/go/src/crypto/tls/conn.go:588
81572026-03-11T20:45:51.486Zcrypto/tls.(*Conn).Read(0xc006f14008, {0xc00781e000, 0x8000, 0xc006bfcc88?})
81582026-03-11T20:45:51.486Z /work/go/src/crypto/tls/conn.go:1370 +0x156
81592026-03-11T20:45:51.486Zbufio.(*Reader).Read(0xc006028ba0, {0xc0038a6200, 0x9, 0x48943d?})
81602026-03-11T20:45:51.486Z /work/go/src/bufio/bufio.go:241 +0x197
81612026-03-11T20:45:51.486Zio.ReadAtLeast({0x507c320, 0xc006028ba0}, {0xc0038a6200, 0x9, 0x9}, 0x9)
81622026-03-11T20:45:51.487Z /work/go/src/io/io.go:335 +0x90
81632026-03-11T20:45:51.487Zio.ReadFull(...)
81642026-03-11T20:45:51.487Z /work/go/src/io/io.go:354
81652026-03-11T20:45:51.487Zgolang.org/x/net/http2.readFrameHeader({0xc0038a6200, 0x9, 0xc0033fabd0?}, {0x507c320?, 0xc006028ba0?})
81662026-03-11T20:45:51.487Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x65
81672026-03-11T20:45:51.487Zgolang.org/x/net/http2.(*Framer).ReadFrame(0xc0038a61c0)
81682026-03-11T20:45:51.487Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:501 +0x85
81692026-03-11T20:45:51.487Zgoogle.golang.org/grpc/internal/transport.(*http2Client).reader(0xc007514000)
81702026-03-11T20:45:51.487Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1498 +0x41d
81712026-03-11T20:45:51.487Zcreated by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 10707
81722026-03-11T20:45:51.487Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:365 +0x184d
81732026-03-11T20:45:51.487Z
81742026-03-11T20:45:51.487Zgoroutine 10746 [sync.Cond.Wait, 59 minutes]:
81752026-03-11T20:45:51.487Zsync.runtime_notifyListWait(0xc0064c5050, 0xf)
81762026-03-11T20:45:51.487Z /work/go/src/runtime/sema.go:569 +0x159
81772026-03-11T20:45:51.487Zsync.(*Cond).Wait(0xc006e3e878?)
81782026-03-11T20:45:51.487Z /work/go/src/sync/cond.go:70 +0x85
81792026-03-11T20:45:51.487Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3080})
81802026-03-11T20:45:51.487Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
81812026-03-11T20:45:51.488Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
81822026-03-11T20:45:51.488Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
81832026-03-11T20:45:51.488Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
81842026-03-11T20:45:51.488Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
81852026-03-11T20:45:51.488Z
81862026-03-11T20:45:51.488Zgoroutine 10445 [select]:
81872026-03-11T20:45:51.488Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents(0xc002c1b1e0, {0x50b1338, 0xc003eef5e0}, 0xc0040cb6b0, 0xc0071254a0)
81882026-03-11T20:45:51.488Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:331 +0xc5
81892026-03-11T20:45:51.488Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func2({0x50b1338?, 0xc003eef5e0?})
81902026-03-11T20:45:51.488Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:288 +0x32
81912026-03-11T20:45:51.488Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
81922026-03-11T20:45:51.488Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
81932026-03-11T20:45:51.489Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
81942026-03-11T20:45:51.489Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
81952026-03-11T20:45:51.489Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10179
81962026-03-11T20:45:51.489Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
81972026-03-11T20:45:51.489Z
81982026-03-11T20:45:51.489Zgoroutine 11147 [chan receive, 59 minutes]:
81992026-03-11T20:45:51.489Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).rangeFeedWithRangeID(0xc001e56008, 0x3d87860?, 0xc000a9c5a0, {0x7fc12ce5a548, 0xc005553270})
82002026-03-11T20:45:51.489Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:249 +0x76b
82012026-03-11T20:45:51.489Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed(...)
82022026-03-11T20:45:51.489Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:149
82032026-03-11T20:45:51.500Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).RangeFeed(0xc000f39208, 0xc000a9c5a0, {0x50e5040, 0xc005553270})
82042026-03-11T20:45:51.500Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:3082 +0x16f
82052026-03-11T20:45:51.500Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).RangeFeed(0xc001eb4380, 0xc000a9c5a0, {0x50e5040, 0xc005553270})
82062026-03-11T20:45:51.500Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/stores.go:216 +0x19a
82072026-03-11T20:45:51.500Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed(0x732ffb0?, 0x40055e0?, {0x50e5040, 0xc005553270})
82082026-03-11T20:45:51.500Z /work/oxidecomputer/cockroach/pkg/server/node.go:1232 +0x28
82092026-03-11T20:45:51.500Zgithub.com/cockroachdb/cockroach/pkg/roachpb._Internal_RangeFeed_Handler({0x40055e0, 0xc004911b08}, {0x50df230, 0xc0040522c0})
82102026-03-11T20:45:51.500Z /work/oxidecomputer/cockroach/pkg/roachpb/api.pb.go:9523 +0x107
82112026-03-11T20:45:51.501Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.StreamServerInterceptor.func14({0x40055e0, 0xc004911b08}, {0x50df9e0, 0xc0043f3e10}, 0xc003453f08, 0x42e42f8)
82122026-03-11T20:45:51.501Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:205 +0x35b
82132026-03-11T20:45:51.501Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x40055e0?, 0xc004911b08?}, {0x50df9e0?, 0xc0043f3e10?})
82142026-03-11T20:45:51.501Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1408 +0x59
82152026-03-11T20:45:51.501Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func4({0x40055e0, 0xc004911b08}, {0x50df9e0, 0xc0043f3e10}, 0xc000a6c918?, 0xc0036e0280)
82162026-03-11T20:45:51.501Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:247 +0x6c
82172026-03-11T20:45:51.501Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x40055e0?, 0xc004911b08?}, {0x50df9e0?, 0xc0043f3e10?})
82182026-03-11T20:45:51.501Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
82192026-03-11T20:45:51.501Zgithub.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({{{0x2c?}}}, {0x40055e0, 0xc004911b08}, {0x50df9e0, 0xc0043f3e10}, 0xc003453f08, 0xc0036e0280)
82202026-03-11T20:45:51.501Z /work/oxidecomputer/cockroach/pkg/rpc/auth.go:98 +0x19b
82212026-03-11T20:45:51.501Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x40055e0?, 0xc004911b08?}, {0x50df9e0?, 0xc0043f3e10?})
82222026-03-11T20:45:51.501Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
82232026-03-11T20:45:51.501Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func2.1({0x66?, 0x40?})
82242026-03-11T20:45:51.501Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:216 +0x27
82252026-03-11T20:45:51.501Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc003f40400, {0x50b1300, 0xc0070e3b60}, {0xc000a6cb28?, 0x415dbb?}, 0xc000a6cac0)
82262026-03-11T20:45:51.501Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:344 +0xcb
82272026-03-11T20:45:51.501Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func2({0x40055e0, 0xc004911b08}, {0x50df9e0, 0xc0043f3e10}, 0xc003453f08, 0xc0036e0280)
82282026-03-11T20:45:51.501Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:215 +0xdf
82292026-03-11T20:45:51.501Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x40055e0?, 0xc004911b08?}, {0x50df9e0?, 0xc0043f3e10?})
82302026-03-11T20:45:51.501Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
82312026-03-11T20:45:51.501Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x40055e0, 0xc004911b08}, {0x50df9e0, 0xc0043f3e10}, 0xc003453f08, 0x42e42f8)
82322026-03-11T20:45:51.501Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1413 +0x108
82332026-03-11T20:45:51.501Zgoogle.golang.org/grpc.(*Server).processStreamingRPC(0xc004feefc0, {0x50f1d80, 0xc001aef6c0}, 0xc0020ea480, 0xc007661ad0, 0x73abba0, 0x0)
82342026-03-11T20:45:51.501Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1549 +0xe51
82352026-03-11T20:45:51.501Zgoogle.golang.org/grpc.(*Server).handleStream(0xc004feefc0, {0x50f1d80, 0xc001aef6c0}, 0xc0020ea480, 0x0)
82362026-03-11T20:45:51.501Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1624 +0x999
82372026-03-11T20:45:51.501Zgoogle.golang.org/grpc.(*Server).serveStreams.func1.2()
82382026-03-11T20:45:51.501Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:922 +0x8d
82392026-03-11T20:45:51.501Zcreated by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 10956
82402026-03-11T20:45:51.501Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:920 +0x226
82412026-03-11T20:45:51.501Z
82422026-03-11T20:45:51.501Zgoroutine 10013 [sync.Cond.Wait]:
82432026-03-11T20:45:51.501Zsync.runtime_notifyListWait(0xc001e22250, 0x379e)
82442026-03-11T20:45:51.502Z /work/go/src/runtime/sema.go:569 +0x159
82452026-03-11T20:45:51.502Zsync.(*Cond).Wait(0x3aac600?)
82462026-03-11T20:45:51.502Z /work/go/src/sync/cond.go:70 +0x85
82472026-03-11T20:45:51.502Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583050})
82482026-03-11T20:45:51.502Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
82492026-03-11T20:45:51.502Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
82502026-03-11T20:45:51.502Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
82512026-03-11T20:45:51.502Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
82522026-03-11T20:45:51.502Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
82532026-03-11T20:45:51.502Z
82542026-03-11T20:45:51.502Zgoroutine 10498 [select]:
82552026-03-11T20:45:51.502Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run(0xc006014c00, {0x50b1300?, 0xc0042308a0?}, 0x2a?, 0xc007ee04b0, 0xc003f40400)
82562026-03-11T20:45:51.502Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:269 +0x4dc
82572026-03-11T20:45:51.502Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).Start.func1({0x50b1300?, 0xc0042308a0?})
82582026-03-11T20:45:51.502Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:222 +0x36
82592026-03-11T20:45:51.502Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
82602026-03-11T20:45:51.502Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
82612026-03-11T20:45:51.502Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10497
82622026-03-11T20:45:51.502Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
82632026-03-11T20:45:51.502Z
82642026-03-11T20:45:51.502Zgoroutine 10425 [select]:
82652026-03-11T20:45:51.502Zgithub.com/cockroachdb/cockroach/pkg/rpc.respStreamClientAdapter.recvInternal(...)
82662026-03-11T20:45:51.502Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:694
82672026-03-11T20:45:51.502Zgithub.com/cockroachdb/cockroach/pkg/rpc.rangeFeedClientAdapter.Recv(...)
82682026-03-11T20:45:51.502Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:723
82692026-03-11T20:45:51.502Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed.func3()
82702026-03-11T20:45:51.502Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:516 +0x25
82712026-03-11T20:45:51.502Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*stuckRangeFeedCanceler).do(0xc007611b30, 0xc00351da50)
82722026-03-11T20:45:51.502Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed_canceler.go:100 +0x1b3
82732026-03-11T20:45:51.502Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed(0xc002d6f408, {0x50b1338?, 0xc007611a40?}, {{0xc005269288, 0x1, 0x8}, {0xc005269290, 0x1, 0x8}}, {0x189be113ed70e8ca, ...}, ...)
82742026-03-11T20:45:51.502Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:515 +0xc98
82752026-03-11T20:45:51.502Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).partialRangeFeed(0xc002d6f408, {0x50b1338, 0xc007611a40}, 0xc0076119f0, {{0xc005269288, 0x1, 0x8}, {0xc005269290, 0x1, 0x8}}, ...)
82762026-03-11T20:45:51.502Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:336 +0x745
82772026-03-11T20:45:51.502Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.1({0x50b1338?, 0xc007611a40?})
82782026-03-11T20:45:51.502Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:120 +0xa9
82792026-03-11T20:45:51.502Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.Group.GoCtx.2()
82802026-03-11T20:45:51.502Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
82812026-03-11T20:45:51.502Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
82822026-03-11T20:45:51.502Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
82832026-03-11T20:45:51.503Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10423
82842026-03-11T20:45:51.503Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
82852026-03-11T20:45:51.503Z
82862026-03-11T20:45:51.503Zgoroutine 11138 [select, 59 minutes]:
82872026-03-11T20:45:51.503Zgoogle.golang.org/grpc.newClientStreamWithParams.func4()
82882026-03-11T20:45:51.503Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:341 +0x8c
82892026-03-11T20:45:51.503Zcreated by google.golang.org/grpc.newClientStreamWithParams in goroutine 11041
82902026-03-11T20:45:51.503Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:340 +0xb2a
82912026-03-11T20:45:51.503Z
82922026-03-11T20:45:51.503Zgoroutine 11077 [IO wait]:
82932026-03-11T20:45:51.503Zinternal/poll.runtime_pollWait(0x7fc12eba3650, 0x72)
82942026-03-11T20:45:51.503Z /work/go/src/runtime/netpoll.go:345 +0x85
82952026-03-11T20:45:51.503Zinternal/poll.(*pollDesc).wait(0xc002dea980?, 0xc003a42000?, 0x0)
82962026-03-11T20:45:51.503Z /work/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
82972026-03-11T20:45:51.503Zinternal/poll.(*pollDesc).waitRead(...)
82982026-03-11T20:45:51.503Z /work/go/src/internal/poll/fd_poll_runtime.go:89
82992026-03-11T20:45:51.503Zinternal/poll.(*FD).Read(0xc002dea980, {0xc003a42000, 0x600, 0x600})
83002026-03-11T20:45:51.503Z /work/go/src/internal/poll/fd_unix.go:164 +0x27a
83012026-03-11T20:45:51.503Znet.(*netFD).Read(0xc002dea980, {0xc003a42000?, 0xc005ab1868?, 0x41e1e5?})
83022026-03-11T20:45:51.503Z /work/go/src/net/fd_posix.go:55 +0x25
83032026-03-11T20:45:51.513Znet.(*conn).Read(0xc0020591f0, {0xc003a42000?, 0x7fc126f9b100?, 0xc02eb6c798?})
83042026-03-11T20:45:51.513Z /work/go/src/net/net.go:185 +0x45
83052026-03-11T20:45:51.513Zgithub.com/cockroachdb/cmux.(*MuxConn).Read(0xc0048adf20, {0xc003a42000, 0x600, 0x600})
83062026-03-11T20:45:51.513Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x14f
83072026-03-11T20:45:51.513Zcrypto/tls.(*atLeastReader).Read(0xc02eb6c798, {0xc003a42000?, 0x0?, 0xc02eb6c798?})
83082026-03-11T20:45:51.513Z /work/go/src/crypto/tls/conn.go:806 +0x3b
83092026-03-11T20:45:51.513Zbytes.(*Buffer).ReadFrom(0xc0019a97b0, {0x5081fe0, 0xc02eb6c798})
83102026-03-11T20:45:51.513Z /work/go/src/bytes/buffer.go:211 +0x98
83112026-03-11T20:45:51.513Zcrypto/tls.(*Conn).readFromUntil(0xc0019a9508, {0x7fc12ce5a2d8, 0xc0048adf20}, 0xc005ab1990?)
83122026-03-11T20:45:51.513Z /work/go/src/crypto/tls/conn.go:828 +0xde
83132026-03-11T20:45:51.513Zcrypto/tls.(*Conn).readRecordOrCCS(0xc0019a9508, 0x0)
83142026-03-11T20:45:51.513Z /work/go/src/crypto/tls/conn.go:626 +0x3cf
83152026-03-11T20:45:51.513Zcrypto/tls.(*Conn).readRecord(...)
83162026-03-11T20:45:51.513Z /work/go/src/crypto/tls/conn.go:588
83172026-03-11T20:45:51.513Zcrypto/tls.(*Conn).Read(0xc0019a9508, {0xc005cc2000, 0x8000, 0x7fc12570daf8?})
83182026-03-11T20:45:51.513Z /work/go/src/crypto/tls/conn.go:1370 +0x156
83192026-03-11T20:45:51.513Zbufio.(*Reader).Read(0xc0057897a0, {0xc004b158c0, 0x9, 0xc0000c3008?})
83202026-03-11T20:45:51.513Z /work/go/src/bufio/bufio.go:241 +0x197
83212026-03-11T20:45:51.513Zio.ReadAtLeast({0x507c320, 0xc0057897a0}, {0xc004b158c0, 0x9, 0x9}, 0x9)
83222026-03-11T20:45:51.513Z /work/go/src/io/io.go:335 +0x90
83232026-03-11T20:45:51.513Zio.ReadFull(...)
83242026-03-11T20:45:51.513Z /work/go/src/io/io.go:354
83252026-03-11T20:45:51.513Zgolang.org/x/net/http2.readFrameHeader({0xc004b158c0, 0x9, 0xc02eb18408?}, {0x507c320?, 0xc0057897a0?})
83262026-03-11T20:45:51.513Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x65
83272026-03-11T20:45:51.513Zgolang.org/x/net/http2.(*Framer).ReadFrame(0xc004b15880)
83282026-03-11T20:45:51.513Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:501 +0x85
83292026-03-11T20:45:51.513Zgoogle.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc00169cb60, 0xc003c925d0, 0x42e6fc8)
83302026-03-11T20:45:51.513Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:611 +0x14e
83312026-03-11T20:45:51.513Zgoogle.golang.org/grpc.(*Server).serveStreams(0xc004feefc0, {0x50f1d80, 0xc00169cb60})
83322026-03-11T20:45:51.513Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:906 +0x114
83332026-03-11T20:45:51.513Zgoogle.golang.org/grpc.(*Server).handleRawConn.func1()
83342026-03-11T20:45:51.513Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:848 +0x45
83352026-03-11T20:45:51.513Zcreated by google.golang.org/grpc.(*Server).handleRawConn in goroutine 11048
83362026-03-11T20:45:51.513Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:847 +0x15b
83372026-03-11T20:45:51.513Z
83382026-03-11T20:45:51.513Zgoroutine 9956 [select, 59 minutes]:
83392026-03-11T20:45:51.514Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initCancelingWorkers.func1({0x50b1300, 0xc007659950})
83402026-03-11T20:45:51.514Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:139 +0xc5
83412026-03-11T20:45:51.514Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
83422026-03-11T20:45:51.514Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
83432026-03-11T20:45:51.514Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
83442026-03-11T20:45:51.514Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
83452026-03-11T20:45:51.514Z
83462026-03-11T20:45:51.514Zgoroutine 10117 [semacquire, 59 minutes]:
83472026-03-11T20:45:51.514Zsync.runtime_Semacquire(0x0?)
83482026-03-11T20:45:51.514Z /work/go/src/runtime/sema.go:62 +0x25
83492026-03-11T20:45:51.514Zsync.(*WaitGroup).Wait(0x0?)
83502026-03-11T20:45:51.514Z /work/go/src/sync/waitgroup.go:116 +0x48
83512026-03-11T20:45:51.514Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Wait(...)
83522026-03-11T20:45:51.514Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:236
83532026-03-11T20:45:51.514Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
83542026-03-11T20:45:51.514Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
83552026-03-11T20:45:51.514Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
83562026-03-11T20:45:51.514Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
83572026-03-11T20:45:51.514Z
83582026-03-11T20:45:51.514Zgoroutine 10043 [sync.Cond.Wait]:
83592026-03-11T20:45:51.514Zsync.runtime_notifyListWait(0xc001e22250, 0x375c)
83602026-03-11T20:45:51.514Z /work/go/src/runtime/sema.go:569 +0x159
83612026-03-11T20:45:51.514Zsync.(*Cond).Wait(0x3aac600?)
83622026-03-11T20:45:51.515Z /work/go/src/sync/cond.go:70 +0x85
83632026-03-11T20:45:51.515Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc0075832f0})
83642026-03-11T20:45:51.515Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
83652026-03-11T20:45:51.515Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
83662026-03-11T20:45:51.515Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
83672026-03-11T20:45:51.515Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
83682026-03-11T20:45:51.515Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
83692026-03-11T20:45:51.515Z
83702026-03-11T20:45:51.515Zgoroutine 10864 [select]:
83712026-03-11T20:45:51.515Zgithub.com/cockroachdb/cockroach/pkg/sql/contention/txnidcache.(*Cache).Start.func1({0xc000fce908?, 0xc006815508?})
83722026-03-11T20:45:51.515Z /work/oxidecomputer/cockroach/pkg/sql/contention/txnidcache/txn_id_cache.go:144 +0x9b
83732026-03-11T20:45:51.515Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
83742026-03-11T20:45:51.515Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
83752026-03-11T20:45:51.515Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
83762026-03-11T20:45:51.515Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
83772026-03-11T20:45:51.515Z
83782026-03-11T20:45:51.515Zgoroutine 9816 [chan receive, 59 minutes]:
83792026-03-11T20:45:51.515Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc005572570?})
83802026-03-11T20:45:51.515Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
83812026-03-11T20:45:51.515Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0xc004046300?, 0x1028cc0?}}, 0xc0040557a8)
83822026-03-11T20:45:51.515Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
83832026-03-11T20:45:51.515Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0xc004055798?)
83842026-03-11T20:45:51.515Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
83852026-03-11T20:45:51.515Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
83862026-03-11T20:45:51.515Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
83872026-03-11T20:45:51.515Z
83882026-03-11T20:45:51.515Zgoroutine 10819 [select, 1 minutes]:
83892026-03-11T20:45:51.515Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).startGossip.func3({0x50b1300, 0xc006fd3e90})
83902026-03-11T20:45:51.515Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:2152 +0x225
83912026-03-11T20:45:51.515Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
83922026-03-11T20:45:51.515Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
83932026-03-11T20:45:51.515Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
83942026-03-11T20:45:51.515Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
83952026-03-11T20:45:51.516Z
83962026-03-11T20:45:51.516Zgoroutine 10040 [sync.Cond.Wait]:
83972026-03-11T20:45:51.516Zsync.runtime_notifyListWait(0xc001e22250, 0x3754)
83982026-03-11T20:45:51.516Z /work/go/src/runtime/sema.go:569 +0x159
83992026-03-11T20:45:51.516Zsync.(*Cond).Wait(0x3aac600?)
84002026-03-11T20:45:51.516Z /work/go/src/sync/cond.go:70 +0x85
84012026-03-11T20:45:51.516Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583260})
84022026-03-11T20:45:51.516Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
84032026-03-11T20:45:51.520Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
84042026-03-11T20:45:51.520Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
84052026-03-11T20:45:51.520Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
84062026-03-11T20:45:51.520Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
84072026-03-11T20:45:51.520Z
84082026-03-11T20:45:51.520Zgoroutine 9951 [select, 59 minutes]:
84092026-03-11T20:45:51.520Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc004cb1808?, 0x0?})
84102026-03-11T20:45:51.521Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
84112026-03-11T20:45:51.521Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
84122026-03-11T20:45:51.521Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
84132026-03-11T20:45:51.521Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
84142026-03-11T20:45:51.521Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
84152026-03-11T20:45:51.521Z
84162026-03-11T20:45:51.521Zgoroutine 9958 [select, 59 minutes]:
84172026-03-11T20:45:51.521Zgithub.com/cockroachdb/cockroach/pkg/server.startPurgeOldSessions.func1({0xc0002a3b08?, 0x0?})
84182026-03-11T20:45:51.521Z /work/oxidecomputer/cockroach/pkg/server/purge_auth_session.go:69 +0x205
84192026-03-11T20:45:51.521Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
84202026-03-11T20:45:51.521Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
84212026-03-11T20:45:51.521Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
84222026-03-11T20:45:51.521Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
84232026-03-11T20:45:51.521Z
84242026-03-11T20:45:51.521Zgoroutine 10708 [select]:
84252026-03-11T20:45:51.521Zgithub.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc000e28800, {0x50b1300, 0xc006d70510}, 0xc006d542d0, {0xc008818b50, 0xf}, 0xc000fae960)
84262026-03-11T20:45:51.521Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:1462 +0x27b
84272026-03-11T20:45:51.521Zgithub.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1({0x50b1300, 0xc006d70510})
84282026-03-11T20:45:51.521Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:1380 +0x85
84292026-03-11T20:45:51.521Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
84302026-03-11T20:45:51.521Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
84312026-03-11T20:45:51.521Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10657
84322026-03-11T20:45:51.521Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
84332026-03-11T20:45:51.521Z
84342026-03-11T20:45:51.521Zgoroutine 9992 [select, 59 minutes]:
84352026-03-11T20:45:51.521Zgithub.com/cockroachdb/cockroach/pkg/server.(*loopbackListener).Accept(0xc0073f21b0)
84362026-03-11T20:45:51.521Z /work/oxidecomputer/cockroach/pkg/server/loopback.go:49 +0x90
84372026-03-11T20:45:51.521Zgoogle.golang.org/grpc.(*Server).Serve(0xc004feefc0, {0x509c250, 0xc0073f21b0})
84382026-03-11T20:45:51.521Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:780 +0x469
84392026-03-11T20:45:51.521Zgithub.com/cockroachdb/cockroach/pkg/server.configureGRPCGateway.func2({0xc0022c2008?, 0x0?})
84402026-03-11T20:45:51.521Z /work/oxidecomputer/cockroach/pkg/server/grpc_gateway.go:94 +0x25
84412026-03-11T20:45:51.521Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
84422026-03-11T20:45:51.521Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
84432026-03-11T20:45:51.521Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
84442026-03-11T20:45:51.522Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
84452026-03-11T20:45:51.522Z
84462026-03-11T20:45:51.522Zgoroutine 8967 [select, 59 minutes]:
84472026-03-11T20:45:51.522Zgithub.com/cockroachdb/cockroach/pkg/util/log.(*fileSink).gcDaemon(0xc006938340, {0x50b1338, 0xc006cfc460})
84482026-03-11T20:45:51.522Z /work/oxidecomputer/cockroach/pkg/util/log/file_log_gc.go:25 +0x79
84492026-03-11T20:45:51.522Zcreated by github.com/cockroachdb/cockroach/pkg/util/log.ApplyConfig in goroutine 9708
84502026-03-11T20:45:51.522Z /work/oxidecomputer/cockroach/pkg/util/log/flags.go:293 +0x10f5
84512026-03-11T20:45:51.522Z
84522026-03-11T20:45:51.522Zgoroutine 11135 [select, 59 minutes]:
84532026-03-11T20:45:51.522Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1({0x50b1338, 0xc0010bd310})
84542026-03-11T20:45:51.522Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:116 +0x157
84552026-03-11T20:45:51.522Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.Group.GoCtx.func3()
84562026-03-11T20:45:51.522Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
84572026-03-11T20:45:51.522Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
84582026-03-11T20:45:51.522Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
84592026-03-11T20:45:51.522Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 11133
84602026-03-11T20:45:51.522Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
84612026-03-11T20:45:51.522Z
84622026-03-11T20:45:51.522Zgoroutine 10751 [sync.Cond.Wait, 59 minutes]:
84632026-03-11T20:45:51.522Zsync.runtime_notifyListWait(0xc0064c5050, 0xb)
84642026-03-11T20:45:51.522Z /work/go/src/runtime/sema.go:569 +0x159
84652026-03-11T20:45:51.522Zsync.(*Cond).Wait(0x50f7930?)
84662026-03-11T20:45:51.522Z /work/go/src/sync/cond.go:70 +0x85
84672026-03-11T20:45:51.522Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3170})
84682026-03-11T20:45:51.522Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
84692026-03-11T20:45:51.522Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
84702026-03-11T20:45:51.522Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
84712026-03-11T20:45:51.522Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
84722026-03-11T20:45:51.522Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
84732026-03-11T20:45:51.522Z
84742026-03-11T20:45:51.522Zgoroutine 10009 [sync.Cond.Wait]:
84752026-03-11T20:45:51.522Zsync.runtime_notifyListWait(0xc001e22250, 0x378f)
84762026-03-11T20:45:51.522Z /work/go/src/runtime/sema.go:569 +0x159
84772026-03-11T20:45:51.522Zsync.(*Cond).Wait(0x3aac600?)
84782026-03-11T20:45:51.522Z /work/go/src/sync/cond.go:70 +0x85
84792026-03-11T20:45:51.522Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007582f90})
84802026-03-11T20:45:51.522Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
84812026-03-11T20:45:51.523Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
84822026-03-11T20:45:51.523Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
84832026-03-11T20:45:51.523Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
84842026-03-11T20:45:51.523Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
84852026-03-11T20:45:51.523Z
84862026-03-11T20:45:51.523Zgoroutine 9836 [select, 59 minutes]:
84872026-03-11T20:45:51.523Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc005957b08?, 0x50b1258?})
84882026-03-11T20:45:51.523Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
84892026-03-11T20:45:51.523Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
84902026-03-11T20:45:51.523Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
84912026-03-11T20:45:51.523Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
84922026-03-11T20:45:51.523Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
84932026-03-11T20:45:51.523Z
84942026-03-11T20:45:51.523Zgoroutine 10901 [select]:
84952026-03-11T20:45:51.523Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).startWriteNodeStatus.func2({0x50b1300, 0xc0076b6870})
84962026-03-11T20:45:51.523Z /work/oxidecomputer/cockroach/pkg/server/node.go:852 +0xf3
84972026-03-11T20:45:51.523Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
84982026-03-11T20:45:51.523Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
84992026-03-11T20:45:51.523Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
85002026-03-11T20:45:51.523Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
85012026-03-11T20:45:51.523Z
85022026-03-11T20:45:51.523Zgoroutine 10131 [select, 59 minutes]:
85032026-03-11T20:45:51.528Zgithub.com/cockroachdb/cockroach/pkg/server.periodicallyPersistHLCUpperBound(0xc006cfc550, 0xc00303e150, 0xc00746b150, 0x42e4010, 0xc0018927e0, 0x0)
85042026-03-11T20:45:51.528Z /work/oxidecomputer/cockroach/pkg/server/clock_monotonicity.go:193 +0x14c
85052026-03-11T20:45:51.528Zgithub.com/cockroachdb/cockroach/pkg/server.(*Server).startPersistingHLCUpperBound.func3({0xc002344008?, 0x0?})
85062026-03-11T20:45:51.528Z /work/oxidecomputer/cockroach/pkg/server/clock_monotonicity.go:268 +0x45
85072026-03-11T20:45:51.528Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
85082026-03-11T20:45:51.528Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
85092026-03-11T20:45:51.528Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
85102026-03-11T20:45:51.528Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
85112026-03-11T20:45:51.528Z
85122026-03-11T20:45:51.528Zgoroutine 10818 [select]:
85132026-03-11T20:45:51.528Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).startGossip.func3({0x50b1300, 0xc006fd3e60})
85142026-03-11T20:45:51.528Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:2152 +0x225
85152026-03-11T20:45:51.528Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
85162026-03-11T20:45:51.528Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
85172026-03-11T20:45:51.528Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
85182026-03-11T20:45:51.528Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
85192026-03-11T20:45:51.528Z
85202026-03-11T20:45:51.528Zgoroutine 10922 [semacquire, 59 minutes]:
85212026-03-11T20:45:51.528Zsync.runtime_Semacquire(0xc006b6b7e8?)
85222026-03-11T20:45:51.528Z /work/go/src/runtime/sema.go:62 +0x25
85232026-03-11T20:45:51.528Zsync.(*WaitGroup).Wait(0x44e7a0?)
85242026-03-11T20:45:51.528Z /work/go/src/sync/waitgroup.go:116 +0x48
85252026-03-11T20:45:51.528Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc007706b00)
85262026-03-11T20:45:51.528Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
85272026-03-11T20:45:51.528Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc007706b00?, {0x50b1338?, 0xc0078ad220?}})
85282026-03-11T20:45:51.528Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
85292026-03-11T20:45:51.528Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed(0xc003d48008, {0x50b1338?, 0xc006b57bd0?}, {0xc00067d500, 0x1, 0xc00067d200?}, {0xc00750cb00?, 0x827a9a0?, 0xc0?}, 0x0, ...)
85302026-03-11T20:45:51.528Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:139 +0x6c5
85312026-03-11T20:45:51.528Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*dbAdapter).RangeFeed(0xc003aac450?, {0x50b1338?, 0xc006b57bd0?}, {0xc00067d500?, 0x50b1300?, 0xc00067d6b0?}, {0x707000000000000?, 0x50b1300?, 0x0?}, 0x0, ...)
85322026-03-11T20:45:51.528Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/db_adapter.go:78 +0x3e
85332026-03-11T20:45:51.528Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func1({0x50b1338?, 0xc006b57bd0?})
85342026-03-11T20:45:51.528Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:285 +0x6c
85352026-03-11T20:45:51.528Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
85362026-03-11T20:45:51.528Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
85372026-03-11T20:45:51.529Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
85382026-03-11T20:45:51.529Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
85392026-03-11T20:45:51.529Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10915
85402026-03-11T20:45:51.529Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
85412026-03-11T20:45:51.529Z
85422026-03-11T20:45:51.529Zgoroutine 10920 [select]:
85432026-03-11T20:45:51.529Zgithub.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10({0x50b1300?, 0xc00067dce0?})
85442026-03-11T20:45:51.529Z /work/oxidecomputer/cockroach/pkg/jobs/registry.go:866 +0x1bd
85452026-03-11T20:45:51.529Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
85462026-03-11T20:45:51.529Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
85472026-03-11T20:45:51.529Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
85482026-03-11T20:45:51.529Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
85492026-03-11T20:45:51.529Z
85502026-03-11T20:45:51.529Zgoroutine 10919 [select, 59 minutes]:
85512026-03-11T20:45:51.530Zgithub.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func9({0x50b1300?, 0xc00067dc80?})
85522026-03-11T20:45:51.530Z /work/oxidecomputer/cockroach/pkg/jobs/registry.go:843 +0x1be
85532026-03-11T20:45:51.530Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
85542026-03-11T20:45:51.530Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
85552026-03-11T20:45:51.530Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
85562026-03-11T20:45:51.530Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
85572026-03-11T20:45:51.530Z
85582026-03-11T20:45:51.530Zgoroutine 10750 [sync.Cond.Wait, 59 minutes]:
85592026-03-11T20:45:51.530Zsync.runtime_notifyListWait(0xc0064c5050, 0x18)
85602026-03-11T20:45:51.530Z /work/go/src/runtime/sema.go:569 +0x159
85612026-03-11T20:45:51.530Zsync.(*Cond).Wait(0x50f7930?)
85622026-03-11T20:45:51.530Z /work/go/src/sync/cond.go:70 +0x85
85632026-03-11T20:45:51.530Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3140})
85642026-03-11T20:45:51.530Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
85652026-03-11T20:45:51.530Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
85662026-03-11T20:45:51.530Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
85672026-03-11T20:45:51.530Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
85682026-03-11T20:45:51.530Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
85692026-03-11T20:45:51.531Z
85702026-03-11T20:45:51.531Zgoroutine 9920 [chan receive, 59 minutes]:
85712026-03-11T20:45:51.531Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc002e5aa50?})
85722026-03-11T20:45:51.531Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
85732026-03-11T20:45:51.531Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x0?, 0x0?}}, 0xc0042427a8)
85742026-03-11T20:45:51.531Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
85752026-03-11T20:45:51.531Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
85762026-03-11T20:45:51.531Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
85772026-03-11T20:45:51.531Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
85782026-03-11T20:45:51.531Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
85792026-03-11T20:45:51.531Z
85802026-03-11T20:45:51.531Zgoroutine 11073 [select, 59 minutes]:
85812026-03-11T20:45:51.531Zgoogle.golang.org/grpc.newClientStreamWithParams.func4()
85822026-03-11T20:45:51.531Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:341 +0x8c
85832026-03-11T20:45:51.531Zcreated by google.golang.org/grpc.newClientStreamWithParams in goroutine 11072
85842026-03-11T20:45:51.531Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:340 +0xb2a
85852026-03-11T20:45:51.531Z
85862026-03-11T20:45:51.531Zgoroutine 10588 [chan receive, 59 minutes]:
85872026-03-11T20:45:51.531Zgithub.com/cockroachdb/cockroach/pkg/server.(*httpServer).start.func1({0x50b1300, 0xc0064c1e90})
85882026-03-11T20:45:51.531Z /work/oxidecomputer/cockroach/pkg/server/server_http.go:226 +0x45
85892026-03-11T20:45:51.531Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
85902026-03-11T20:45:51.531Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
85912026-03-11T20:45:51.531Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
85922026-03-11T20:45:51.531Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
85932026-03-11T20:45:51.531Z
85942026-03-11T20:45:51.531Zgoroutine 9948 [select, 59 minutes]:
85952026-03-11T20:45:51.531Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc004cb0f08?, 0x0?})
85962026-03-11T20:45:51.531Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
85972026-03-11T20:45:51.531Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
85982026-03-11T20:45:51.531Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
85992026-03-11T20:45:51.531Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
86002026-03-11T20:45:51.531Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
86012026-03-11T20:45:51.531Z
86022026-03-11T20:45:51.531Zgoroutine 10195 [select]:
86032026-03-11T20:45:51.536Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).startWriteNodeStatus.func2({0x50b1300, 0xc0018cfe90})
86042026-03-11T20:45:51.536Z /work/oxidecomputer/cockroach/pkg/server/node.go:852 +0xf3
86052026-03-11T20:45:51.536Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
86062026-03-11T20:45:51.536Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
86072026-03-11T20:45:51.536Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
86082026-03-11T20:45:51.536Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
86092026-03-11T20:45:51.536Z
86102026-03-11T20:45:51.536Zgoroutine 10088 [sync.Cond.Wait]:
86112026-03-11T20:45:51.536Zsync.runtime_notifyListWait(0xc001e22250, 0x378e)
86122026-03-11T20:45:51.536Z /work/go/src/runtime/sema.go:569 +0x159
86132026-03-11T20:45:51.536Zsync.(*Cond).Wait(0x3aac600?)
86142026-03-11T20:45:51.536Z /work/go/src/sync/cond.go:70 +0x85
86152026-03-11T20:45:51.536Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583b60})
86162026-03-11T20:45:51.536Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
86172026-03-11T20:45:51.536Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
86182026-03-11T20:45:51.536Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
86192026-03-11T20:45:51.536Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
86202026-03-11T20:45:51.536Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
86212026-03-11T20:45:51.536Z
86222026-03-11T20:45:51.536Zgoroutine 11070 [select, 59 minutes]:
86232026-03-11T20:45:51.536Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1({0x50b1338, 0xc00540c280})
86242026-03-11T20:45:51.536Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:116 +0x157
86252026-03-11T20:45:51.536Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.Group.GoCtx.func3()
86262026-03-11T20:45:51.536Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
86272026-03-11T20:45:51.536Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
86282026-03-11T20:45:51.536Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
86292026-03-11T20:45:51.536Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 11068
86302026-03-11T20:45:51.536Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
86312026-03-11T20:45:51.536Z
86322026-03-11T20:45:51.536Zgoroutine 11148 [chan receive, 59 minutes]:
86332026-03-11T20:45:51.536Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).rangeFeedWithRangeID(0xc001ded408, 0x3d87860?, 0xc0020ea7e0, {0x7fc12ce5a548, 0xc003c49ba0})
86342026-03-11T20:45:51.536Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:249 +0x76b
86352026-03-11T20:45:51.536Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed(...)
86362026-03-11T20:45:51.536Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:149
86372026-03-11T20:45:51.536Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).RangeFeed(0xc000f39208, 0xc0020ea7e0, {0x50e5040, 0xc003c49ba0})
86382026-03-11T20:45:51.537Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:3082 +0x16f
86392026-03-11T20:45:51.537Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).RangeFeed(0xc001eb4380, 0xc0020ea7e0, {0x50e5040, 0xc003c49ba0})
86402026-03-11T20:45:51.537Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/stores.go:216 +0x19a
86412026-03-11T20:45:51.537Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed(0x732ffb0?, 0x40055e0?, {0x50e5040, 0xc003c49ba0})
86422026-03-11T20:45:51.537Z /work/oxidecomputer/cockroach/pkg/server/node.go:1232 +0x28
86432026-03-11T20:45:51.537Zgithub.com/cockroachdb/cockroach/pkg/roachpb._Internal_RangeFeed_Handler({0x40055e0, 0xc004911b08}, {0x50df230, 0xc006519bc0})
86442026-03-11T20:45:51.537Z /work/oxidecomputer/cockroach/pkg/roachpb/api.pb.go:9523 +0x107
86452026-03-11T20:45:51.537Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.StreamServerInterceptor.func14({0x40055e0, 0xc004911b08}, {0x50df9e0, 0xc007b38000}, 0xc00212a420, 0x42e42f8)
86462026-03-11T20:45:51.537Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:205 +0x35b
86472026-03-11T20:45:51.537Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x40055e0?, 0xc004911b08?}, {0x50df9e0?, 0xc007b38000?})
86482026-03-11T20:45:51.537Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1408 +0x59
86492026-03-11T20:45:51.537Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func4({0x40055e0, 0xc004911b08}, {0x50df9e0, 0xc007b38000}, 0x30?, 0xc0043cfec0)
86502026-03-11T20:45:51.537Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:247 +0x6c
86512026-03-11T20:45:51.537Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x40055e0?, 0xc004911b08?}, {0x50df9e0?, 0xc007b38000?})
86522026-03-11T20:45:51.537Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
86532026-03-11T20:45:51.537Zgithub.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({{{0x3ac98c0?}}}, {0x40055e0, 0xc004911b08}, {0x50df9e0, 0xc007b38000}, 0xc00212a420, 0xc0043cfec0)
86542026-03-11T20:45:51.537Z /work/oxidecomputer/cockroach/pkg/rpc/auth.go:98 +0x19b
86552026-03-11T20:45:51.537Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x40055e0?, 0xc004911b08?}, {0x50df9e0?, 0xc007b38000?})
86562026-03-11T20:45:51.537Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
86572026-03-11T20:45:51.537Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func2.1({0x66?, 0x40?})
86582026-03-11T20:45:51.537Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:216 +0x27
86592026-03-11T20:45:51.537Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc003f40400, {0x50b1300, 0xc006eea240}, {0xc003bd2b28?, 0x415dbb?}, 0xc003bd2ac0)
86602026-03-11T20:45:51.537Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:344 +0xcb
86612026-03-11T20:45:51.537Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func2({0x40055e0, 0xc004911b08}, {0x50df9e0, 0xc007b38000}, 0xc00212a420, 0xc0043cfec0)
86622026-03-11T20:45:51.537Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:215 +0xdf
86632026-03-11T20:45:51.537Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x40055e0?, 0xc004911b08?}, {0x50df9e0?, 0xc007b38000?})
86642026-03-11T20:45:51.537Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
86652026-03-11T20:45:51.537Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x40055e0, 0xc004911b08}, {0x50df9e0, 0xc007b38000}, 0xc00212a420, 0x42e42f8)
86662026-03-11T20:45:51.537Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1413 +0x108
86672026-03-11T20:45:51.537Zgoogle.golang.org/grpc.(*Server).processStreamingRPC(0xc004feefc0, {0x50f1d80, 0xc001aef6c0}, 0xc0020ea5a0, 0xc007661ad0, 0x73abba0, 0x0)
86682026-03-11T20:45:51.537Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1549 +0xe51
86692026-03-11T20:45:51.537Zgoogle.golang.org/grpc.(*Server).handleStream(0xc004feefc0, {0x50f1d80, 0xc001aef6c0}, 0xc0020ea5a0, 0x0)
86702026-03-11T20:45:51.537Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1624 +0x999
86712026-03-11T20:45:51.537Zgoogle.golang.org/grpc.(*Server).serveStreams.func1.2()
86722026-03-11T20:45:51.537Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:922 +0x8d
86732026-03-11T20:45:51.537Zcreated by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 10956
86742026-03-11T20:45:51.537Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:920 +0x226
86752026-03-11T20:45:51.538Z
86762026-03-11T20:45:51.538Zgoroutine 11156 [select]:
86772026-03-11T20:45:51.538Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).outputLoop(0xc004b49180, {0x50b1338, 0xc000f22050})
86782026-03-11T20:45:51.538Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:310 +0x247
86792026-03-11T20:45:51.538Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).runOutputLoop(0xc004b49180, {0x50b1300, 0xc0063c78f0}, 0x1b?)
86802026-03-11T20:45:51.538Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:335 +0xba
86812026-03-11T20:45:51.538Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run.func1({0x50b1300?, 0xc0063c78f0?})
86822026-03-11T20:45:51.538Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:297 +0x46
86832026-03-11T20:45:51.538Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
86842026-03-11T20:45:51.538Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
86852026-03-11T20:45:51.538Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10489
86862026-03-11T20:45:51.538Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
86872026-03-11T20:45:51.538Z
86882026-03-11T20:45:51.539Zgoroutine 11120 [semacquire, 59 minutes]:
86892026-03-11T20:45:51.539Zsync.runtime_Semacquire(0xc0022904b0?)
86902026-03-11T20:45:51.539Z /work/go/src/runtime/sema.go:62 +0x25
86912026-03-11T20:45:51.539Zsync.(*WaitGroup).Wait(0x44e7a0?)
86922026-03-11T20:45:51.539Z /work/go/src/sync/waitgroup.go:116 +0x48
86932026-03-11T20:45:51.539Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc004b39280)
86942026-03-11T20:45:51.539Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
86952026-03-11T20:45:51.539Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc004b39280?, {0x50b1338?, 0xc004c69540?}})
86962026-03-11T20:45:51.539Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
86972026-03-11T20:45:51.539Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed(0xc003d48008, {0x50b1338?, 0xc00218ee60?}, {0xc0078b6750, 0x1, 0xc0054cf050?}, {0xc007661ad0?, 0x55b21e1?, 0xc0?}, 0x0, ...)
86982026-03-11T20:45:51.539Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:139 +0x6c5
86992026-03-11T20:45:51.539Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*dbAdapter).RangeFeed(0xc0044e9608?, {0x50b1338?, 0xc00218ee60?}, {0xc0078b6750?, 0x50b1300?, 0xc0054cf140?}, {0x7070000050b1300?, 0x50b1300?, 0x0?}, 0x0, ...)
87002026-03-11T20:45:51.539Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/db_adapter.go:78 +0x3e
87012026-03-11T20:45:51.539Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func1({0x50b1338?, 0xc00218ee60?})
87022026-03-11T20:45:51.539Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:285 +0x6c
87032026-03-11T20:45:51.543Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
87042026-03-11T20:45:51.543Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
87052026-03-11T20:45:51.543Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
87062026-03-11T20:45:51.543Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
87072026-03-11T20:45:51.543Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10908
87082026-03-11T20:45:51.543Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
87092026-03-11T20:45:51.543Z
87102026-03-11T20:45:51.543Zgoroutine 9024 [select]:
87112026-03-11T20:45:51.543Zgithub.com/cockroachdb/cockroach/pkg/sql/contention.(*eventStore).startResolver.func1({0x50b1300, 0xc007659530})
87122026-03-11T20:45:51.543Z /work/oxidecomputer/cockroach/pkg/sql/contention/event_store.go:219 +0x1f4
87132026-03-11T20:45:51.543Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
87142026-03-11T20:45:51.543Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
87152026-03-11T20:45:51.543Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
87162026-03-11T20:45:51.543Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
87172026-03-11T20:45:51.544Z
87182026-03-11T20:45:51.544Zgoroutine 9062 [IO wait]:
87192026-03-11T20:45:51.544Zinternal/poll.runtime_pollWait(0x7fc12ea8f898, 0x72)
87202026-03-11T20:45:51.544Z /work/go/src/runtime/netpoll.go:345 +0x85
87212026-03-11T20:45:51.544Zinternal/poll.(*pollDesc).wait(0xc00692d500?, 0xc007682000?, 0x0)
87222026-03-11T20:45:51.544Z /work/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
87232026-03-11T20:45:51.544Zinternal/poll.(*pollDesc).waitRead(...)
87242026-03-11T20:45:51.544Z /work/go/src/internal/poll/fd_poll_runtime.go:89
87252026-03-11T20:45:51.544Zinternal/poll.(*FD).Read(0xc00692d500, {0xc007682000, 0x600, 0x600})
87262026-03-11T20:45:51.544Z /work/go/src/internal/poll/fd_unix.go:164 +0x27a
87272026-03-11T20:45:51.544Znet.(*netFD).Read(0xc00692d500, {0xc007682000?, 0xc005ab2868?, 0x41e1e5?})
87282026-03-11T20:45:51.544Z /work/go/src/net/fd_posix.go:55 +0x25
87292026-03-11T20:45:51.544Znet.(*conn).Read(0xc004e7cb88, {0xc007682000?, 0x7fc126f226b8?, 0xc03142bb78?})
87302026-03-11T20:45:51.544Z /work/go/src/net/net.go:185 +0x45
87312026-03-11T20:45:51.544Zgithub.com/cockroachdb/cmux.(*MuxConn).Read(0xc003cf0780, {0xc007682000, 0x600, 0x600})
87322026-03-11T20:45:51.544Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x14f
87332026-03-11T20:45:51.544Zcrypto/tls.(*atLeastReader).Read(0xc03142bb78, {0xc007682000?, 0x0?, 0xc03142bb78?})
87342026-03-11T20:45:51.544Z /work/go/src/crypto/tls/conn.go:806 +0x3b
87352026-03-11T20:45:51.544Zbytes.(*Buffer).ReadFrom(0xc0065ff0b0, {0x5081fe0, 0xc03142bb78})
87362026-03-11T20:45:51.544Z /work/go/src/bytes/buffer.go:211 +0x98
87372026-03-11T20:45:51.544Zcrypto/tls.(*Conn).readFromUntil(0xc0065fee08, {0x7fc12ce5a2d8, 0xc003cf0780}, 0xc005ab2990?)
87382026-03-11T20:45:51.544Z /work/go/src/crypto/tls/conn.go:828 +0xde
87392026-03-11T20:45:51.544Zcrypto/tls.(*Conn).readRecordOrCCS(0xc0065fee08, 0x0)
87402026-03-11T20:45:51.544Z /work/go/src/crypto/tls/conn.go:626 +0x3cf
87412026-03-11T20:45:51.544Zcrypto/tls.(*Conn).readRecord(...)
87422026-03-11T20:45:51.544Z /work/go/src/crypto/tls/conn.go:588
87432026-03-11T20:45:51.544Zcrypto/tls.(*Conn).Read(0xc0065fee08, {0xc007836000, 0x8000, 0x40dc20?})
87442026-03-11T20:45:51.545Z /work/go/src/crypto/tls/conn.go:1370 +0x156
87452026-03-11T20:45:51.545Zbufio.(*Reader).Read(0xc00625e5a0, {0xc0038cc120, 0x9, 0x0?})
87462026-03-11T20:45:51.545Z /work/go/src/bufio/bufio.go:241 +0x197
87472026-03-11T20:45:51.545Zio.ReadAtLeast({0x507c320, 0xc00625e5a0}, {0xc0038cc120, 0x9, 0x9}, 0x9)
87482026-03-11T20:45:51.545Z /work/go/src/io/io.go:335 +0x90
87492026-03-11T20:45:51.545Zio.ReadFull(...)
87502026-03-11T20:45:51.545Z /work/go/src/io/io.go:354
87512026-03-11T20:45:51.545Zgolang.org/x/net/http2.readFrameHeader({0xc0038cc120, 0x9, 0xc005ab2e10?}, {0x507c320?, 0xc00625e5a0?})
87522026-03-11T20:45:51.545Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x65
87532026-03-11T20:45:51.545Zgolang.org/x/net/http2.(*Framer).ReadFrame(0xc0038cc0e0)
87542026-03-11T20:45:51.545Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:501 +0x85
87552026-03-11T20:45:51.545Zgoogle.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc007738000, 0xc0076b6510, 0x42e6fc8)
87562026-03-11T20:45:51.545Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:611 +0x14e
87572026-03-11T20:45:51.545Zgoogle.golang.org/grpc.(*Server).serveStreams(0xc004feefc0, {0x50f1d80, 0xc007738000})
87582026-03-11T20:45:51.545Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:906 +0x114
87592026-03-11T20:45:51.545Zgoogle.golang.org/grpc.(*Server).handleRawConn.func1()
87602026-03-11T20:45:51.545Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:848 +0x45
87612026-03-11T20:45:51.545Zcreated by google.golang.org/grpc.(*Server).handleRawConn in goroutine 10710
87622026-03-11T20:45:51.545Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:847 +0x15b
87632026-03-11T20:45:51.545Z
87642026-03-11T20:45:51.545Zgoroutine 10752 [sync.Cond.Wait, 59 minutes]:
87652026-03-11T20:45:51.545Zsync.runtime_notifyListWait(0xc0064c5050, 0x11)
87662026-03-11T20:45:51.545Z /work/go/src/runtime/sema.go:569 +0x159
87672026-03-11T20:45:51.545Zsync.(*Cond).Wait(0x50f7930?)
87682026-03-11T20:45:51.545Z /work/go/src/sync/cond.go:70 +0x85
87692026-03-11T20:45:51.545Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd31a0})
87702026-03-11T20:45:51.545Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
87712026-03-11T20:45:51.545Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
87722026-03-11T20:45:51.545Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
87732026-03-11T20:45:51.545Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
87742026-03-11T20:45:51.545Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
87752026-03-11T20:45:51.545Z
87762026-03-11T20:45:51.545Zgoroutine 10753 [sync.Cond.Wait, 59 minutes]:
87772026-03-11T20:45:51.545Zsync.runtime_notifyListWait(0xc0064c5050, 0x13)
87782026-03-11T20:45:51.545Z /work/go/src/runtime/sema.go:569 +0x159
87792026-03-11T20:45:51.545Zsync.(*Cond).Wait(0x50b1300?)
87802026-03-11T20:45:51.545Z /work/go/src/sync/cond.go:70 +0x85
87812026-03-11T20:45:51.545Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd31d0})
87822026-03-11T20:45:51.545Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
87832026-03-11T20:45:51.546Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
87842026-03-11T20:45:51.546Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
87852026-03-11T20:45:51.546Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
87862026-03-11T20:45:51.546Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
87872026-03-11T20:45:51.546Z
87882026-03-11T20:45:51.546Zgoroutine 10725 [sync.Cond.Wait, 59 minutes]:
87892026-03-11T20:45:51.546Zsync.runtime_notifyListWait(0xc0064c5050, 0x1a)
87902026-03-11T20:45:51.546Z /work/go/src/runtime/sema.go:569 +0x159
87912026-03-11T20:45:51.546Zsync.(*Cond).Wait(0xc000f39208?)
87922026-03-11T20:45:51.546Z /work/go/src/sync/cond.go:70 +0x85
87932026-03-11T20:45:51.546Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd2c90})
87942026-03-11T20:45:51.546Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
87952026-03-11T20:45:51.546Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
87962026-03-11T20:45:51.546Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
87972026-03-11T20:45:51.546Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
87982026-03-11T20:45:51.546Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
87992026-03-11T20:45:51.546Z
88002026-03-11T20:45:51.546Zgoroutine 10724 [sync.Cond.Wait, 59 minutes]:
88012026-03-11T20:45:51.546Zsync.runtime_notifyListWait(0xc0064c5050, 0x17)
88022026-03-11T20:45:51.546Z /work/go/src/runtime/sema.go:569 +0x159
88032026-03-11T20:45:51.550Zsync.(*Cond).Wait(0xc0083c6ab0?)
88042026-03-11T20:45:51.550Z /work/go/src/sync/cond.go:70 +0x85
88052026-03-11T20:45:51.550Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd2c60})
88062026-03-11T20:45:51.550Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
88072026-03-11T20:45:51.550Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
88082026-03-11T20:45:51.550Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
88092026-03-11T20:45:51.550Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
88102026-03-11T20:45:51.551Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
88112026-03-11T20:45:51.551Z
88122026-03-11T20:45:51.551Zgoroutine 11061 [select]:
88132026-03-11T20:45:51.551Zgoogle.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc002092000, 0x1)
88142026-03-11T20:45:51.551Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:407 +0x113
88152026-03-11T20:45:51.551Zgoogle.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00612f920)
88162026-03-11T20:45:51.551Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:534 +0x7b
88172026-03-11T20:45:51.551Zgoogle.golang.org/grpc/internal/transport.newHTTP2Client.func3()
88182026-03-11T20:45:51.551Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:415 +0x67
88192026-03-11T20:45:51.551Zcreated by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 11046
88202026-03-11T20:45:51.551Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:413 +0x1dcb
88212026-03-11T20:45:51.551Z
88222026-03-11T20:45:51.551Zgoroutine 10997 [select]:
88232026-03-11T20:45:51.551Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/closedts/sidetransport.(*Sender).Run.func2({0x50b1300, 0xc003294780})
88242026-03-11T20:45:51.551Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/closedts/sidetransport/sender.go:251 +0x1ab
88252026-03-11T20:45:51.551Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
88262026-03-11T20:45:51.551Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
88272026-03-11T20:45:51.551Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
88282026-03-11T20:45:51.551Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
88292026-03-11T20:45:51.551Z
88302026-03-11T20:45:51.551Zgoroutine 10996 [select, 59 minutes]:
88312026-03-11T20:45:51.551Zgithub.com/cockroachdb/cockroach/pkg/sql/scheduledlogging.(*CaptureIndexUsageStatsLoggingScheduler).start.func1({0x50b1300, 0xc0032942d0})
88322026-03-11T20:45:51.551Z /work/oxidecomputer/cockroach/pkg/sql/scheduledlogging/captured_index_usage_stats.go:144 +0xeb
88332026-03-11T20:45:51.551Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
88342026-03-11T20:45:51.551Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
88352026-03-11T20:45:51.551Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
88362026-03-11T20:45:51.551Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
88372026-03-11T20:45:51.551Z
88382026-03-11T20:45:51.551Zgoroutine 9932 [select, 59 minutes]:
88392026-03-11T20:45:51.551Zgithub.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowScheduler).Start.func1({0xc003c6e908?, 0xc00296e210?})
88402026-03-11T20:45:51.551Z /work/oxidecomputer/cockroach/pkg/sql/flowinfra/flow_scheduler.go:331 +0x25c
88412026-03-11T20:45:51.551Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
88422026-03-11T20:45:51.551Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
88432026-03-11T20:45:51.551Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
88442026-03-11T20:45:51.551Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
88452026-03-11T20:45:51.551Z
88462026-03-11T20:45:51.551Zgoroutine 10968 [select]:
88472026-03-11T20:45:51.552Zgoogle.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0078adb30, 0x1)
88482026-03-11T20:45:51.552Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:407 +0x113
88492026-03-11T20:45:51.552Zgoogle.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0036d7440)
88502026-03-11T20:45:51.552Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:534 +0x7b
88512026-03-11T20:45:51.552Zgoogle.golang.org/grpc/internal/transport.newHTTP2Client.func3()
88522026-03-11T20:45:51.552Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:415 +0x67
88532026-03-11T20:45:51.552Zcreated by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 10962
88542026-03-11T20:45:51.552Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:413 +0x1dcb
88552026-03-11T20:45:51.552Z
88562026-03-11T20:45:51.553Zgoroutine 10967 [IO wait]:
88572026-03-11T20:45:51.553Zinternal/poll.runtime_pollWait(0x7fc12c3be2d8, 0x72)
88582026-03-11T20:45:51.553Z /work/go/src/runtime/netpoll.go:345 +0x85
88592026-03-11T20:45:51.553Zinternal/poll.(*pollDesc).wait(0xc006377a80?, 0xc0061fe300?, 0x0)
88602026-03-11T20:45:51.553Z /work/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
88612026-03-11T20:45:51.553Zinternal/poll.(*pollDesc).waitRead(...)
88622026-03-11T20:45:51.553Z /work/go/src/internal/poll/fd_poll_runtime.go:89
88632026-03-11T20:45:51.553Zinternal/poll.(*FD).Read(0xc006377a80, {0xc0061fe300, 0x700, 0x700})
88642026-03-11T20:45:51.553Z /work/go/src/internal/poll/fd_unix.go:164 +0x27a
88652026-03-11T20:45:51.553Znet.(*netFD).Read(0xc006377a80, {0xc0061fe300?, 0x7fc12572c978?, 0xc027f44b10?})
88662026-03-11T20:45:51.553Z /work/go/src/net/fd_posix.go:55 +0x25
88672026-03-11T20:45:51.553Znet.(*conn).Read(0xc003c32388, {0xc0061fe300?, 0xc0029bf8e0?, 0x415dbb?})
88682026-03-11T20:45:51.553Z /work/go/src/net/net.go:185 +0x45
88692026-03-11T20:45:51.553Zcrypto/tls.(*atLeastReader).Read(0xc027f44b10, {0xc0061fe300?, 0x0?, 0xc027f44b10?})
88702026-03-11T20:45:51.553Z /work/go/src/crypto/tls/conn.go:806 +0x3b
88712026-03-11T20:45:51.553Zbytes.(*Buffer).ReadFrom(0xc006f157b0, {0x5081fe0, 0xc027f44b10})
88722026-03-11T20:45:51.553Z /work/go/src/bytes/buffer.go:211 +0x98
88732026-03-11T20:45:51.553Zcrypto/tls.(*Conn).readFromUntil(0xc006f15508, {0x50821c0, 0xc003c32388}, 0xc0029bf928?)
88742026-03-11T20:45:51.553Z /work/go/src/crypto/tls/conn.go:828 +0xde
88752026-03-11T20:45:51.553Zcrypto/tls.(*Conn).readRecordOrCCS(0xc006f15508, 0x0)
88762026-03-11T20:45:51.553Z /work/go/src/crypto/tls/conn.go:626 +0x3cf
88772026-03-11T20:45:51.553Zcrypto/tls.(*Conn).readRecord(...)
88782026-03-11T20:45:51.554Z /work/go/src/crypto/tls/conn.go:588
88792026-03-11T20:45:51.554Zcrypto/tls.(*Conn).Read(0xc006f15508, {0xc003f1a000, 0x8000, 0xc0029bfc88?})
88802026-03-11T20:45:51.554Z /work/go/src/crypto/tls/conn.go:1370 +0x156
88812026-03-11T20:45:51.554Zbufio.(*Reader).Read(0xc0036d72c0, {0xc000d397e0, 0x9, 0x8?})
88822026-03-11T20:45:51.554Z /work/go/src/bufio/bufio.go:241 +0x197
88832026-03-11T20:45:51.554Zio.ReadAtLeast({0x507c320, 0xc0036d72c0}, {0xc000d397e0, 0x9, 0x9}, 0x9)
88842026-03-11T20:45:51.554Z /work/go/src/io/io.go:335 +0x90
88852026-03-11T20:45:51.554Zio.ReadFull(...)
88862026-03-11T20:45:51.554Z /work/go/src/io/io.go:354
88872026-03-11T20:45:51.554Zgolang.org/x/net/http2.readFrameHeader({0xc000d397e0, 0x9, 0xc00426d050?}, {0x507c320?, 0xc0036d72c0?})
88882026-03-11T20:45:51.554Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x65
88892026-03-11T20:45:51.554Zgolang.org/x/net/http2.(*Framer).ReadFrame(0xc000d397a0)
88902026-03-11T20:45:51.554Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:501 +0x85
88912026-03-11T20:45:51.554Zgoogle.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0066c23c0)
88922026-03-11T20:45:51.554Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1498 +0x41d
88932026-03-11T20:45:51.554Zcreated by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 10962
88942026-03-11T20:45:51.554Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:365 +0x184d
88952026-03-11T20:45:51.554Z
88962026-03-11T20:45:51.554Zgoroutine 3679883 [select]:
88972026-03-11T20:45:51.554Zgithub.com/cockroachdb/cockroach/pkg/sql/stats.(*Refresher).Start.func1.1({0x50b1300, 0xc009486d20})
88982026-03-11T20:45:51.554Z /work/oxidecomputer/cockroach/pkg/sql/stats/automatic_stats.go:410 +0x145
88992026-03-11T20:45:51.554Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
89002026-03-11T20:45:51.554Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
89012026-03-11T20:45:51.554Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10865
89022026-03-11T20:45:51.554Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
89032026-03-11T20:45:51.564Z
89042026-03-11T20:45:51.564Zgoroutine 10973 [chan receive, 59 minutes]:
89052026-03-11T20:45:51.564Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).rangeFeedWithRangeID(0xc001ded408, 0x3d87860?, 0xc004e62c60, {0x7fc12ce5a548, 0xc006e38030})
89062026-03-11T20:45:51.564Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:249 +0x76b
89072026-03-11T20:45:51.564Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed(...)
89082026-03-11T20:45:51.565Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:149
89092026-03-11T20:45:51.565Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).RangeFeed(0xc000f39208, 0xc004e62c60, {0x50e5040, 0xc006e38030})
89102026-03-11T20:45:51.565Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:3082 +0x16f
89112026-03-11T20:45:51.565Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).RangeFeed(0xc001eb4380, 0xc004e62c60, {0x50e5040, 0xc006e38030})
89122026-03-11T20:45:51.565Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/stores.go:216 +0x19a
89132026-03-11T20:45:51.565Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed(0x732ffb0?, 0x40055e0?, {0x50e5040, 0xc006e38030})
89142026-03-11T20:45:51.565Z /work/oxidecomputer/cockroach/pkg/server/node.go:1232 +0x28
89152026-03-11T20:45:51.565Zgithub.com/cockroachdb/cockroach/pkg/roachpb._Internal_RangeFeed_Handler({0x40055e0, 0xc004911b08}, {0x50df230, 0xc00827bae0})
89162026-03-11T20:45:51.565Z /work/oxidecomputer/cockroach/pkg/roachpb/api.pb.go:9523 +0x107
89172026-03-11T20:45:51.565Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.StreamServerInterceptor.func14({0x40055e0, 0xc004911b08}, {0x50df9e0, 0xc0044ffad0}, 0xc00583c288, 0x42e42f8)
89182026-03-11T20:45:51.565Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:205 +0x35b
89192026-03-11T20:45:51.565Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x40055e0?, 0xc004911b08?}, {0x50df9e0?, 0xc0044ffad0?})
89202026-03-11T20:45:51.565Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1408 +0x59
89212026-03-11T20:45:51.565Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func4({0x40055e0, 0xc004911b08}, {0x50df9e0, 0xc0044ffad0}, 0x15b74f6?, 0xc0077075c0)
89222026-03-11T20:45:51.566Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:247 +0x6c
89232026-03-11T20:45:51.566Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x40055e0?, 0xc004911b08?}, {0x50df9e0?, 0xc0044ffad0?})
89242026-03-11T20:45:51.566Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
89252026-03-11T20:45:51.566Zgithub.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({{{0x10238e0?}}}, {0x40055e0, 0xc004911b08}, {0x50df9e0, 0xc0044ffad0}, 0xc00583c288, 0xc0077075c0)
89262026-03-11T20:45:51.566Z /work/oxidecomputer/cockroach/pkg/rpc/auth.go:98 +0x19b
89272026-03-11T20:45:51.566Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x40055e0?, 0xc004911b08?}, {0x50df9e0?, 0xc0044ffad0?})
89282026-03-11T20:45:51.566Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
89292026-03-11T20:45:51.566Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func2.1({0x66?, 0x40?})
89302026-03-11T20:45:51.566Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:216 +0x27
89312026-03-11T20:45:51.566Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc003f40400, {0x50b1300, 0xc0032c9e00}, {0xc002c81b28?, 0x415dbb?}, 0xc002c81ac0)
89322026-03-11T20:45:51.566Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:344 +0xcb
89332026-03-11T20:45:51.566Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewServer.func2({0x40055e0, 0xc004911b08}, {0x50df9e0, 0xc0044ffad0}, 0xc00583c288, 0xc0077075c0)
89342026-03-11T20:45:51.566Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:215 +0xdf
89352026-03-11T20:45:51.566Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2.1({0x40055e0?, 0xc004911b08?}, {0x50df9e0?, 0xc0044ffad0?})
89362026-03-11T20:45:51.566Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1411 +0x7a
89372026-03-11T20:45:51.566Zgoogle.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x40055e0, 0xc004911b08}, {0x50df9e0, 0xc0044ffad0}, 0xc00583c288, 0x42e42f8)
89382026-03-11T20:45:51.566Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1413 +0x108
89392026-03-11T20:45:51.566Zgoogle.golang.org/grpc.(*Server).processStreamingRPC(0xc004feefc0, {0x50f1d80, 0xc001aef6c0}, 0xc004e62b40, 0xc007661ad0, 0x73abba0, 0x0)
89402026-03-11T20:45:51.566Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1549 +0xe51
89412026-03-11T20:45:51.566Zgoogle.golang.org/grpc.(*Server).handleStream(0xc004feefc0, {0x50f1d80, 0xc001aef6c0}, 0xc004e62b40, 0x0)
89422026-03-11T20:45:51.566Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:1624 +0x999
89432026-03-11T20:45:51.566Zgoogle.golang.org/grpc.(*Server).serveStreams.func1.2()
89442026-03-11T20:45:51.567Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:922 +0x8d
89452026-03-11T20:45:51.567Zcreated by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 10956
89462026-03-11T20:45:51.567Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:920 +0x226
89472026-03-11T20:45:51.567Z
89482026-03-11T20:45:51.567Zgoroutine 9759 [select]:
89492026-03-11T20:45:51.567Zgithub.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc000e28800, {0x50b1300, 0xc007b06a50}, 0xc007b0a410, {0xc0041a99e0, 0xf}, 0xc0061442a0)
89502026-03-11T20:45:51.567Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:1462 +0x27b
89512026-03-11T20:45:51.567Zgithub.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1({0x50b1300, 0xc007b06a50})
89522026-03-11T20:45:51.567Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:1380 +0x85
89532026-03-11T20:45:51.567Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
89542026-03-11T20:45:51.567Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
89552026-03-11T20:45:51.567Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9755
89562026-03-11T20:45:51.567Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
89572026-03-11T20:45:51.568Z
89582026-03-11T20:45:51.568Zgoroutine 11187 [select]:
89592026-03-11T20:45:51.568Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).outputLoop(0xc004de28c0, {0x50b1338, 0xc001435860})
89602026-03-11T20:45:51.568Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:310 +0x247
89612026-03-11T20:45:51.568Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).runOutputLoop(0xc004de28c0, {0x50b1300, 0xc0070e3e00}, 0x1b?)
89622026-03-11T20:45:51.568Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:335 +0xba
89632026-03-11T20:45:51.568Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run.func1({0x50b1300?, 0xc0070e3e00?})
89642026-03-11T20:45:51.568Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:297 +0x46
89652026-03-11T20:45:51.568Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
89662026-03-11T20:45:51.568Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
89672026-03-11T20:45:51.568Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 11080
89682026-03-11T20:45:51.568Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
89692026-03-11T20:45:51.568Z
89702026-03-11T20:45:51.568Zgoroutine 11186 [select, 59 minutes]:
89712026-03-11T20:45:51.568Zgithub.com/cockroachdb/cockroach/pkg/util/tracing.newTracingClientStream.func2()
89722026-03-11T20:45:51.568Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:386 +0x67
89732026-03-11T20:45:51.568Zcreated by github.com/cockroachdb/cockroach/pkg/util/tracing.newTracingClientStream in goroutine 11104
89742026-03-11T20:45:51.568Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:385 +0x125
89752026-03-11T20:45:51.568Z
89762026-03-11T20:45:51.568Zgoroutine 10773 [sync.Cond.Wait, 59 minutes]:
89772026-03-11T20:45:51.568Zsync.runtime_notifyListWait(0xc0064c5050, 0x36)
89782026-03-11T20:45:51.568Z /work/go/src/runtime/sema.go:569 +0x159
89792026-03-11T20:45:51.568Zsync.(*Cond).Wait(0x416165?)
89802026-03-11T20:45:51.568Z /work/go/src/sync/cond.go:70 +0x85
89812026-03-11T20:45:51.568Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3590})
89822026-03-11T20:45:51.568Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
89832026-03-11T20:45:51.569Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
89842026-03-11T20:45:51.569Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
89852026-03-11T20:45:51.569Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
89862026-03-11T20:45:51.569Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
89872026-03-11T20:45:51.569Z
89882026-03-11T20:45:51.569Zgoroutine 10772 [sync.Cond.Wait, 59 minutes]:
89892026-03-11T20:45:51.569Zsync.runtime_notifyListWait(0xc0064c5050, 0x37)
89902026-03-11T20:45:51.569Z /work/go/src/runtime/sema.go:569 +0x159
89912026-03-11T20:45:51.569Zsync.(*Cond).Wait(0x0?)
89922026-03-11T20:45:51.569Z /work/go/src/sync/cond.go:70 +0x85
89932026-03-11T20:45:51.569Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3560})
89942026-03-11T20:45:51.569Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
89952026-03-11T20:45:51.569Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
89962026-03-11T20:45:51.569Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
89972026-03-11T20:45:51.569Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
89982026-03-11T20:45:51.569Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
89992026-03-11T20:45:51.569Z
90002026-03-11T20:45:51.569Zgoroutine 10771 [sync.Cond.Wait, 59 minutes]:
90012026-03-11T20:45:51.569Zsync.runtime_notifyListWait(0xc0064c5050, 0x35)
90022026-03-11T20:45:51.569Z /work/go/src/runtime/sema.go:569 +0x159
90032026-03-11T20:45:51.573Zsync.(*Cond).Wait(0x7fc12ce971b8?)
90042026-03-11T20:45:51.574Z /work/go/src/sync/cond.go:70 +0x85
90052026-03-11T20:45:51.574Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3530})
90062026-03-11T20:45:51.574Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
90072026-03-11T20:45:51.574Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
90082026-03-11T20:45:51.574Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
90092026-03-11T20:45:51.574Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
90102026-03-11T20:45:51.574Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
90112026-03-11T20:45:51.574Z
90122026-03-11T20:45:51.574Zgoroutine 10770 [sync.Cond.Wait, 59 minutes]:
90132026-03-11T20:45:51.574Zsync.runtime_notifyListWait(0xc0064c5050, 0x33)
90142026-03-11T20:45:51.574Z /work/go/src/runtime/sema.go:569 +0x159
90152026-03-11T20:45:51.574Zsync.(*Cond).Wait(0x50b1300?)
90162026-03-11T20:45:51.575Z /work/go/src/sync/cond.go:70 +0x85
90172026-03-11T20:45:51.575Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3500})
90182026-03-11T20:45:51.575Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
90192026-03-11T20:45:51.575Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
90202026-03-11T20:45:51.575Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
90212026-03-11T20:45:51.575Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
90222026-03-11T20:45:51.575Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
90232026-03-11T20:45:51.575Z
90242026-03-11T20:45:51.575Zgoroutine 10774 [sync.Cond.Wait, 59 minutes]:
90252026-03-11T20:45:51.575Zsync.runtime_notifyListWait(0xc0064c5050, 0x38)
90262026-03-11T20:45:51.575Z /work/go/src/runtime/sema.go:569 +0x159
90272026-03-11T20:45:51.575Zsync.(*Cond).Wait(0x18?)
90282026-03-11T20:45:51.575Z /work/go/src/sync/cond.go:70 +0x85
90292026-03-11T20:45:51.575Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd35c0})
90302026-03-11T20:45:51.575Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
90312026-03-11T20:45:51.575Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
90322026-03-11T20:45:51.575Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
90332026-03-11T20:45:51.575Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
90342026-03-11T20:45:51.575Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
90352026-03-11T20:45:51.575Z
90362026-03-11T20:45:51.575Zgoroutine 10768 [sync.Cond.Wait, 59 minutes]:
90372026-03-11T20:45:51.575Zsync.runtime_notifyListWait(0xc0064c5050, 0x2f)
90382026-03-11T20:45:51.575Z /work/go/src/runtime/sema.go:569 +0x159
90392026-03-11T20:45:51.575Zsync.(*Cond).Wait(0xc005dc2e30?)
90402026-03-11T20:45:51.575Z /work/go/src/sync/cond.go:70 +0x85
90412026-03-11T20:45:51.575Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd34a0})
90422026-03-11T20:45:51.575Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
90432026-03-11T20:45:51.576Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
90442026-03-11T20:45:51.576Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
90452026-03-11T20:45:51.576Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
90462026-03-11T20:45:51.576Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
90472026-03-11T20:45:51.576Z
90482026-03-11T20:45:51.576Zgoroutine 10769 [sync.Cond.Wait, 59 minutes]:
90492026-03-11T20:45:51.576Zsync.runtime_notifyListWait(0xc0064c5050, 0x32)
90502026-03-11T20:45:51.576Z /work/go/src/runtime/sema.go:569 +0x159
90512026-03-11T20:45:51.576Zsync.(*Cond).Wait(0xffffffffffffffff?)
90522026-03-11T20:45:51.576Z /work/go/src/sync/cond.go:70 +0x85
90532026-03-11T20:45:51.576Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd34d0})
90542026-03-11T20:45:51.576Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
90552026-03-11T20:45:51.576Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
90562026-03-11T20:45:51.576Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
90572026-03-11T20:45:51.576Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
90582026-03-11T20:45:51.576Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
90592026-03-11T20:45:51.576Z
90602026-03-11T20:45:51.577Zgoroutine 10767 [sync.Cond.Wait, 59 minutes]:
90612026-03-11T20:45:51.577Zsync.runtime_notifyListWait(0xc0064c5050, 0x31)
90622026-03-11T20:45:51.577Z /work/go/src/runtime/sema.go:569 +0x159
90632026-03-11T20:45:51.577Zsync.(*Cond).Wait(0x1d3ab80?)
90642026-03-11T20:45:51.577Z /work/go/src/sync/cond.go:70 +0x85
90652026-03-11T20:45:51.577Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3470})
90662026-03-11T20:45:51.577Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
90672026-03-11T20:45:51.577Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
90682026-03-11T20:45:51.577Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
90692026-03-11T20:45:51.577Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
90702026-03-11T20:45:51.577Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
90712026-03-11T20:45:51.577Z
90722026-03-11T20:45:51.577Zgoroutine 10764 [sync.Cond.Wait, 59 minutes]:
90732026-03-11T20:45:51.577Zsync.runtime_notifyListWait(0xc0064c5050, 0x30)
90742026-03-11T20:45:51.577Z /work/go/src/runtime/sema.go:569 +0x159
90752026-03-11T20:45:51.577Zsync.(*Cond).Wait(0x50f7930?)
90762026-03-11T20:45:51.577Z /work/go/src/sync/cond.go:70 +0x85
90772026-03-11T20:45:51.578Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd33e0})
90782026-03-11T20:45:51.578Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
90792026-03-11T20:45:51.578Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
90802026-03-11T20:45:51.578Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
90812026-03-11T20:45:51.578Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
90822026-03-11T20:45:51.578Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
90832026-03-11T20:45:51.578Z
90842026-03-11T20:45:51.578Zgoroutine 10766 [sync.Cond.Wait, 59 minutes]:
90852026-03-11T20:45:51.578Zsync.runtime_notifyListWait(0xc0064c5050, 0x2c)
90862026-03-11T20:45:51.578Z /work/go/src/runtime/sema.go:569 +0x159
90872026-03-11T20:45:51.578Zsync.(*Cond).Wait(0x1d3ab80?)
90882026-03-11T20:45:51.578Z /work/go/src/sync/cond.go:70 +0x85
90892026-03-11T20:45:51.578Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3440})
90902026-03-11T20:45:51.578Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
90912026-03-11T20:45:51.578Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
90922026-03-11T20:45:51.578Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
90932026-03-11T20:45:51.578Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
90942026-03-11T20:45:51.578Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
90952026-03-11T20:45:51.578Z
90962026-03-11T20:45:51.578Zgoroutine 10765 [sync.Cond.Wait, 59 minutes]:
90972026-03-11T20:45:51.578Zsync.runtime_notifyListWait(0xc0064c5050, 0x34)
90982026-03-11T20:45:51.578Z /work/go/src/runtime/sema.go:569 +0x159
90992026-03-11T20:45:51.578Zsync.(*Cond).Wait(0xc006e3e878?)
91002026-03-11T20:45:51.578Z /work/go/src/sync/cond.go:70 +0x85
91012026-03-11T20:45:51.578Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3410})
91022026-03-11T20:45:51.578Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
91032026-03-11T20:45:51.583Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
91042026-03-11T20:45:51.583Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
91052026-03-11T20:45:51.583Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
91062026-03-11T20:45:51.583Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
91072026-03-11T20:45:51.583Z
91082026-03-11T20:45:51.583Zgoroutine 10763 [sync.Cond.Wait, 59 minutes]:
91092026-03-11T20:45:51.583Zsync.runtime_notifyListWait(0xc0064c5050, 0x2e)
91102026-03-11T20:45:51.583Z /work/go/src/runtime/sema.go:569 +0x159
91112026-03-11T20:45:51.583Zsync.(*Cond).Wait(0xc002d08008?)
91122026-03-11T20:45:51.583Z /work/go/src/sync/cond.go:70 +0x85
91132026-03-11T20:45:51.583Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd33b0})
91142026-03-11T20:45:51.583Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
91152026-03-11T20:45:51.583Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
91162026-03-11T20:45:51.583Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
91172026-03-11T20:45:51.583Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
91182026-03-11T20:45:51.583Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
91192026-03-11T20:45:51.584Z
91202026-03-11T20:45:51.584Zgoroutine 10762 [sync.Cond.Wait, 59 minutes]:
91212026-03-11T20:45:51.584Zsync.runtime_notifyListWait(0xc0064c5050, 0x23)
91222026-03-11T20:45:51.584Z /work/go/src/runtime/sema.go:569 +0x159
91232026-03-11T20:45:51.584Zsync.(*Cond).Wait(0x50f7930?)
91242026-03-11T20:45:51.584Z /work/go/src/sync/cond.go:70 +0x85
91252026-03-11T20:45:51.584Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3380})
91262026-03-11T20:45:51.584Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
91272026-03-11T20:45:51.584Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
91282026-03-11T20:45:51.584Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
91292026-03-11T20:45:51.584Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
91302026-03-11T20:45:51.585Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
91312026-03-11T20:45:51.585Z
91322026-03-11T20:45:51.585Zgoroutine 10761 [sync.Cond.Wait, 59 minutes]:
91332026-03-11T20:45:51.585Zsync.runtime_notifyListWait(0xc0064c5050, 0x28)
91342026-03-11T20:45:51.585Z /work/go/src/runtime/sema.go:569 +0x159
91352026-03-11T20:45:51.585Zsync.(*Cond).Wait(0x10000c002291620?)
91362026-03-11T20:45:51.585Z /work/go/src/sync/cond.go:70 +0x85
91372026-03-11T20:45:51.585Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3350})
91382026-03-11T20:45:51.585Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
91392026-03-11T20:45:51.585Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
91402026-03-11T20:45:51.585Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
91412026-03-11T20:45:51.585Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
91422026-03-11T20:45:51.585Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
91432026-03-11T20:45:51.585Z
91442026-03-11T20:45:51.585Zgoroutine 10760 [sync.Cond.Wait, 59 minutes]:
91452026-03-11T20:45:51.585Zsync.runtime_notifyListWait(0xc0064c5050, 0x25)
91462026-03-11T20:45:51.585Z /work/go/src/runtime/sema.go:569 +0x159
91472026-03-11T20:45:51.586Zsync.(*Cond).Wait(0x50f7930?)
91482026-03-11T20:45:51.586Z /work/go/src/sync/cond.go:70 +0x85
91492026-03-11T20:45:51.586Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3320})
91502026-03-11T20:45:51.586Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
91512026-03-11T20:45:51.586Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
91522026-03-11T20:45:51.586Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
91532026-03-11T20:45:51.586Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
91542026-03-11T20:45:51.586Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
91552026-03-11T20:45:51.586Z
91562026-03-11T20:45:51.586Zgoroutine 10759 [sync.Cond.Wait, 59 minutes]:
91572026-03-11T20:45:51.586Zsync.runtime_notifyListWait(0xc0064c5050, 0x19)
91582026-03-11T20:45:51.586Z /work/go/src/runtime/sema.go:569 +0x159
91592026-03-11T20:45:51.586Zsync.(*Cond).Wait(0x50f7930?)
91602026-03-11T20:45:51.586Z /work/go/src/sync/cond.go:70 +0x85
91612026-03-11T20:45:51.586Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd32f0})
91622026-03-11T20:45:51.586Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
91632026-03-11T20:45:51.586Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
91642026-03-11T20:45:51.586Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
91652026-03-11T20:45:51.586Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
91662026-03-11T20:45:51.587Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
91672026-03-11T20:45:51.587Z
91682026-03-11T20:45:51.587Zgoroutine 10758 [sync.Cond.Wait, 59 minutes]:
91692026-03-11T20:45:51.587Zsync.runtime_notifyListWait(0xc0064c5050, 0x20)
91702026-03-11T20:45:51.587Z /work/go/src/runtime/sema.go:569 +0x159
91712026-03-11T20:45:51.587Zsync.(*Cond).Wait(0xc006e3e878?)
91722026-03-11T20:45:51.587Z /work/go/src/sync/cond.go:70 +0x85
91732026-03-11T20:45:51.587Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd32c0})
91742026-03-11T20:45:51.587Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
91752026-03-11T20:45:51.587Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
91762026-03-11T20:45:51.588Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
91772026-03-11T20:45:51.588Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
91782026-03-11T20:45:51.588Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
91792026-03-11T20:45:51.588Z
91802026-03-11T20:45:51.588Zgoroutine 10728 [sync.Cond.Wait, 59 minutes]:
91812026-03-11T20:45:51.588Zsync.runtime_notifyListWait(0xc0064c5050, 0x27)
91822026-03-11T20:45:51.588Z /work/go/src/runtime/sema.go:569 +0x159
91832026-03-11T20:45:51.588Zsync.(*Cond).Wait(0x50f7930?)
91842026-03-11T20:45:51.588Z /work/go/src/sync/cond.go:70 +0x85
91852026-03-11T20:45:51.588Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd2d20})
91862026-03-11T20:45:51.588Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
91872026-03-11T20:45:51.588Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
91882026-03-11T20:45:51.588Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
91892026-03-11T20:45:51.588Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
91902026-03-11T20:45:51.588Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
91912026-03-11T20:45:51.588Z
91922026-03-11T20:45:51.588Zgoroutine 10727 [sync.Cond.Wait, 59 minutes]:
91932026-03-11T20:45:51.588Zsync.runtime_notifyListWait(0xc0064c5050, 0x21)
91942026-03-11T20:45:51.588Z /work/go/src/runtime/sema.go:569 +0x159
91952026-03-11T20:45:51.588Zsync.(*Cond).Wait(0xc006e3e878?)
91962026-03-11T20:45:51.588Z /work/go/src/sync/cond.go:70 +0x85
91972026-03-11T20:45:51.588Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd2cf0})
91982026-03-11T20:45:51.588Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
91992026-03-11T20:45:51.588Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
92002026-03-11T20:45:51.588Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
92012026-03-11T20:45:51.588Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
92022026-03-11T20:45:51.588Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
92032026-03-11T20:45:51.592Z
92042026-03-11T20:45:51.592Zgoroutine 10726 [sync.Cond.Wait, 59 minutes]:
92052026-03-11T20:45:51.592Zsync.runtime_notifyListWait(0xc0064c5050, 0x1e)
92062026-03-11T20:45:51.592Z /work/go/src/runtime/sema.go:569 +0x159
92072026-03-11T20:45:51.592Zsync.(*Cond).Wait(0x50b1300?)
92082026-03-11T20:45:51.592Z /work/go/src/sync/cond.go:70 +0x85
92092026-03-11T20:45:51.592Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd2cc0})
92102026-03-11T20:45:51.592Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
92112026-03-11T20:45:51.592Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
92122026-03-11T20:45:51.592Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
92132026-03-11T20:45:51.593Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
92142026-03-11T20:45:51.593Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
92152026-03-11T20:45:51.593Z
92162026-03-11T20:45:51.593Zgoroutine 10789 [sync.Cond.Wait, 59 minutes]:
92172026-03-11T20:45:51.593Zsync.runtime_notifyListWait(0xc0064c5050, 0x53)
92182026-03-11T20:45:51.593Z /work/go/src/runtime/sema.go:569 +0x159
92192026-03-11T20:45:51.593Zsync.(*Cond).Wait(0x50f7930?)
92202026-03-11T20:45:51.593Z /work/go/src/sync/cond.go:70 +0x85
92212026-03-11T20:45:51.593Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3890})
92222026-03-11T20:45:51.593Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
92232026-03-11T20:45:51.593Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
92242026-03-11T20:45:51.593Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
92252026-03-11T20:45:51.593Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
92262026-03-11T20:45:51.593Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
92272026-03-11T20:45:51.593Z
92282026-03-11T20:45:51.593Zgoroutine 10788 [sync.Cond.Wait, 59 minutes]:
92292026-03-11T20:45:51.593Zsync.runtime_notifyListWait(0xc0064c5050, 0x51)
92302026-03-11T20:45:51.593Z /work/go/src/runtime/sema.go:569 +0x159
92312026-03-11T20:45:51.593Zsync.(*Cond).Wait(0x50f7930?)
92322026-03-11T20:45:51.593Z /work/go/src/sync/cond.go:70 +0x85
92332026-03-11T20:45:51.593Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3860})
92342026-03-11T20:45:51.593Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
92352026-03-11T20:45:51.593Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
92362026-03-11T20:45:51.593Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
92372026-03-11T20:45:51.593Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
92382026-03-11T20:45:51.593Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
92392026-03-11T20:45:51.593Z
92402026-03-11T20:45:51.593Zgoroutine 10787 [sync.Cond.Wait, 59 minutes]:
92412026-03-11T20:45:51.594Zsync.runtime_notifyListWait(0xc0064c5050, 0x50)
92422026-03-11T20:45:51.594Z /work/go/src/runtime/sema.go:569 +0x159
92432026-03-11T20:45:51.594Zsync.(*Cond).Wait(0x50f7930?)
92442026-03-11T20:45:51.594Z /work/go/src/sync/cond.go:70 +0x85
92452026-03-11T20:45:51.594Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3830})
92462026-03-11T20:45:51.594Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
92472026-03-11T20:45:51.594Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
92482026-03-11T20:45:51.594Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
92492026-03-11T20:45:51.594Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
92502026-03-11T20:45:51.594Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
92512026-03-11T20:45:51.594Z
92522026-03-11T20:45:51.594Zgoroutine 10786 [sync.Cond.Wait, 59 minutes]:
92532026-03-11T20:45:51.594Zsync.runtime_notifyListWait(0xc0064c5050, 0x4d)
92542026-03-11T20:45:51.594Z /work/go/src/runtime/sema.go:569 +0x159
92552026-03-11T20:45:51.594Zsync.(*Cond).Wait(0x50f7930?)
92562026-03-11T20:45:51.594Z /work/go/src/sync/cond.go:70 +0x85
92572026-03-11T20:45:51.594Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3800})
92582026-03-11T20:45:51.594Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
92592026-03-11T20:45:51.594Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
92602026-03-11T20:45:51.594Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
92612026-03-11T20:45:51.594Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
92622026-03-11T20:45:51.594Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
92632026-03-11T20:45:51.594Z
92642026-03-11T20:45:51.595Zgoroutine 10729 [sync.Cond.Wait, 59 minutes]:
92652026-03-11T20:45:51.595Zsync.runtime_notifyListWait(0xc0064c5050, 0x2a)
92662026-03-11T20:45:51.595Z /work/go/src/runtime/sema.go:569 +0x159
92672026-03-11T20:45:51.595Zsync.(*Cond).Wait(0xc000f39208?)
92682026-03-11T20:45:51.595Z /work/go/src/sync/cond.go:70 +0x85
92692026-03-11T20:45:51.595Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd2d50})
92702026-03-11T20:45:51.595Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
92712026-03-11T20:45:51.595Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
92722026-03-11T20:45:51.595Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
92732026-03-11T20:45:51.595Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
92742026-03-11T20:45:51.595Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
92752026-03-11T20:45:51.595Z
92762026-03-11T20:45:51.595Zgoroutine 11309 [select, 59 minutes]:
92772026-03-11T20:45:51.595Zgithub.com/cockroachdb/cockroach/pkg/util/tracing.newTracingClientStream.func2()
92782026-03-11T20:45:51.595Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:386 +0x67
92792026-03-11T20:45:51.595Zcreated by github.com/cockroachdb/cockroach/pkg/util/tracing.newTracingClientStream in goroutine 11307
92802026-03-11T20:45:51.595Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:385 +0x125
92812026-03-11T20:45:51.595Z
92822026-03-11T20:45:51.595Zgoroutine 10820 [select, 1 minutes]:
92832026-03-11T20:45:51.595Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).startGossip.func3({0x50b1300, 0xc006fd3ec0})
92842026-03-11T20:45:51.596Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:2152 +0x225
92852026-03-11T20:45:51.596Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
92862026-03-11T20:45:51.596Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
92872026-03-11T20:45:51.596Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
92882026-03-11T20:45:51.596Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
92892026-03-11T20:45:51.596Z
92902026-03-11T20:45:51.596Zgoroutine 10883 [semacquire, 59 minutes]:
92912026-03-11T20:45:51.596Zsync.runtime_Semacquire(0xc00349a198?)
92922026-03-11T20:45:51.596Z /work/go/src/runtime/sema.go:62 +0x25
92932026-03-11T20:45:51.596Zsync.(*WaitGroup).Wait(0x44e7a0?)
92942026-03-11T20:45:51.596Z /work/go/src/sync/waitgroup.go:116 +0x48
92952026-03-11T20:45:51.596Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc007534d00)
92962026-03-11T20:45:51.597Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
92972026-03-11T20:45:51.597Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc007534d00?, {0x50b1338?, 0xc00540c1e0?}})
92982026-03-11T20:45:51.597Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
92992026-03-11T20:45:51.597Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait({0x50b1300?, 0xc0033fb9e0?}, {0xc005831d78, 0x2, 0xc005831c70?})
93002026-03-11T20:45:51.597Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:193 +0xff
93012026-03-11T20:45:51.597Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0xc00426f8c0, {0x50b1300, 0xc0033fb9e0}, 0xc0033fb830)
93022026-03-11T20:45:51.597Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:291 +0x59c
93032026-03-11T20:45:51.603Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).Start.func2({0x50b1300, 0xc0033fb950})
93042026-03-11T20:45:51.603Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:215 +0x2f3
93052026-03-11T20:45:51.603Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
93062026-03-11T20:45:51.603Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
93072026-03-11T20:45:51.603Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10903
93082026-03-11T20:45:51.603Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
93092026-03-11T20:45:51.603Z
93102026-03-11T20:45:51.603Zgoroutine 10906 [select, 59 minutes]:
93112026-03-11T20:45:51.603Zgithub.com/cockroachdb/cockroach/pkg/sql/sqlliveness/slstorage.(*Storage).deleteSessionsLoop(0xc004a02b40, {0x50b1300?, 0xc007b07ec0?})
93122026-03-11T20:45:51.603Z /work/oxidecomputer/cockroach/pkg/sql/sqlliveness/slstorage/slstorage.go:354 +0x165
93132026-03-11T20:45:51.603Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
93142026-03-11T20:45:51.603Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
93152026-03-11T20:45:51.603Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
93162026-03-11T20:45:51.603Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
93172026-03-11T20:45:51.603Z
93182026-03-11T20:45:51.603Zgoroutine 10858 [select, 59 minutes]:
93192026-03-11T20:45:51.603Zgithub.com/cockroachdb/cockroach/pkg/sql/gcjob/gcjobnotifier.(*Notifier).run(0xc001c38060, {0x2a6a?, 0x0?})
93202026-03-11T20:45:51.603Z /work/oxidecomputer/cockroach/pkg/sql/gcjob/gcjobnotifier/notifier.go:145 +0xc8
93212026-03-11T20:45:51.603Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
93222026-03-11T20:45:51.603Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
93232026-03-11T20:45:51.604Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
93242026-03-11T20:45:51.604Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
93252026-03-11T20:45:51.604Z
93262026-03-11T20:45:51.604Zgoroutine 10099 [sync.Cond.Wait]:
93272026-03-11T20:45:51.604Zsync.runtime_notifyListWait(0xc001e22250, 0x379c)
93282026-03-11T20:45:51.604Z /work/go/src/runtime/sema.go:569 +0x159
93292026-03-11T20:45:51.604Zsync.(*Cond).Wait(0x3aac600?)
93302026-03-11T20:45:51.604Z /work/go/src/sync/cond.go:70 +0x85
93312026-03-11T20:45:51.604Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583d70})
93322026-03-11T20:45:51.604Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
93332026-03-11T20:45:51.604Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
93342026-03-11T20:45:51.604Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
93352026-03-11T20:45:51.604Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
93362026-03-11T20:45:51.604Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
93372026-03-11T20:45:51.604Z
93382026-03-11T20:45:51.604Zgoroutine 10908 [semacquire, 59 minutes]:
93392026-03-11T20:45:51.604Zsync.runtime_Semacquire(0xc0043217b8?)
93402026-03-11T20:45:51.604Z /work/go/src/runtime/sema.go:62 +0x25
93412026-03-11T20:45:51.604Zsync.(*WaitGroup).Wait(0x44e7a0?)
93422026-03-11T20:45:51.604Z /work/go/src/sync/waitgroup.go:116 +0x48
93432026-03-11T20:45:51.604Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc0010f48c0)
93442026-03-11T20:45:51.604Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
93452026-03-11T20:45:51.604Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc0010f48c0?, {0x50b1338?, 0xc00218ee60?}})
93462026-03-11T20:45:51.604Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
93472026-03-11T20:45:51.604Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait({0x50b1300?, 0xc0078b6810?}, {0xc00596bd78, 0x2, 0xc00596bc70?})
93482026-03-11T20:45:51.604Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:193 +0xff
93492026-03-11T20:45:51.604Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0xc001131600, {0x50b1300, 0xc0078b6810}, 0xc007b07f50)
93502026-03-11T20:45:51.604Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:291 +0x59c
93512026-03-11T20:45:51.605Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).Start.func2({0x50b1300, 0xc0078b67b0})
93522026-03-11T20:45:51.605Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:215 +0x2f3
93532026-03-11T20:45:51.605Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
93542026-03-11T20:45:51.605Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
93552026-03-11T20:45:51.605Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
93562026-03-11T20:45:51.605Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
93572026-03-11T20:45:51.605Z
93582026-03-11T20:45:51.605Zgoroutine 10907 [select]:
93592026-03-11T20:45:51.605Zgithub.com/cockroachdb/cockroach/pkg/sql/sqlliveness/slinstance.(*Instance).heartbeatLoop(0xc005e087e0, {0x50b1338, 0xc007702190})
93602026-03-11T20:45:51.605Z /work/oxidecomputer/cockroach/pkg/sql/sqlliveness/slinstance/slinstance.go:244 +0x187
93612026-03-11T20:45:51.605Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
93622026-03-11T20:45:51.605Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
93632026-03-11T20:45:51.605Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
93642026-03-11T20:45:51.605Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
93652026-03-11T20:45:51.605Z
93662026-03-11T20:45:51.605Zgoroutine 10144 [select]:
93672026-03-11T20:45:51.605Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run(0xc009020000, {0x50b1300?, 0xc0040cb470?}, 0x10?, 0xc003560ff0, 0xc003f40400)
93682026-03-11T20:45:51.605Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:269 +0x4dc
93692026-03-11T20:45:51.606Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).Start.func1({0x50b1300?, 0xc0040cb470?})
93702026-03-11T20:45:51.606Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:222 +0x36
93712026-03-11T20:45:51.606Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
93722026-03-11T20:45:51.606Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
93732026-03-11T20:45:51.606Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10140
93742026-03-11T20:45:51.606Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
93752026-03-11T20:45:51.606Z
93762026-03-11T20:45:51.606Zgoroutine 10140 [chan receive, 59 minutes]:
93772026-03-11T20:45:51.606Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).rangeFeedWithRangeID(0xc001b49008, 0x3f3de40?, 0xc0036ae360, {0x7fc12cefd140, 0xc0021d8520})
93782026-03-11T20:45:51.606Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:249 +0x76b
93792026-03-11T20:45:51.606Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed(...)
93802026-03-11T20:45:51.606Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:149
93812026-03-11T20:45:51.606Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).RangeFeed(0xc000f39208, 0xc0036ae360, {0x50e5b30, 0xc0021d8520})
93822026-03-11T20:45:51.606Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:3082 +0x16f
93832026-03-11T20:45:51.607Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).RangeFeed(0xc001eb4380, 0xc0036ae360, {0x50e5b30, 0xc0021d8520})
93842026-03-11T20:45:51.607Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/stores.go:216 +0x19a
93852026-03-11T20:45:51.607Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed(0x3f3de40?, 0xc00225b930?, {0x50e5b30, 0xc0021d8520})
93862026-03-11T20:45:51.607Z /work/oxidecomputer/cockroach/pkg/server/node.go:1232 +0x28
93872026-03-11T20:45:51.607Zgithub.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed.func1()
93882026-03-11T20:45:51.607Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:753 +0xa7
93892026-03-11T20:45:51.607Zcreated by github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed in goroutine 10139
93902026-03-11T20:45:51.607Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:750 +0x17c
93912026-03-11T20:45:51.607Z
93922026-03-11T20:45:51.607Zgoroutine 10038 [sync.Cond.Wait]:
93932026-03-11T20:45:51.608Zsync.runtime_notifyListWait(0xc001e22250, 0x374e)
93942026-03-11T20:45:51.608Z /work/go/src/runtime/sema.go:569 +0x159
93952026-03-11T20:45:51.608Zsync.(*Cond).Wait(0x3aac600?)
93962026-03-11T20:45:51.608Z /work/go/src/sync/cond.go:70 +0x85
93972026-03-11T20:45:51.608Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583200})
93982026-03-11T20:45:51.608Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
93992026-03-11T20:45:51.608Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
94002026-03-11T20:45:51.608Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
94012026-03-11T20:45:51.608Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
94022026-03-11T20:45:51.608Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
94032026-03-11T20:45:51.612Z
94042026-03-11T20:45:51.612Zgoroutine 10134 [select, 1 minutes]:
94052026-03-11T20:45:51.612Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/protectedts/ptcache.(*Cache).periodicallyRefreshProtectedtsCache(0xc00274f600, {0x50b1300, 0xc003ebcb40})
94062026-03-11T20:45:51.612Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/protectedts/ptcache/cache.go:175 +0x216
94072026-03-11T20:45:51.612Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
94082026-03-11T20:45:51.612Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
94092026-03-11T20:45:51.613Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
94102026-03-11T20:45:51.613Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
94112026-03-11T20:45:51.613Z
94122026-03-11T20:45:51.613Zgoroutine 10055 [sync.Cond.Wait]:
94132026-03-11T20:45:51.613Zsync.runtime_notifyListWait(0xc001e22250, 0x376c)
94142026-03-11T20:45:51.613Z /work/go/src/runtime/sema.go:569 +0x159
94152026-03-11T20:45:51.613Zsync.(*Cond).Wait(0x3aac600?)
94162026-03-11T20:45:51.613Z /work/go/src/sync/cond.go:70 +0x85
94172026-03-11T20:45:51.613Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583530})
94182026-03-11T20:45:51.613Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
94192026-03-11T20:45:51.613Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
94202026-03-11T20:45:51.613Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
94212026-03-11T20:45:51.613Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
94222026-03-11T20:45:51.613Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
94232026-03-11T20:45:51.613Z
94242026-03-11T20:45:51.613Zgoroutine 9888 [select]:
94252026-03-11T20:45:51.613Zgithub.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1()
94262026-03-11T20:45:51.613Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:71 +0xcb
94272026-03-11T20:45:51.613Zcreated by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 9708
94282026-03-11T20:45:51.613Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:66 +0x58
94292026-03-11T20:45:51.613Z
94302026-03-11T20:45:51.613Zgoroutine 10133 [select]:
94312026-03-11T20:45:51.613Zgithub.com/cockroachdb/cockroach/pkg/server.startSampleEnvironment.func1({0x50b1300, 0xc003ebca80})
94322026-03-11T20:45:51.613Z /work/oxidecomputer/cockroach/pkg/server/env_sampler.go:138 +0x1ad
94332026-03-11T20:45:51.613Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
94342026-03-11T20:45:51.614Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
94352026-03-11T20:45:51.614Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
94362026-03-11T20:45:51.614Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
94372026-03-11T20:45:51.614Z
94382026-03-11T20:45:51.614Zgoroutine 10087 [sync.Cond.Wait]:
94392026-03-11T20:45:51.614Zsync.runtime_notifyListWait(0xc001e22250, 0x3786)
94402026-03-11T20:45:51.614Z /work/go/src/runtime/sema.go:569 +0x159
94412026-03-11T20:45:51.614Zsync.(*Cond).Wait(0x3aac600?)
94422026-03-11T20:45:51.614Z /work/go/src/sync/cond.go:70 +0x85
94432026-03-11T20:45:51.614Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583b30})
94442026-03-11T20:45:51.614Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
94452026-03-11T20:45:51.614Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
94462026-03-11T20:45:51.614Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
94472026-03-11T20:45:51.614Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
94482026-03-11T20:45:51.614Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
94492026-03-11T20:45:51.614Z
94502026-03-11T20:45:51.614Zgoroutine 10052 [sync.Cond.Wait]:
94512026-03-11T20:45:51.614Zsync.runtime_notifyListWait(0xc001e22250, 0x375b)
94522026-03-11T20:45:51.614Z /work/go/src/runtime/sema.go:569 +0x159
94532026-03-11T20:45:51.614Zsync.(*Cond).Wait(0x3aac600?)
94542026-03-11T20:45:51.614Z /work/go/src/sync/cond.go:70 +0x85
94552026-03-11T20:45:51.614Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc0075834a0})
94562026-03-11T20:45:51.614Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
94572026-03-11T20:45:51.614Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
94582026-03-11T20:45:51.614Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
94592026-03-11T20:45:51.614Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
94602026-03-11T20:45:51.615Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
94612026-03-11T20:45:51.615Z
94622026-03-11T20:45:51.615Zgoroutine 10053 [sync.Cond.Wait]:
94632026-03-11T20:45:51.615Zsync.runtime_notifyListWait(0xc001e22250, 0x3753)
94642026-03-11T20:45:51.615Z /work/go/src/runtime/sema.go:569 +0x159
94652026-03-11T20:45:51.615Zsync.(*Cond).Wait(0x3aac600?)
94662026-03-11T20:45:51.615Z /work/go/src/sync/cond.go:70 +0x85
94672026-03-11T20:45:51.615Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc0075834d0})
94682026-03-11T20:45:51.615Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
94692026-03-11T20:45:51.615Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
94702026-03-11T20:45:51.615Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
94712026-03-11T20:45:51.615Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
94722026-03-11T20:45:51.615Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
94732026-03-11T20:45:51.615Z
94742026-03-11T20:45:51.615Zgoroutine 10054 [sync.Cond.Wait]:
94752026-03-11T20:45:51.615Zsync.runtime_notifyListWait(0xc001e22250, 0x3764)
94762026-03-11T20:45:51.615Z /work/go/src/runtime/sema.go:569 +0x159
94772026-03-11T20:45:51.615Zsync.(*Cond).Wait(0x3aac600?)
94782026-03-11T20:45:51.615Z /work/go/src/sync/cond.go:70 +0x85
94792026-03-11T20:45:51.615Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583500})
94802026-03-11T20:45:51.615Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
94812026-03-11T20:45:51.615Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
94822026-03-11T20:45:51.615Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
94832026-03-11T20:45:51.615Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
94842026-03-11T20:45:51.615Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
94852026-03-11T20:45:51.615Z
94862026-03-11T20:45:51.615Zgoroutine 10440 [select, 59 minutes]:
94872026-03-11T20:45:51.616Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1({0x50b1338, 0xc006bc6b40})
94882026-03-11T20:45:51.616Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:116 +0x157
94892026-03-11T20:45:51.616Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.Group.GoCtx.func3()
94902026-03-11T20:45:51.616Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
94912026-03-11T20:45:51.616Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
94922026-03-11T20:45:51.616Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
94932026-03-11T20:45:51.616Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10462
94942026-03-11T20:45:51.616Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
94952026-03-11T20:45:51.616Z
94962026-03-11T20:45:51.616Zgoroutine 9936 [select]:
94972026-03-11T20:45:51.616Zgithub.com/cockroachdb/cockroach/pkg/sql/contention/txnidcache.(*Cache).Start.func1({0xc005178008?, 0xc00296e0c0?})
94982026-03-11T20:45:51.616Z /work/oxidecomputer/cockroach/pkg/sql/contention/txnidcache/txn_id_cache.go:144 +0x9b
94992026-03-11T20:45:51.616Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
95002026-03-11T20:45:51.616Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
95012026-03-11T20:45:51.616Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
95022026-03-11T20:45:51.616Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
95032026-03-11T20:45:51.623Z
95042026-03-11T20:45:51.623Zgoroutine 10105 [sync.Cond.Wait]:
95052026-03-11T20:45:51.623Zsync.runtime_notifyListWait(0xc001e22250, 0x374b)
95062026-03-11T20:45:51.623Z /work/go/src/runtime/sema.go:569 +0x159
95072026-03-11T20:45:51.623Zsync.(*Cond).Wait(0x3aac600?)
95082026-03-11T20:45:51.623Z /work/go/src/sync/cond.go:70 +0x85
95092026-03-11T20:45:51.623Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583e90})
95102026-03-11T20:45:51.623Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
95112026-03-11T20:45:51.624Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
95122026-03-11T20:45:51.624Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
95132026-03-11T20:45:51.624Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
95142026-03-11T20:45:51.624Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
95152026-03-11T20:45:51.624Z
95162026-03-11T20:45:51.624Zgoroutine 10101 [sync.Cond.Wait]:
95172026-03-11T20:45:51.624Zsync.runtime_notifyListWait(0xc001e22250, 0x3750)
95182026-03-11T20:45:51.624Z /work/go/src/runtime/sema.go:569 +0x159
95192026-03-11T20:45:51.624Zsync.(*Cond).Wait(0x3aac600?)
95202026-03-11T20:45:51.624Z /work/go/src/sync/cond.go:70 +0x85
95212026-03-11T20:45:51.624Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583dd0})
95222026-03-11T20:45:51.624Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
95232026-03-11T20:45:51.624Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
95242026-03-11T20:45:51.624Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
95252026-03-11T20:45:51.624Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
95262026-03-11T20:45:51.624Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
95272026-03-11T20:45:51.624Z
95282026-03-11T20:45:51.624Zgoroutine 10100 [sync.Cond.Wait]:
95292026-03-11T20:45:51.624Zsync.runtime_notifyListWait(0xc001e22250, 0x3749)
95302026-03-11T20:45:51.624Z /work/go/src/runtime/sema.go:569 +0x159
95312026-03-11T20:45:51.624Zsync.(*Cond).Wait(0x3aac600?)
95322026-03-11T20:45:51.624Z /work/go/src/sync/cond.go:70 +0x85
95332026-03-11T20:45:51.624Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583da0})
95342026-03-11T20:45:51.624Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
95352026-03-11T20:45:51.624Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
95362026-03-11T20:45:51.625Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
95372026-03-11T20:45:51.625Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
95382026-03-11T20:45:51.625Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
95392026-03-11T20:45:51.625Z
95402026-03-11T20:45:51.625Zgoroutine 10626 [select]:
95412026-03-11T20:45:51.625Zgithub.com/cockroachdb/cockroach/pkg/jobs.(*jobScheduler).runDaemon.func1({0x50b1300?, 0xc006858840?})
95422026-03-11T20:45:51.625Z /work/oxidecomputer/cockroach/pkg/jobs/job_scheduler.go:443 +0x293
95432026-03-11T20:45:51.625Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
95442026-03-11T20:45:51.625Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
95452026-03-11T20:45:51.625Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
95462026-03-11T20:45:51.625Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
95472026-03-11T20:45:51.625Z
95482026-03-11T20:45:51.625Zgoroutine 3501265 [sync.Cond.Wait]:
95492026-03-11T20:45:51.625Zsync.runtime_notifyListWait(0xc0120ea5e8, 0x449)
95502026-03-11T20:45:51.625Z /work/go/src/runtime/sema.go:569 +0x159
95512026-03-11T20:45:51.625Zsync.(*Cond).Wait(0xc0120ea508?)
95522026-03-11T20:45:51.625Z /work/go/src/sync/cond.go:70 +0x85
95532026-03-11T20:45:51.625Zgithub.com/cockroachdb/pebble/record.(*flusherCond).Wait(...)
95542026-03-11T20:45:51.626Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/record/log_writer.go:193
95552026-03-11T20:45:51.626Zgithub.com/cockroachdb/pebble/record.(*LogWriter).flushLoop(0xc0120ea508, {0x3ac7a60?, 0x77577e0?})
95562026-03-11T20:45:51.626Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/record/log_writer.go:396 +0x475
95572026-03-11T20:45:51.626Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fba00?, 0xc003d1a1a0?, 0xc004feefc0?}}, 0xc0020c17c0)
95582026-03-11T20:45:51.626Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
95592026-03-11T20:45:51.626Zgithub.com/cockroachdb/pebble/record.NewLogWriter.func2()
95602026-03-11T20:45:51.626Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/record/log_writer.go:314 +0x56
95612026-03-11T20:45:51.626Zcreated by github.com/cockroachdb/pebble/record.NewLogWriter in goroutine 10013
95622026-03-11T20:45:51.626Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/record/log_writer.go:313 +0x2c9
95632026-03-11T20:45:51.626Z
95642026-03-11T20:45:51.626Zgoroutine 10090 [sync.Cond.Wait]:
95652026-03-11T20:45:51.626Zsync.runtime_notifyListWait(0xc001e22250, 0x3791)
95662026-03-11T20:45:51.626Z /work/go/src/runtime/sema.go:569 +0x159
95672026-03-11T20:45:51.626Zsync.(*Cond).Wait(0x3aac600?)
95682026-03-11T20:45:51.626Z /work/go/src/sync/cond.go:70 +0x85
95692026-03-11T20:45:51.626Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583bc0})
95702026-03-11T20:45:51.626Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
95712026-03-11T20:45:51.626Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
95722026-03-11T20:45:51.626Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
95732026-03-11T20:45:51.626Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
95742026-03-11T20:45:51.627Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
95752026-03-11T20:45:51.627Z
95762026-03-11T20:45:51.628Zgoroutine 10092 [sync.Cond.Wait]:
95772026-03-11T20:45:51.628Zsync.runtime_notifyListWait(0xc001e22250, 0x3795)
95782026-03-11T20:45:51.628Z /work/go/src/runtime/sema.go:569 +0x159
95792026-03-11T20:45:51.628Zsync.(*Cond).Wait(0x3aac600?)
95802026-03-11T20:45:51.628Z /work/go/src/sync/cond.go:70 +0x85
95812026-03-11T20:45:51.628Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583c20})
95822026-03-11T20:45:51.628Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
95832026-03-11T20:45:51.628Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
95842026-03-11T20:45:51.628Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
95852026-03-11T20:45:51.628Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
95862026-03-11T20:45:51.628Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
95872026-03-11T20:45:51.628Z
95882026-03-11T20:45:51.628Zgoroutine 10923 [select]:
95892026-03-11T20:45:51.629Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents(0xc00750cb00, {0x50b1338, 0xc006b57bd0}, 0xc00067d200, 0xc001384d20)
95902026-03-11T20:45:51.629Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:331 +0xc5
95912026-03-11T20:45:51.629Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func2({0x50b1338?, 0xc006b57bd0?})
95922026-03-11T20:45:51.629Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:288 +0x32
95932026-03-11T20:45:51.629Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
95942026-03-11T20:45:51.629Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
95952026-03-11T20:45:51.629Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
95962026-03-11T20:45:51.629Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
95972026-03-11T20:45:51.629Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10915
95982026-03-11T20:45:51.629Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
95992026-03-11T20:45:51.630Z
96002026-03-11T20:45:51.630Zgoroutine 10832 [select]:
96012026-03-11T20:45:51.630Zgithub.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue.func2()
96022026-03-11T20:45:51.630Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:366 +0x85
96032026-03-11T20:45:51.635Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue in goroutine 9708
96042026-03-11T20:45:51.635Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:363 +0x3ed
96052026-03-11T20:45:51.635Z
96062026-03-11T20:45:51.635Zgoroutine 10178 [select]:
96072026-03-11T20:45:51.635Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).outputLoop(0xc001ef70a0, {0x50b1338, 0xc00225bc20})
96082026-03-11T20:45:51.635Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:310 +0x247
96092026-03-11T20:45:51.635Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).runOutputLoop(0xc001ef70a0, {0x50b1300, 0xc0040cb560}, 0xc009328ae0?)
96102026-03-11T20:45:51.636Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:335 +0xba
96112026-03-11T20:45:51.636Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run.func1({0x50b1300?, 0xc0040cb560?})
96122026-03-11T20:45:51.636Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:297 +0x46
96132026-03-11T20:45:51.636Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
96142026-03-11T20:45:51.636Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
96152026-03-11T20:45:51.636Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10144
96162026-03-11T20:45:51.636Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
96172026-03-11T20:45:51.636Z
96182026-03-11T20:45:51.636Zgoroutine 10002 [select, 45 minutes]:
96192026-03-11T20:45:51.636Zgithub.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run(0xc002c1a420, {0x50b1300, 0xc007b275c0})
96202026-03-11T20:45:51.636Z /work/oxidecomputer/cockroach/pkg/internal/client/requestbatcher/batcher.go:475 +0x326
96212026-03-11T20:45:51.636Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
96222026-03-11T20:45:51.636Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
96232026-03-11T20:45:51.636Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
96242026-03-11T20:45:51.636Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
96252026-03-11T20:45:51.636Z
96262026-03-11T20:45:51.636Zgoroutine 9821 [chan receive, 59 minutes]:
96272026-03-11T20:45:51.636Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc0075822d0?})
96282026-03-11T20:45:51.636Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
96292026-03-11T20:45:51.636Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0xc003032588?, 0x1?}}, 0xc00405afa8)
96302026-03-11T20:45:51.636Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
96312026-03-11T20:45:51.636Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
96322026-03-11T20:45:51.636Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
96332026-03-11T20:45:51.636Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
96342026-03-11T20:45:51.636Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
96352026-03-11T20:45:51.636Z
96362026-03-11T20:45:51.637Zgoroutine 10391 [select]:
96372026-03-11T20:45:51.637Zgithub.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigsqlwatcher.(*SQLWatcher).watch(0xc00438e550, {0x50b1300, 0xc0036207b0}, {0x18?, 0x3cb5ec0?, 0x0?}, 0xc006e563f0)
96382026-03-11T20:45:51.637Z /work/oxidecomputer/cockroach/pkg/spanconfig/spanconfigsqlwatcher/sqlwatcher.go:159 +0x5ff
96392026-03-11T20:45:51.637Zgithub.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigsqlwatcher.(*SQLWatcher).WatchForSQLUpdates(0xc0039669f0?, {0x50b1300?, 0xc0036207b0?}, {0xc000d93a40?, 0x71de5a0?, 0xc0?}, 0x40b86e8?)
96402026-03-11T20:45:51.637Z /work/oxidecomputer/cockroach/pkg/spanconfig/spanconfigsqlwatcher/sqlwatcher.go:89 +0x1d
96412026-03-11T20:45:51.637Zgithub.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigreconciler.(*incrementalReconciler).reconcile(...)
96422026-03-11T20:45:51.637Z /work/oxidecomputer/cockroach/pkg/spanconfig/spanconfigreconciler/reconciler.go:453
96432026-03-11T20:45:51.637Zgithub.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigreconciler.(*Reconciler).Reconcile(0xc003966990, {0x50b1300, 0xc0036207b0}, {0x6fc23ac00?, 0xcccccccd?, 0xcc?}, {0x50ba398, 0xc0090ef3e0}, 0xc006168af0)
96442026-03-11T20:45:51.637Z /work/oxidecomputer/cockroach/pkg/spanconfig/spanconfigreconciler/reconciler.go:186 +0x419
96452026-03-11T20:45:51.637Zgithub.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigjob.(*resumer).Resume(0xc004dfc678, {0x50b1300, 0xc0036207b0}, {0x3e5fa00?, 0xc003c82bc8})
96462026-03-11T20:45:51.637Z /work/oxidecomputer/cockroach/pkg/spanconfig/spanconfigjob/job.go:162 +0xb8a
96472026-03-11T20:45:51.637Zgithub.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func2(0x50b1300?, 0x50b1300?, 0xc0324fd9d0, {0x5091770?, 0xc004dfc678?}, {0x50b1300?, 0xc0036207b0?}, {0x3e5fa00?, 0xc003c82bc8?})
96482026-03-11T20:45:51.637Z /work/oxidecomputer/cockroach/pkg/jobs/registry.go:1243 +0xc2
96492026-03-11T20:45:51.638Zgithub.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine(0xc002e9ac88, {0x50b1300, 0xc0031f7d10}, {0x3e5fa00, 0xc003c82bc8}, {0x5091770, 0xc004dfc678}, 0xc000a43380, {0xc007d305a8, 0x7}, ...)
96502026-03-11T20:45:51.638Z /work/oxidecomputer/cockroach/pkg/jobs/registry.go:1244 +0x67f
96512026-03-11T20:45:51.638Zgithub.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob(0xc002e9ac88, {0x50b1300, 0xc002685c50}, {0x5091770, 0xc004dfc678}, 0xc000a43380, {0xc007d305a8, 0x7}, {0xc006141908, 0x17})
96522026-03-11T20:45:51.638Z /work/oxidecomputer/cockroach/pkg/jobs/adopt.go:415 +0x570
96532026-03-11T20:45:51.638Zgithub.com/cockroachdb/cockroach/pkg/jobs.(*Registry).resumeJob.func1({0xc00361a308?, 0x17eee95?})
96542026-03-11T20:45:51.638Z /work/oxidecomputer/cockroach/pkg/jobs/adopt.go:336 +0xee
96552026-03-11T20:45:51.638Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
96562026-03-11T20:45:51.638Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
96572026-03-11T20:45:51.638Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10388
96582026-03-11T20:45:51.638Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
96592026-03-11T20:45:51.638Z
96602026-03-11T20:45:51.638Zgoroutine 9812 [chan receive, 59 minutes]:
96612026-03-11T20:45:51.638Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc0036f8540?})
96622026-03-11T20:45:51.638Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
96632026-03-11T20:45:51.638Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0xc001fee900?, 0xc001fee940?}}, 0xc004055fa8)
96642026-03-11T20:45:51.638Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
96652026-03-11T20:45:51.638Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
96662026-03-11T20:45:51.638Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
96672026-03-11T20:45:51.638Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
96682026-03-11T20:45:51.638Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
96692026-03-11T20:45:51.638Z
96702026-03-11T20:45:51.638Zgoroutine 9784 [chan receive, 59 minutes]:
96712026-03-11T20:45:51.638Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc005572240?})
96722026-03-11T20:45:51.638Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
96732026-03-11T20:45:51.638Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x0?, 0x0?}}, 0xc0000bafa8)
96742026-03-11T20:45:51.638Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
96752026-03-11T20:45:51.639Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
96762026-03-11T20:45:51.639Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
96772026-03-11T20:45:51.639Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
96782026-03-11T20:45:51.639Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
96792026-03-11T20:45:51.639Z
96802026-03-11T20:45:51.639Zgoroutine 9799 [select]:
96812026-03-11T20:45:51.639Zgithub.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
96822026-03-11T20:45:51.639Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:439 +0x1c9
96832026-03-11T20:45:51.639Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 9708
96842026-03-11T20:45:51.639Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:412 +0x4f
96852026-03-11T20:45:51.639Z
96862026-03-11T20:45:51.639Zgoroutine 8998 [semacquire, 59 minutes]:
96872026-03-11T20:45:51.639Zsync.runtime_Semacquire(0xc005cf1890?)
96882026-03-11T20:45:51.640Z /work/go/src/runtime/sema.go:62 +0x25
96892026-03-11T20:45:51.640Zsync.(*WaitGroup).Wait(0x44e7a0?)
96902026-03-11T20:45:51.640Z /work/go/src/sync/waitgroup.go:116 +0x48
96912026-03-11T20:45:51.640Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc006048f00)
96922026-03-11T20:45:51.640Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
96932026-03-11T20:45:51.640Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc006048f00?, {0x50b1338?, 0xc003a62af0?}})
96942026-03-11T20:45:51.640Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
96952026-03-11T20:45:51.640Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait({0x50b1300?, 0xc006c0aa50?}, {0xc005a55d78, 0x2, 0xc005a55c70?})
96962026-03-11T20:45:51.640Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:193 +0xff
96972026-03-11T20:45:51.640Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0xc007576000, {0x50b1300, 0xc006c0aa50}, 0xc006c0a870)
96982026-03-11T20:45:51.640Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:291 +0x59c
96992026-03-11T20:45:51.640Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).Start.func2({0x50b1300, 0xc006c0a9c0})
97002026-03-11T20:45:51.640Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:215 +0x2f3
97012026-03-11T20:45:51.640Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
97022026-03-11T20:45:51.640Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
97032026-03-11T20:45:51.648Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 8997
97042026-03-11T20:45:51.648Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
97052026-03-11T20:45:51.648Z
97062026-03-11T20:45:51.648Zgoroutine 9996 [chan receive, 59 minutes]:
97072026-03-11T20:45:51.648Zgithub.com/cockroachdb/cmux.muxListener.Accept(...)
97082026-03-11T20:45:51.648Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184
97092026-03-11T20:45:51.648Zgoogle.golang.org/grpc.(*Server).Serve(0xc004feefc0, {0x509c4c0, 0xc0059e37b8})
97102026-03-11T20:45:51.648Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:780 +0x469
97112026-03-11T20:45:51.648Zgithub.com/cockroachdb/cockroach/pkg/server.startListenRPCAndSQL.func5.1({0xc0022c2608?, 0x0?})
97122026-03-11T20:45:51.648Z /work/oxidecomputer/cockroach/pkg/server/start_listen.go:134 +0x25
97132026-03-11T20:45:51.648Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
97142026-03-11T20:45:51.648Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
97152026-03-11T20:45:51.648Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
97162026-03-11T20:45:51.648Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
97172026-03-11T20:45:51.648Z
97182026-03-11T20:45:51.648Zgoroutine 10587 [chan receive, 59 minutes]:
97192026-03-11T20:45:51.648Zgithub.com/cockroachdb/cockroach/pkg/util/netutil.MakeServer.func2({0xc006e54008?, 0xc006c743c0?})
97202026-03-11T20:45:51.648Z /work/oxidecomputer/cockroach/pkg/util/netutil/net.go:113 +0x49
97212026-03-11T20:45:51.649Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
97222026-03-11T20:45:51.649Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
97232026-03-11T20:45:51.649Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
97242026-03-11T20:45:51.649Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
97252026-03-11T20:45:51.649Z
97262026-03-11T20:45:51.649Zgoroutine 9842 [select, 59 minutes]:
97272026-03-11T20:45:51.649Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc006b64f08?, 0x42e6fc8?})
97282026-03-11T20:45:51.649Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
97292026-03-11T20:45:51.649Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
97302026-03-11T20:45:51.649Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
97312026-03-11T20:45:51.649Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
97322026-03-11T20:45:51.649Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
97332026-03-11T20:45:51.649Z
97342026-03-11T20:45:51.649Zgoroutine 10589 [IO wait, 59 minutes]:
97352026-03-11T20:45:51.649Zinternal/poll.runtime_pollWait(0x7fc12c3bdd08, 0x72)
97362026-03-11T20:45:51.650Z /work/go/src/runtime/netpoll.go:345 +0x85
97372026-03-11T20:45:51.650Zinternal/poll.(*pollDesc).wait(0xd?, 0x1?, 0x0)
97382026-03-11T20:45:51.650Z /work/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
97392026-03-11T20:45:51.650Zinternal/poll.(*pollDesc).waitRead(...)
97402026-03-11T20:45:51.650Z /work/go/src/internal/poll/fd_poll_runtime.go:89
97412026-03-11T20:45:51.650Zinternal/poll.(*FD).Accept(0xc00667e500)
97422026-03-11T20:45:51.650Z /work/go/src/internal/poll/fd_unix.go:611 +0x2ac
97432026-03-11T20:45:51.650Znet.(*netFD).accept(0xc00667e500)
97442026-03-11T20:45:51.650Z /work/go/src/net/fd_unix.go:172 +0x29
97452026-03-11T20:45:51.650Znet.(*TCPListener).accept(0xc006634f60)
97462026-03-11T20:45:51.650Z /work/go/src/net/tcpsock_posix.go:159 +0x1e
97472026-03-11T20:45:51.650Znet.(*TCPListener).Accept(0xc006634f60)
97482026-03-11T20:45:51.650Z /work/go/src/net/tcpsock.go:327 +0x30
97492026-03-11T20:45:51.650Zgithub.com/cockroachdb/cmux.(*cMux).Serve(0xc0064c4240)
97502026-03-11T20:45:51.650Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:124 +0x97
97512026-03-11T20:45:51.650Zgithub.com/cockroachdb/cockroach/pkg/server.(*httpServer).start.func2({0xc006e54608?, 0xc0064c1500?})
97522026-03-11T20:45:51.650Z /work/oxidecomputer/cockroach/pkg/server/server_http.go:243 +0x27
97532026-03-11T20:45:51.650Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
97542026-03-11T20:45:51.650Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
97552026-03-11T20:45:51.650Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
97562026-03-11T20:45:51.650Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
97572026-03-11T20:45:51.650Z
97582026-03-11T20:45:51.650Zgoroutine 10136 [select]:
97592026-03-11T20:45:51.650Zgithub.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
97602026-03-11T20:45:51.650Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:439 +0x1c9
97612026-03-11T20:45:51.650Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 9708
97622026-03-11T20:45:51.650Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:412 +0x4f
97632026-03-11T20:45:51.650Z
97642026-03-11T20:45:51.651Zgoroutine 10137 [select]:
97652026-03-11T20:45:51.651Zgithub.com/cockroachdb/cockroach/pkg/util/admission.(*StoreGrantCoordinators).SetPebbleMetricsProvider.func1()
97662026-03-11T20:45:51.651Z /work/oxidecomputer/cockroach/pkg/util/admission/granter.go:1261 +0xd5
97672026-03-11T20:45:51.651Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.(*StoreGrantCoordinators).SetPebbleMetricsProvider in goroutine 9708
97682026-03-11T20:45:51.651Z /work/oxidecomputer/cockroach/pkg/util/admission/granter.go:1256 +0x28d
97692026-03-11T20:45:51.651Z
97702026-03-11T20:45:51.651Zgoroutine 9889 [sync.Cond.Wait]:
97712026-03-11T20:45:51.651Zsync.runtime_notifyListWait(0xc0034e39e8, 0x646)
97722026-03-11T20:45:51.651Z /work/go/src/runtime/sema.go:569 +0x159
97732026-03-11T20:45:51.651Zsync.(*Cond).Wait(0xc0034e3908?)
97742026-03-11T20:45:51.651Z /work/go/src/sync/cond.go:70 +0x85
97752026-03-11T20:45:51.651Zgithub.com/cockroachdb/pebble/record.(*flusherCond).Wait(...)
97762026-03-11T20:45:51.651Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/record/log_writer.go:193
97772026-03-11T20:45:51.651Zgithub.com/cockroachdb/pebble/record.(*LogWriter).flushLoop(0xc0034e3908, {0x3ac7a60?, 0x77577e0?})
97782026-03-11T20:45:51.651Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/record/log_writer.go:396 +0x475
97792026-03-11T20:45:51.651Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fba00?, 0xc00431be00?, 0xc0036fea08?}}, 0xc003b23fc0)
97802026-03-11T20:45:51.651Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
97812026-03-11T20:45:51.651Zgithub.com/cockroachdb/pebble/record.NewLogWriter.func2()
97822026-03-11T20:45:51.651Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/record/log_writer.go:314 +0x56
97832026-03-11T20:45:51.651Zcreated by github.com/cockroachdb/pebble/record.NewLogWriter in goroutine 9708
97842026-03-11T20:45:51.651Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/record/log_writer.go:313 +0x2c9
97852026-03-11T20:45:51.651Z
97862026-03-11T20:45:51.651Zgoroutine 10833 [select]:
97872026-03-11T20:45:51.651Zgithub.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
97882026-03-11T20:45:51.651Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:439 +0x1c9
97892026-03-11T20:45:51.651Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 9708
97902026-03-11T20:45:51.652Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:412 +0x4f
97912026-03-11T20:45:51.652Z
97922026-03-11T20:45:51.652Zgoroutine 3679765 [select]:
97932026-03-11T20:45:51.652Zgithub.com/cockroachdb/cockroach/pkg/sql/stats.(*Refresher).Start.func1.1({0x50b1300, 0xc0312bb050})
97942026-03-11T20:45:51.652Z /work/oxidecomputer/cockroach/pkg/sql/stats/automatic_stats.go:410 +0x145
97952026-03-11T20:45:51.652Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
97962026-03-11T20:45:51.652Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
97972026-03-11T20:45:51.652Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9937
97982026-03-11T20:45:51.652Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
97992026-03-11T20:45:51.652Z
98002026-03-11T20:45:51.652Zgoroutine 10135 [select]:
98012026-03-11T20:45:51.652Zgithub.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue.func2()
98022026-03-11T20:45:51.652Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:366 +0x85
98032026-03-11T20:45:51.657Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue in goroutine 9708
98042026-03-11T20:45:51.657Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:363 +0x3ed
98052026-03-11T20:45:51.657Z
98062026-03-11T20:45:51.657Zgoroutine 10107 [sync.Cond.Wait]:
98072026-03-11T20:45:51.657Zsync.runtime_notifyListWait(0xc001e22250, 0x3752)
98082026-03-11T20:45:51.657Z /work/go/src/runtime/sema.go:569 +0x159
98092026-03-11T20:45:51.657Zsync.(*Cond).Wait(0x3aac600?)
98102026-03-11T20:45:51.657Z /work/go/src/sync/cond.go:70 +0x85
98112026-03-11T20:45:51.657Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583ef0})
98122026-03-11T20:45:51.657Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
98132026-03-11T20:45:51.658Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
98142026-03-11T20:45:51.658Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
98152026-03-11T20:45:51.658Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
98162026-03-11T20:45:51.658Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
98172026-03-11T20:45:51.658Z
98182026-03-11T20:45:51.658Zgoroutine 10106 [sync.Cond.Wait]:
98192026-03-11T20:45:51.658Zsync.runtime_notifyListWait(0xc001e22250, 0x374f)
98202026-03-11T20:45:51.658Z /work/go/src/runtime/sema.go:569 +0x159
98212026-03-11T20:45:51.658Zsync.(*Cond).Wait(0x3aac600?)
98222026-03-11T20:45:51.658Z /work/go/src/sync/cond.go:70 +0x85
98232026-03-11T20:45:51.658Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583ec0})
98242026-03-11T20:45:51.658Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
98252026-03-11T20:45:51.658Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
98262026-03-11T20:45:51.658Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
98272026-03-11T20:45:51.658Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
98282026-03-11T20:45:51.658Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
98292026-03-11T20:45:51.659Z
98302026-03-11T20:45:51.659Zgoroutine 9830 [select]:
98312026-03-11T20:45:51.659Zgithub.com/cockroachdb/cockroach/pkg/sql/contention.(*eventStore).startResolver.func1({0x50b1300, 0xc00485e090})
98322026-03-11T20:45:51.659Z /work/oxidecomputer/cockroach/pkg/sql/contention/event_store.go:219 +0x1f4
98332026-03-11T20:45:51.659Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
98342026-03-11T20:45:51.659Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
98352026-03-11T20:45:51.659Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
98362026-03-11T20:45:51.659Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
98372026-03-11T20:45:51.659Z
98382026-03-11T20:45:51.659Zgoroutine 9831 [select, 59 minutes]:
98392026-03-11T20:45:51.660Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc005956608?, 0x1?})
98402026-03-11T20:45:51.660Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
98412026-03-11T20:45:51.660Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
98422026-03-11T20:45:51.660Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
98432026-03-11T20:45:51.660Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
98442026-03-11T20:45:51.660Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
98452026-03-11T20:45:51.660Z
98462026-03-11T20:45:51.660Zgoroutine 10464 [select, 59 minutes]:
98472026-03-11T20:45:51.660Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1({0x50b1338, 0xc0076843c0})
98482026-03-11T20:45:51.660Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:116 +0x157
98492026-03-11T20:45:51.660Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.Group.GoCtx.func3()
98502026-03-11T20:45:51.660Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
98512026-03-11T20:45:51.660Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
98522026-03-11T20:45:51.660Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
98532026-03-11T20:45:51.660Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10458
98542026-03-11T20:45:51.660Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
98552026-03-11T20:45:51.660Z
98562026-03-11T20:45:51.660Zgoroutine 10585 [select, 59 minutes]:
98572026-03-11T20:45:51.660Zdatabase/sql.(*DB).connectionOpener(0xc006e3e1a0, {0x50b1338, 0xc0066216d0})
98582026-03-11T20:45:51.660Z /work/go/src/database/sql/sql.go:1246 +0x87
98592026-03-11T20:45:51.660Zcreated by database/sql.OpenDB in goroutine 9708
98602026-03-11T20:45:51.660Z /work/go/src/database/sql/sql.go:824 +0x14c
98612026-03-11T20:45:51.660Z
98622026-03-11T20:45:51.660Zgoroutine 9892 [semacquire, 59 minutes]:
98632026-03-11T20:45:51.660Zsync.runtime_Semacquire(0xc003358150?)
98642026-03-11T20:45:51.660Z /work/go/src/runtime/sema.go:62 +0x25
98652026-03-11T20:45:51.661Zsync.(*WaitGroup).Wait(0x44e7a0?)
98662026-03-11T20:45:51.661Z /work/go/src/sync/waitgroup.go:116 +0x48
98672026-03-11T20:45:51.661Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc000662040)
98682026-03-11T20:45:51.661Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
98692026-03-11T20:45:51.661Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc000662040?, {0x50b1338?, 0xc003aae050?}})
98702026-03-11T20:45:51.661Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
98712026-03-11T20:45:51.661Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed(0xc003d48008, {0x50b1338?, 0xc000afc0f0?}, {0xc007659a70, 0x1, 0xc0076599b0?}, {0xc003cd4dc0?, 0x4d56000?, 0xc0?}, 0x0, ...)
98722026-03-11T20:45:51.661Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:139 +0x6c5
98732026-03-11T20:45:51.661Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*dbAdapter).RangeFeed(0x4093f67?, {0x50b1338?, 0xc000afc0f0?}, {0xc007659a70?, 0x50b1300?, 0xc007659b00?}, {0x707000000000000?, 0x50b1300?, 0x0?}, 0x0, ...)
98742026-03-11T20:45:51.661Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/db_adapter.go:78 +0x3e
98752026-03-11T20:45:51.661Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func1({0x50b1338?, 0xc000afc0f0?})
98762026-03-11T20:45:51.661Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:285 +0x6c
98772026-03-11T20:45:51.661Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
98782026-03-11T20:45:51.661Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
98792026-03-11T20:45:51.661Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
98802026-03-11T20:45:51.661Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
98812026-03-11T20:45:51.661Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9957
98822026-03-11T20:45:51.661Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
98832026-03-11T20:45:51.661Z
98842026-03-11T20:45:51.661Zgoroutine 10377 [select]:
98852026-03-11T20:45:51.661Zgithub.com/cockroachdb/cockroach/pkg/kv/kvprober.(*Prober).Start.func1.1({0x50b1300, 0xc009328d20})
98862026-03-11T20:45:51.661Z /work/oxidecomputer/cockroach/pkg/kv/kvprober/kvprober.go:293 +0x314
98872026-03-11T20:45:51.661Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
98882026-03-11T20:45:51.661Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
98892026-03-11T20:45:51.661Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
98902026-03-11T20:45:51.661Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
98912026-03-11T20:45:51.661Z
98922026-03-11T20:45:51.662Zgoroutine 10256 [select]:
98932026-03-11T20:45:51.662Zgithub.com/cockroachdb/cockroach/pkg/rpc.respStreamClientAdapter.recvInternal(...)
98942026-03-11T20:45:51.662Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:694
98952026-03-11T20:45:51.662Zgithub.com/cockroachdb/cockroach/pkg/rpc.rangeFeedClientAdapter.Recv(...)
98962026-03-11T20:45:51.662Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:723
98972026-03-11T20:45:51.662Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed.func3()
98982026-03-11T20:45:51.662Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:516 +0x25
98992026-03-11T20:45:51.662Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*stuckRangeFeedCanceler).do(0xc002228730, 0xc001955a50)
99002026-03-11T20:45:51.662Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed_canceler.go:100 +0x1b3
99012026-03-11T20:45:51.662Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed(0xc002d6f408, {0x50b1338?, 0xc006bc6b40?}, {{0xc005269330, 0x1, 0x8}, {0xc005269338, 0x1, 0x8}}, {0x189be113ed70e8ca, ...}, ...)
99022026-03-11T20:45:51.662Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:515 +0xc98
99032026-03-11T20:45:51.666Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).partialRangeFeed(0xc002d6f408, {0x50b1338, 0xc006bc6b40}, 0xc006bc6af0, {{0xc005269330, 0x1, 0x8}, {0xc005269338, 0x1, 0x8}}, ...)
99042026-03-11T20:45:51.666Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:336 +0x745
99052026-03-11T20:45:51.666Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.1({0x50b1338?, 0xc006bc6b40?})
99062026-03-11T20:45:51.666Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:120 +0xa9
99072026-03-11T20:45:51.666Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.Group.GoCtx.2()
99082026-03-11T20:45:51.666Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
99092026-03-11T20:45:51.666Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
99102026-03-11T20:45:51.666Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
99112026-03-11T20:45:51.666Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10440
99122026-03-11T20:45:51.666Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
99132026-03-11T20:45:51.666Z
99142026-03-11T20:45:51.666Zgoroutine 10862 [select, 1 minutes]:
99152026-03-11T20:45:51.667Zgithub.com/cockroachdb/cockroach/pkg/sql/sqlstats/persistedsqlstats.(*jobMonitor).start.func1({0x50b1300, 0xc00067ce70})
99162026-03-11T20:45:51.667Z /work/oxidecomputer/cockroach/pkg/sql/sqlstats/persistedsqlstats/scheduled_job_monitor.go:96 +0x296
99172026-03-11T20:45:51.667Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
99182026-03-11T20:45:51.667Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
99192026-03-11T20:45:51.667Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
99202026-03-11T20:45:51.667Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
99212026-03-11T20:45:51.667Z
99222026-03-11T20:45:51.667Zgoroutine 10863 [select, 59 minutes]:
99232026-03-11T20:45:51.667Zgithub.com/cockroachdb/cockroach/pkg/sql/sqlstats/sslocal.(*SQLStats).Start.func1({0x50b1300, 0xc00067cf90})
99242026-03-11T20:45:51.667Z /work/oxidecomputer/cockroach/pkg/sql/sqlstats/sslocal/sslocal_provider.go:80 +0x306
99252026-03-11T20:45:51.667Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
99262026-03-11T20:45:51.667Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
99272026-03-11T20:45:51.667Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
99282026-03-11T20:45:51.667Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
99292026-03-11T20:45:51.667Z
99302026-03-11T20:45:51.667Zgoroutine 9846 [select, 59 minutes]:
99312026-03-11T20:45:51.667Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc0073bc308?, 0xc00043bf0c?})
99322026-03-11T20:45:51.667Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
99332026-03-11T20:45:51.667Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
99342026-03-11T20:45:51.667Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
99352026-03-11T20:45:51.667Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
99362026-03-11T20:45:51.667Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
99372026-03-11T20:45:51.667Z
99382026-03-11T20:45:51.667Zgoroutine 9826 [sync.Cond.Wait, 59 minutes]:
99392026-03-11T20:45:51.667Zsync.runtime_notifyListWait(0xc0034defe8, 0x0)
99402026-03-11T20:45:51.668Z /work/go/src/runtime/sema.go:569 +0x159
99412026-03-11T20:45:51.668Zsync.(*Cond).Wait(0x8?)
99422026-03-11T20:45:51.668Z /work/go/src/sync/cond.go:70 +0x85
99432026-03-11T20:45:51.668Zgithub.com/cockroachdb/pebble/record.(*flusherCond).Wait(...)
99442026-03-11T20:45:51.668Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/record/log_writer.go:193
99452026-03-11T20:45:51.668Zgithub.com/cockroachdb/pebble/record.(*LogWriter).flushLoop(0xc0034def08, {0x3ac7a60?, 0x77577e0?})
99462026-03-11T20:45:51.668Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/record/log_writer.go:396 +0x475
99472026-03-11T20:45:51.668Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fba00?, 0xc00061d7d0?, 0x44d1fd?}}, 0xc00061d7c0)
99482026-03-11T20:45:51.668Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
99492026-03-11T20:45:51.668Zgithub.com/cockroachdb/pebble/record.NewLogWriter.func2()
99502026-03-11T20:45:51.668Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/record/log_writer.go:314 +0x56
99512026-03-11T20:45:51.668Zcreated by github.com/cockroachdb/pebble/record.NewLogWriter in goroutine 9708
99522026-03-11T20:45:51.668Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/record/log_writer.go:313 +0x2c9
99532026-03-11T20:45:51.668Z
99542026-03-11T20:45:51.668Zgoroutine 10829 [select, 1 minutes]:
99552026-03-11T20:45:51.668Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/reports.(*Reporter).Start.func2({0x50b1300?, 0xc007164360?})
99562026-03-11T20:45:51.668Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/reports/reporter.go:165 +0x305
99572026-03-11T20:45:51.668Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
99582026-03-11T20:45:51.668Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
99592026-03-11T20:45:51.668Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
99602026-03-11T20:45:51.668Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
99612026-03-11T20:45:51.668Z
99622026-03-11T20:45:51.668Zgoroutine 10798 [sync.Cond.Wait, 59 minutes]:
99632026-03-11T20:45:51.668Zsync.runtime_notifyListWait(0xc0064c5050, 0x3d)
99642026-03-11T20:45:51.668Z /work/go/src/runtime/sema.go:569 +0x159
99652026-03-11T20:45:51.668Zsync.(*Cond).Wait(0x0?)
99662026-03-11T20:45:51.668Z /work/go/src/sync/cond.go:70 +0x85
99672026-03-11T20:45:51.669Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3a40})
99682026-03-11T20:45:51.669Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
99692026-03-11T20:45:51.669Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
99702026-03-11T20:45:51.669Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
99712026-03-11T20:45:51.669Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
99722026-03-11T20:45:51.669Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
99732026-03-11T20:45:51.669Z
99742026-03-11T20:45:51.669Zgoroutine 9979 [select]:
99752026-03-11T20:45:51.669Znet.(*pipe).read(0xc00184c080, {0xc001694c00, 0x600, 0x5?})
99762026-03-11T20:45:51.669Z /work/go/src/net/pipe.go:159 +0x155
99772026-03-11T20:45:51.669Znet.(*pipe).Read(0xc0042b41c0?, {0xc001694c00?, 0xc005aa7908?, 0x43d505?})
99782026-03-11T20:45:51.669Z /work/go/src/net/pipe.go:142 +0x1c
99792026-03-11T20:45:51.669Zcrypto/tls.(*atLeastReader).Read(0xc0283c2a38, {0xc001694c00?, 0x0?, 0xc0283c2a38?})
99802026-03-11T20:45:51.669Z /work/go/src/crypto/tls/conn.go:806 +0x3b
99812026-03-11T20:45:51.669Zbytes.(*Buffer).ReadFrom(0xc0042b42b0, {0x5081fe0, 0xc0283c2a38})
99822026-03-11T20:45:51.669Z /work/go/src/bytes/buffer.go:211 +0x98
99832026-03-11T20:45:51.669Zcrypto/tls.(*Conn).readFromUntil(0xc0042b4008, {0x7fc12c3c80b0, 0xc00184c080}, 0xc005aa7990?)
99842026-03-11T20:45:51.669Z /work/go/src/crypto/tls/conn.go:828 +0xde
99852026-03-11T20:45:51.669Zcrypto/tls.(*Conn).readRecordOrCCS(0xc0042b4008, 0x0)
99862026-03-11T20:45:51.669Z /work/go/src/crypto/tls/conn.go:626 +0x3cf
99872026-03-11T20:45:51.670Zcrypto/tls.(*Conn).readRecord(...)
99882026-03-11T20:45:51.670Z /work/go/src/crypto/tls/conn.go:588
99892026-03-11T20:45:51.670Zcrypto/tls.(*Conn).Read(0xc0042b4008, {0xc001cbc000, 0x8000, 0x7fc128bbe268?})
99902026-03-11T20:45:51.670Z /work/go/src/crypto/tls/conn.go:1370 +0x156
99912026-03-11T20:45:51.670Zbufio.(*Reader).Read(0xc001be49c0, {0xc000e43000, 0x9, 0xc0047f2808?})
99922026-03-11T20:45:51.670Z /work/go/src/bufio/bufio.go:241 +0x197
99932026-03-11T20:45:51.670Zio.ReadAtLeast({0x507c320, 0xc001be49c0}, {0xc000e43000, 0x9, 0x9}, 0x9)
99942026-03-11T20:45:51.670Z /work/go/src/io/io.go:335 +0x90
99952026-03-11T20:45:51.670Zio.ReadFull(...)
99962026-03-11T20:45:51.670Z /work/go/src/io/io.go:354
99972026-03-11T20:45:51.670Zgolang.org/x/net/http2.readFrameHeader({0xc000e43000, 0x9, 0xc0280c22e8?}, {0x507c320?, 0xc001be49c0?})
99982026-03-11T20:45:51.670Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x65
99992026-03-11T20:45:51.670Zgolang.org/x/net/http2.(*Framer).ReadFrame(0xc000e42fc0)
100002026-03-11T20:45:51.670Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:501 +0x85
100012026-03-11T20:45:51.670Zgoogle.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc001d0e1a0, 0xc007fdb5c0, 0x42e6fc8)
100022026-03-11T20:45:51.670Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:611 +0x14e
100032026-03-11T20:45:51.675Zgoogle.golang.org/grpc.(*Server).serveStreams(0xc004feefc0, {0x50f1d80, 0xc001d0e1a0})
100042026-03-11T20:45:51.675Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:906 +0x114
100052026-03-11T20:45:51.675Zgoogle.golang.org/grpc.(*Server).handleRawConn.func1()
100062026-03-11T20:45:51.675Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:848 +0x45
100072026-03-11T20:45:51.675Zcreated by google.golang.org/grpc.(*Server).handleRawConn in goroutine 9974
100082026-03-11T20:45:51.675Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:847 +0x15b
100092026-03-11T20:45:51.675Z
100102026-03-11T20:45:51.675Zgoroutine 10827 [select, 1 minutes]:
100112026-03-11T20:45:51.675Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).startGossiping.func1({0x50b1300, 0xc007164270})
100122026-03-11T20:45:51.675Z /work/oxidecomputer/cockroach/pkg/server/node.go:713 +0x1be
100132026-03-11T20:45:51.675Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
100142026-03-11T20:45:51.675Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
100152026-03-11T20:45:51.675Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
100162026-03-11T20:45:51.675Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
100172026-03-11T20:45:51.675Z
100182026-03-11T20:45:51.675Zgoroutine 10828 [select, 59 minutes]:
100192026-03-11T20:45:51.675Zgithub.com/cockroachdb/cockroach/pkg/server.periodicallyPersistHLCUpperBound(0xc00438e690, 0xc001fb1b90, 0xc00675b870, 0x42e4010, 0xc008191da0, 0x0)
100202026-03-11T20:45:51.675Z /work/oxidecomputer/cockroach/pkg/server/clock_monotonicity.go:193 +0x14c
100212026-03-11T20:45:51.675Zgithub.com/cockroachdb/cockroach/pkg/server.(*Server).startPersistingHLCUpperBound.func3({0xc007166008?, 0x0?})
100222026-03-11T20:45:51.675Z /work/oxidecomputer/cockroach/pkg/server/clock_monotonicity.go:268 +0x45
100232026-03-11T20:45:51.675Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
100242026-03-11T20:45:51.675Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
100252026-03-11T20:45:51.676Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
100262026-03-11T20:45:51.676Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
100272026-03-11T20:45:51.676Z
100282026-03-11T20:45:51.676Zgoroutine 9818 [chan receive, 59 minutes]:
100292026-03-11T20:45:51.676Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc005572630?})
100302026-03-11T20:45:51.676Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
100312026-03-11T20:45:51.676Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0xc002280180?, 0xc004054fb0?}}, 0xc004054fa8)
100322026-03-11T20:45:51.676Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
100332026-03-11T20:45:51.676Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x50681f8?)
100342026-03-11T20:45:51.676Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
100352026-03-11T20:45:51.677Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
100362026-03-11T20:45:51.677Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
100372026-03-11T20:45:51.677Z
100382026-03-11T20:45:51.677Zgoroutine 10102 [sync.Cond.Wait]:
100392026-03-11T20:45:51.677Zsync.runtime_notifyListWait(0xc001e22250, 0x3747)
100402026-03-11T20:45:51.677Z /work/go/src/runtime/sema.go:569 +0x159
100412026-03-11T20:45:51.677Zsync.(*Cond).Wait(0x3aac600?)
100422026-03-11T20:45:51.677Z /work/go/src/sync/cond.go:70 +0x85
100432026-03-11T20:45:51.677Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583e00})
100442026-03-11T20:45:51.677Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
100452026-03-11T20:45:51.677Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
100462026-03-11T20:45:51.677Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
100472026-03-11T20:45:51.677Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
100482026-03-11T20:45:51.677Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
100492026-03-11T20:45:51.677Z
100502026-03-11T20:45:51.677Zgoroutine 9935 [select, 59 minutes]:
100512026-03-11T20:45:51.677Zgithub.com/cockroachdb/cockroach/pkg/sql/sqlstats/sslocal.(*SQLStats).Start.func1({0x50b1300, 0xc003456c90})
100522026-03-11T20:45:51.677Z /work/oxidecomputer/cockroach/pkg/sql/sqlstats/sslocal/sslocal_provider.go:80 +0x306
100532026-03-11T20:45:51.677Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
100542026-03-11T20:45:51.677Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
100552026-03-11T20:45:51.677Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
100562026-03-11T20:45:51.677Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
100572026-03-11T20:45:51.677Z
100582026-03-11T20:45:51.677Zgoroutine 10826 [select, 9 minutes]:
100592026-03-11T20:45:51.677Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.KVAdmissionControllerImpl.SetTenantWeightProvider.func1()
100602026-03-11T20:45:51.677Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:3809 +0xd5
100612026-03-11T20:45:51.677Zcreated by github.com/cockroachdb/cockroach/pkg/kv/kvserver.KVAdmissionControllerImpl.SetTenantWeightProvider in goroutine 9708
100622026-03-11T20:45:51.677Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:3802 +0xcc
100632026-03-11T20:45:51.678Z
100642026-03-11T20:45:51.678Zgoroutine 10825 [select]:
100652026-03-11T20:45:51.678Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).startComputePeriodicMetrics.func1({0x50b1300, 0xc007164240})
100662026-03-11T20:45:51.678Z /work/oxidecomputer/cockroach/pkg/server/node.go:748 +0xff
100672026-03-11T20:45:51.678Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
100682026-03-11T20:45:51.678Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
100692026-03-11T20:45:51.678Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
100702026-03-11T20:45:51.678Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
100712026-03-11T20:45:51.678Z
100722026-03-11T20:45:51.678Zgoroutine 9860 [chan receive, 59 minutes]:
100732026-03-11T20:45:51.678Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewContext.func2({0xc001bb4008?, 0x0?})
100742026-03-11T20:45:51.678Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:541 +0x3f
100752026-03-11T20:45:51.678Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
100762026-03-11T20:45:51.678Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
100772026-03-11T20:45:51.678Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
100782026-03-11T20:45:51.678Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
100792026-03-11T20:45:51.678Z
100802026-03-11T20:45:51.678Zgoroutine 9933 [select, 1 minutes]:
100812026-03-11T20:45:51.678Zgithub.com/cockroachdb/cockroach/pkg/sql/sqlstats/persistedsqlstats.(*PersistedSQLStats).startSQLStatsFlushLoop.func1({0x50b1300, 0xc003456c00})
100822026-03-11T20:45:51.678Z /work/oxidecomputer/cockroach/pkg/sql/sqlstats/persistedsqlstats/provider.go:137 +0x236
100832026-03-11T20:45:51.678Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
100842026-03-11T20:45:51.678Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
100852026-03-11T20:45:51.678Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
100862026-03-11T20:45:51.678Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
100872026-03-11T20:45:51.678Z
100882026-03-11T20:45:51.678Zgoroutine 9931 [select, 29 minutes]:
100892026-03-11T20:45:51.679Zgithub.com/cockroachdb/cockroach/pkg/sql.(*TemporaryObjectCleaner).Start.func1({0x50b1300, 0xc003456b40})
100902026-03-11T20:45:51.679Z /work/oxidecomputer/cockroach/pkg/sql/temporary_schema.go:632 +0x22e
100912026-03-11T20:45:51.679Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
100922026-03-11T20:45:51.679Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
100932026-03-11T20:45:51.679Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
100942026-03-11T20:45:51.679Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
100952026-03-11T20:45:51.679Z
100962026-03-11T20:45:51.679Zgoroutine 10095 [sync.Cond.Wait]:
100972026-03-11T20:45:51.679Zsync.runtime_notifyListWait(0xc001e22250, 0x3796)
100982026-03-11T20:45:51.679Z /work/go/src/runtime/sema.go:569 +0x159
100992026-03-11T20:45:51.679Zsync.(*Cond).Wait(0x3aac600?)
101002026-03-11T20:45:51.679Z /work/go/src/sync/cond.go:70 +0x85
101012026-03-11T20:45:51.679Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583cb0})
101022026-03-11T20:45:51.679Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
101032026-03-11T20:45:51.684Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
101042026-03-11T20:45:51.684Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
101052026-03-11T20:45:51.684Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
101062026-03-11T20:45:51.684Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
101072026-03-11T20:45:51.684Z
101082026-03-11T20:45:51.684Zgoroutine 10097 [sync.Cond.Wait]:
101092026-03-11T20:45:51.684Zsync.runtime_notifyListWait(0xc001e22250, 0x3744)
101102026-03-11T20:45:51.684Z /work/go/src/runtime/sema.go:569 +0x159
101112026-03-11T20:45:51.684Zsync.(*Cond).Wait(0x3aac600?)
101122026-03-11T20:45:51.684Z /work/go/src/sync/cond.go:70 +0x85
101132026-03-11T20:45:51.684Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583d10})
101142026-03-11T20:45:51.684Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
101152026-03-11T20:45:51.684Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
101162026-03-11T20:45:51.684Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
101172026-03-11T20:45:51.684Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
101182026-03-11T20:45:51.684Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
101192026-03-11T20:45:51.685Z
101202026-03-11T20:45:51.685Zgoroutine 9934 [select, 1 minutes]:
101212026-03-11T20:45:51.685Zgithub.com/cockroachdb/cockroach/pkg/sql/sqlstats/persistedsqlstats.(*jobMonitor).start.func1({0x50b1300, 0xc003456c30})
101222026-03-11T20:45:51.685Z /work/oxidecomputer/cockroach/pkg/sql/sqlstats/persistedsqlstats/scheduled_job_monitor.go:96 +0x296
101232026-03-11T20:45:51.685Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
101242026-03-11T20:45:51.685Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
101252026-03-11T20:45:51.685Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
101262026-03-11T20:45:51.685Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
101272026-03-11T20:45:51.685Z
101282026-03-11T20:45:51.685Zgoroutine 10096 [sync.Cond.Wait]:
101292026-03-11T20:45:51.685Zsync.runtime_notifyListWait(0xc001e22250, 0x3798)
101302026-03-11T20:45:51.685Z /work/go/src/runtime/sema.go:569 +0x159
101312026-03-11T20:45:51.685Zsync.(*Cond).Wait(0x3aac600?)
101322026-03-11T20:45:51.685Z /work/go/src/sync/cond.go:70 +0x85
101332026-03-11T20:45:51.685Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583ce0})
101342026-03-11T20:45:51.685Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
101352026-03-11T20:45:51.685Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
101362026-03-11T20:45:51.685Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
101372026-03-11T20:45:51.685Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
101382026-03-11T20:45:51.685Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
101392026-03-11T20:45:51.685Z
101402026-03-11T20:45:51.685Zgoroutine 9845 [select, 59 minutes]:
101412026-03-11T20:45:51.685Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc006b65b08?, 0xc0004fba00?})
101422026-03-11T20:45:51.685Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
101432026-03-11T20:45:51.685Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
101442026-03-11T20:45:51.685Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
101452026-03-11T20:45:51.685Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
101462026-03-11T20:45:51.685Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
101472026-03-11T20:45:51.686Z
101482026-03-11T20:45:51.686Zgoroutine 9911 [chan receive, 59 minutes]:
101492026-03-11T20:45:51.686Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc002e5a6c0?})
101502026-03-11T20:45:51.686Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
101512026-03-11T20:45:51.686Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x0?, 0x0?}}, 0xc004245fa8)
101522026-03-11T20:45:51.686Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
101532026-03-11T20:45:51.686Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
101542026-03-11T20:45:51.686Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
101552026-03-11T20:45:51.686Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
101562026-03-11T20:45:51.686Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
101572026-03-11T20:45:51.686Z
101582026-03-11T20:45:51.686Zgoroutine 10627 [select, 59 minutes]:
101592026-03-11T20:45:51.686Zgithub.com/cockroachdb/cockroach/pkg/sql/scheduledlogging.(*CaptureIndexUsageStatsLoggingScheduler).start.func1({0x50b1300, 0xc006858870})
101602026-03-11T20:45:51.686Z /work/oxidecomputer/cockroach/pkg/sql/scheduledlogging/captured_index_usage_stats.go:144 +0xeb
101612026-03-11T20:45:51.686Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
101622026-03-11T20:45:51.686Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
101632026-03-11T20:45:51.686Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
101642026-03-11T20:45:51.686Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
101652026-03-11T20:45:51.686Z
101662026-03-11T20:45:51.686Zgoroutine 9820 [chan receive, 59 minutes]:
101672026-03-11T20:45:51.686Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc0055726c0?})
101682026-03-11T20:45:51.686Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
101692026-03-11T20:45:51.686Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x1000000050b1258?, 0x50f1d80?}}, 0xc0040587a8)
101702026-03-11T20:45:51.686Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
101712026-03-11T20:45:51.686Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
101722026-03-11T20:45:51.687Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
101732026-03-11T20:45:51.687Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
101742026-03-11T20:45:51.687Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
101752026-03-11T20:45:51.687Z
101762026-03-11T20:45:51.687Zgoroutine 10779 [sync.Cond.Wait, 59 minutes]:
101772026-03-11T20:45:51.687Zsync.runtime_notifyListWait(0xc0064c5050, 0x3e)
101782026-03-11T20:45:51.687Z /work/go/src/runtime/sema.go:569 +0x159
101792026-03-11T20:45:51.687Zsync.(*Cond).Wait(0x3a30e80?)
101802026-03-11T20:45:51.687Z /work/go/src/sync/cond.go:70 +0x85
101812026-03-11T20:45:51.687Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd36b0})
101822026-03-11T20:45:51.687Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
101832026-03-11T20:45:51.687Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
101842026-03-11T20:45:51.687Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
101852026-03-11T20:45:51.687Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
101862026-03-11T20:45:51.687Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
101872026-03-11T20:45:51.687Z
101882026-03-11T20:45:51.687Zgoroutine 10780 [sync.Cond.Wait, 59 minutes]:
101892026-03-11T20:45:51.687Zsync.runtime_notifyListWait(0xc0064c5050, 0x3f)
101902026-03-11T20:45:51.687Z /work/go/src/runtime/sema.go:569 +0x159
101912026-03-11T20:45:51.687Zsync.(*Cond).Wait(0x50f7930?)
101922026-03-11T20:45:51.687Z /work/go/src/sync/cond.go:70 +0x85
101932026-03-11T20:45:51.687Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd36e0})
101942026-03-11T20:45:51.687Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
101952026-03-11T20:45:51.687Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
101962026-03-11T20:45:51.687Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
101972026-03-11T20:45:51.687Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
101982026-03-11T20:45:51.687Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
101992026-03-11T20:45:51.688Z
102002026-03-11T20:45:51.688Zgoroutine 10731 [sync.Cond.Wait, 59 minutes]:
102012026-03-11T20:45:51.688Zsync.runtime_notifyListWait(0xc0064c5050, 0x2d)
102022026-03-11T20:45:51.688Z /work/go/src/runtime/sema.go:569 +0x159
102032026-03-11T20:45:51.697Zsync.(*Cond).Wait(0xc006e3e878?)
102042026-03-11T20:45:51.697Z /work/go/src/sync/cond.go:70 +0x85
102052026-03-11T20:45:51.697Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd2db0})
102062026-03-11T20:45:51.697Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
102072026-03-11T20:45:51.697Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
102082026-03-11T20:45:51.697Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
102092026-03-11T20:45:51.697Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
102102026-03-11T20:45:51.697Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
102112026-03-11T20:45:51.697Z
102122026-03-11T20:45:51.697Zgoroutine 10730 [sync.Cond.Wait, 59 minutes]:
102132026-03-11T20:45:51.697Zsync.runtime_notifyListWait(0xc0064c5050, 0x2b)
102142026-03-11T20:45:51.697Z /work/go/src/runtime/sema.go:569 +0x159
102152026-03-11T20:45:51.697Zsync.(*Cond).Wait(0xc000f39208?)
102162026-03-11T20:45:51.697Z /work/go/src/sync/cond.go:70 +0x85
102172026-03-11T20:45:51.697Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd2d80})
102182026-03-11T20:45:51.698Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
102192026-03-11T20:45:51.698Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
102202026-03-11T20:45:51.698Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
102212026-03-11T20:45:51.698Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
102222026-03-11T20:45:51.698Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
102232026-03-11T20:45:51.698Z
102242026-03-11T20:45:51.698Zgoroutine 10963 [select]:
102252026-03-11T20:45:51.698Zgithub.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc000e28800, {0x50b1300, 0xc00426c2d0}, 0xc0078ad3b0, {0xc0041a99e0, 0xf}, 0xc001385500)
102262026-03-11T20:45:51.698Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:1462 +0x27b
102272026-03-11T20:45:51.698Zgithub.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1({0x50b1300, 0xc00426c2d0})
102282026-03-11T20:45:51.698Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:1380 +0x85
102292026-03-11T20:45:51.698Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
102302026-03-11T20:45:51.698Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
102312026-03-11T20:45:51.698Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10928
102322026-03-11T20:45:51.698Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
102332026-03-11T20:45:51.698Z
102342026-03-11T20:45:51.698Zgoroutine 10576 [select]:
102352026-03-11T20:45:51.698Zgoogle.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc006ec68c0, 0x1)
102362026-03-11T20:45:51.698Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:407 +0x113
102372026-03-11T20:45:51.698Zgoogle.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc006029f20)
102382026-03-11T20:45:51.698Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:534 +0x7b
102392026-03-11T20:45:51.698Zgoogle.golang.org/grpc/internal/transport.NewServerTransport.func2()
102402026-03-11T20:45:51.698Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:332 +0xc6
102412026-03-11T20:45:51.698Zcreated by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 10669
102422026-03-11T20:45:51.698Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:329 +0x184c
102432026-03-11T20:45:51.698Z
102442026-03-11T20:45:51.698Zgoroutine 9841 [select, 59 minutes]:
102452026-03-11T20:45:51.699Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc006b64c08?, 0xc0036218c0?})
102462026-03-11T20:45:51.699Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
102472026-03-11T20:45:51.699Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
102482026-03-11T20:45:51.699Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
102492026-03-11T20:45:51.699Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
102502026-03-11T20:45:51.699Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
102512026-03-11T20:45:51.699Z
102522026-03-11T20:45:51.699Zgoroutine 10628 [select]:
102532026-03-11T20:45:51.699Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/closedts/sidetransport.(*Sender).Run.func2({0x50b1300, 0xc006858960})
102542026-03-11T20:45:51.699Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/closedts/sidetransport/sender.go:251 +0x1ab
102552026-03-11T20:45:51.699Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
102562026-03-11T20:45:51.699Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
102572026-03-11T20:45:51.699Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
102582026-03-11T20:45:51.699Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
102592026-03-11T20:45:51.699Z
102602026-03-11T20:45:51.699Zgoroutine 10310 [semacquire, 59 minutes]:
102612026-03-11T20:45:51.699Zsync.runtime_Semacquire(0xc003452090?)
102622026-03-11T20:45:51.699Z /work/go/src/runtime/sema.go:62 +0x25
102632026-03-11T20:45:51.699Zsync.(*WaitGroup).Wait(0x44e7a0?)
102642026-03-11T20:45:51.699Z /work/go/src/sync/waitgroup.go:116 +0x48
102652026-03-11T20:45:51.699Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc00527d640)
102662026-03-11T20:45:51.699Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
102672026-03-11T20:45:51.699Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc00527d640?, {0x50b1338?, 0xc002340910?}})
102682026-03-11T20:45:51.699Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
102692026-03-11T20:45:51.699Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed(0xc002d6f408, {0x50b1338?, 0xc0023407d0?}, {0xc003456f30, 0x1, 0xc003456e10?}, {0xc00356a420?, 0x21d9740?, 0xc0?}, 0x0, ...)
102702026-03-11T20:45:51.699Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:139 +0x6c5
102712026-03-11T20:45:51.699Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*dbAdapter).RangeFeed(0x406e704?, {0x50b1338?, 0xc0023407d0?}, {0xc003456f30?, 0x50b1300?, 0xc003456fc0?}, {0x707000000000000?, 0x50b1300?, 0x0?}, 0x0, ...)
102722026-03-11T20:45:51.700Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/db_adapter.go:78 +0x3e
102732026-03-11T20:45:51.700Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func1({0x50b1338?, 0xc0023407d0?})
102742026-03-11T20:45:51.700Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:285 +0x6c
102752026-03-11T20:45:51.700Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
102762026-03-11T20:45:51.700Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
102772026-03-11T20:45:51.700Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
102782026-03-11T20:45:51.700Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
102792026-03-11T20:45:51.700Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10323
102802026-03-11T20:45:51.700Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
102812026-03-11T20:45:51.700Z
102822026-03-11T20:45:51.701Zgoroutine 9076 [select]:
102832026-03-11T20:45:51.701Zgoogle.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc0078ac0f0, {0xc0078b00e8, 0x5, 0x5})
102842026-03-11T20:45:51.701Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:179 +0x8a
102852026-03-11T20:45:51.701Zgoogle.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc0078ac0f0, {0xc0078b00e8?, 0xc0075ba000?, 0xc005c617d8?})
102862026-03-11T20:45:51.701Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:173 +0x15b
102872026-03-11T20:45:51.701Zgoogle.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0078b6180, {0xc0078b00e8?, 0xc005c61850?, 0xa4f485?})
102882026-03-11T20:45:51.701Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:485 +0x2c
102892026-03-11T20:45:51.701Zio.ReadAtLeast({0x5086380, 0xc0078b6180}, {0xc0078b00e8, 0x5, 0x5}, 0x5)
102902026-03-11T20:45:51.701Z /work/go/src/io/io.go:335 +0x90
102912026-03-11T20:45:51.701Zio.ReadFull(...)
102922026-03-11T20:45:51.701Z /work/go/src/io/io.go:354
102932026-03-11T20:45:51.701Zgoogle.golang.org/grpc/internal/transport.(*Stream).Read(0xc0078e8000, {0xc0078b00e8, 0x5, 0x5})
102942026-03-11T20:45:51.701Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:469 +0x96
102952026-03-11T20:45:51.701Zgoogle.golang.org/grpc.(*parser).recvMsg(0xc0078b00d8, 0x7fffffff)
102962026-03-11T20:45:51.702Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:559 +0x42
102972026-03-11T20:45:51.702Zgoogle.golang.org/grpc.recvAndDecompress(0x36?, 0xc0078e8000, {0x0, 0x0}, 0x7fffffff, 0xc0315adb00, {0x50a8e40, 0x77577e0})
102982026-03-11T20:45:51.702Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:690 +0x5a
102992026-03-11T20:45:51.702Zgoogle.golang.org/grpc.recv(0xc0063ce000?, {0x7fc12cff4640, 0xc0004ce590}, 0x38?, {0x0?, 0x0?}, {0x3efb0a0, 0xc0315a2cd0}, 0xc005c61a48?, 0xc0315adb00, ...)
103002026-03-11T20:45:51.702Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:756 +0x68
103012026-03-11T20:45:51.702Zgoogle.golang.org/grpc.(*serverStream).RecvMsg(0xc0063ce000, {0x3efb0a0, 0xc0315a2cd0})
103022026-03-11T20:45:51.702Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:1583 +0x16a
103032026-03-11T20:45:51.708Zgithub.com/cockroachdb/cockroach/pkg/gossip.(*gossipGossipServer).Recv(0xc006e341d0)
103042026-03-11T20:45:51.708Z /work/oxidecomputer/cockroach/pkg/gossip/gossip.pb.go:679 +0x46
103052026-03-11T20:45:51.708Zgithub.com/cockroachdb/cockroach/pkg/gossip.(*server).gossipReceiver(0xc006d3e300, {0x50b1300, 0xc0078b6450}, 0xc003000000, 0xc0078ae140, 0xc0315adf00)
103062026-03-11T20:45:51.708Z /work/oxidecomputer/cockroach/pkg/gossip/server.go:331 +0xa7f
103072026-03-11T20:45:51.708Zgithub.com/cockroachdb/cockroach/pkg/gossip.(*server).Gossip.func3({0x50b1300?, 0xc0078b6450?})
103082026-03-11T20:45:51.708Z /work/oxidecomputer/cockroach/pkg/gossip/server.go:139 +0x74
103092026-03-11T20:45:51.708Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
103102026-03-11T20:45:51.709Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
103112026-03-11T20:45:51.709Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9075
103122026-03-11T20:45:51.709Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
103132026-03-11T20:45:51.709Z
103142026-03-11T20:45:51.709Zgoroutine 10776 [sync.Cond.Wait, 59 minutes]:
103152026-03-11T20:45:51.709Zsync.runtime_notifyListWait(0xc0064c5050, 0x3a)
103162026-03-11T20:45:51.709Z /work/go/src/runtime/sema.go:569 +0x159
103172026-03-11T20:45:51.709Zsync.(*Cond).Wait(0x50f7930?)
103182026-03-11T20:45:51.709Z /work/go/src/sync/cond.go:70 +0x85
103192026-03-11T20:45:51.709Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3620})
103202026-03-11T20:45:51.709Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
103212026-03-11T20:45:51.709Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
103222026-03-11T20:45:51.709Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
103232026-03-11T20:45:51.709Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
103242026-03-11T20:45:51.709Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
103252026-03-11T20:45:51.709Z
103262026-03-11T20:45:51.709Zgoroutine 10456 [semacquire, 59 minutes]:
103272026-03-11T20:45:51.709Zsync.runtime_Semacquire(0xc006e56630?)
103282026-03-11T20:45:51.709Z /work/go/src/runtime/sema.go:62 +0x25
103292026-03-11T20:45:51.709Zsync.(*WaitGroup).Wait(0x44e7a0?)
103302026-03-11T20:45:51.709Z /work/go/src/sync/waitgroup.go:116 +0x48
103312026-03-11T20:45:51.709Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc005ed4f80)
103322026-03-11T20:45:51.709Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
103332026-03-11T20:45:51.709Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc005ed4f80?, {0x50b1338?, 0xc0076842d0?}})
103342026-03-11T20:45:51.710Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
103352026-03-11T20:45:51.710Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait({0x50b1300?, 0xc0074950b0?}, {0xc004342d78, 0x2, 0x4000000000000000?})
103362026-03-11T20:45:51.710Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:193 +0xff
103372026-03-11T20:45:51.710Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0xc003283340, {0x50b1300, 0xc0074950b0}, 0xc007494b10)
103382026-03-11T20:45:51.710Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:291 +0x59c
103392026-03-11T20:45:51.710Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).Start.func2({0x50b1300, 0xc007494c00})
103402026-03-11T20:45:51.710Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:215 +0x2f3
103412026-03-11T20:45:51.710Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
103422026-03-11T20:45:51.710Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
103432026-03-11T20:45:51.710Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10391
103442026-03-11T20:45:51.710Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
103452026-03-11T20:45:51.710Z
103462026-03-11T20:45:51.710Zgoroutine 9917 [chan receive, 59 minutes]:
103472026-03-11T20:45:51.710Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc0036f8810?})
103482026-03-11T20:45:51.710Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
103492026-03-11T20:45:51.710Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x0?, 0x0?}}, 0xc00465afa8)
103502026-03-11T20:45:51.710Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
103512026-03-11T20:45:51.710Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
103522026-03-11T20:45:51.711Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
103532026-03-11T20:45:51.711Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
103542026-03-11T20:45:51.711Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
103552026-03-11T20:45:51.711Z
103562026-03-11T20:45:51.711Zgoroutine 10471 [semacquire, 59 minutes]:
103572026-03-11T20:45:51.711Zsync.runtime_Semacquire(0xc005cf18f0?)
103582026-03-11T20:45:51.711Z /work/go/src/runtime/sema.go:62 +0x25
103592026-03-11T20:45:51.711Zsync.(*WaitGroup).Wait(0x44e7a0?)
103602026-03-11T20:45:51.711Z /work/go/src/sync/waitgroup.go:116 +0x48
103612026-03-11T20:45:51.711Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc006048f40)
103622026-03-11T20:45:51.711Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
103632026-03-11T20:45:51.711Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc006048f40?, {0x50b1338?, 0xc003a62b90?}})
103642026-03-11T20:45:51.711Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
103652026-03-11T20:45:51.711Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed(0xc002d6f408, {0x50b1338?, 0xc003a62af0?}, {0xc006c0a930, 0x1, 0xc005269338?}, {0x1?, 0x8?, 0x0?}, 0x0, ...)
103662026-03-11T20:45:51.711Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:139 +0x6c5
103672026-03-11T20:45:51.711Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*dbAdapter).RangeFeed(0x160fbbd6b224ee?, {0x50b1338?, 0xc003a62af0?}, {0xc006c0a930?, 0x0?, 0xc0038f8a08?}, {0x50b1300?, 0x74c1a40?, 0xc0?}, 0x0, ...)
103682026-03-11T20:45:51.711Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/db_adapter.go:78 +0x3e
103692026-03-11T20:45:51.711Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func1({0x50b1338?, 0xc003a62af0?})
103702026-03-11T20:45:51.711Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:285 +0x6c
103712026-03-11T20:45:51.711Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
103722026-03-11T20:45:51.711Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
103732026-03-11T20:45:51.711Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
103742026-03-11T20:45:51.711Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
103752026-03-11T20:45:51.711Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 8998
103762026-03-11T20:45:51.711Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
103772026-03-11T20:45:51.711Z
103782026-03-11T20:45:51.711Zgoroutine 10455 [semacquire, 59 minutes]:
103792026-03-11T20:45:51.711Zsync.runtime_Semacquire(0xc006e56600?)
103802026-03-11T20:45:51.711Z /work/go/src/runtime/sema.go:62 +0x25
103812026-03-11T20:45:51.711Zsync.(*WaitGroup).Wait(0x44e7a0?)
103822026-03-11T20:45:51.711Z /work/go/src/sync/waitgroup.go:116 +0x48
103832026-03-11T20:45:51.711Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc005ed4f40)
103842026-03-11T20:45:51.711Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
103852026-03-11T20:45:51.711Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc005ed4f40?, {0x50b1338?, 0xc007684280?}})
103862026-03-11T20:45:51.711Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
103872026-03-11T20:45:51.711Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait({0x50b1300?, 0xc007494f60?}, {0xc009689d78, 0x2, 0x4000000000000000?})
103882026-03-11T20:45:51.711Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:193 +0xff
103892026-03-11T20:45:51.711Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0xc0032831e0, {0x50b1300, 0xc007494f60}, 0xc0074949c0)
103902026-03-11T20:45:51.712Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:291 +0x59c
103912026-03-11T20:45:51.712Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).Start.func2({0x50b1300, 0xc007494ab0})
103922026-03-11T20:45:51.712Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:215 +0x2f3
103932026-03-11T20:45:51.712Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
103942026-03-11T20:45:51.712Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
103952026-03-11T20:45:51.712Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10391
103962026-03-11T20:45:51.712Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
103972026-03-11T20:45:51.712Z
103982026-03-11T20:45:51.712Zgoroutine 9829 [select]:
103992026-03-11T20:45:51.712Zgithub.com/cockroachdb/cockroach/pkg/sql/contention.(*eventStore).startEventIntake.func3({0xc005956008?, 0x0?})
104002026-03-11T20:45:51.712Z /work/oxidecomputer/cockroach/pkg/sql/contention/event_store.go:188 +0xa5
104012026-03-11T20:45:51.713Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
104022026-03-11T20:45:51.713Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
104032026-03-11T20:45:51.719Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
104042026-03-11T20:45:51.719Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
104052026-03-11T20:45:51.719Z
104062026-03-11T20:45:51.719Zgoroutine 9913 [chan receive, 59 minutes]:
104072026-03-11T20:45:51.719Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc002e5a090?})
104082026-03-11T20:45:51.719Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
104092026-03-11T20:45:51.719Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x0?, 0x0?}}, 0xc004658fa8)
104102026-03-11T20:45:51.719Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
104112026-03-11T20:45:51.719Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
104122026-03-11T20:45:51.719Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
104132026-03-11T20:45:51.719Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
104142026-03-11T20:45:51.719Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
104152026-03-11T20:45:51.719Z
104162026-03-11T20:45:51.719Zgoroutine 10458 [semacquire, 59 minutes]:
104172026-03-11T20:45:51.719Zsync.runtime_Semacquire(0xc006e56690?)
104182026-03-11T20:45:51.719Z /work/go/src/runtime/sema.go:62 +0x25
104192026-03-11T20:45:51.719Zsync.(*WaitGroup).Wait(0x44e7a0?)
104202026-03-11T20:45:51.719Z /work/go/src/sync/waitgroup.go:116 +0x48
104212026-03-11T20:45:51.719Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc005ed4fc0)
104222026-03-11T20:45:51.719Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
104232026-03-11T20:45:51.719Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc005ed4fc0?, {0x50b1338?, 0xc0076843c0?}})
104242026-03-11T20:45:51.719Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
104252026-03-11T20:45:51.719Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed(0xc002d6f408, {0x50b1338?, 0xc007684230?}, {0xc007494cf0, 0x1, 0xc007494c60?}, {0xc0032834a0?, 0xd8ada0?, 0xc0?}, 0x1, ...)
104262026-03-11T20:45:51.720Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:139 +0x6c5
104272026-03-11T20:45:51.720Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*dbAdapter).RangeFeed(0xc007494e10?, {0x50b1338?, 0xc007684230?}, {0xc007494cf0?, 0x50b1300?, 0xc007494d50?}, {0x707000000000000?, 0x50b1300?, 0x0?}, 0x1, ...)
104282026-03-11T20:45:51.720Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/db_adapter.go:78 +0x3e
104292026-03-11T20:45:51.720Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func1({0x50b1338?, 0xc007684230?})
104302026-03-11T20:45:51.720Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:285 +0x6c
104312026-03-11T20:45:51.720Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
104322026-03-11T20:45:51.720Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
104332026-03-11T20:45:51.720Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
104342026-03-11T20:45:51.720Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
104352026-03-11T20:45:51.720Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10457
104362026-03-11T20:45:51.720Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
104372026-03-11T20:45:51.720Z
104382026-03-11T20:45:51.720Zgoroutine 10325 [select]:
104392026-03-11T20:45:51.720Zgithub.com/cockroachdb/cockroach/pkg/sql/catalog/lease.(*Manager).PeriodicallyRefreshSomeLeases.func1({0x50b1300, 0xc0034570e0})
104402026-03-11T20:45:51.720Z /work/oxidecomputer/cockroach/pkg/sql/catalog/lease/lease.go:1151 +0x2df
104412026-03-11T20:45:51.720Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
104422026-03-11T20:45:51.720Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
104432026-03-11T20:45:51.720Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
104442026-03-11T20:45:51.720Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
104452026-03-11T20:45:51.720Z
104462026-03-11T20:45:51.720Zgoroutine 10837 [select]:
104472026-03-11T20:45:51.720Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.(*Watcher).Run(0xc003a03290, {0x50b1338, 0xc001f488c0})
104482026-03-11T20:45:51.720Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:319 +0x955
104492026-03-11T20:45:51.720Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.Start.func1({0x50b1300?, 0xc003c30660?})
104502026-03-11T20:45:51.720Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:189 +0x2d1
104512026-03-11T20:45:51.720Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
104522026-03-11T20:45:51.720Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
104532026-03-11T20:45:51.720Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
104542026-03-11T20:45:51.720Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
104552026-03-11T20:45:51.720Z
104562026-03-11T20:45:51.720Zgoroutine 9817 [chan receive, 59 minutes]:
104572026-03-11T20:45:51.720Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc0055725d0?})
104582026-03-11T20:45:51.720Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
104592026-03-11T20:45:51.720Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x1024180?, 0xc001147130?}}, 0xc004059fa8)
104602026-03-11T20:45:51.720Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
104612026-03-11T20:45:51.721Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
104622026-03-11T20:45:51.721Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
104632026-03-11T20:45:51.721Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
104642026-03-11T20:45:51.721Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
104652026-03-11T20:45:51.721Z
104662026-03-11T20:45:51.721Zgoroutine 10459 [select]:
104672026-03-11T20:45:51.721Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents(0xc0032834a0, {0x50b1338, 0xc007684230}, 0xc007494c60, 0xc002070c60)
104682026-03-11T20:45:51.721Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:331 +0xc5
104692026-03-11T20:45:51.721Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func2({0x50b1338?, 0xc007684230?})
104702026-03-11T20:45:51.721Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:288 +0x32
104712026-03-11T20:45:51.721Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
104722026-03-11T20:45:51.721Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
104732026-03-11T20:45:51.721Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
104742026-03-11T20:45:51.721Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
104752026-03-11T20:45:51.721Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10457
104762026-03-11T20:45:51.721Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
104772026-03-11T20:45:51.721Z
104782026-03-11T20:45:51.721Zgoroutine 10590 [chan receive, 59 minutes]:
104792026-03-11T20:45:51.721Zgithub.com/cockroachdb/cmux.muxListener.Accept(...)
104802026-03-11T20:45:51.721Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184
104812026-03-11T20:45:51.721Znet/http.(*Server).Serve(0xc006c6cd20, {0x509c4c0, 0xc006e57cb0})
104822026-03-11T20:45:51.721Z /work/go/src/net/http/server.go:3260 +0x33e
104832026-03-11T20:45:51.721Zgithub.com/cockroachdb/cockroach/pkg/server.(*httpServer).start.func3({0xc006e54908?, 0x0?})
104842026-03-11T20:45:51.721Z /work/oxidecomputer/cockroach/pkg/server/server_http.go:264 +0x126
104852026-03-11T20:45:51.721Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
104862026-03-11T20:45:51.721Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
104872026-03-11T20:45:51.721Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
104882026-03-11T20:45:51.721Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
104892026-03-11T20:45:51.721Z
104902026-03-11T20:45:51.721Zgoroutine 10473 [select, 59 minutes]:
104912026-03-11T20:45:51.721Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1({0x50b1338, 0xc003a62b90})
104922026-03-11T20:45:51.721Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:116 +0x157
104932026-03-11T20:45:51.721Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.Group.GoCtx.func3()
104942026-03-11T20:45:51.721Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
104952026-03-11T20:45:51.721Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
104962026-03-11T20:45:51.721Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
104972026-03-11T20:45:51.721Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10471
104982026-03-11T20:45:51.721Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
104992026-03-11T20:45:51.722Z
105002026-03-11T20:45:51.722Zgoroutine 10048 [sync.Cond.Wait]:
105012026-03-11T20:45:51.722Zsync.runtime_notifyListWait(0xc001e22250, 0x378d)
105022026-03-11T20:45:51.722Z /work/go/src/runtime/sema.go:569 +0x159
105032026-03-11T20:45:51.726Zsync.(*Cond).Wait(0x3aac600?)
105042026-03-11T20:45:51.726Z /work/go/src/sync/cond.go:70 +0x85
105052026-03-11T20:45:51.726Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc0075833e0})
105062026-03-11T20:45:51.726Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
105072026-03-11T20:45:51.726Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
105082026-03-11T20:45:51.727Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
105092026-03-11T20:45:51.727Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
105102026-03-11T20:45:51.727Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
105112026-03-11T20:45:51.727Z
105122026-03-11T20:45:51.727Zgoroutine 9004 [chan receive, 59 minutes]:
105132026-03-11T20:45:51.727Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).rangeFeedWithRangeID(0xc001d48008, 0x3f3de40?, 0xc000f157a0, {0x7fc12cefd140, 0xc00271c900})
105142026-03-11T20:45:51.727Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:249 +0x76b
105152026-03-11T20:45:51.727Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed(...)
105162026-03-11T20:45:51.727Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:149
105172026-03-11T20:45:51.727Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).RangeFeed(0xc000f39208, 0xc000f157a0, {0x50e5b30, 0xc00271c900})
105182026-03-11T20:45:51.727Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:3082 +0x16f
105192026-03-11T20:45:51.727Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).RangeFeed(0xc001eb4380, 0xc000f157a0, {0x50e5b30, 0xc00271c900})
105202026-03-11T20:45:51.727Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/stores.go:216 +0x19a
105212026-03-11T20:45:51.727Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed(0x3f3de40?, 0xc000ebd9d0?, {0x50e5b30, 0xc00271c900})
105222026-03-11T20:45:51.727Z /work/oxidecomputer/cockroach/pkg/server/node.go:1232 +0x28
105232026-03-11T20:45:51.727Zgithub.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed.func1()
105242026-03-11T20:45:51.727Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:753 +0xa7
105252026-03-11T20:45:51.727Zcreated by github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed in goroutine 9003
105262026-03-11T20:45:51.727Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:750 +0x17c
105272026-03-11T20:45:51.727Z
105282026-03-11T20:45:51.727Zgoroutine 10830 [select]:
105292026-03-11T20:45:51.727Zgithub.com/cockroachdb/cockroach/pkg/server.startSampleEnvironment.func1({0x50b1300, 0xc007164390})
105302026-03-11T20:45:51.727Z /work/oxidecomputer/cockroach/pkg/server/env_sampler.go:138 +0x1ad
105312026-03-11T20:45:51.727Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
105322026-03-11T20:45:51.727Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
105332026-03-11T20:45:51.728Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
105342026-03-11T20:45:51.728Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
105352026-03-11T20:45:51.728Z
105362026-03-11T20:45:51.728Zgoroutine 10831 [select, 1 minutes]:
105372026-03-11T20:45:51.728Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/protectedts/ptcache.(*Cache).periodicallyRefreshProtectedtsCache(0xc004a47970, {0x50b1300, 0xc0071643c0})
105382026-03-11T20:45:51.728Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/protectedts/ptcache/cache.go:175 +0x216
105392026-03-11T20:45:51.728Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
105402026-03-11T20:45:51.728Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
105412026-03-11T20:45:51.728Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
105422026-03-11T20:45:51.728Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
105432026-03-11T20:45:51.728Z
105442026-03-11T20:45:51.728Zgoroutine 9232 [select]:
105452026-03-11T20:45:51.728Zgoogle.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc007514000)
105462026-03-11T20:45:51.728Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1572 +0x137
105472026-03-11T20:45:51.728Zcreated by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 10707
105482026-03-11T20:45:51.728Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:360 +0x180b
105492026-03-11T20:45:51.728Z
105502026-03-11T20:45:51.728Zgoroutine 10115 [sync.Cond.Wait]:
105512026-03-11T20:45:51.728Zsync.runtime_notifyListWait(0xc001e22250, 0x375f)
105522026-03-11T20:45:51.728Z /work/go/src/runtime/sema.go:569 +0x159
105532026-03-11T20:45:51.728Zsync.(*Cond).Wait(0x3aac600?)
105542026-03-11T20:45:51.728Z /work/go/src/sync/cond.go:70 +0x85
105552026-03-11T20:45:51.728Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc003ebc180})
105562026-03-11T20:45:51.728Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
105572026-03-11T20:45:51.729Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
105582026-03-11T20:45:51.729Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
105592026-03-11T20:45:51.729Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
105602026-03-11T20:45:51.729Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
105612026-03-11T20:45:51.729Z
105622026-03-11T20:45:51.729Zgoroutine 9906 [chan receive, 59 minutes]:
105632026-03-11T20:45:51.729Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc002e5a2a0?})
105642026-03-11T20:45:51.729Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
105652026-03-11T20:45:51.729Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x0?, 0x0?}}, 0xc0042437a8)
105662026-03-11T20:45:51.729Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
105672026-03-11T20:45:51.729Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
105682026-03-11T20:45:51.729Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
105692026-03-11T20:45:51.729Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
105702026-03-11T20:45:51.729Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
105712026-03-11T20:45:51.729Z
105722026-03-11T20:45:51.729Zgoroutine 10139 [select]:
105732026-03-11T20:45:51.729Zgithub.com/cockroachdb/cockroach/pkg/rpc.respStreamClientAdapter.recvInternal(...)
105742026-03-11T20:45:51.729Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:694
105752026-03-11T20:45:51.729Zgithub.com/cockroachdb/cockroach/pkg/rpc.rangeFeedClientAdapter.Recv(...)
105762026-03-11T20:45:51.729Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:723
105772026-03-11T20:45:51.729Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed.func3()
105782026-03-11T20:45:51.730Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:516 +0x25
105792026-03-11T20:45:51.730Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*stuckRangeFeedCanceler).do(0xc00225b860, 0xc0032b3a50)
105802026-03-11T20:45:51.730Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed_canceler.go:100 +0x1b3
105812026-03-11T20:45:51.730Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed(0xc002d6f408, {0x50b1338?, 0xc005d142d0?}, {{0xc0088180c0, 0x1, 0x8}, {0xc0088180c8, 0x1, 0x8}}, {0x189be113e84e8e2e, ...}, ...)
105822026-03-11T20:45:51.730Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:515 +0xc98
105832026-03-11T20:45:51.730Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).partialRangeFeed(0xc002d6f408, {0x50b1338, 0xc005d142d0}, 0xc005d14280, {{0xc0088180c0, 0x1, 0x8}, {0xc0088180c8, 0x1, 0x8}}, ...)
105842026-03-11T20:45:51.730Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:336 +0x745
105852026-03-11T20:45:51.730Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.1({0x50b1338?, 0xc005d142d0?})
105862026-03-11T20:45:51.730Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:120 +0xa9
105872026-03-11T20:45:51.730Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.Group.GoCtx.2()
105882026-03-11T20:45:51.730Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
105892026-03-11T20:45:51.730Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
105902026-03-11T20:45:51.730Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
105912026-03-11T20:45:51.730Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9711
105922026-03-11T20:45:51.730Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
105932026-03-11T20:45:51.730Z
105942026-03-11T20:45:51.730Zgoroutine 11124 [select, 59 minutes]:
105952026-03-11T20:45:51.730Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1({0x50b1338, 0xc0013d28c0})
105962026-03-11T20:45:51.730Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:116 +0x157
105972026-03-11T20:45:51.730Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.Group.GoCtx.func3()
105982026-03-11T20:45:51.730Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
105992026-03-11T20:45:51.730Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
106002026-03-11T20:45:51.730Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
106012026-03-11T20:45:51.730Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 11122
106022026-03-11T20:45:51.730Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
106032026-03-11T20:45:51.737Z
106042026-03-11T20:45:51.737Zgoroutine 10080 [sync.Cond.Wait]:
106052026-03-11T20:45:51.737Zsync.runtime_notifyListWait(0xc001e22250, 0x3792)
106062026-03-11T20:45:51.737Z /work/go/src/runtime/sema.go:569 +0x159
106072026-03-11T20:45:51.737Zsync.(*Cond).Wait(0x3aac600?)
106082026-03-11T20:45:51.737Z /work/go/src/sync/cond.go:70 +0x85
106092026-03-11T20:45:51.737Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc0075839e0})
106102026-03-11T20:45:51.737Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
106112026-03-11T20:45:51.738Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
106122026-03-11T20:45:51.738Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
106132026-03-11T20:45:51.738Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
106142026-03-11T20:45:51.738Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
106152026-03-11T20:45:51.738Z
106162026-03-11T20:45:51.738Zgoroutine 10822 [select, 59 minutes]:
106172026-03-11T20:45:51.738Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).startLeaseRenewer.func1({0x50b1300, 0xc006fd3f20})
106182026-03-11T20:45:51.738Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:2256 +0x2ac
106192026-03-11T20:45:51.738Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
106202026-03-11T20:45:51.738Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
106212026-03-11T20:45:51.738Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
106222026-03-11T20:45:51.738Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
106232026-03-11T20:45:51.738Z
106242026-03-11T20:45:51.738Zgoroutine 10877 [select, 59 minutes]:
106252026-03-11T20:45:51.738Zdatabase/sql.(*DB).connectionOpener(0xc0070ba8f0, {0x50b1338, 0xc004e01180})
106262026-03-11T20:45:51.738Z /work/go/src/database/sql/sql.go:1246 +0x87
106272026-03-11T20:45:51.738Zcreated by database/sql.OpenDB in goroutine 9708
106282026-03-11T20:45:51.738Z /work/go/src/database/sql/sql.go:824 +0x14c
106292026-03-11T20:45:51.738Z
106302026-03-11T20:45:51.738Zgoroutine 9919 [chan receive, 59 minutes]:
106312026-03-11T20:45:51.738Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc002e5ab40?})
106322026-03-11T20:45:51.738Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
106332026-03-11T20:45:51.738Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x0?, 0x0?}}, 0xc00465bfa8)
106342026-03-11T20:45:51.738Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
106352026-03-11T20:45:51.738Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
106362026-03-11T20:45:51.738Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
106372026-03-11T20:45:51.738Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
106382026-03-11T20:45:51.738Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
106392026-03-11T20:45:51.738Z
106402026-03-11T20:45:51.738Zgoroutine 9003 [select]:
106412026-03-11T20:45:51.738Zgithub.com/cockroachdb/cockroach/pkg/rpc.respStreamClientAdapter.recvInternal(...)
106422026-03-11T20:45:51.739Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:694
106432026-03-11T20:45:51.739Zgithub.com/cockroachdb/cockroach/pkg/rpc.rangeFeedClientAdapter.Recv(...)
106442026-03-11T20:45:51.739Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:723
106452026-03-11T20:45:51.739Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed.func3()
106462026-03-11T20:45:51.739Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:516 +0x25
106472026-03-11T20:45:51.739Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*stuckRangeFeedCanceler).do(0xc000ebd900, 0xc003d85a50)
106482026-03-11T20:45:51.739Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed_canceler.go:100 +0x1b3
106492026-03-11T20:45:51.739Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed(0xc002d6f408, {0x50b1338?, 0xc0076843c0?}, {{0xc0052693c0, 0x1, 0x8}, {0xc0052693c8, 0x1, 0x8}}, {0x189be113ed70e8ca, ...}, ...)
106502026-03-11T20:45:51.739Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:515 +0xc98
106512026-03-11T20:45:51.739Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).partialRangeFeed(0xc002d6f408, {0x50b1338, 0xc0076843c0}, 0xc007684370, {{0xc0052693c0, 0x1, 0x8}, {0xc0052693c8, 0x1, 0x8}}, ...)
106522026-03-11T20:45:51.739Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:336 +0x745
106532026-03-11T20:45:51.739Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.1({0x50b1338?, 0xc0076843c0?})
106542026-03-11T20:45:51.739Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:120 +0xa9
106552026-03-11T20:45:51.739Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.Group.GoCtx.2()
106562026-03-11T20:45:51.739Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
106572026-03-11T20:45:51.739Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
106582026-03-11T20:45:51.739Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
106592026-03-11T20:45:51.739Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10464
106602026-03-11T20:45:51.739Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
106612026-03-11T20:45:51.739Z
106622026-03-11T20:45:51.739Zgoroutine 9918 [chan receive, 59 minutes]:
106632026-03-11T20:45:51.739Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc0036f86c0?})
106642026-03-11T20:45:51.739Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
106652026-03-11T20:45:51.739Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x0?, 0x0?}}, 0xc00465b7a8)
106662026-03-11T20:45:51.739Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
106672026-03-11T20:45:51.739Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
106682026-03-11T20:45:51.739Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
106692026-03-11T20:45:51.739Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
106702026-03-11T20:45:51.739Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
106712026-03-11T20:45:51.739Z
106722026-03-11T20:45:51.739Zgoroutine 11129 [select]:
106732026-03-11T20:45:51.739Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).outputLoop(0xc004d86c40, {0x50b1338, 0xc0013d3040})
106742026-03-11T20:45:51.739Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:310 +0x247
106752026-03-11T20:45:51.739Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).runOutputLoop(0xc004d86c40, {0x50b1300, 0xc006f64c60}, 0x1b?)
106762026-03-11T20:45:51.739Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:335 +0xba
106772026-03-11T20:45:51.740Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run.func1({0x50b1300?, 0xc006f64c60?})
106782026-03-11T20:45:51.740Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:297 +0x46
106792026-03-11T20:45:51.740Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
106802026-03-11T20:45:51.740Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
106812026-03-11T20:45:51.740Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10334
106822026-03-11T20:45:51.740Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
106832026-03-11T20:45:51.740Z
106842026-03-11T20:45:51.740Zgoroutine 10792 [sync.Cond.Wait, 59 minutes]:
106852026-03-11T20:45:51.740Zsync.runtime_notifyListWait(0xc0064c5050, 0x5d)
106862026-03-11T20:45:51.740Z /work/go/src/runtime/sema.go:569 +0x159
106872026-03-11T20:45:51.740Zsync.(*Cond).Wait(0xc006e3e878?)
106882026-03-11T20:45:51.740Z /work/go/src/sync/cond.go:70 +0x85
106892026-03-11T20:45:51.740Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3920})
106902026-03-11T20:45:51.740Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
106912026-03-11T20:45:51.740Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
106922026-03-11T20:45:51.740Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
106932026-03-11T20:45:51.740Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
106942026-03-11T20:45:51.740Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
106952026-03-11T20:45:51.740Z
106962026-03-11T20:45:51.740Zgoroutine 9914 [chan receive, 59 minutes]:
106972026-03-11T20:45:51.740Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc0036f8750?})
106982026-03-11T20:45:51.740Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
106992026-03-11T20:45:51.740Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x0?, 0x0?}}, 0xc0046597a8)
107002026-03-11T20:45:51.740Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
107012026-03-11T20:45:51.740Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
107022026-03-11T20:45:51.740Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
107032026-03-11T20:45:51.745Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
107042026-03-11T20:45:51.745Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
107052026-03-11T20:45:51.745Z
107062026-03-11T20:45:51.745Zgoroutine 11126 [select]:
107072026-03-11T20:45:51.745Zgoogle.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc0013d2c30, {0xc005dbe538, 0x5, 0x5})
107082026-03-11T20:45:51.745Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:191 +0x95
107092026-03-11T20:45:51.745Zgoogle.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc0013d2c30, {0xc005dbe538?, 0xc004400060?, 0xc007691048?})
107102026-03-11T20:45:51.746Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:171 +0x12d
107112026-03-11T20:45:51.746Zgoogle.golang.org/grpc/internal/transport.(*transportReader).Read(0xc006f647b0, {0xc005dbe538?, 0xc0076910c0?, 0xa40865?})
107122026-03-11T20:45:51.746Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:485 +0x2c
107132026-03-11T20:45:51.746Zio.ReadAtLeast({0x5086380, 0xc006f647b0}, {0xc005dbe538, 0x5, 0x5}, 0x5)
107142026-03-11T20:45:51.746Z /work/go/src/io/io.go:335 +0x90
107152026-03-11T20:45:51.746Zio.ReadFull(...)
107162026-03-11T20:45:51.746Z /work/go/src/io/io.go:354
107172026-03-11T20:45:51.746Zgoogle.golang.org/grpc/internal/transport.(*Stream).Read(0xc001c5f680, {0xc005dbe538, 0x5, 0x5})
107182026-03-11T20:45:51.746Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:469 +0x96
107192026-03-11T20:45:51.746Zgoogle.golang.org/grpc.(*parser).recvMsg(0xc005dbe528, 0x7fffffff)
107202026-03-11T20:45:51.746Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:559 +0x42
107212026-03-11T20:45:51.746Zgoogle.golang.org/grpc.recvAndDecompress(0x60?, 0xc001c5f680, {0x0, 0x0}, 0x7fffffff, 0xc007691378, {0x50a8e40, 0x77577e0})
107222026-03-11T20:45:51.746Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:690 +0x5a
107232026-03-11T20:45:51.746Zgoogle.golang.org/grpc.recv(0x50900a0?, {0x7fc12cff4640, 0xc0004ce590}, 0xc007691300?, {0x0?, 0x0?}, {0x3f6ba40, 0xc01a6c46e0}, 0x10?, 0xc007691378, ...)
107242026-03-11T20:45:51.746Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/rpc_util.go:756 +0x68
107252026-03-11T20:45:51.746Zgoogle.golang.org/grpc.(*csAttempt).recvMsg(0xc0064322c0, {0x3f6ba40, 0xc01a6c46e0}, 0x10?)
107262026-03-11T20:45:51.746Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:983 +0x27e
107272026-03-11T20:45:51.746Zgoogle.golang.org/grpc.(*clientStream).RecvMsg.func1(0x41e1e5?)
107282026-03-11T20:45:51.746Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:834 +0x1f
107292026-03-11T20:45:51.746Zgoogle.golang.org/grpc.(*clientStream).withRetry(0xc001c5efc0, 0xc0076914a8, 0xc007691498)
107302026-03-11T20:45:51.746Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:688 +0x2ee
107312026-03-11T20:45:51.746Zgoogle.golang.org/grpc.(*clientStream).RecvMsg(0xc001c5efc0, {0x3f6ba40?, 0xc01a6c46e0?})
107322026-03-11T20:45:51.746Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/stream.go:833 +0x107
107332026-03-11T20:45:51.746Zgithub.com/cockroachdb/cockroach/pkg/util/tracing.(*tracingClientStream).RecvMsg(0xc0062f7ce0, {0x3f6ba40?, 0xc01a6c46e0?})
107342026-03-11T20:45:51.746Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:441 +0x31
107352026-03-11T20:45:51.746Zgithub.com/cockroachdb/cockroach/pkg/roachpb.(*internalRangeFeedClient).Recv(0xc000829580)
107362026-03-11T20:45:51.746Z /work/oxidecomputer/cockroach/pkg/roachpb/api.pb.go:9289 +0x46
107372026-03-11T20:45:51.746Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed.func3()
107382026-03-11T20:45:51.747Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:516 +0x25
107392026-03-11T20:45:51.747Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*stuckRangeFeedCanceler).do(0xc0013d2aa0, 0xc007691a50)
107402026-03-11T20:45:51.747Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed_canceler.go:100 +0x1b3
107412026-03-11T20:45:51.747Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed(0xc003d48008, {0x50b1338?, 0xc0013d28c0?}, {{0xc004ab65b0, 0x1, 0x8}, {0xc004ab65b8, 0x1, 0x8}}, {0x189be113f3d7da4a, ...}, ...)
107422026-03-11T20:45:51.747Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:515 +0xc98
107432026-03-11T20:45:51.747Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).partialRangeFeed(0xc003d48008, {0x50b1338, 0xc0013d28c0}, 0xc0013d2870, {{0xc004ab65b0, 0x1, 0x8}, {0xc004ab65b8, 0x1, 0x8}}, ...)
107442026-03-11T20:45:51.747Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:336 +0x745
107452026-03-11T20:45:51.747Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.1({0x50b1338?, 0xc0013d28c0?})
107462026-03-11T20:45:51.747Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:120 +0xa9
107472026-03-11T20:45:51.747Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1.Group.GoCtx.2()
107482026-03-11T20:45:51.747Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
107492026-03-11T20:45:51.747Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
107502026-03-11T20:45:51.747Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
107512026-03-11T20:45:51.747Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 11124
107522026-03-11T20:45:51.747Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
107532026-03-11T20:45:51.747Z
107542026-03-11T20:45:51.747Zgoroutine 9822 [chan receive, 59 minutes]:
107552026-03-11T20:45:51.747Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc005572960?})
107562026-03-11T20:45:51.747Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
107572026-03-11T20:45:51.748Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x0?, 0x0?}}, 0xc00405b7a8)
107582026-03-11T20:45:51.748Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
107592026-03-11T20:45:51.748Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
107602026-03-11T20:45:51.748Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
107612026-03-11T20:45:51.748Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
107622026-03-11T20:45:51.748Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
107632026-03-11T20:45:51.748Z
107642026-03-11T20:45:51.748Zgoroutine 10109 [sync.Cond.Wait]:
107652026-03-11T20:45:51.748Zsync.runtime_notifyListWait(0xc001e22250, 0x3763)
107662026-03-11T20:45:51.748Z /work/go/src/runtime/sema.go:569 +0x159
107672026-03-11T20:45:51.748Zsync.(*Cond).Wait(0x3aac600?)
107682026-03-11T20:45:51.748Z /work/go/src/sync/cond.go:70 +0x85
107692026-03-11T20:45:51.748Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583f50})
107702026-03-11T20:45:51.748Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
107712026-03-11T20:45:51.748Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
107722026-03-11T20:45:51.748Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
107732026-03-11T20:45:51.748Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
107742026-03-11T20:45:51.748Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
107752026-03-11T20:45:51.748Z
107762026-03-11T20:45:51.748Zgoroutine 10838 [semacquire, 59 minutes]:
107772026-03-11T20:45:51.748Zsync.runtime_Semacquire(0xc005dbe318?)
107782026-03-11T20:45:51.748Z /work/go/src/runtime/sema.go:62 +0x25
107792026-03-11T20:45:51.748Zsync.(*WaitGroup).Wait(0x44e7a0?)
107802026-03-11T20:45:51.748Z /work/go/src/sync/waitgroup.go:116 +0x48
107812026-03-11T20:45:51.749Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc000ee6b00)
107822026-03-11T20:45:51.749Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
107832026-03-11T20:45:51.749Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc000ee6b00?, {0x50b1338?, 0xc0013d2820?}})
107842026-03-11T20:45:51.749Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
107852026-03-11T20:45:51.749Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait({0x50b1300?, 0xc003c315c0?}, {0xc006077d78, 0x2, 0xc006077c70?})
107862026-03-11T20:45:51.749Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:193 +0xff
107872026-03-11T20:45:51.749Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0xc00491e2c0, {0x50b1300, 0xc003c315c0}, 0xc003c31320)
107882026-03-11T20:45:51.749Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:291 +0x59c
107892026-03-11T20:45:51.749Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).Start.func2({0x50b1300, 0xc003c31530})
107902026-03-11T20:45:51.749Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:215 +0x2f3
107912026-03-11T20:45:51.749Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
107922026-03-11T20:45:51.749Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
107932026-03-11T20:45:51.749Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10837
107942026-03-11T20:45:51.749Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
107952026-03-11T20:45:51.749Z
107962026-03-11T20:45:51.749Zgoroutine 11123 [select]:
107972026-03-11T20:45:51.749Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents(0xc00491e2c0, {0x50b1338, 0xc0013d2820}, 0xc003c31320, 0xc004a22420)
107982026-03-11T20:45:51.749Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:331 +0xc5
107992026-03-11T20:45:51.749Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func2({0x50b1338?, 0xc0013d2820?})
108002026-03-11T20:45:51.749Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:288 +0x32
108012026-03-11T20:45:51.749Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
108022026-03-11T20:45:51.750Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
108032026-03-11T20:45:51.755Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
108042026-03-11T20:45:51.755Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
108052026-03-11T20:45:51.755Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10838
108062026-03-11T20:45:51.755Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
108072026-03-11T20:45:51.755Z
108082026-03-11T20:45:51.755Zgoroutine 10791 [sync.Cond.Wait, 59 minutes]:
108092026-03-11T20:45:51.755Zsync.runtime_notifyListWait(0xc0064c5050, 0x58)
108102026-03-11T20:45:51.755Z /work/go/src/runtime/sema.go:569 +0x159
108112026-03-11T20:45:51.755Zsync.(*Cond).Wait(0x50b1300?)
108122026-03-11T20:45:51.755Z /work/go/src/sync/cond.go:70 +0x85
108132026-03-11T20:45:51.755Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd38f0})
108142026-03-11T20:45:51.755Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
108152026-03-11T20:45:51.755Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
108162026-03-11T20:45:51.755Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
108172026-03-11T20:45:51.755Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
108182026-03-11T20:45:51.755Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
108192026-03-11T20:45:51.755Z
108202026-03-11T20:45:51.755Zgoroutine 9854 [chan receive, 59 minutes]:
108212026-03-11T20:45:51.755Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc007582b10?})
108222026-03-11T20:45:51.755Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
108232026-03-11T20:45:51.755Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x0?, 0x0?}}, 0xc000ac9fa8)
108242026-03-11T20:45:51.756Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
108252026-03-11T20:45:51.756Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
108262026-03-11T20:45:51.756Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
108272026-03-11T20:45:51.756Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
108282026-03-11T20:45:51.756Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
108292026-03-11T20:45:51.756Z
108302026-03-11T20:45:51.756Zgoroutine 9853 [select]:
108312026-03-11T20:45:51.756Zgithub.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).startTickerLocked.func1()
108322026-03-11T20:45:51.756Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:365 +0xd2
108332026-03-11T20:45:51.756Zcreated by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).startTickerLocked in goroutine 9708
108342026-03-11T20:45:51.756Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:360 +0x6e
108352026-03-11T20:45:51.756Z
108362026-03-11T20:45:51.756Zgoroutine 9823 [select]:
108372026-03-11T20:45:51.756Zgithub.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1()
108382026-03-11T20:45:51.756Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:71 +0xcb
108392026-03-11T20:45:51.756Zcreated by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 9708
108402026-03-11T20:45:51.756Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:66 +0x58
108412026-03-11T20:45:51.756Z
108422026-03-11T20:45:51.756Zgoroutine 9825 [select]:
108432026-03-11T20:45:51.756Zgithub.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1()
108442026-03-11T20:45:51.756Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:71 +0xcb
108452026-03-11T20:45:51.756Zcreated by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 9708
108462026-03-11T20:45:51.757Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:66 +0x58
108472026-03-11T20:45:51.757Z
108482026-03-11T20:45:51.757Zgoroutine 9709 [semacquire, 59 minutes]:
108492026-03-11T20:45:51.757Zsync.runtime_Semacquire(0xc00212a0f0?)
108502026-03-11T20:45:51.757Z /work/go/src/runtime/sema.go:62 +0x25
108512026-03-11T20:45:51.757Zsync.(*WaitGroup).Wait(0x44e7a0?)
108522026-03-11T20:45:51.757Z /work/go/src/sync/waitgroup.go:116 +0x48
108532026-03-11T20:45:51.757Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc001f30040)
108542026-03-11T20:45:51.757Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
108552026-03-11T20:45:51.757Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc001f30040?, {0x50b1338?, 0xc005d142d0?}})
108562026-03-11T20:45:51.757Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
108572026-03-11T20:45:51.757Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed(0xc002d6f408, {0x50b1338?, 0xc005d14230?}, {0xc00485e8a0, 0x1, 0xc000617708?}, {0xc0006176f0?, 0x479f93?, 0x0?}, 0x0, ...)
108582026-03-11T20:45:51.757Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:139 +0x6c5
108592026-03-11T20:45:51.757Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*dbAdapter).RangeFeed(0xc0006177d0?, {0x50b1338?, 0xc005d14230?}, {0xc00485e8a0?, 0x0?, 0x0?}, {0x50b1300?, 0x36f8b01?, 0xc0?}, 0x0, ...)
108602026-03-11T20:45:51.757Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/db_adapter.go:78 +0x3e
108612026-03-11T20:45:51.757Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func1({0x50b1338?, 0xc005d14230?})
108622026-03-11T20:45:51.757Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:285 +0x6c
108632026-03-11T20:45:51.757Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
108642026-03-11T20:45:51.757Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
108652026-03-11T20:45:51.757Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
108662026-03-11T20:45:51.758Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
108672026-03-11T20:45:51.758Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9851
108682026-03-11T20:45:51.758Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
108692026-03-11T20:45:51.758Z
108702026-03-11T20:45:51.758Zgoroutine 10800 [sync.Cond.Wait, 59 minutes]:
108712026-03-11T20:45:51.758Zsync.runtime_notifyListWait(0xc0064c5050, 0x44)
108722026-03-11T20:45:51.758Z /work/go/src/runtime/sema.go:569 +0x159
108732026-03-11T20:45:51.758Zsync.(*Cond).Wait(0x0?)
108742026-03-11T20:45:51.758Z /work/go/src/sync/cond.go:70 +0x85
108752026-03-11T20:45:51.758Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3aa0})
108762026-03-11T20:45:51.758Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
108772026-03-11T20:45:51.758Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
108782026-03-11T20:45:51.759Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
108792026-03-11T20:45:51.759Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
108802026-03-11T20:45:51.759Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
108812026-03-11T20:45:51.759Z
108822026-03-11T20:45:51.759Zgoroutine 10823 [select]:
108832026-03-11T20:45:51.759Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).startRangefeedUpdater.func1({0x50b1300, 0xc006fd3f50})
108842026-03-11T20:45:51.759Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:2307 +0x28a
108852026-03-11T20:45:51.759Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
108862026-03-11T20:45:51.759Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
108872026-03-11T20:45:51.759Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
108882026-03-11T20:45:51.759Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
108892026-03-11T20:45:51.759Z
108902026-03-11T20:45:51.759Zgoroutine 10797 [sync.Cond.Wait, 59 minutes]:
108912026-03-11T20:45:51.759Zsync.runtime_notifyListWait(0xc0064c5050, 0x5f)
108922026-03-11T20:45:51.759Z /work/go/src/runtime/sema.go:569 +0x159
108932026-03-11T20:45:51.759Zsync.(*Cond).Wait(0xc001e87e60?)
108942026-03-11T20:45:51.760Z /work/go/src/sync/cond.go:70 +0x85
108952026-03-11T20:45:51.760Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3a10})
108962026-03-11T20:45:51.760Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
108972026-03-11T20:45:51.760Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
108982026-03-11T20:45:51.760Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
108992026-03-11T20:45:51.760Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
109002026-03-11T20:45:51.760Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
109012026-03-11T20:45:51.760Z
109022026-03-11T20:45:51.760Zgoroutine 10778 [sync.Cond.Wait, 59 minutes]:
109032026-03-11T20:45:51.764Zsync.runtime_notifyListWait(0xc0064c5050, 0x3c)
109042026-03-11T20:45:51.764Z /work/go/src/runtime/sema.go:569 +0x159
109052026-03-11T20:45:51.764Zsync.(*Cond).Wait(0x50f7930?)
109062026-03-11T20:45:51.764Z /work/go/src/sync/cond.go:70 +0x85
109072026-03-11T20:45:51.764Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3680})
109082026-03-11T20:45:51.764Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
109092026-03-11T20:45:51.764Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
109102026-03-11T20:45:51.764Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
109112026-03-11T20:45:51.764Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
109122026-03-11T20:45:51.764Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
109132026-03-11T20:45:51.765Z
109142026-03-11T20:45:51.765Zgoroutine 9910 [chan receive, 59 minutes]:
109152026-03-11T20:45:51.765Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc002e5a630?})
109162026-03-11T20:45:51.765Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
109172026-03-11T20:45:51.765Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x0?, 0x0?}}, 0xc0042457a8)
109182026-03-11T20:45:51.765Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
109192026-03-11T20:45:51.765Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
109202026-03-11T20:45:51.765Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
109212026-03-11T20:45:51.765Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
109222026-03-11T20:45:51.765Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
109232026-03-11T20:45:51.765Z
109242026-03-11T20:45:51.765Zgoroutine 10737 [sync.Cond.Wait, 59 minutes]:
109252026-03-11T20:45:51.765Zsync.runtime_notifyListWait(0xc0064c5050, 0xe)
109262026-03-11T20:45:51.765Z /work/go/src/runtime/sema.go:569 +0x159
109272026-03-11T20:45:51.765Zsync.(*Cond).Wait(0x18?)
109282026-03-11T20:45:51.765Z /work/go/src/sync/cond.go:70 +0x85
109292026-03-11T20:45:51.765Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd2ed0})
109302026-03-11T20:45:51.765Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
109312026-03-11T20:45:51.765Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
109322026-03-11T20:45:51.765Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
109332026-03-11T20:45:51.765Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
109342026-03-11T20:45:51.765Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
109352026-03-11T20:45:51.765Z
109362026-03-11T20:45:51.765Zgoroutine 10736 [sync.Cond.Wait, 59 minutes]:
109372026-03-11T20:45:51.765Zsync.runtime_notifyListWait(0xc0064c5050, 0xa)
109382026-03-11T20:45:51.765Z /work/go/src/runtime/sema.go:569 +0x159
109392026-03-11T20:45:51.765Zsync.(*Cond).Wait(0x50f7930?)
109402026-03-11T20:45:51.765Z /work/go/src/sync/cond.go:70 +0x85
109412026-03-11T20:45:51.765Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd2ea0})
109422026-03-11T20:45:51.765Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
109432026-03-11T20:45:51.765Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
109442026-03-11T20:45:51.765Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
109452026-03-11T20:45:51.765Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
109462026-03-11T20:45:51.765Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
109472026-03-11T20:45:51.765Z
109482026-03-11T20:45:51.765Zgoroutine 10802 [sync.Cond.Wait, 59 minutes]:
109492026-03-11T20:45:51.766Zsync.runtime_notifyListWait(0xc0064c5050, 0x47)
109502026-03-11T20:45:51.766Z /work/go/src/runtime/sema.go:569 +0x159
109512026-03-11T20:45:51.766Zsync.(*Cond).Wait(0x0?)
109522026-03-11T20:45:51.766Z /work/go/src/sync/cond.go:70 +0x85
109532026-03-11T20:45:51.766Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3b00})
109542026-03-11T20:45:51.766Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
109552026-03-11T20:45:51.766Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
109562026-03-11T20:45:51.766Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
109572026-03-11T20:45:51.766Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
109582026-03-11T20:45:51.766Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
109592026-03-11T20:45:51.766Z
109602026-03-11T20:45:51.766Zgoroutine 10824 [select, 1 minutes]:
109612026-03-11T20:45:51.766Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*StoreRebalancer).Start.func1({0x50b1300, 0xc007164090})
109622026-03-11T20:45:51.766Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store_rebalancer.go:235 +0x1c5
109632026-03-11T20:45:51.766Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
109642026-03-11T20:45:51.766Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
109652026-03-11T20:45:51.766Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
109662026-03-11T20:45:51.766Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
109672026-03-11T20:45:51.766Z
109682026-03-11T20:45:51.766Zgoroutine 10301 [select, 59 minutes]:
109692026-03-11T20:45:51.766Zgoogle.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc007863380)
109702026-03-11T20:45:51.766Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:112 +0x6d
109712026-03-11T20:45:51.766Zcreated by google.golang.org/grpc.newCCBalancerWrapper in goroutine 9708
109722026-03-11T20:45:51.766Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:73 +0x22b
109732026-03-11T20:45:51.766Z
109742026-03-11T20:45:51.766Zgoroutine 9915 [chan receive, 59 minutes]:
109752026-03-11T20:45:51.766Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc002e5ad20?})
109762026-03-11T20:45:51.766Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
109772026-03-11T20:45:51.766Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x0?, 0x0?}}, 0xc004659fa8)
109782026-03-11T20:45:51.766Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
109792026-03-11T20:45:51.766Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
109802026-03-11T20:45:51.766Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
109812026-03-11T20:45:51.766Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
109822026-03-11T20:45:51.767Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
109832026-03-11T20:45:51.767Z
109842026-03-11T20:45:51.767Zgoroutine 10799 [sync.Cond.Wait, 59 minutes]:
109852026-03-11T20:45:51.767Zsync.runtime_notifyListWait(0xc0064c5050, 0x43)
109862026-03-11T20:45:51.767Z /work/go/src/runtime/sema.go:569 +0x159
109872026-03-11T20:45:51.767Zsync.(*Cond).Wait(0x0?)
109882026-03-11T20:45:51.767Z /work/go/src/sync/cond.go:70 +0x85
109892026-03-11T20:45:51.767Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3a70})
109902026-03-11T20:45:51.767Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
109912026-03-11T20:45:51.767Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
109922026-03-11T20:45:51.767Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
109932026-03-11T20:45:51.767Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
109942026-03-11T20:45:51.767Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
109952026-03-11T20:45:51.767Z
109962026-03-11T20:45:51.767Zgoroutine 10790 [sync.Cond.Wait, 59 minutes]:
109972026-03-11T20:45:51.767Zsync.runtime_notifyListWait(0xc0064c5050, 0x56)
109982026-03-11T20:45:51.767Z /work/go/src/runtime/sema.go:569 +0x159
109992026-03-11T20:45:51.767Zsync.(*Cond).Wait(0xc006e3e878?)
110002026-03-11T20:45:51.768Z /work/go/src/sync/cond.go:70 +0x85
110012026-03-11T20:45:51.768Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd38c0})
110022026-03-11T20:45:51.768Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
110032026-03-11T20:45:51.772Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
110042026-03-11T20:45:51.772Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
110052026-03-11T20:45:51.772Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
110062026-03-11T20:45:51.772Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
110072026-03-11T20:45:51.772Z
110082026-03-11T20:45:51.772Zgoroutine 9909 [chan receive, 59 minutes]:
110092026-03-11T20:45:51.772Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc002e5a540?})
110102026-03-11T20:45:51.772Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
110112026-03-11T20:45:51.772Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x0?, 0x0?}}, 0xc004244fa8)
110122026-03-11T20:45:51.773Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
110132026-03-11T20:45:51.773Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
110142026-03-11T20:45:51.773Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
110152026-03-11T20:45:51.773Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
110162026-03-11T20:45:51.773Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
110172026-03-11T20:45:51.773Z
110182026-03-11T20:45:51.773Zgoroutine 10657 [select]:
110192026-03-11T20:45:51.773Zgithub.com/cockroachdb/cockroach/pkg/gossip.(*client).gossip(0xc00740e410, {0x50b1300, 0xc006d70120}, 0xc003d1ab60, {0x50ebb00, 0xc00082acd0}, 0xc003dd7580, 0xc0060db9b0)
110202026-03-11T20:45:51.773Z /work/oxidecomputer/cockroach/pkg/gossip/client.go:364 +0x415
110212026-03-11T20:45:51.773Zgithub.com/cockroachdb/cockroach/pkg/gossip.(*client).startLocked.func1({0x50b1300, 0xc006d70120})
110222026-03-11T20:45:51.773Z /work/oxidecomputer/cockroach/pkg/gossip/client.go:125 +0x3d2
110232026-03-11T20:45:51.773Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
110242026-03-11T20:45:51.773Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
110252026-03-11T20:45:51.773Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10696
110262026-03-11T20:45:51.773Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
110272026-03-11T20:45:51.773Z
110282026-03-11T20:45:51.773Zgoroutine 10866 [select]:
110292026-03-11T20:45:51.773Zgoogle.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000e8e5f0, 0x1)
110302026-03-11T20:45:51.773Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:407 +0x113
110312026-03-11T20:45:51.773Zgoogle.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc006028c00)
110322026-03-11T20:45:51.773Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:534 +0x7b
110332026-03-11T20:45:51.773Zgoogle.golang.org/grpc/internal/transport.newHTTP2Client.func3()
110342026-03-11T20:45:51.773Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:415 +0x67
110352026-03-11T20:45:51.773Zcreated by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 10707
110362026-03-11T20:45:51.773Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:413 +0x1dcb
110372026-03-11T20:45:51.773Z
110382026-03-11T20:45:51.773Zgoroutine 9908 [chan receive, 59 minutes]:
110392026-03-11T20:45:51.773Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc002e5a3f0?})
110402026-03-11T20:45:51.773Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
110412026-03-11T20:45:51.773Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x0?, 0x0?}}, 0xc0042447a8)
110422026-03-11T20:45:51.773Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
110432026-03-11T20:45:51.773Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
110442026-03-11T20:45:51.773Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
110452026-03-11T20:45:51.773Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
110462026-03-11T20:45:51.773Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
110472026-03-11T20:45:51.773Z
110482026-03-11T20:45:51.774Zgoroutine 10794 [sync.Cond.Wait, 59 minutes]:
110492026-03-11T20:45:51.774Zsync.runtime_notifyListWait(0xc0064c5050, 0x5b)
110502026-03-11T20:45:51.774Z /work/go/src/runtime/sema.go:569 +0x159
110512026-03-11T20:45:51.774Zsync.(*Cond).Wait(0xc006e3e878?)
110522026-03-11T20:45:51.774Z /work/go/src/sync/cond.go:70 +0x85
110532026-03-11T20:45:51.774Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3980})
110542026-03-11T20:45:51.774Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
110552026-03-11T20:45:51.774Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
110562026-03-11T20:45:51.774Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
110572026-03-11T20:45:51.774Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
110582026-03-11T20:45:51.774Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
110592026-03-11T20:45:51.774Z
110602026-03-11T20:45:51.774Zgoroutine 10426 [chan receive, 59 minutes]:
110612026-03-11T20:45:51.774Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).rangeFeedWithRangeID(0xc001ded408, 0x3f3de40?, 0xc005dff8c0, {0x7fc12cefd140, 0xc005dbb420})
110622026-03-11T20:45:51.774Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:249 +0x76b
110632026-03-11T20:45:51.774Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed(...)
110642026-03-11T20:45:51.774Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:149
110652026-03-11T20:45:51.774Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).RangeFeed(0xc000f39208, 0xc005dff8c0, {0x50e5b30, 0xc005dbb420})
110662026-03-11T20:45:51.774Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:3082 +0x16f
110672026-03-11T20:45:51.774Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).RangeFeed(0xc001eb4380, 0xc005dff8c0, {0x50e5b30, 0xc005dbb420})
110682026-03-11T20:45:51.774Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/stores.go:216 +0x19a
110692026-03-11T20:45:51.774Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed(0x3f3de40?, 0xc007611c00?, {0x50e5b30, 0xc005dbb420})
110702026-03-11T20:45:51.774Z /work/oxidecomputer/cockroach/pkg/server/node.go:1232 +0x28
110712026-03-11T20:45:51.774Zgithub.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed.func1()
110722026-03-11T20:45:51.774Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:753 +0xa7
110732026-03-11T20:45:51.774Zcreated by github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed in goroutine 10425
110742026-03-11T20:45:51.774Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:750 +0x17c
110752026-03-11T20:45:51.774Z
110762026-03-11T20:45:51.774Zgoroutine 9063 [select, 59 minutes]:
110772026-03-11T20:45:51.774Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*baseQueue).processLoop.func2({0x50b1300, 0xc0076b6630})
110782026-03-11T20:45:51.774Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/queue.go:823 +0xff
110792026-03-11T20:45:51.774Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
110802026-03-11T20:45:51.774Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
110812026-03-11T20:45:51.774Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10821
110822026-03-11T20:45:51.774Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
110832026-03-11T20:45:51.775Z
110842026-03-11T20:45:51.775Zgoroutine 10805 [sync.Cond.Wait, 59 minutes]:
110852026-03-11T20:45:51.775Zsync.runtime_notifyListWait(0xc0064c5050, 0x4a)
110862026-03-11T20:45:51.775Z /work/go/src/runtime/sema.go:569 +0x159
110872026-03-11T20:45:51.775Zsync.(*Cond).Wait(0x0?)
110882026-03-11T20:45:51.775Z /work/go/src/sync/cond.go:70 +0x85
110892026-03-11T20:45:51.775Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3b90})
110902026-03-11T20:45:51.775Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
110912026-03-11T20:45:51.775Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
110922026-03-11T20:45:51.775Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
110932026-03-11T20:45:51.775Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
110942026-03-11T20:45:51.775Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
110952026-03-11T20:45:51.775Z
110962026-03-11T20:45:51.775Zgoroutine 10804 [sync.Cond.Wait, 59 minutes]:
110972026-03-11T20:45:51.775Zsync.runtime_notifyListWait(0xc0064c5050, 0x49)
110982026-03-11T20:45:51.775Z /work/go/src/runtime/sema.go:569 +0x159
110992026-03-11T20:45:51.775Zsync.(*Cond).Wait(0x0?)
111002026-03-11T20:45:51.775Z /work/go/src/sync/cond.go:70 +0x85
111012026-03-11T20:45:51.775Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3b60})
111022026-03-11T20:45:51.775Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
111032026-03-11T20:45:51.781Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
111042026-03-11T20:45:51.781Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
111052026-03-11T20:45:51.781Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
111062026-03-11T20:45:51.781Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
111072026-03-11T20:45:51.781Z
111082026-03-11T20:45:51.781Zgoroutine 9783 [chan receive, 59 minutes]:
111092026-03-11T20:45:51.781Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc005572300?})
111102026-03-11T20:45:51.781Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
111112026-03-11T20:45:51.781Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x50f8001?, 0x1000000010000?}}, 0xc0000bc7a8)
111122026-03-11T20:45:51.781Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
111132026-03-11T20:45:51.781Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
111142026-03-11T20:45:51.781Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
111152026-03-11T20:45:51.781Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
111162026-03-11T20:45:51.781Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
111172026-03-11T20:45:51.781Z
111182026-03-11T20:45:51.781Zgoroutine 8973 [chan receive, 59 minutes]:
111192026-03-11T20:45:51.781Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc007582150?})
111202026-03-11T20:45:51.782Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
111212026-03-11T20:45:51.782Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x1000000000001?, 0xc002270a00?}}, 0xc0004e47a8)
111222026-03-11T20:45:51.782Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
111232026-03-11T20:45:51.782Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
111242026-03-11T20:45:51.782Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
111252026-03-11T20:45:51.782Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
111262026-03-11T20:45:51.782Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
111272026-03-11T20:45:51.782Z
111282026-03-11T20:45:51.782Zgoroutine 10680 [chan receive, 59 minutes]:
111292026-03-11T20:45:51.782Zgithub.com/cockroachdb/cockroach/pkg/server.startListenRPCAndSQL.func1({0x50b1300, 0xc006e62630})
111302026-03-11T20:45:51.782Z /work/oxidecomputer/cockroach/pkg/server/start_listen.go:64 +0x45
111312026-03-11T20:45:51.782Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
111322026-03-11T20:45:51.782Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
111332026-03-11T20:45:51.782Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
111342026-03-11T20:45:51.782Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
111352026-03-11T20:45:51.782Z
111362026-03-11T20:45:51.782Zgoroutine 8970 [select]:
111372026-03-11T20:45:51.782Zgithub.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).startTickerLocked.func1()
111382026-03-11T20:45:51.782Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:365 +0xd2
111392026-03-11T20:45:51.783Zcreated by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).startTickerLocked in goroutine 9708
111402026-03-11T20:45:51.783Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:360 +0x6e
111412026-03-11T20:45:51.783Z
111422026-03-11T20:45:51.783Zgoroutine 9780 [chan receive, 59 minutes]:
111432026-03-11T20:45:51.783Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc0055721b0?})
111442026-03-11T20:45:51.783Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
111452026-03-11T20:45:51.783Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x1000000010000?, 0xc00540c140?}}, 0xc002be7fa8)
111462026-03-11T20:45:51.783Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
111472026-03-11T20:45:51.783Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
111482026-03-11T20:45:51.783Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
111492026-03-11T20:45:51.783Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
111502026-03-11T20:45:51.783Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
111512026-03-11T20:45:51.783Z
111522026-03-11T20:45:51.783Zgoroutine 10809 [sync.Cond.Wait, 59 minutes]:
111532026-03-11T20:45:51.783Zsync.runtime_notifyListWait(0xc0064c5050, 0x41)
111542026-03-11T20:45:51.783Z /work/go/src/runtime/sema.go:569 +0x159
111552026-03-11T20:45:51.783Zsync.(*Cond).Wait(0x0?)
111562026-03-11T20:45:51.783Z /work/go/src/sync/cond.go:70 +0x85
111572026-03-11T20:45:51.783Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3c50})
111582026-03-11T20:45:51.783Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
111592026-03-11T20:45:51.783Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
111602026-03-11T20:45:51.783Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
111612026-03-11T20:45:51.783Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
111622026-03-11T20:45:51.783Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
111632026-03-11T20:45:51.784Z
111642026-03-11T20:45:51.784Zgoroutine 10491 [select]:
111652026-03-11T20:45:51.784Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).outputLoop(0xc002759260, {0x50b1338, 0xc005f39cc0})
111662026-03-11T20:45:51.784Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:310 +0x247
111672026-03-11T20:45:51.784Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).runOutputLoop(0xc002759260, {0x50b1300, 0xc0070e37a0}, 0xc0020cf9e0?)
111682026-03-11T20:45:51.784Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:335 +0xba
111692026-03-11T20:45:51.784Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run.func1({0x50b1300?, 0xc0070e37a0?})
111702026-03-11T20:45:51.784Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:297 +0x46
111712026-03-11T20:45:51.784Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
111722026-03-11T20:45:51.784Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
111732026-03-11T20:45:51.784Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10489
111742026-03-11T20:45:51.784Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
111752026-03-11T20:45:51.784Z
111762026-03-11T20:45:51.784Zgoroutine 10492 [semacquire, 59 minutes]:
111772026-03-11T20:45:51.784Zsync.runtime_Semacquire(0xc007ee03d8?)
111782026-03-11T20:45:51.784Z /work/go/src/runtime/sema.go:62 +0x25
111792026-03-11T20:45:51.784Zsync.(*WaitGroup).Wait(0x44e7a0?)
111802026-03-11T20:45:51.784Z /work/go/src/sync/waitgroup.go:116 +0x48
111812026-03-11T20:45:51.784Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc006042f80)
111822026-03-11T20:45:51.784Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
111832026-03-11T20:45:51.784Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc006042f80?, {0x50b1338?, 0xc005f39e00?}})
111842026-03-11T20:45:51.784Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
111852026-03-11T20:45:51.784Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed(0xc002d6f408, {0x50b1338?, 0xc005f39d60?}, {0xc00081b0e0, 0x1, 0x1d2a580?}, {0xc007ee01f8?, 0x60855b8?, 0xc0?}, 0x0, ...)
111862026-03-11T20:45:51.784Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:139 +0x6c5
111872026-03-11T20:45:51.784Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*dbAdapter).RangeFeed(0x50b1300?, {0x50b1338?, 0xc005f39d60?}, {0xc00081b0e0?, 0x50b1300?, 0xc0070e3710?}, {0x707000000000001?, 0x50b1300?, 0x0?}, 0x0, ...)
111882026-03-11T20:45:51.784Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/db_adapter.go:78 +0x3e
111892026-03-11T20:45:51.784Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func1({0x50b1338?, 0xc005f39d60?})
111902026-03-11T20:45:51.784Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:285 +0x6c
111912026-03-11T20:45:51.784Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
111922026-03-11T20:45:51.784Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
111932026-03-11T20:45:51.784Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
111942026-03-11T20:45:51.784Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
111952026-03-11T20:45:51.784Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10202
111962026-03-11T20:45:51.784Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
111972026-03-11T20:45:51.784Z
111982026-03-11T20:45:51.784Zgoroutine 10103 [sync.Cond.Wait]:
111992026-03-11T20:45:51.784Zsync.runtime_notifyListWait(0xc001e22250, 0x3745)
112002026-03-11T20:45:51.784Z /work/go/src/runtime/sema.go:569 +0x159
112012026-03-11T20:45:51.785Zsync.(*Cond).Wait(0x3aac600?)
112022026-03-11T20:45:51.785Z /work/go/src/sync/cond.go:70 +0x85
112032026-03-11T20:45:51.790Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583e30})
112042026-03-11T20:45:51.790Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
112052026-03-11T20:45:51.790Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
112062026-03-11T20:45:51.790Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
112072026-03-11T20:45:51.790Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
112082026-03-11T20:45:51.790Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
112092026-03-11T20:45:51.791Z
112102026-03-11T20:45:51.791Zgoroutine 10494 [select, 59 minutes]:
112112026-03-11T20:45:51.791Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.func1({0x50b1338, 0xc005f39e00})
112122026-03-11T20:45:51.791Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:116 +0x157
112132026-03-11T20:45:51.791Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed.Group.GoCtx.func3()
112142026-03-11T20:45:51.791Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
112152026-03-11T20:45:51.791Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
112162026-03-11T20:45:51.791Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
112172026-03-11T20:45:51.791Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10492
112182026-03-11T20:45:51.791Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
112192026-03-11T20:45:51.791Z
112202026-03-11T20:45:51.791Zgoroutine 10851 [select]:
112212026-03-11T20:45:51.791Zgoogle.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc0066c21e0)
112222026-03-11T20:45:51.791Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1572 +0x137
112232026-03-11T20:45:51.792Zcreated by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 9758
112242026-03-11T20:45:51.792Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:360 +0x180b
112252026-03-11T20:45:51.792Z
112262026-03-11T20:45:51.792Zgoroutine 9802 [select]:
112272026-03-11T20:45:51.792Zgithub.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue.func2()
112282026-03-11T20:45:51.792Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:366 +0x85
112292026-03-11T20:45:51.792Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue in goroutine 9708
112302026-03-11T20:45:51.793Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:363 +0x3ed
112312026-03-11T20:45:51.793Z
112322026-03-11T20:45:51.793Zgoroutine 10629 [select, 59 minutes]:
112332026-03-11T20:45:51.793Zgithub.com/cockroachdb/cockroach/pkg/server.(*Server).startAttemptUpgrade.func1({0x50b1300?, 0xc006858990?})
112342026-03-11T20:45:51.793Z /work/oxidecomputer/cockroach/pkg/server/auto_upgrade.go:45 +0x1cf
112352026-03-11T20:45:51.793Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
112362026-03-11T20:45:51.793Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
112372026-03-11T20:45:51.793Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
112382026-03-11T20:45:51.793Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
112392026-03-11T20:45:51.793Z
112402026-03-11T20:45:51.793Zgoroutine 9813 [chan receive, 59 minutes]:
112412026-03-11T20:45:51.793Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc0055729c0?})
112422026-03-11T20:45:51.793Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
112432026-03-11T20:45:51.793Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0xc00350de60?, 0xc00350dea0?}}, 0xc00405a7a8)
112442026-03-11T20:45:51.793Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
112452026-03-11T20:45:51.793Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
112462026-03-11T20:45:51.793Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
112472026-03-11T20:45:51.793Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
112482026-03-11T20:45:51.793Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
112492026-03-11T20:45:51.793Z
112502026-03-11T20:45:51.793Zgoroutine 9801 [select]:
112512026-03-11T20:45:51.793Zgithub.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
112522026-03-11T20:45:51.793Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:439 +0x1c9
112532026-03-11T20:45:51.793Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 9708
112542026-03-11T20:45:51.793Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:412 +0x4f
112552026-03-11T20:45:51.793Z
112562026-03-11T20:45:51.793Zgoroutine 11053 [semacquire, 59 minutes]:
112572026-03-11T20:45:51.793Zsync.runtime_Semacquire(0xc008c3ef18?)
112582026-03-11T20:45:51.793Z /work/go/src/runtime/sema.go:62 +0x25
112592026-03-11T20:45:51.793Zsync.(*WaitGroup).Wait(0x44e7a0?)
112602026-03-11T20:45:51.793Z /work/go/src/sync/waitgroup.go:116 +0x48
112612026-03-11T20:45:51.793Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc0044a7980)
112622026-03-11T20:45:51.793Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
112632026-03-11T20:45:51.793Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc0044a7980?, {0x50b1338?, 0xc0032344b0?}})
112642026-03-11T20:45:51.793Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
112652026-03-11T20:45:51.793Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed(0xc002d6f408, {0x50b1338?, 0xc003234370?}, {0xc0066f4ab0, 0x1, 0xb2d05e00?}, {0xc007661ad0?, 0x55b21e1?, 0xc0?}, 0x0, ...)
112662026-03-11T20:45:51.793Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:139 +0x6c5
112672026-03-11T20:45:51.794Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*dbAdapter).RangeFeed(0x227ed3ae1b6c5a?, {0x50b1338?, 0xc003234370?}, {0xc0066f4ab0?, 0x0?, 0xc002c43508?}, {0x50b1300?, 0x556abd0?, 0xc0?}, 0x0, ...)
112682026-03-11T20:45:51.794Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/db_adapter.go:78 +0x3e
112692026-03-11T20:45:51.794Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func1({0x50b1338?, 0xc003234370?})
112702026-03-11T20:45:51.794Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:285 +0x6c
112712026-03-11T20:45:51.794Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
112722026-03-11T20:45:51.794Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
112732026-03-11T20:45:51.794Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
112742026-03-11T20:45:51.794Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
112752026-03-11T20:45:51.794Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10557
112762026-03-11T20:45:51.794Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
112772026-03-11T20:45:51.794Z
112782026-03-11T20:45:51.794Zgoroutine 9807 [chan receive, 59 minutes]:
112792026-03-11T20:45:51.794Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc0036f8180?})
112802026-03-11T20:45:51.794Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
112812026-03-11T20:45:51.794Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0xc00350db00?, 0xc00350db40?}}, 0xc004057fa8)
112822026-03-11T20:45:51.794Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
112832026-03-11T20:45:51.794Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
112842026-03-11T20:45:51.794Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
112852026-03-11T20:45:51.794Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
112862026-03-11T20:45:51.794Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
112872026-03-11T20:45:51.794Z
112882026-03-11T20:45:51.794Zgoroutine 10687 [chan receive, 59 minutes]:
112892026-03-11T20:45:51.794Zgithub.com/cockroachdb/cmux.muxListener.Accept(...)
112902026-03-11T20:45:51.794Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184
112912026-03-11T20:45:51.794Zgoogle.golang.org/grpc.(*Server).Serve(0xc0076b8000, {0x509c4c0, 0xc006e57e90})
112922026-03-11T20:45:51.794Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/server.go:780 +0x469
112932026-03-11T20:45:51.794Zgithub.com/cockroachdb/cockroach/pkg/server.startListenRPCAndSQL.func5.1({0xc006e9c008?, 0xc0083c6870?})
112942026-03-11T20:45:51.794Z /work/oxidecomputer/cockroach/pkg/server/start_listen.go:134 +0x25
112952026-03-11T20:45:51.794Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
112962026-03-11T20:45:51.794Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
112972026-03-11T20:45:51.794Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
112982026-03-11T20:45:51.794Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
112992026-03-11T20:45:51.794Z
113002026-03-11T20:45:51.794Zgoroutine 11054 [select]:
113012026-03-11T20:45:51.794Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents(0xc008050580, {0x50b1338, 0xc003234370}, 0xc0066f4a20, 0xc006834660)
113022026-03-11T20:45:51.794Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:331 +0xc5
113032026-03-11T20:45:51.802Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func2({0x50b1338?, 0xc003234370?})
113042026-03-11T20:45:51.802Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:288 +0x32
113052026-03-11T20:45:51.802Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
113062026-03-11T20:45:51.802Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
113072026-03-11T20:45:51.802Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
113082026-03-11T20:45:51.802Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
113092026-03-11T20:45:51.802Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10557
113102026-03-11T20:45:51.802Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
113112026-03-11T20:45:51.802Z
113122026-03-11T20:45:51.802Zgoroutine 9855 [chan receive, 59 minutes]:
113132026-03-11T20:45:51.802Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc007582b70?})
113142026-03-11T20:45:51.802Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
113152026-03-11T20:45:51.802Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0xc00350dd40?, 0xc00350dd80?}}, 0xc000e097a8)
113162026-03-11T20:45:51.802Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
113172026-03-11T20:45:51.802Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
113182026-03-11T20:45:51.802Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
113192026-03-11T20:45:51.802Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
113202026-03-11T20:45:51.802Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
113212026-03-11T20:45:51.802Z
113222026-03-11T20:45:51.802Zgoroutine 10334 [select]:
113232026-03-11T20:45:51.802Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run(0xc005b6e000, {0x50b1300?, 0xc0006a94a0?}, 0x6?, 0xc0023085a0, 0xc003f40400)
113242026-03-11T20:45:51.802Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:269 +0x4dc
113252026-03-11T20:45:51.803Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).Start.func1({0x50b1300?, 0xc0006a94a0?})
113262026-03-11T20:45:51.803Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:222 +0x36
113272026-03-11T20:45:51.803Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
113282026-03-11T20:45:51.803Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
113292026-03-11T20:45:51.803Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10333
113302026-03-11T20:45:51.803Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
113312026-03-11T20:45:51.803Z
113322026-03-11T20:45:51.803Zgoroutine 10859 [select, 29 minutes]:
113332026-03-11T20:45:51.803Zgithub.com/cockroachdb/cockroach/pkg/sql.(*TemporaryObjectCleaner).Start.func1({0x50b1300, 0xc00067cb70})
113342026-03-11T20:45:51.803Z /work/oxidecomputer/cockroach/pkg/sql/temporary_schema.go:632 +0x22e
113352026-03-11T20:45:51.803Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
113362026-03-11T20:45:51.803Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
113372026-03-11T20:45:51.804Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
113382026-03-11T20:45:51.804Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
113392026-03-11T20:45:51.804Z
113402026-03-11T20:45:51.804Zgoroutine 9808 [chan receive, 59 minutes]:
113412026-03-11T20:45:51.804Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc0036f8210?})
113422026-03-11T20:45:51.804Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
113432026-03-11T20:45:51.804Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x1000000010000?, 0xc000d0a410?}}, 0xc0040567a8)
113442026-03-11T20:45:51.804Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
113452026-03-11T20:45:51.804Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
113462026-03-11T20:45:51.804Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
113472026-03-11T20:45:51.804Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
113482026-03-11T20:45:51.804Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
113492026-03-11T20:45:51.804Z
113502026-03-11T20:45:51.804Zgoroutine 9806 [select]:
113512026-03-11T20:45:51.804Zgithub.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).startTickerLocked.func1()
113522026-03-11T20:45:51.804Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:365 +0xd2
113532026-03-11T20:45:51.804Zcreated by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFS).startTickerLocked in goroutine 9708
113542026-03-11T20:45:51.804Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:360 +0x6e
113552026-03-11T20:45:51.804Z
113562026-03-11T20:45:51.804Zgoroutine 8976 [chan receive, 59 minutes]:
113572026-03-11T20:45:51.804Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc0055723c0?})
113582026-03-11T20:45:51.804Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
113592026-03-11T20:45:51.804Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x10000c000ee6780?, 0xc003477290?}}, 0xc002be87a8)
113602026-03-11T20:45:51.804Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
113612026-03-11T20:45:51.804Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
113622026-03-11T20:45:51.804Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
113632026-03-11T20:45:51.804Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
113642026-03-11T20:45:51.804Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
113652026-03-11T20:45:51.804Z
113662026-03-11T20:45:51.804Zgoroutine 8997 [select]:
113672026-03-11T20:45:51.804Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.(*Watcher).Run(0xc0023b94d0, {0x50b1338, 0xc000ebc690})
113682026-03-11T20:45:51.804Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:319 +0x955
113692026-03-11T20:45:51.804Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.Start.func1({0x50b1300?, 0xc006c0a7b0?})
113702026-03-11T20:45:51.804Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:189 +0x2d1
113712026-03-11T20:45:51.804Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
113722026-03-11T20:45:51.804Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
113732026-03-11T20:45:51.805Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
113742026-03-11T20:45:51.805Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
113752026-03-11T20:45:51.805Z
113762026-03-11T20:45:51.805Zgoroutine 10331 [semacquire, 59 minutes]:
113772026-03-11T20:45:51.805Zsync.runtime_Semacquire(0xc004f352c0?)
113782026-03-11T20:45:51.805Z /work/go/src/runtime/sema.go:62 +0x25
113792026-03-11T20:45:51.805Zsync.(*WaitGroup).Wait(0x44e7a0?)
113802026-03-11T20:45:51.805Z /work/go/src/sync/waitgroup.go:116 +0x48
113812026-03-11T20:45:51.805Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc005d7af00)
113822026-03-11T20:45:51.805Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
113832026-03-11T20:45:51.805Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc005d7af00?, {0x50b1338?, 0xc007611c70?}})
113842026-03-11T20:45:51.805Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
113852026-03-11T20:45:51.805Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait({0x50b1300?, 0xc003457ad0?}, {0xc0059abd78, 0x2, 0xc0059abc70?})
113862026-03-11T20:45:51.805Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:193 +0xff
113872026-03-11T20:45:51.805Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0xc00356a580, {0x50b1300, 0xc003457ad0}, 0xc003457800)
113882026-03-11T20:45:51.805Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:291 +0x59c
113892026-03-11T20:45:51.805Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).Start.func2({0x50b1300, 0xc0034579e0})
113902026-03-11T20:45:51.805Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:215 +0x2f3
113912026-03-11T20:45:51.805Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
113922026-03-11T20:45:51.805Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
113932026-03-11T20:45:51.805Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10330
113942026-03-11T20:45:51.805Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
113952026-03-11T20:45:51.805Z
113962026-03-11T20:45:51.805Zgoroutine 10333 [chan receive, 59 minutes]:
113972026-03-11T20:45:51.805Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).rangeFeedWithRangeID(0xc001ded408, 0x3f3de40?, 0xc000d7a120, {0x7fc12cefd140, 0xc004052300})
113982026-03-11T20:45:51.805Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:249 +0x76b
113992026-03-11T20:45:51.805Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).RangeFeed(...)
114002026-03-11T20:45:51.805Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/replica_rangefeed.go:149
114012026-03-11T20:45:51.805Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).RangeFeed(0xc000f39208, 0xc000d7a120, {0x50e5b30, 0xc004052300})
114022026-03-11T20:45:51.805Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/store.go:3082 +0x16f
114032026-03-11T20:45:51.810Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).RangeFeed(0xc001eb4380, 0xc000d7a120, {0x50e5b30, 0xc004052300})
114042026-03-11T20:45:51.810Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/stores.go:216 +0x19a
114052026-03-11T20:45:51.810Zgithub.com/cockroachdb/cockroach/pkg/server.(*Node).RangeFeed(0x3f3de40?, 0xc0022ba530?, {0x50e5b30, 0xc004052300})
114062026-03-11T20:45:51.810Z /work/oxidecomputer/cockroach/pkg/server/node.go:1232 +0x28
114072026-03-11T20:45:51.810Zgithub.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed.func1()
114082026-03-11T20:45:51.810Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:753 +0xa7
114092026-03-11T20:45:51.810Zcreated by github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.RangeFeed in goroutine 10332
114102026-03-11T20:45:51.810Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:750 +0x17c
114112026-03-11T20:45:51.810Z
114122026-03-11T20:45:51.810Zgoroutine 10682 [chan receive, 59 minutes]:
114132026-03-11T20:45:51.810Zgithub.com/cockroachdb/cockroach/pkg/server.configureGRPCGateway.func1({0xc006e55508?, 0xc003a4b110?})
114142026-03-11T20:45:51.810Z /work/oxidecomputer/cockroach/pkg/server/grpc_gateway.go:86 +0x31
114152026-03-11T20:45:51.810Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
114162026-03-11T20:45:51.810Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
114172026-03-11T20:45:51.810Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
114182026-03-11T20:45:51.810Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
114192026-03-11T20:45:51.810Z
114202026-03-11T20:45:51.810Zgoroutine 10327 [select, 59 minutes]:
114212026-03-11T20:45:51.810Zgithub.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func9({0x50b1300?, 0xc0034572c0?})
114222026-03-11T20:45:51.810Z /work/oxidecomputer/cockroach/pkg/jobs/registry.go:843 +0x1be
114232026-03-11T20:45:51.810Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
114242026-03-11T20:45:51.810Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
114252026-03-11T20:45:51.810Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
114262026-03-11T20:45:51.810Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
114272026-03-11T20:45:51.810Z
114282026-03-11T20:45:51.810Zgoroutine 10808 [sync.Cond.Wait, 59 minutes]:
114292026-03-11T20:45:51.810Zsync.runtime_notifyListWait(0xc0064c5050, 0x52)
114302026-03-11T20:45:51.810Z /work/go/src/runtime/sema.go:569 +0x159
114312026-03-11T20:45:51.811Zsync.(*Cond).Wait(0x0?)
114322026-03-11T20:45:51.811Z /work/go/src/sync/cond.go:70 +0x85
114332026-03-11T20:45:51.811Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3c20})
114342026-03-11T20:45:51.811Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
114352026-03-11T20:45:51.811Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
114362026-03-11T20:45:51.811Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
114372026-03-11T20:45:51.812Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
114382026-03-11T20:45:51.812Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
114392026-03-11T20:45:51.812Z
114402026-03-11T20:45:51.812Zgoroutine 10330 [select]:
114412026-03-11T20:45:51.812Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.(*Watcher).Run(0xc002d04000, {0x50b1338, 0xc0022ba320})
114422026-03-11T20:45:51.812Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:319 +0x955
114432026-03-11T20:45:51.812Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.Start.func1({0x50b1300?, 0xc0034577a0?})
114442026-03-11T20:45:51.812Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:189 +0x2d1
114452026-03-11T20:45:51.812Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
114462026-03-11T20:45:51.812Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
114472026-03-11T20:45:51.812Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
114482026-03-11T20:45:51.812Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
114492026-03-11T20:45:51.812Z
114502026-03-11T20:45:51.812Zgoroutine 10329 [select, 9 minutes]:
114512026-03-11T20:45:51.812Zgithub.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigmanager.(*Manager).run(0xc00438e5a0, {0x50b1300, 0xc003457350})
114522026-03-11T20:45:51.812Z /work/oxidecomputer/cockroach/pkg/spanconfig/spanconfigmanager/manager.go:159 +0x3ba
114532026-03-11T20:45:51.812Zgithub.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigmanager.(*Manager).Start.func1({0x50b1300?, 0xc003457350?})
114542026-03-11T20:45:51.812Z /work/oxidecomputer/cockroach/pkg/spanconfig/spanconfigmanager/manager.go:101 +0x2a
114552026-03-11T20:45:51.812Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
114562026-03-11T20:45:51.812Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
114572026-03-11T20:45:51.812Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
114582026-03-11T20:45:51.813Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
114592026-03-11T20:45:51.813Z
114602026-03-11T20:45:51.813Zgoroutine 9997 [IO wait, 59 minutes]:
114612026-03-11T20:45:51.813Zinternal/poll.runtime_pollWait(0x7fc12c3bdb18, 0x72)
114622026-03-11T20:45:51.813Z /work/go/src/runtime/netpoll.go:345 +0x85
114632026-03-11T20:45:51.813Zinternal/poll.(*pollDesc).wait(0x8?, 0x1?, 0x0)
114642026-03-11T20:45:51.813Z /work/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
114652026-03-11T20:45:51.813Zinternal/poll.(*pollDesc).waitRead(...)
114662026-03-11T20:45:51.813Z /work/go/src/internal/poll/fd_poll_runtime.go:89
114672026-03-11T20:45:51.813Zinternal/poll.(*FD).Accept(0xc003f40380)
114682026-03-11T20:45:51.813Z /work/go/src/internal/poll/fd_unix.go:611 +0x2ac
114692026-03-11T20:45:51.813Znet.(*netFD).accept(0xc003f40380)
114702026-03-11T20:45:51.813Z /work/go/src/net/fd_unix.go:172 +0x29
114712026-03-11T20:45:51.813Znet.(*TCPListener).accept(0xc0079ce5c0)
114722026-03-11T20:45:51.813Z /work/go/src/net/tcpsock_posix.go:159 +0x1e
114732026-03-11T20:45:51.813Znet.(*TCPListener).Accept(0xc0079ce5c0)
114742026-03-11T20:45:51.813Z /work/go/src/net/tcpsock.go:327 +0x30
114752026-03-11T20:45:51.813Zgithub.com/cockroachdb/cmux.(*cMux).Serve(0xc0034cbc00)
114762026-03-11T20:45:51.813Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:124 +0x97
114772026-03-11T20:45:51.813Zgithub.com/cockroachdb/cockroach/pkg/server.startListenRPCAndSQL.func5.2.1()
114782026-03-11T20:45:51.813Z /work/oxidecomputer/cockroach/pkg/server/start_listen.go:139 +0x27
114792026-03-11T20:45:51.813Zsync.(*Once).doSlow(0xc0018b66d8?, 0xc0018b6718?)
114802026-03-11T20:45:51.813Z /work/go/src/sync/once.go:74 +0xc2
114812026-03-11T20:45:51.813Zsync.(*Once).Do(...)
114822026-03-11T20:45:51.813Z /work/go/src/sync/once.go:65
114832026-03-11T20:45:51.813Zgithub.com/cockroachdb/cockroach/pkg/server.startListenRPCAndSQL.func5.2({0xc0022c2908?, 0x0?})
114842026-03-11T20:45:51.813Z /work/oxidecomputer/cockroach/pkg/server/start_listen.go:138 +0x4f
114852026-03-11T20:45:51.813Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
114862026-03-11T20:45:51.813Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
114872026-03-11T20:45:51.813Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
114882026-03-11T20:45:51.813Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
114892026-03-11T20:45:51.813Z
114902026-03-11T20:45:51.813Zgoroutine 10324 [select, 59 minutes]:
114912026-03-11T20:45:51.813Zgithub.com/cockroachdb/cockroach/pkg/sql/catalog/lease.(*Manager).RefreshLeases.func1({0x50b1300, 0xc0034570b0})
114922026-03-11T20:45:51.813Z /work/oxidecomputer/cockroach/pkg/sql/catalog/lease/lease.go:1041 +0x105
114932026-03-11T20:45:51.813Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
114942026-03-11T20:45:51.813Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
114952026-03-11T20:45:51.813Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
114962026-03-11T20:45:51.814Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
114972026-03-11T20:45:51.814Z
114982026-03-11T20:45:51.814Zgoroutine 10489 [select]:
114992026-03-11T20:45:51.814Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run(0xc006014a80, {0x50b1300?, 0xc0070e36e0?}, 0x2b?, 0xc007ee01e0, 0xc003f40400)
115002026-03-11T20:45:51.814Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:269 +0x4dc
115012026-03-11T20:45:51.814Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).Start.func1({0x50b1300?, 0xc0070e36e0?})
115022026-03-11T20:45:51.814Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:222 +0x36
115032026-03-11T20:45:51.819Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
115042026-03-11T20:45:51.819Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
115052026-03-11T20:45:51.819Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10488
115062026-03-11T20:45:51.819Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
115072026-03-11T20:45:51.819Z
115082026-03-11T20:45:51.819Zgoroutine 10428 [semacquire, 59 minutes]:
115092026-03-11T20:45:51.819Zsync.runtime_Semacquire(0xc004f35320?)
115102026-03-11T20:45:51.819Z /work/go/src/runtime/sema.go:62 +0x25
115112026-03-11T20:45:51.819Zsync.(*WaitGroup).Wait(0x44e7a0?)
115122026-03-11T20:45:51.819Z /work/go/src/sync/waitgroup.go:116 +0x48
115132026-03-11T20:45:51.820Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc005d7af40)
115142026-03-11T20:45:51.820Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
115152026-03-11T20:45:51.820Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc005d7af40?, {0x50b1338?, 0xc007611e00?}})
115162026-03-11T20:45:51.820Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
115172026-03-11T20:45:51.820Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed(0xc002d6f408, {0x50b1338?, 0xc007611c70?}, {0xc0034578f0, 0x1, 0x1d2a580?}, {0xc007ee04c8?, 0x60cc6b8?, 0xc0?}, 0x0, ...)
115182026-03-11T20:45:51.820Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:139 +0x6c5
115192026-03-11T20:45:51.820Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*dbAdapter).RangeFeed(0x50b1300?, {0x50b1338?, 0xc007611c70?}, {0xc0034578f0?, 0x50b1300?, 0xc0042308d0?}, {0x707000000000001?, 0x50b1300?, 0x0?}, 0x0, ...)
115202026-03-11T20:45:51.820Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/db_adapter.go:78 +0x3e
115212026-03-11T20:45:51.820Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func1({0x50b1338?, 0xc007611c70?})
115222026-03-11T20:45:51.820Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:285 +0x6c
115232026-03-11T20:45:51.820Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
115242026-03-11T20:45:51.820Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
115252026-03-11T20:45:51.821Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
115262026-03-11T20:45:51.821Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
115272026-03-11T20:45:51.821Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10331
115282026-03-11T20:45:51.821Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
115292026-03-11T20:45:51.821Z
115302026-03-11T20:45:51.821Zgoroutine 10681 [chan receive, 59 minutes]:
115312026-03-11T20:45:51.821Zgithub.com/cockroachdb/cockroach/pkg/server.startListenRPCAndSQL.func3({0xc006e55208?, 0xc0066f4990?})
115322026-03-11T20:45:51.821Z /work/oxidecomputer/cockroach/pkg/server/start_listen.go:108 +0x3a
115332026-03-11T20:45:51.821Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
115342026-03-11T20:45:51.821Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
115352026-03-11T20:45:51.821Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
115362026-03-11T20:45:51.821Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
115372026-03-11T20:45:51.821Z
115382026-03-11T20:45:51.821Zgoroutine 10197 [select]:
115392026-03-11T20:45:51.821Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.(*Watcher).Run(0xc0023b9560, {0x50b1338, 0xc0023400f0})
115402026-03-11T20:45:51.821Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:319 +0x955
115412026-03-11T20:45:51.821Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.Start.func1({0x50b1300?, 0xc0018cfef0?})
115422026-03-11T20:45:51.822Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:189 +0x2d1
115432026-03-11T20:45:51.822Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
115442026-03-11T20:45:51.822Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
115452026-03-11T20:45:51.822Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
115462026-03-11T20:45:51.822Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
115472026-03-11T20:45:51.822Z
115482026-03-11T20:45:51.822Zgoroutine 9777 [select]:
115492026-03-11T20:45:51.822Zgithub.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run(0xc002c1a2c0, {0x50b1300, 0xc007b27530})
115502026-03-11T20:45:51.822Z /work/oxidecomputer/cockroach/pkg/internal/client/requestbatcher/batcher.go:475 +0x326
115512026-03-11T20:45:51.822Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
115522026-03-11T20:45:51.822Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
115532026-03-11T20:45:51.822Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
115542026-03-11T20:45:51.822Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
115552026-03-11T20:45:51.822Z
115562026-03-11T20:45:51.822Zgoroutine 8971 [chan receive, 59 minutes]:
115572026-03-11T20:45:51.822Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc007582090?})
115582026-03-11T20:45:51.822Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
115592026-03-11T20:45:51.822Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x1000000010000?, 0xc002270a50?}}, 0xc0004e37a8)
115602026-03-11T20:45:51.822Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
115612026-03-11T20:45:51.822Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
115622026-03-11T20:45:51.822Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
115632026-03-11T20:45:51.822Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
115642026-03-11T20:45:51.822Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
115652026-03-11T20:45:51.822Z
115662026-03-11T20:45:51.822Zgoroutine 8969 [select, 59 minutes]:
115672026-03-11T20:45:51.822Zgithub.com/cockroachdb/cockroach/pkg/util/log.(*fileSink).gcDaemon(0xc0069384e0, {0x50b1338, 0xc006cfc460})
115682026-03-11T20:45:51.822Z /work/oxidecomputer/cockroach/pkg/util/log/file_log_gc.go:25 +0x79
115692026-03-11T20:45:51.822Zcreated by github.com/cockroachdb/cockroach/pkg/util/log.ApplyConfig in goroutine 9708
115702026-03-11T20:45:51.822Z /work/oxidecomputer/cockroach/pkg/util/log/flags.go:293 +0x10f5
115712026-03-11T20:45:51.822Z
115722026-03-11T20:45:51.822Zgoroutine 9781 [chan receive, 59 minutes]:
115732026-03-11T20:45:51.822Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc005572060?})
115742026-03-11T20:45:51.822Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
115752026-03-11T20:45:51.822Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0xc000f95b00?, 0xc000f95b40?}}, 0xc002becfa8)
115762026-03-11T20:45:51.822Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
115772026-03-11T20:45:51.822Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
115782026-03-11T20:45:51.822Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
115792026-03-11T20:45:51.823Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
115802026-03-11T20:45:51.823Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
115812026-03-11T20:45:51.823Z
115822026-03-11T20:45:51.823Zgoroutine 10323 [semacquire, 59 minutes]:
115832026-03-11T20:45:51.823Zsync.runtime_Semacquire(0xc003452030?)
115842026-03-11T20:45:51.823Z /work/go/src/runtime/sema.go:62 +0x25
115852026-03-11T20:45:51.823Zsync.(*WaitGroup).Wait(0x44e7a0?)
115862026-03-11T20:45:51.823Z /work/go/src/sync/waitgroup.go:116 +0x48
115872026-03-11T20:45:51.823Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc00527d5c0)
115882026-03-11T20:45:51.823Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
115892026-03-11T20:45:51.823Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc00527d5c0?, {0x50b1338?, 0xc0023407d0?}})
115902026-03-11T20:45:51.823Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
115912026-03-11T20:45:51.823Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait({0x50b1300?, 0xc00296f3e0?}, {0xc003bc0d78, 0x2, 0x4000000000000000?})
115922026-03-11T20:45:51.823Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:193 +0xff
115932026-03-11T20:45:51.823Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0xc00356a420, {0x50b1300, 0xc00296f3e0}, 0xc003456e10)
115942026-03-11T20:45:51.823Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:291 +0x59c
115952026-03-11T20:45:51.823Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).Start.func2({0x50b1300, 0xc003456fc0})
115962026-03-11T20:45:51.823Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:215 +0x2f3
115972026-03-11T20:45:51.823Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
115982026-03-11T20:45:51.823Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
115992026-03-11T20:45:51.823Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
116002026-03-11T20:45:51.823Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
116012026-03-11T20:45:51.823Z
116022026-03-11T20:45:51.823Zgoroutine 9995 [chan receive, 59 minutes]:
116032026-03-11T20:45:51.830Zgithub.com/cockroachdb/cockroach/pkg/server.configureGRPCGateway.func5({0x50b1300, 0xc0073f2540})
116042026-03-11T20:45:51.830Z /work/oxidecomputer/cockroach/pkg/server/grpc_gateway.go:127 +0x45
116052026-03-11T20:45:51.830Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
116062026-03-11T20:45:51.830Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
116072026-03-11T20:45:51.830Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
116082026-03-11T20:45:51.830Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
116092026-03-11T20:45:51.830Z
116102026-03-11T20:45:51.830Zgoroutine 10328 [select]:
116112026-03-11T20:45:51.830Zgithub.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func10({0x50b1300?, 0xc003457320?})
116122026-03-11T20:45:51.830Z /work/oxidecomputer/cockroach/pkg/jobs/registry.go:866 +0x1bd
116132026-03-11T20:45:51.830Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
116142026-03-11T20:45:51.830Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
116152026-03-11T20:45:51.830Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
116162026-03-11T20:45:51.830Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
116172026-03-11T20:45:51.830Z
116182026-03-11T20:45:51.830Zgoroutine 8972 [chan receive, 59 minutes]:
116192026-03-11T20:45:51.830Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc0075820f0?})
116202026-03-11T20:45:51.830Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
116212026-03-11T20:45:51.830Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0xc0022598c0?, 0xc002259900?}}, 0xc0004e3fa8)
116222026-03-11T20:45:51.830Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
116232026-03-11T20:45:51.830Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
116242026-03-11T20:45:51.830Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
116252026-03-11T20:45:51.830Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
116262026-03-11T20:45:51.830Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
116272026-03-11T20:45:51.830Z
116282026-03-11T20:45:51.830Zgoroutine 10852 [IO wait]:
116292026-03-11T20:45:51.830Zinternal/poll.runtime_pollWait(0x7fc12eba3840, 0x72)
116302026-03-11T20:45:51.830Z /work/go/src/runtime/netpoll.go:345 +0x85
116312026-03-11T20:45:51.830Zinternal/poll.(*pollDesc).wait(0xc007878180?, 0xc0061fce00?, 0x0)
116322026-03-11T20:45:51.830Z /work/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
116332026-03-11T20:45:51.830Zinternal/poll.(*pollDesc).waitRead(...)
116342026-03-11T20:45:51.830Z /work/go/src/internal/poll/fd_poll_runtime.go:89
116352026-03-11T20:45:51.831Zinternal/poll.(*FD).Read(0xc007878180, {0xc0061fce00, 0x700, 0x700})
116362026-03-11T20:45:51.831Z /work/go/src/internal/poll/fd_unix.go:164 +0x27a
116372026-03-11T20:45:51.831Znet.(*netFD).Read(0xc007878180, {0xc0061fce00?, 0x7fc12c37d2e0?, 0xc01b084ed0?})
116382026-03-11T20:45:51.831Z /work/go/src/net/fd_posix.go:55 +0x25
116392026-03-11T20:45:51.831Znet.(*conn).Read(0xc004e7c000, {0xc0061fce00?, 0xc005aa08e0?, 0x415dbb?})
116402026-03-11T20:45:51.832Z /work/go/src/net/net.go:185 +0x45
116412026-03-11T20:45:51.832Zcrypto/tls.(*atLeastReader).Read(0xc01b084ed0, {0xc0061fce00?, 0x0?, 0xc01b084ed0?})
116422026-03-11T20:45:51.832Z /work/go/src/crypto/tls/conn.go:806 +0x3b
116432026-03-11T20:45:51.832Zbytes.(*Buffer).ReadFrom(0xc002ce2630, {0x5081fe0, 0xc01b084ed0})
116442026-03-11T20:45:51.832Z /work/go/src/bytes/buffer.go:211 +0x98
116452026-03-11T20:45:51.832Zcrypto/tls.(*Conn).readFromUntil(0xc002ce2388, {0x50821c0, 0xc004e7c000}, 0xc005aa0928?)
116462026-03-11T20:45:51.832Z /work/go/src/crypto/tls/conn.go:828 +0xde
116472026-03-11T20:45:51.832Zcrypto/tls.(*Conn).readRecordOrCCS(0xc002ce2388, 0x0)
116482026-03-11T20:45:51.832Z /work/go/src/crypto/tls/conn.go:626 +0x3cf
116492026-03-11T20:45:51.832Zcrypto/tls.(*Conn).readRecord(...)
116502026-03-11T20:45:51.832Z /work/go/src/crypto/tls/conn.go:588
116512026-03-11T20:45:51.832Zcrypto/tls.(*Conn).Read(0xc002ce2388, {0xc000718000, 0x8000, 0xc008f03560?})
116522026-03-11T20:45:51.832Z /work/go/src/crypto/tls/conn.go:1370 +0x156
116532026-03-11T20:45:51.832Zbufio.(*Reader).Read(0xc005ff9b00, {0xc000d38660, 0x9, 0x0?})
116542026-03-11T20:45:51.832Z /work/go/src/bufio/bufio.go:241 +0x197
116552026-03-11T20:45:51.832Zio.ReadAtLeast({0x507c320, 0xc005ff9b00}, {0xc000d38660, 0x9, 0x9}, 0x9)
116562026-03-11T20:45:51.832Z /work/go/src/io/io.go:335 +0x90
116572026-03-11T20:45:51.832Zio.ReadFull(...)
116582026-03-11T20:45:51.832Z /work/go/src/io/io.go:354
116592026-03-11T20:45:51.832Zgolang.org/x/net/http2.readFrameHeader({0xc000d38660, 0x9, 0x0?}, {0x507c320?, 0xc005ff9b00?})
116602026-03-11T20:45:51.832Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x65
116612026-03-11T20:45:51.832Zgolang.org/x/net/http2.(*Framer).ReadFrame(0xc000d38620)
116622026-03-11T20:45:51.832Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:501 +0x85
116632026-03-11T20:45:51.832Zgoogle.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0066c21e0)
116642026-03-11T20:45:51.832Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1498 +0x41d
116652026-03-11T20:45:51.832Zcreated by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 9758
116662026-03-11T20:45:51.832Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:365 +0x184d
116672026-03-11T20:45:51.832Z
116682026-03-11T20:45:51.832Zgoroutine 9800 [select]:
116692026-03-11T20:45:51.832Zgithub.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue.func2()
116702026-03-11T20:45:51.833Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:366 +0x85
116712026-03-11T20:45:51.833Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.makeWorkQueue in goroutine 9708
116722026-03-11T20:45:51.833Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:363 +0x3ed
116732026-03-11T20:45:51.833Z
116742026-03-11T20:45:51.833Zgoroutine 9937 [select]:
116752026-03-11T20:45:51.833Zgithub.com/cockroachdb/cockroach/pkg/sql/stats.(*Refresher).Start.func1({0x50b1300, 0xc003456d20})
116762026-03-11T20:45:51.833Z /work/oxidecomputer/cockroach/pkg/sql/stats/automatic_stats.go:374 +0x1b9
116772026-03-11T20:45:51.833Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
116782026-03-11T20:45:51.833Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
116792026-03-11T20:45:51.833Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
116802026-03-11T20:45:51.833Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
116812026-03-11T20:45:51.833Z
116822026-03-11T20:45:51.834Zgoroutine 9989 [chan receive, 59 minutes]:
116832026-03-11T20:45:51.834Zgithub.com/cockroachdb/cockroach/pkg/server.startListenRPCAndSQL.func1({0x50b1300, 0xc0073f2030})
116842026-03-11T20:45:51.834Z /work/oxidecomputer/cockroach/pkg/server/start_listen.go:64 +0x45
116852026-03-11T20:45:51.834Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
116862026-03-11T20:45:51.834Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
116872026-03-11T20:45:51.834Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
116882026-03-11T20:45:51.834Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
116892026-03-11T20:45:51.834Z
116902026-03-11T20:45:51.834Zgoroutine 10807 [sync.Cond.Wait, 59 minutes]:
116912026-03-11T20:45:51.834Zsync.runtime_notifyListWait(0xc0064c5050, 0x4f)
116922026-03-11T20:45:51.834Z /work/go/src/runtime/sema.go:569 +0x159
116932026-03-11T20:45:51.834Zsync.(*Cond).Wait(0x0?)
116942026-03-11T20:45:51.834Z /work/go/src/sync/cond.go:70 +0x85
116952026-03-11T20:45:51.834Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3bf0})
116962026-03-11T20:45:51.834Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
116972026-03-11T20:45:51.834Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
116982026-03-11T20:45:51.834Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
116992026-03-11T20:45:51.834Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
117002026-03-11T20:45:51.834Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
117012026-03-11T20:45:51.834Z
117022026-03-11T20:45:51.834Zgoroutine 11133 [semacquire, 59 minutes]:
117032026-03-11T20:45:51.841Zsync.runtime_Semacquire(0xc005dbfd40?)
117042026-03-11T20:45:51.841Z /work/go/src/runtime/sema.go:62 +0x25
117052026-03-11T20:45:51.841Zsync.(*WaitGroup).Wait(0x44e7a0?)
117062026-03-11T20:45:51.841Z /work/go/src/sync/waitgroup.go:116 +0x48
117072026-03-11T20:45:51.841Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc000e48800)
117082026-03-11T20:45:51.841Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
117092026-03-11T20:45:51.841Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc000e48800?, {0x50b1338?, 0xc0010bd310?}})
117102026-03-11T20:45:51.841Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
117112026-03-11T20:45:51.841Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeed(0xc003d48008, {0x50b1338?, 0xc0010bcc30?}, {0xc0055739e0, 0x1, 0xc001b78808?}, {0xc007661ad0?, 0x55b21e1?, 0xc0?}, 0x0, ...)
117122026-03-11T20:45:51.841Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:139 +0x6c5
117132026-03-11T20:45:51.841Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*dbAdapter).RangeFeed(0x15d98bf63f157f?, {0x50b1338?, 0xc0010bcc30?}, {0xc0055739e0?, 0x0?, 0xc003152008?}, {0x50b1300?, 0x6eaf110?, 0xc0?}, 0x0, ...)
117142026-03-11T20:45:51.842Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/db_adapter.go:78 +0x3e
117152026-03-11T20:45:51.842Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func1({0x50b1338?, 0xc0010bcc30?})
117162026-03-11T20:45:51.842Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:285 +0x6c
117172026-03-11T20:45:51.842Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
117182026-03-11T20:45:51.842Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
117192026-03-11T20:45:51.842Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
117202026-03-11T20:45:51.842Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
117212026-03-11T20:45:51.842Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 10934
117222026-03-11T20:45:51.842Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
117232026-03-11T20:45:51.842Z
117242026-03-11T20:45:51.842Zgoroutine 9797 [select]:
117252026-03-11T20:45:51.842Zgithub.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs.func1()
117262026-03-11T20:45:51.842Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:439 +0x1c9
117272026-03-11T20:45:51.842Zcreated by github.com/cockroachdb/cockroach/pkg/util/admission.(*WorkQueue).startClosingEpochs in goroutine 9708
117282026-03-11T20:45:51.842Z /work/oxidecomputer/cockroach/pkg/util/admission/work_queue.go:412 +0x4f
117292026-03-11T20:45:51.842Z
117302026-03-11T20:45:51.842Zgoroutine 10179 [semacquire, 59 minutes]:
117312026-03-11T20:45:51.842Zsync.runtime_Semacquire(0xc0034d77d0?)
117322026-03-11T20:45:51.842Z /work/go/src/runtime/sema.go:62 +0x25
117332026-03-11T20:45:51.842Zsync.(*WaitGroup).Wait(0x44e7a0?)
117342026-03-11T20:45:51.842Z /work/go/src/sync/waitgroup.go:116 +0x48
117352026-03-11T20:45:51.842Zgolang.org/x/sync/errgroup.(*Group).Wait(0xc004827f00)
117362026-03-11T20:45:51.842Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:56 +0x25
117372026-03-11T20:45:51.842Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.Wait({0xc004827f00?, {0x50b1338?, 0xc003eef5e0?}})
117382026-03-11T20:45:51.842Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:145 +0x47
117392026-03-11T20:45:51.842Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait({0x50b1300?, 0xc0040cb8c0?}, {0xc0058edd78, 0x2, 0xc0058edc70?})
117402026-03-11T20:45:51.842Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:193 +0xff
117412026-03-11T20:45:51.842Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run(0xc002c1b1e0, {0x50b1300, 0xc0040cb8c0}, 0xc0040cb6b0)
117422026-03-11T20:45:51.842Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:291 +0x59c
117432026-03-11T20:45:51.842Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).Start.func2({0x50b1300, 0xc0040cb800})
117442026-03-11T20:45:51.842Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:215 +0x2f3
117452026-03-11T20:45:51.842Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
117462026-03-11T20:45:51.843Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
117472026-03-11T20:45:51.843Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10197
117482026-03-11T20:45:51.843Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
117492026-03-11T20:45:51.843Z
117502026-03-11T20:45:51.843Zgoroutine 9832 [select, 59 minutes]:
117512026-03-11T20:45:51.843Zgithub.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1({0xc005956908?, 0xc004f0bce0?})
117522026-03-11T20:45:51.843Z /work/oxidecomputer/cockroach/pkg/sql/distsql_running.go:113 +0xf7
117532026-03-11T20:45:51.843Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
117542026-03-11T20:45:51.843Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
117552026-03-11T20:45:51.843Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
117562026-03-11T20:45:51.843Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
117572026-03-11T20:45:51.843Z
117582026-03-11T20:45:51.843Zgoroutine 9793 [chan receive, 59 minutes]:
117592026-03-11T20:45:51.843Zgithub.com/cockroachdb/cockroach/pkg/rpc.NewContext.func2({0xc00588c608?, 0x0?})
117602026-03-11T20:45:51.844Z /work/oxidecomputer/cockroach/pkg/rpc/context.go:541 +0x3f
117612026-03-11T20:45:51.844Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
117622026-03-11T20:45:51.844Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
117632026-03-11T20:45:51.844Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
117642026-03-11T20:45:51.844Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
117652026-03-11T20:45:51.844Z
117662026-03-11T20:45:51.844Zgoroutine 10077 [sync.Cond.Wait]:
117672026-03-11T20:45:51.844Zsync.runtime_notifyListWait(0xc001e22250, 0x378b)
117682026-03-11T20:45:51.844Z /work/go/src/runtime/sema.go:569 +0x159
117692026-03-11T20:45:51.844Zsync.(*Cond).Wait(0x3aac600?)
117702026-03-11T20:45:51.844Z /work/go/src/sync/cond.go:70 +0x85
117712026-03-11T20:45:51.844Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583950})
117722026-03-11T20:45:51.844Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
117732026-03-11T20:45:51.844Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
117742026-03-11T20:45:51.844Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
117752026-03-11T20:45:51.844Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
117762026-03-11T20:45:51.844Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
117772026-03-11T20:45:51.844Z
117782026-03-11T20:45:51.844Zgoroutine 8975 [chan receive, 59 minutes]:
117792026-03-11T20:45:51.844Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop.func1({0x50b1300?, 0xc007582210?})
117802026-03-11T20:45:51.844Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:300 +0x55
117812026-03-11T20:45:51.845Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fbb00?, 0x0?, 0x0?}}, 0xc002be67a8)
117822026-03-11T20:45:51.845Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
117832026-03-11T20:45:51.845Zgithub.com/cockroachdb/pebble.(*tableCacheShard).releaseLoop(0x0?)
117842026-03-11T20:45:51.845Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:299 +0x52
117852026-03-11T20:45:51.845Zcreated by github.com/cockroachdb/pebble.(*tableCacheShard).init in goroutine 9708
117862026-03-11T20:45:51.845Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/table_cache.go:291 +0xd6
117872026-03-11T20:45:51.845Z
117882026-03-11T20:45:51.845Zgoroutine 9924 [select]:
117892026-03-11T20:45:51.845Zgithub.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1()
117902026-03-11T20:45:51.845Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:71 +0xcb
117912026-03-11T20:45:51.845Zcreated by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 9708
117922026-03-11T20:45:51.845Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:66 +0x58
117932026-03-11T20:45:51.845Z
117942026-03-11T20:45:51.845Zgoroutine 10664 [select]:
117952026-03-11T20:45:51.845Zgoogle.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc0067249c0)
117962026-03-11T20:45:51.845Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:1131 +0x205
117972026-03-11T20:45:51.845Zcreated by google.golang.org/grpc/internal/transport.NewServerTransport in goroutine 10600
117982026-03-11T20:45:51.845Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:341 +0x188e
117992026-03-11T20:45:51.845Z
118002026-03-11T20:45:51.845Zgoroutine 10654 [select]:
118012026-03-11T20:45:51.845Znet.(*pipe).read(0xc006756800, {0xc004b36a00, 0x700, 0x5?})
118022026-03-11T20:45:51.845Z /work/go/src/net/pipe.go:159 +0x155
118032026-03-11T20:45:51.849Znet.(*pipe).Read(0xc0066d9dc0?, {0xc004b36a00?, 0x90?, 0x515b808?})
118042026-03-11T20:45:51.850Z /work/go/src/net/pipe.go:142 +0x1c
118052026-03-11T20:45:51.850Zcrypto/tls.(*atLeastReader).Read(0xc02928da28, {0xc004b36a00?, 0x0?, 0xc02928da28?})
118062026-03-11T20:45:51.850Z /work/go/src/crypto/tls/conn.go:806 +0x3b
118072026-03-11T20:45:51.850Zbytes.(*Buffer).ReadFrom(0xc0066d9eb0, {0x5081fe0, 0xc02928da28})
118082026-03-11T20:45:51.850Z /work/go/src/bytes/buffer.go:211 +0x98
118092026-03-11T20:45:51.850Zcrypto/tls.(*Conn).readFromUntil(0xc0066d9c08, {0x7fc12c3c80b0, 0xc006756800}, 0xc005a9a958?)
118102026-03-11T20:45:51.850Z /work/go/src/crypto/tls/conn.go:828 +0xde
118112026-03-11T20:45:51.850Zcrypto/tls.(*Conn).readRecordOrCCS(0xc0066d9c08, 0x0)
118122026-03-11T20:45:51.850Z /work/go/src/crypto/tls/conn.go:626 +0x3cf
118132026-03-11T20:45:51.850Zcrypto/tls.(*Conn).readRecord(...)
118142026-03-11T20:45:51.850Z /work/go/src/crypto/tls/conn.go:588
118152026-03-11T20:45:51.850Zcrypto/tls.(*Conn).Read(0xc0066d9c08, {0xc006f5a000, 0x8000, 0xc005a9aca8?})
118162026-03-11T20:45:51.850Z /work/go/src/crypto/tls/conn.go:1370 +0x156
118172026-03-11T20:45:51.850Zbufio.(*Reader).Read(0xc006233aa0, {0xc0034ceac0, 0x9, 0xc005a9ad30?})
118182026-03-11T20:45:51.850Z /work/go/src/bufio/bufio.go:241 +0x197
118192026-03-11T20:45:51.850Zio.ReadAtLeast({0x507c320, 0xc006233aa0}, {0xc0034ceac0, 0x9, 0x9}, 0x9)
118202026-03-11T20:45:51.850Z /work/go/src/io/io.go:335 +0x90
118212026-03-11T20:45:51.850Zio.ReadFull(...)
118222026-03-11T20:45:51.850Z /work/go/src/io/io.go:354
118232026-03-11T20:45:51.850Zgolang.org/x/net/http2.readFrameHeader({0xc0034ceac0, 0x9, 0x3c27d00?}, {0x507c320?, 0xc006233aa0?})
118242026-03-11T20:45:51.850Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x65
118252026-03-11T20:45:51.850Zgolang.org/x/net/http2.(*Framer).ReadFrame(0xc0034cea80)
118262026-03-11T20:45:51.850Z /work/oxidecomputer/cockroach/vendor/golang.org/x/net/http2/frame.go:501 +0x85
118272026-03-11T20:45:51.850Zgoogle.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0066c25a0)
118282026-03-11T20:45:51.850Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1498 +0x41d
118292026-03-11T20:45:51.850Zcreated by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 10685
118302026-03-11T20:45:51.850Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:365 +0x184d
118312026-03-11T20:45:51.851Z
118322026-03-11T20:45:51.851Zgoroutine 10322 [select]:
118332026-03-11T20:45:51.851Zgithub.com/cockroachdb/cockroach/pkg/sql/stmtdiagnostics.(*Registry).poll(0xc000390310, {0x50b1300, 0xc003456db0})
118342026-03-11T20:45:51.851Z /work/oxidecomputer/cockroach/pkg/sql/stmtdiagnostics/statement_diagnostics.go:181 +0x2b5
118352026-03-11T20:45:51.851Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
118362026-03-11T20:45:51.851Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
118372026-03-11T20:45:51.851Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
118382026-03-11T20:45:51.851Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
118392026-03-11T20:45:51.851Z
118402026-03-11T20:45:51.851Zgoroutine 9710 [select]:
118412026-03-11T20:45:51.851Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).processEvents(0xc003283ce0, {0x50b1338, 0xc005d14230}, 0xc00485e720, 0xc00168a060)
118422026-03-11T20:45:51.852Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:331 +0xc5
118432026-03-11T20:45:51.852Zgithub.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed.(*RangeFeed).run.func2({0x50b1338?, 0xc005d14230?})
118442026-03-11T20:45:51.852Z /work/oxidecomputer/cockroach/pkg/kv/kvclient/rangefeed/rangefeed.go:288 +0x32
118452026-03-11T20:45:51.852Zgithub.com/cockroachdb/cockroach/pkg/util/ctxgroup.GoAndWait.Group.GoCtx.func1()
118462026-03-11T20:45:51.852Z /work/oxidecomputer/cockroach/pkg/util/ctxgroup/ctxgroup.go:169 +0x1f
118472026-03-11T20:45:51.852Zgolang.org/x/sync/errgroup.(*Group).Go.func1()
118482026-03-11T20:45:51.852Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:78 +0x56
118492026-03-11T20:45:51.852Zcreated by golang.org/x/sync/errgroup.(*Group).Go in goroutine 9851
118502026-03-11T20:45:51.852Z /work/oxidecomputer/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:75 +0x96
118512026-03-11T20:45:51.852Z
118522026-03-11T20:45:51.853Zgoroutine 9925 [sync.Cond.Wait, 59 minutes]:
118532026-03-11T20:45:51.853Zsync.runtime_notifyListWait(0xc0036eeae8, 0x0)
118542026-03-11T20:45:51.853Z /work/go/src/runtime/sema.go:569 +0x159
118552026-03-11T20:45:51.853Zsync.(*Cond).Wait(0x8?)
118562026-03-11T20:45:51.853Z /work/go/src/sync/cond.go:70 +0x85
118572026-03-11T20:45:51.853Zgithub.com/cockroachdb/pebble/record.(*flusherCond).Wait(...)
118582026-03-11T20:45:51.853Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/record/log_writer.go:193
118592026-03-11T20:45:51.853Zgithub.com/cockroachdb/pebble/record.(*LogWriter).flushLoop(0xc0036eea08, {0x3ac7a60?, 0x77577e0?})
118602026-03-11T20:45:51.853Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/record/log_writer.go:396 +0x475
118612026-03-11T20:45:51.853Zruntime/pprof.Do({0x50b1258?, 0x77577e0?}, {{0xc0004fba00?, 0x0?, 0x0?}}, 0xc004656fc0)
118622026-03-11T20:45:51.853Z /work/go/src/runtime/pprof/runtime.go:51 +0x9d
118632026-03-11T20:45:51.853Zgithub.com/cockroachdb/pebble/record.NewLogWriter.func2()
118642026-03-11T20:45:51.853Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/record/log_writer.go:314 +0x56
118652026-03-11T20:45:51.853Zcreated by github.com/cockroachdb/pebble/record.NewLogWriter in goroutine 9708
118662026-03-11T20:45:51.853Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/record/log_writer.go:313 +0x2c9
118672026-03-11T20:45:51.853Z
118682026-03-11T20:45:51.853Zgoroutine 10196 [select]:
118692026-03-11T20:45:51.854Zgithub.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigkvsubscriber.(*KVSubscriber).Start.func1({0x50b1300, 0xc0018cfec0})
118702026-03-11T20:45:51.854Z /work/oxidecomputer/cockroach/pkg/spanconfig/spanconfigkvsubscriber/kvsubscriber.go:268 +0x227
118712026-03-11T20:45:51.854Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
118722026-03-11T20:45:51.854Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
118732026-03-11T20:45:51.854Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
118742026-03-11T20:45:51.854Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
118752026-03-11T20:45:51.854Z
118762026-03-11T20:45:51.854Zgoroutine 9005 [select]:
118772026-03-11T20:45:51.854Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run(0xc001140d80, {0x50b1300?, 0xc002016ff0?}, 0x1c?, 0xc001b334a0, 0xc003f40400)
118782026-03-11T20:45:51.854Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:269 +0x4dc
118792026-03-11T20:45:51.854Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).Start.func1({0x50b1300?, 0xc002016ff0?})
118802026-03-11T20:45:51.854Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:222 +0x36
118812026-03-11T20:45:51.854Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
118822026-03-11T20:45:51.854Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
118832026-03-11T20:45:51.854Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9004
118842026-03-11T20:45:51.854Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
118852026-03-11T20:45:51.854Z
118862026-03-11T20:45:51.854Zgoroutine 10584 [IO wait, 59 minutes]:
118872026-03-11T20:45:51.854Zinternal/poll.runtime_pollWait(0x7fc12ce622f0, 0x72)
118882026-03-11T20:45:51.854Z /work/go/src/runtime/netpoll.go:345 +0x85
118892026-03-11T20:45:51.854Zinternal/poll.(*pollDesc).wait(0xc?, 0x0?, 0x0)
118902026-03-11T20:45:51.854Z /work/go/src/internal/poll/fd_poll_runtime.go:84 +0x27
118912026-03-11T20:45:51.855Zinternal/poll.(*pollDesc).waitRead(...)
118922026-03-11T20:45:51.855Z /work/go/src/internal/poll/fd_poll_runtime.go:89
118932026-03-11T20:45:51.855Zinternal/poll.(*FD).Accept(0xc005cfc500)
118942026-03-11T20:45:51.855Z /work/go/src/internal/poll/fd_unix.go:611 +0x2ac
118952026-03-11T20:45:51.855Znet.(*netFD).accept(0xc005cfc500)
118962026-03-11T20:45:51.855Z /work/go/src/net/fd_unix.go:172 +0x29
118972026-03-11T20:45:51.855Znet.(*TCPListener).accept(0xc0062f67e0)
118982026-03-11T20:45:51.855Z /work/go/src/net/tcpsock_posix.go:159 +0x1e
118992026-03-11T20:45:51.855Znet.(*TCPListener).Accept(0xc0062f67e0)
119002026-03-11T20:45:51.855Z /work/go/src/net/tcpsock.go:327 +0x30
119012026-03-11T20:45:51.855Zgithub.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith(0xc001c9f178, {0x50b1300, 0xc0064c1920}, 0xc003f40400, {0x50a4470, 0xc0062f67e0}, 0xc0066b3c38)
119022026-03-11T20:45:51.855Z /work/oxidecomputer/cockroach/pkg/util/netutil/net.go:138 +0x77
119032026-03-11T20:45:51.863Zgithub.com/cockroachdb/cockroach/pkg/server.(*SQLServer).startServeSQL.func1({0x50b1300, 0xc0064c1920})
119042026-03-11T20:45:51.863Z /work/oxidecomputer/cockroach/pkg/server/server_sql.go:1374 +0xcb
119052026-03-11T20:45:51.863Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
119062026-03-11T20:45:51.863Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
119072026-03-11T20:45:51.863Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
119082026-03-11T20:45:51.863Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
119092026-03-11T20:45:51.863Z
119102026-03-11T20:45:51.863Zgoroutine 9023 [select]:
119112026-03-11T20:45:51.863Zgithub.com/cockroachdb/cockroach/pkg/sql/contention.(*eventStore).startEventIntake.func3({0xc001795b08?, 0xc002be9720?})
119122026-03-11T20:45:51.863Z /work/oxidecomputer/cockroach/pkg/sql/contention/event_store.go:188 +0xa5
119132026-03-11T20:45:51.863Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
119142026-03-11T20:45:51.863Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
119152026-03-11T20:45:51.863Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
119162026-03-11T20:45:51.863Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
119172026-03-11T20:45:51.863Z
119182026-03-11T20:45:51.863Zgoroutine 9794 [select, 59 minutes]:
119192026-03-11T20:45:51.863Zgithub.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler.func1()
119202026-03-11T20:45:51.863Z /work/oxidecomputer/cockroach/pkg/security/certificate_manager.go:241 +0x113
119212026-03-11T20:45:51.863Zcreated by github.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler in goroutine 9708
119222026-03-11T20:45:51.863Z /work/oxidecomputer/cockroach/pkg/security/certificate_manager.go:238 +0x7d
119232026-03-11T20:45:51.863Z
119242026-03-11T20:45:51.863Zgoroutine 10684 [select, 59 minutes]:
119252026-03-11T20:45:51.863Zgoogle.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0064c4740)
119262026-03-11T20:45:51.863Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:112 +0x6d
119272026-03-11T20:45:51.863Zcreated by google.golang.org/grpc.newCCBalancerWrapper in goroutine 9708
119282026-03-11T20:45:51.864Z /work/oxidecomputer/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:73 +0x22b
119292026-03-11T20:45:51.864Z
119302026-03-11T20:45:51.864Zgoroutine 9787 [select]:
119312026-03-11T20:45:51.864Zgithub.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker.func1()
119322026-03-11T20:45:51.864Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:71 +0xcb
119332026-03-11T20:45:51.864Zcreated by github.com/cockroachdb/pebble/vfs.(*diskHealthCheckingFile).startTicker in goroutine 9708
119342026-03-11T20:45:51.864Z /work/oxidecomputer/cockroach/vendor/github.com/cockroachdb/pebble/vfs/disk_health.go:66 +0x58
119352026-03-11T20:45:51.864Z
119362026-03-11T20:45:51.864Zgoroutine 11128 [select, 59 minutes]:
119372026-03-11T20:45:51.864Zgithub.com/cockroachdb/cockroach/pkg/util/tracing.newTracingClientStream.func2()
119382026-03-11T20:45:51.865Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:386 +0x67
119392026-03-11T20:45:51.865Zcreated by github.com/cockroachdb/cockroach/pkg/util/tracing.newTracingClientStream in goroutine 11126
119402026-03-11T20:45:51.865Z /work/oxidecomputer/cockroach/pkg/util/tracing/grpc_interceptor.go:385 +0x125
119412026-03-11T20:45:51.865Z
119422026-03-11T20:45:51.865Zgoroutine 10810 [sync.Cond.Wait, 59 minutes]:
119432026-03-11T20:45:51.865Zsync.runtime_notifyListWait(0xc0064c5050, 0x54)
119442026-03-11T20:45:51.865Z /work/go/src/runtime/sema.go:569 +0x159
119452026-03-11T20:45:51.865Zsync.(*Cond).Wait(0x0?)
119462026-03-11T20:45:51.865Z /work/go/src/sync/cond.go:70 +0x85
119472026-03-11T20:45:51.865Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3c80})
119482026-03-11T20:45:51.865Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
119492026-03-11T20:45:51.865Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
119502026-03-11T20:45:51.865Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
119512026-03-11T20:45:51.865Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
119522026-03-11T20:45:51.865Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
119532026-03-11T20:45:51.865Z
119542026-03-11T20:45:51.865Zgoroutine 10806 [sync.Cond.Wait, 59 minutes]:
119552026-03-11T20:45:51.865Zsync.runtime_notifyListWait(0xc0064c5050, 0x4e)
119562026-03-11T20:45:51.865Z /work/go/src/runtime/sema.go:569 +0x159
119572026-03-11T20:45:51.865Zsync.(*Cond).Wait(0x0?)
119582026-03-11T20:45:51.865Z /work/go/src/sync/cond.go:70 +0x85
119592026-03-11T20:45:51.865Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc006e3e820, {0x50b1300, 0xc006fd3bc0})
119602026-03-11T20:45:51.865Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
119612026-03-11T20:45:51.865Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
119622026-03-11T20:45:51.865Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
119632026-03-11T20:45:51.865Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
119642026-03-11T20:45:51.865Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
119652026-03-11T20:45:51.866Z
119662026-03-11T20:45:51.866Zgoroutine 10427 [select]:
119672026-03-11T20:45:51.866Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).outputLoop(0xc002210ee0, {0x50b1338, 0xc007611c20})
119682026-03-11T20:45:51.866Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:310 +0x247
119692026-03-11T20:45:51.866Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*registration).runOutputLoop(0xc002210ee0, {0x50b1300, 0xc007182720}, 0x1?)
119702026-03-11T20:45:51.866Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/registry.go:335 +0xba
119712026-03-11T20:45:51.866Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*Processor).run.func1({0x50b1300?, 0xc007182720?})
119722026-03-11T20:45:51.866Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/rangefeed/processor.go:297 +0x46
119732026-03-11T20:45:51.866Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
119742026-03-11T20:45:51.866Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
119752026-03-11T20:45:51.866Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10334
119762026-03-11T20:45:51.866Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
119772026-03-11T20:45:51.866Z
119782026-03-11T20:45:51.866Zgoroutine 10078 [sync.Cond.Wait]:
119792026-03-11T20:45:51.866Zsync.runtime_notifyListWait(0xc001e22250, 0x3782)
119802026-03-11T20:45:51.867Z /work/go/src/runtime/sema.go:569 +0x159
119812026-03-11T20:45:51.867Zsync.(*Cond).Wait(0x3aac600?)
119822026-03-11T20:45:51.867Z /work/go/src/sync/cond.go:70 +0x85
119832026-03-11T20:45:51.867Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc007583980})
119842026-03-11T20:45:51.867Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
119852026-03-11T20:45:51.867Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
119862026-03-11T20:45:51.867Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
119872026-03-11T20:45:51.867Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
119882026-03-11T20:45:51.867Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
119892026-03-11T20:45:51.867Z
119902026-03-11T20:45:51.867Zgoroutine 10079 [sync.Cond.Wait]:
119912026-03-11T20:45:51.867Zsync.runtime_notifyListWait(0xc001e22250, 0x3762)
119922026-03-11T20:45:51.867Z /work/go/src/runtime/sema.go:569 +0x159
119932026-03-11T20:45:51.867Zsync.(*Cond).Wait(0x3aac600?)
119942026-03-11T20:45:51.867Z /work/go/src/sync/cond.go:70 +0x85
119952026-03-11T20:45:51.867Zgithub.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).worker(0xc0031e6b60, {0x50b1300, 0xc0075839b0})
119962026-03-11T20:45:51.867Z /work/oxidecomputer/cockroach/pkg/kv/kvserver/scheduler.go:275 +0x2ef
119972026-03-11T20:45:51.867Zgithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
119982026-03-11T20:45:51.867Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:494 +0x151
119992026-03-11T20:45:51.867Zcreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9708
120002026-03-11T20:45:51.867Z /work/oxidecomputer/cockroach/pkg/util/stop/stopper.go:483 +0x3fe
120012026-03-11T20:45:51.867ZFAIL github.com/cockroachdb/cockroach/pkg/migration/migrationmanager 3600.138s
120022026-03-11T20:45:51.867Zok github.com/cockroachdb/cockroach/pkg/migration/migrations 44.537s
120032026-03-11T20:45:51.873Zok github.com/cockroachdb/cockroach/pkg/roachpb 0.409s
120042026-03-11T20:45:51.873Zok github.com/cockroachdb/cockroach/pkg/roachprod/cloud 0.006s
120052026-03-11T20:45:51.873Zok github.com/cockroachdb/cockroach/pkg/roachprod/config 0.006s
120062026-03-11T20:45:51.873Zok github.com/cockroachdb/cockroach/pkg/roachprod/install 0.109s
120072026-03-11T20:45:51.873Zok github.com/cockroachdb/cockroach/pkg/roachprod/ssh 0.001s
120082026-03-11T20:45:51.873Zok github.com/cockroachdb/cockroach/pkg/roachprod/vm 0.005s
120092026-03-11T20:45:51.873Zok github.com/cockroachdb/cockroach/pkg/rpc 2.842s
120102026-03-11T20:45:51.873Zok github.com/cockroachdb/cockroach/pkg/rpc/nodedialer 1.611s
120112026-03-11T20:45:51.873Zok github.com/cockroachdb/cockroach/pkg/security 2.964s
120122026-03-11T20:45:51.873Zok github.com/cockroachdb/cockroach/pkg/security/certmgr 0.061s
120132026-03-11T20:45:51.873Zok github.com/cockroachdb/cockroach/pkg/security/sessionrevival 0.036s
120142026-03-11T20:45:51.873Zok github.com/cockroachdb/cockroach/pkg/server 235.412s
120152026-03-11T20:45:51.873Zok github.com/cockroachdb/cockroach/pkg/server/debug 0.097s
120162026-03-11T20:45:51.873Zok github.com/cockroachdb/cockroach/pkg/server/debug/goroutineui 0.013s
120172026-03-11T20:45:51.873Zok github.com/cockroachdb/cockroach/pkg/server/debug/pprofui 0.093s
120182026-03-11T20:45:51.873Zok github.com/cockroachdb/cockroach/pkg/server/diagnostics 0.801s
120192026-03-11T20:45:51.873Zok github.com/cockroachdb/cockroach/pkg/server/dumpstore 0.008s
120202026-03-11T20:45:51.873Zok github.com/cockroachdb/cockroach/pkg/server/goroutinedumper 0.058s
120212026-03-11T20:45:51.873Zok github.com/cockroachdb/cockroach/pkg/server/heapprofiler 0.045s
120222026-03-11T20:45:51.873Zok github.com/cockroachdb/cockroach/pkg/server/pgurl 0.027s
120232026-03-11T20:45:51.874Zok github.com/cockroachdb/cockroach/pkg/server/serverpb 0.052s
120242026-03-11T20:45:51.874Zok github.com/cockroachdb/cockroach/pkg/server/settingswatcher 1.509s
120252026-03-11T20:45:51.874Zok github.com/cockroachdb/cockroach/pkg/server/status 10.680s
120262026-03-11T20:45:51.874Zok github.com/cockroachdb/cockroach/pkg/server/systemconfigwatcher 0.579s
120272026-03-11T20:45:51.874Zok github.com/cockroachdb/cockroach/pkg/server/telemetry 0.012s
120282026-03-11T20:45:51.874Zok github.com/cockroachdb/cockroach/pkg/server/tenantsettingswatcher 1.066s
120292026-03-11T20:45:51.874Zok github.com/cockroachdb/cockroach/pkg/server/tracedumper 0.244s
120302026-03-11T20:45:51.874Zok github.com/cockroachdb/cockroach/pkg/settings 0.021s
120312026-03-11T20:45:51.874Zok github.com/cockroachdb/cockroach/pkg/spanconfig 0.066s
120322026-03-11T20:45:51.874Zok github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigkvaccessor 2.160s
120332026-03-11T20:45:51.874Zok github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigkvsubscriber 5.886s
120342026-03-11T20:45:51.874Zok github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigmanager 5.850s
120352026-03-11T20:45:51.874Zok github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigptsreader 0.017s
120362026-03-11T20:45:51.874Zok github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigreconciler 0.026s [no tests to run]
120372026-03-11T20:45:51.874Zok github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigsqltranslator 0.025s [no tests to run]
120382026-03-11T20:45:51.874Zok github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigsqlwatcher 0.382s
120392026-03-11T20:45:51.874Zok github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigstore 7.170s
120402026-03-11T20:45:51.874Zok github.com/cockroachdb/cockroach/pkg/spanconfig/spanconfigtestutils 0.012s
120412026-03-11T20:45:51.874Zok github.com/cockroachdb/cockroach/pkg/sql 354.668s
120422026-03-11T20:45:51.874Zok github.com/cockroachdb/cockroach/pkg/sql/backfill 0.037s
120432026-03-11T20:45:51.874Zok github.com/cockroachdb/cockroach/pkg/sql/catalog 0.224s
120442026-03-11T20:45:51.874Zok github.com/cockroachdb/cockroach/pkg/sql/catalog/bootstrap 0.019s
120452026-03-11T20:45:51.874Zok github.com/cockroachdb/cockroach/pkg/sql/catalog/catalogkeys 0.017s
120462026-03-11T20:45:51.874Zok github.com/cockroachdb/cockroach/pkg/sql/catalog/catformat 0.018s
120472026-03-11T20:45:51.875Zok github.com/cockroachdb/cockroach/pkg/sql/catalog/catpb 0.050s
120482026-03-11T20:45:51.875Zok github.com/cockroachdb/cockroach/pkg/sql/catalog/catprivilege 0.070s
120492026-03-11T20:45:51.875Zok github.com/cockroachdb/cockroach/pkg/sql/catalog/colinfo 0.022s
120502026-03-11T20:45:51.875Zok github.com/cockroachdb/cockroach/pkg/sql/catalog/dbdesc 0.045s
120512026-03-11T20:45:51.875Zok github.com/cockroachdb/cockroach/pkg/sql/catalog/descpb 0.027s
120522026-03-11T20:45:51.875Zok github.com/cockroachdb/cockroach/pkg/sql/catalog/descs 2.052s
120532026-03-11T20:45:51.875Zok github.com/cockroachdb/cockroach/pkg/sql/catalog/hydratedtables 0.029s
120542026-03-11T20:45:51.875Zok github.com/cockroachdb/cockroach/pkg/sql/catalog/internal/validate 0.048s
120552026-03-11T20:45:51.875Zok github.com/cockroachdb/cockroach/pkg/sql/catalog/lease 18.508s
120562026-03-11T20:45:51.876Zok github.com/cockroachdb/cockroach/pkg/sql/catalog/multiregion 0.027s
120572026-03-11T20:45:51.876Zok github.com/cockroachdb/cockroach/pkg/sql/catalog/nstree 0.038s
120582026-03-11T20:45:51.876Zok github.com/cockroachdb/cockroach/pkg/sql/catalog/resolver 0.066s
120592026-03-11T20:45:51.876Zok github.com/cockroachdb/cockroach/pkg/sql/catalog/schemadesc 0.106s
120602026-03-11T20:45:51.876Zok github.com/cockroachdb/cockroach/pkg/sql/catalog/schemaexpr 0.110s
120612026-03-11T20:45:51.876Zok github.com/cockroachdb/cockroach/pkg/sql/catalog/seqexpr 0.035s
120622026-03-11T20:45:51.876Zok github.com/cockroachdb/cockroach/pkg/sql/catalog/systemschema 0.047s
120632026-03-11T20:45:51.876Zok github.com/cockroachdb/cockroach/pkg/sql/catalog/tabledesc 2.000s
120642026-03-11T20:45:51.876Zok github.com/cockroachdb/cockroach/pkg/sql/catalog/typedesc 0.037s
120652026-03-11T20:45:51.876Zok github.com/cockroachdb/cockroach/pkg/sql/colcontainer 1.016s
120662026-03-11T20:45:51.876Zok github.com/cockroachdb/cockroach/pkg/sql/colconv 0.024s
120672026-03-11T20:45:51.876Zok github.com/cockroachdb/cockroach/pkg/sql/colencoding 0.066s
120682026-03-11T20:45:51.876Zok github.com/cockroachdb/cockroach/pkg/sql/colexec 29.669s
120692026-03-11T20:45:51.876Zok github.com/cockroachdb/cockroach/pkg/sql/colexec/colbuilder 0.428s
120702026-03-11T20:45:51.876Zok github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecagg 0.494s
120712026-03-11T20:45:51.876Zok github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecargs 0.469s
120722026-03-11T20:45:51.876Zok github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecbase 0.842s
120732026-03-11T20:45:51.876Zok github.com/cockroachdb/cockroach/pkg/sql/colexec/colexeccmp 0.251s
120742026-03-11T20:45:51.876Zok github.com/cockroachdb/cockroach/pkg/sql/colexec/colexechash 0.500s
120752026-03-11T20:45:51.876Zok github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecjoin 0.615s
120762026-03-11T20:45:51.876Zok github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecproj 1.070s
120772026-03-11T20:45:51.876Zok github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecsel 0.626s
120782026-03-11T20:45:51.876Zok github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecspan 0.474s
120792026-03-11T20:45:51.876Zok github.com/cockroachdb/cockroach/pkg/sql/colexec/colexectestutils 0.189s
120802026-03-11T20:45:51.876Zok github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecutils 0.664s
120812026-03-11T20:45:51.876Zok github.com/cockroachdb/cockroach/pkg/sql/colexec/colexecwindow 0.906s
120822026-03-11T20:45:51.876Zok github.com/cockroachdb/cockroach/pkg/sql/colexec/execgen 0.018s
120832026-03-11T20:45:51.877Zok github.com/cockroachdb/cockroach/pkg/sql/colexec/execgen/cmd/execgen 0.227s
120842026-03-11T20:45:51.877Zok github.com/cockroachdb/cockroach/pkg/sql/colexecerror 0.013s
120852026-03-11T20:45:51.877Zok github.com/cockroachdb/cockroach/pkg/sql/colexecop 0.469s
120862026-03-11T20:45:51.877Zok github.com/cockroachdb/cockroach/pkg/sql/colfetcher 0.251s
120872026-03-11T20:45:51.877Zok github.com/cockroachdb/cockroach/pkg/sql/colflow 1.858s
120882026-03-11T20:45:51.877Zok github.com/cockroachdb/cockroach/pkg/sql/colflow/colrpc 0.189s
120892026-03-11T20:45:51.877Zok github.com/cockroachdb/cockroach/pkg/sql/colmem 0.120s
120902026-03-11T20:45:51.877Zok github.com/cockroachdb/cockroach/pkg/sql/contention 0.141s
120912026-03-11T20:45:51.877Zok github.com/cockroachdb/cockroach/pkg/sql/contention/contentionutils 1.026s
120922026-03-11T20:45:51.877Zok github.com/cockroachdb/cockroach/pkg/sql/contention/txnidcache 0.321s
120932026-03-11T20:45:51.877Zok github.com/cockroachdb/cockroach/pkg/sql/contentionpb 0.021s
120942026-03-11T20:45:51.877Zok github.com/cockroachdb/cockroach/pkg/sql/covering 0.004s
120952026-03-11T20:45:51.877Zok github.com/cockroachdb/cockroach/pkg/sql/delegate 0.043s
120962026-03-11T20:45:51.877Zok github.com/cockroachdb/cockroach/pkg/sql/distsql 3.946s
120972026-03-11T20:45:51.877Zok github.com/cockroachdb/cockroach/pkg/sql/doctor 0.160s
120982026-03-11T20:45:51.877Zok github.com/cockroachdb/cockroach/pkg/sql/enum 0.841s
120992026-03-11T20:45:51.877Zok github.com/cockroachdb/cockroach/pkg/sql/execinfra 0.364s
121002026-03-11T20:45:51.877Zok github.com/cockroachdb/cockroach/pkg/sql/execinfrapb 0.072s
121012026-03-11T20:45:51.877Zok github.com/cockroachdb/cockroach/pkg/sql/execstats 0.430s
121022026-03-11T20:45:51.877Zok github.com/cockroachdb/cockroach/pkg/sql/flowinfra 15.626s
121032026-03-11T20:45:51.883Zok github.com/cockroachdb/cockroach/pkg/sql/gcjob 0.478s
121042026-03-11T20:45:51.883Zok github.com/cockroachdb/cockroach/pkg/sql/gcjob/gcjobnotifier 0.125s
121052026-03-11T20:45:51.883Zok github.com/cockroachdb/cockroach/pkg/sql/gcjob_test 13.921s
121062026-03-11T20:45:51.883Zok github.com/cockroachdb/cockroach/pkg/sql/idxusage 0.062s
121072026-03-11T20:45:51.883Zok github.com/cockroachdb/cockroach/pkg/sql/importer 144.575s
121082026-03-11T20:45:51.883Zok github.com/cockroachdb/cockroach/pkg/sql/inverted 0.147s
121092026-03-11T20:45:51.883Zok github.com/cockroachdb/cockroach/pkg/sql/kvstreamer 11.069s
121102026-03-11T20:45:51.883Zok github.com/cockroachdb/cockroach/pkg/sql/lex 0.129s
121112026-03-11T20:45:51.883Zok github.com/cockroachdb/cockroach/pkg/sql/lexbase 0.267s
121122026-03-11T20:45:51.883Zok github.com/cockroachdb/cockroach/pkg/sql/logictest 671.672s
121132026-03-11T20:45:51.883Zok github.com/cockroachdb/cockroach/pkg/sql/oidext 0.002s
121142026-03-11T20:45:51.883Zok github.com/cockroachdb/cockroach/pkg/sql/opt 0.039s
121152026-03-11T20:45:51.883Zok github.com/cockroachdb/cockroach/pkg/sql/opt/bench 0.125s [no tests to run]
121162026-03-11T20:45:51.883Zok github.com/cockroachdb/cockroach/pkg/sql/opt/cat 0.125s
121172026-03-11T20:45:51.883Zok github.com/cockroachdb/cockroach/pkg/sql/opt/constraint 0.115s
121182026-03-11T20:45:51.883Zok github.com/cockroachdb/cockroach/pkg/sql/opt/distribution 0.104s
121192026-03-11T20:45:51.884Zok github.com/cockroachdb/cockroach/pkg/sql/opt/exec/execbuilder 32.579s
121202026-03-11T20:45:51.884Zok github.com/cockroachdb/cockroach/pkg/sql/opt/exec/explain 0.661s
121212026-03-11T20:45:51.884Zok github.com/cockroachdb/cockroach/pkg/sql/opt/idxconstraint 0.161s
121222026-03-11T20:45:51.884Zok github.com/cockroachdb/cockroach/pkg/sql/opt/indexrec 0.258s
121232026-03-11T20:45:51.884Zok github.com/cockroachdb/cockroach/pkg/sql/opt/invertedexpr 0.220s
121242026-03-11T20:45:51.884Zok github.com/cockroachdb/cockroach/pkg/sql/opt/invertedidx 0.203s
121252026-03-11T20:45:51.884Zok github.com/cockroachdb/cockroach/pkg/sql/opt/memo 1.625s
121262026-03-11T20:45:51.884Zok github.com/cockroachdb/cockroach/pkg/sql/opt/norm 1.001s
121272026-03-11T20:45:51.884Zok github.com/cockroachdb/cockroach/pkg/sql/opt/opbench 0.927s
121282026-03-11T20:45:51.884Zok github.com/cockroachdb/cockroach/pkg/sql/opt/optbuilder 0.839s
121292026-03-11T20:45:51.884Zok github.com/cockroachdb/cockroach/pkg/sql/opt/optgen/cmd/optfmt 0.058s
121302026-03-11T20:45:51.884Zok github.com/cockroachdb/cockroach/pkg/sql/opt/optgen/cmd/optgen 0.027s
121312026-03-11T20:45:51.884Zok github.com/cockroachdb/cockroach/pkg/sql/opt/optgen/exprgen 0.122s
121322026-03-11T20:45:51.884Zok github.com/cockroachdb/cockroach/pkg/sql/opt/optgen/lang 0.034s
121332026-03-11T20:45:51.884Zok github.com/cockroachdb/cockroach/pkg/sql/opt/ordering 0.066s
121342026-03-11T20:45:51.884Zok github.com/cockroachdb/cockroach/pkg/sql/opt/partialidx 0.084s
121352026-03-11T20:45:51.884Zok github.com/cockroachdb/cockroach/pkg/sql/opt/partition 0.105s
121362026-03-11T20:45:51.884Zok github.com/cockroachdb/cockroach/pkg/sql/opt/props 0.572s
121372026-03-11T20:45:51.884Zok github.com/cockroachdb/cockroach/pkg/sql/opt/props/physical 0.075s
121382026-03-11T20:45:51.884Zok github.com/cockroachdb/cockroach/pkg/sql/opt/testutils 0.039s
121392026-03-11T20:45:51.884Zok github.com/cockroachdb/cockroach/pkg/sql/opt/testutils/opttester 0.104s
121402026-03-11T20:45:51.884Zok github.com/cockroachdb/cockroach/pkg/sql/opt/testutils/testcat 0.131s
121412026-03-11T20:45:51.884Zok github.com/cockroachdb/cockroach/pkg/sql/opt/xform 3.185s
121422026-03-11T20:45:51.884Zok github.com/cockroachdb/cockroach/pkg/sql/parser 0.494s
121432026-03-11T20:45:51.884Zok github.com/cockroachdb/cockroach/pkg/sql/pgwire 31.934s
121442026-03-11T20:45:51.884Zok github.com/cockroachdb/cockroach/pkg/sql/pgwire/hba 0.056s
121452026-03-11T20:45:51.884Zok github.com/cockroachdb/cockroach/pkg/sql/pgwire/identmap 0.052s
121462026-03-11T20:45:51.884Zok github.com/cockroachdb/cockroach/pkg/sql/pgwire/pgerror 0.024s
121472026-03-11T20:45:51.885Zok github.com/cockroachdb/cockroach/pkg/sql/pgwire/pgwirecancel 1.994s
121482026-03-11T20:45:51.885Zok github.com/cockroachdb/cockroach/pkg/sql/physicalplan 9.941s
121492026-03-11T20:45:51.885Zok github.com/cockroachdb/cockroach/pkg/sql/physicalplan/replicaoracle 0.034s
121502026-03-11T20:45:51.885Zok github.com/cockroachdb/cockroach/pkg/sql/privilege 0.006s
121512026-03-11T20:45:51.885Zok github.com/cockroachdb/cockroach/pkg/sql/protoreflect 0.058s
121522026-03-11T20:45:51.885Zok github.com/cockroachdb/cockroach/pkg/sql/querycache 0.214s
121532026-03-11T20:45:51.885Zok github.com/cockroachdb/cockroach/pkg/sql/randgen 0.693s
121542026-03-11T20:45:51.885Zok github.com/cockroachdb/cockroach/pkg/sql/row 3.557s
121552026-03-11T20:45:51.885Zok github.com/cockroachdb/cockroach/pkg/sql/rowcontainer 0.919s
121562026-03-11T20:45:51.885Zok github.com/cockroachdb/cockroach/pkg/sql/rowenc 0.946s
121572026-03-11T20:45:51.886Zok github.com/cockroachdb/cockroach/pkg/sql/rowenc/keyside 0.460s
121582026-03-11T20:45:51.886Zok github.com/cockroachdb/cockroach/pkg/sql/rowenc/valueside 0.375s
121592026-03-11T20:45:51.886Zok github.com/cockroachdb/cockroach/pkg/sql/rowexec 8.653s
121602026-03-11T20:45:51.886Zok github.com/cockroachdb/cockroach/pkg/sql/rowflow 1.093s
121612026-03-11T20:45:51.886Zok github.com/cockroachdb/cockroach/pkg/sql/scanner 0.058s
121622026-03-11T20:45:51.886Zok github.com/cockroachdb/cockroach/pkg/sql/scheduledlogging 3.368s
121632026-03-11T20:45:51.886Zok github.com/cockroachdb/cockroach/pkg/sql/schemachange 4.982s
121642026-03-11T20:45:51.886Zok github.com/cockroachdb/cockroach/pkg/sql/schemachanger 46.843s
121652026-03-11T20:45:51.887Zok github.com/cockroachdb/cockroach/pkg/sql/schemachanger/corpus 0.057s
121662026-03-11T20:45:51.887Zok github.com/cockroachdb/cockroach/pkg/sql/schemachanger/rel 0.063s
121672026-03-11T20:45:51.887Zok github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scbuild 5.734s
121682026-03-11T20:45:51.887Zok github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scdecomp 1.501s
121692026-03-11T20:45:51.887Zok github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scdeps 0.071s
121702026-03-11T20:45:51.887Zok github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scexec 1.851s
121712026-03-11T20:45:51.887Zok github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scplan 3.245s
121722026-03-11T20:45:51.887Zok github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scplan/internal/opgen 0.072s
121732026-03-11T20:45:51.887Zok github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scplan/internal/rules 0.083s
121742026-03-11T20:45:51.887Zok github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scplan/internal/scgraph 0.105s
121752026-03-11T20:45:51.887Zok github.com/cockroachdb/cockroach/pkg/sql/schemachanger/screl 0.096s
121762026-03-11T20:45:51.887Zok github.com/cockroachdb/cockroach/pkg/sql/sem/builtins 4.073s
121772026-03-11T20:45:51.887Zok github.com/cockroachdb/cockroach/pkg/sql/sem/tree 1.333s
121782026-03-11T20:45:51.887Zok github.com/cockroachdb/cockroach/pkg/sql/sem/tree/eval_test 0.607s
121792026-03-11T20:45:51.888Zok github.com/cockroachdb/cockroach/pkg/sql/sessiondata 0.009s
121802026-03-11T20:45:51.888Zok github.com/cockroachdb/cockroach/pkg/sql/sessioninit 0.731s
121812026-03-11T20:45:51.888Zok github.com/cockroachdb/cockroach/pkg/sql/span 0.578s
121822026-03-11T20:45:51.888Zok github.com/cockroachdb/cockroach/pkg/sql/sqlinstance/instanceprovider 0.047s
121832026-03-11T20:45:51.888Zok github.com/cockroachdb/cockroach/pkg/sql/sqlinstance/instancestorage 1.502s
121842026-03-11T20:45:51.888Zok github.com/cockroachdb/cockroach/pkg/sql/sqlliveness/slinstance 0.037s
121852026-03-11T20:45:51.888Zok github.com/cockroachdb/cockroach/pkg/sql/sqlliveness/slstorage 1.088s
121862026-03-11T20:45:51.888Zok github.com/cockroachdb/cockroach/pkg/sql/sqlstats/persistedsqlstats 16.683s
121872026-03-11T20:45:51.888Zok github.com/cockroachdb/cockroach/pkg/sql/sqlstats/persistedsqlstats/sqlstatsutil 0.024s
121882026-03-11T20:45:51.888Zok github.com/cockroachdb/cockroach/pkg/sql/sqlstats/sslocal 2.921s
121892026-03-11T20:45:51.888Zok github.com/cockroachdb/cockroach/pkg/sql/stats 13.053s
121902026-03-11T20:45:51.888Zok github.com/cockroachdb/cockroach/pkg/sql/stmtdiagnostics 4.032s
121912026-03-11T20:45:51.888Zok github.com/cockroachdb/cockroach/pkg/sql/tests 32.702s
121922026-03-11T20:45:51.888Zok github.com/cockroachdb/cockroach/pkg/sql/ttl/ttljob 13.643s
121932026-03-11T20:45:51.888Zok github.com/cockroachdb/cockroach/pkg/sql/types 0.197s
121942026-03-11T20:45:51.888Zok github.com/cockroachdb/cockroach/pkg/startupmigrations 1.135s
121952026-03-11T20:45:51.888Zok github.com/cockroachdb/cockroach/pkg/startupmigrations/leasemanager 1.009s
121962026-03-11T20:45:51.888Zok github.com/cockroachdb/cockroach/pkg/storage 5.135s
121972026-03-11T20:45:51.888Zok github.com/cockroachdb/cockroach/pkg/storage/enginepb 0.012s
121982026-03-11T20:45:51.888Zok github.com/cockroachdb/cockroach/pkg/storage/fs 0.027s
121992026-03-11T20:45:51.888Zok github.com/cockroachdb/cockroach/pkg/storage/metamorphic 33.841s
122002026-03-11T20:45:51.888Zok github.com/cockroachdb/cockroach/pkg/testutils 0.163s
122012026-03-11T20:45:51.888Zok github.com/cockroachdb/cockroach/pkg/testutils/floatcmp 0.004s
122022026-03-11T20:45:51.888Zok github.com/cockroachdb/cockroach/pkg/testutils/keysutils 0.026s
122032026-03-11T20:45:52.136Zok github.com/cockroachdb/cockroach/pkg/testutils/lint/passes/errwrap 0.685s
122042026-03-11T20:45:52.136Zok github.com/cockroachdb/cockroach/pkg/testutils/lint/passes/fmtsafe 0.728s
122052026-03-11T20:45:52.136Zok github.com/cockroachdb/cockroach/pkg/testutils/lint/passes/forbiddenmethod 0.189s
122062026-03-11T20:45:52.136Zok github.com/cockroachdb/cockroach/pkg/testutils/lint/passes/hash 0.638s
122072026-03-11T20:45:52.136Zok github.com/cockroachdb/cockroach/pkg/testutils/lint/passes/leaktestcall 0.682s
122082026-03-11T20:45:52.136Zok github.com/cockroachdb/cockroach/pkg/testutils/lint/passes/nilness 0.171s
122092026-03-11T20:45:52.136Zok github.com/cockroachdb/cockroach/pkg/testutils/lint/passes/nocopy 0.237s
122102026-03-11T20:45:52.136Zok github.com/cockroachdb/cockroach/pkg/testutils/lint/passes/passesutil 0.187s
122112026-03-11T20:45:52.136Zok github.com/cockroachdb/cockroach/pkg/testutils/lint/passes/returnerrcheck 0.515s
122122026-03-11T20:45:52.136Zok github.com/cockroachdb/cockroach/pkg/testutils/lint/passes/timer 0.364s
122132026-03-11T20:45:52.136Zok github.com/cockroachdb/cockroach/pkg/testutils/lint/passes/unconvert 0.122s
122142026-03-11T20:45:52.136Zok github.com/cockroachdb/cockroach/pkg/testutils/sqlutils 0.813s
122152026-03-11T20:45:52.136Zok github.com/cockroachdb/cockroach/pkg/testutils/testcluster 13.850s
122162026-03-11T20:45:52.137Zok github.com/cockroachdb/cockroach/pkg/testutils/zerofields 0.005s
122172026-03-11T20:45:52.137Zok github.com/cockroachdb/cockroach/pkg/ts 6.200s
122182026-03-11T20:45:52.137Zok github.com/cockroachdb/cockroach/pkg/ts/testmodel 0.015s
122192026-03-11T20:45:52.137Zok github.com/cockroachdb/cockroach/pkg/util 0.225s
122202026-03-11T20:45:52.137Zok github.com/cockroachdb/cockroach/pkg/util/admission 4.353s
122212026-03-11T20:45:52.137Zok github.com/cockroachdb/cockroach/pkg/util/binfetcher 0.033s
122222026-03-11T20:45:52.137Zok github.com/cockroachdb/cockroach/pkg/util/bitarray 0.012s
122232026-03-11T20:45:52.137Zok github.com/cockroachdb/cockroach/pkg/util/buildutil 0.011s
122242026-03-11T20:45:52.137Zok github.com/cockroachdb/cockroach/pkg/util/cache 0.005s
122252026-03-11T20:45:52.137Zok github.com/cockroachdb/cockroach/pkg/util/caller 0.010s
122262026-03-11T20:45:52.137Zok github.com/cockroachdb/cockroach/pkg/util/cgroups 0.024s
122272026-03-11T20:45:52.137Zok github.com/cockroachdb/cockroach/pkg/util/circuit 0.143s
122282026-03-11T20:45:52.137Zok github.com/cockroachdb/cockroach/pkg/util/cloudinfo 0.020s
122292026-03-11T20:45:52.137Zok github.com/cockroachdb/cockroach/pkg/util/contextutil 0.054s
122302026-03-11T20:45:52.137Zok github.com/cockroachdb/cockroach/pkg/util/ctxgroup 0.017s
122312026-03-11T20:45:52.137Zok github.com/cockroachdb/cockroach/pkg/util/duration 0.008s
122322026-03-11T20:45:52.137Zok github.com/cockroachdb/cockroach/pkg/util/encoding 0.560s
122332026-03-11T20:45:52.137Zok github.com/cockroachdb/cockroach/pkg/util/encoding/csv 0.009s
122342026-03-11T20:45:52.137Zok github.com/cockroachdb/cockroach/pkg/util/envutil 0.004s
122352026-03-11T20:45:52.137Zok github.com/cockroachdb/cockroach/pkg/util/errorutil 0.018s
122362026-03-11T20:45:52.137Zok github.com/cockroachdb/cockroach/pkg/util/errorutil/unimplemented 0.002s
122372026-03-11T20:45:52.137Zok github.com/cockroachdb/cockroach/pkg/util/flagutil 0.005s
122382026-03-11T20:45:52.137Zok github.com/cockroachdb/cockroach/pkg/util/fsm 0.003s
122392026-03-11T20:45:52.137Zok github.com/cockroachdb/cockroach/pkg/util/fuzzystrmatch 0.001s
122402026-03-11T20:45:52.137Zok github.com/cockroachdb/cockroach/pkg/util/goschedstats 0.012s
122412026-03-11T20:45:52.137Zok github.com/cockroachdb/cockroach/pkg/util/grpcutil 0.165s
122422026-03-11T20:45:52.137Zok github.com/cockroachdb/cockroach/pkg/util/hlc 0.093s
122432026-03-11T20:45:52.138Zok github.com/cockroachdb/cockroach/pkg/util/httputil 0.039s
122442026-03-11T20:45:52.138Zok github.com/cockroachdb/cockroach/pkg/util/humanizeutil 0.030s
122452026-03-11T20:45:52.138Zok github.com/cockroachdb/cockroach/pkg/util/interval 1.800s
122462026-03-11T20:45:52.138Zok github.com/cockroachdb/cockroach/pkg/util/interval/generic 11.659s
122472026-03-11T20:45:52.138Zok github.com/cockroachdb/cockroach/pkg/util/ipaddr 0.024s
122482026-03-11T20:45:52.138Zok github.com/cockroachdb/cockroach/pkg/util/json 0.649s
122492026-03-11T20:45:52.138Zok github.com/cockroachdb/cockroach/pkg/util/limit 1.125s
122502026-03-11T20:45:52.138Zok github.com/cockroachdb/cockroach/pkg/util/log 2.971s
122512026-03-11T20:45:52.138Zok github.com/cockroachdb/cockroach/pkg/util/log/eventpb 0.020s
122522026-03-11T20:45:52.138Zok github.com/cockroachdb/cockroach/pkg/util/log/logconfig 0.036s
122532026-03-11T20:45:52.138Zok github.com/cockroachdb/cockroach/pkg/util/log/logcrash 0.398s
122542026-03-11T20:45:52.138Zok github.com/cockroachdb/cockroach/pkg/util/log/testshout 0.007s
122552026-03-11T20:45:52.138Zok github.com/cockroachdb/cockroach/pkg/util/metric 0.021s
122562026-03-11T20:45:52.138Zok github.com/cockroachdb/cockroach/pkg/util/metric/aggmetric 0.018s
122572026-03-11T20:45:52.138Zok github.com/cockroachdb/cockroach/pkg/util/mon 2.949s
122582026-03-11T20:45:52.138Zok github.com/cockroachdb/cockroach/pkg/util/netutil 0.009s
122592026-03-11T20:45:52.138Zok github.com/cockroachdb/cockroach/pkg/util/netutil/addr 0.027s
122602026-03-11T20:45:52.139Zok github.com/cockroachdb/cockroach/pkg/util/optional 0.007s
122612026-03-11T20:45:52.139Zok github.com/cockroachdb/cockroach/pkg/util/pretty 0.002s
122622026-03-11T20:45:52.139Zok github.com/cockroachdb/cockroach/pkg/util/protoutil 0.013s
122632026-03-11T20:45:52.139Zok github.com/cockroachdb/cockroach/pkg/util/quotapool 0.106s
122642026-03-11T20:45:52.139Zok github.com/cockroachdb/cockroach/pkg/util/randutil 0.018s
122652026-03-11T20:45:52.139Zok github.com/cockroachdb/cockroach/pkg/util/retry 0.010s
122662026-03-11T20:45:52.139Zok github.com/cockroachdb/cockroach/pkg/util/ring 0.019s
122672026-03-11T20:45:52.139Zok github.com/cockroachdb/cockroach/pkg/util/sdnotify 0.005s
122682026-03-11T20:45:52.139Zok github.com/cockroachdb/cockroach/pkg/util/search 0.004s
122692026-03-11T20:45:52.139Zok github.com/cockroachdb/cockroach/pkg/util/shuffle 0.003s
122702026-03-11T20:45:52.139Zok github.com/cockroachdb/cockroach/pkg/util/span 0.015s
122712026-03-11T20:45:52.139Zok github.com/cockroachdb/cockroach/pkg/util/startup 0.089s
122722026-03-11T20:45:52.139Zok github.com/cockroachdb/cockroach/pkg/util/stop 0.460s
122732026-03-11T20:45:52.139Zok github.com/cockroachdb/cockroach/pkg/util/stringarena 0.009s [no tests to run]
122742026-03-11T20:45:52.139Zok github.com/cockroachdb/cockroach/pkg/util/syncutil 0.824s
122752026-03-11T20:45:52.139Zok github.com/cockroachdb/cockroach/pkg/util/syncutil/singleflight 0.012s
122762026-03-11T20:45:52.139Zok github.com/cockroachdb/cockroach/pkg/util/sysutil 0.036s
122772026-03-11T20:45:52.140Zok github.com/cockroachdb/cockroach/pkg/util/targz 0.003s
122782026-03-11T20:45:52.140Zok github.com/cockroachdb/cockroach/pkg/util/timeofday 0.004s
122792026-03-11T20:45:52.140Zok github.com/cockroachdb/cockroach/pkg/util/timetz 0.007s
122802026-03-11T20:45:52.140Zok github.com/cockroachdb/cockroach/pkg/util/timeutil 1.557s
122812026-03-11T20:45:52.140Zok github.com/cockroachdb/cockroach/pkg/util/timeutil/pgdate 0.094s
122822026-03-11T20:45:52.140Zok github.com/cockroachdb/cockroach/pkg/util/tracing 0.534s
122832026-03-11T20:45:52.140Zok github.com/cockroachdb/cockroach/pkg/util/tracing/collector 0.527s
122842026-03-11T20:45:52.140Zok github.com/cockroachdb/cockroach/pkg/util/tracing/service 0.016s
122852026-03-11T20:45:52.140Zok github.com/cockroachdb/cockroach/pkg/util/treeprinter 0.001s
122862026-03-11T20:45:52.140Zok github.com/cockroachdb/cockroach/pkg/util/uint128 0.002s
122872026-03-11T20:45:52.140Zok github.com/cockroachdb/cockroach/pkg/util/ulid 1.557s
122882026-03-11T20:45:52.140Zok github.com/cockroachdb/cockroach/pkg/util/unique 0.001s
122892026-03-11T20:45:52.140Zok github.com/cockroachdb/cockroach/pkg/util/uuid 0.006s
122902026-03-11T20:45:52.141Zok github.com/cockroachdb/cockroach/pkg/util/version 0.002s
122912026-03-11T20:45:52.141Zok github.com/cockroachdb/cockroach/pkg/workload 0.083s
122922026-03-11T20:45:52.141Zok github.com/cockroachdb/cockroach/pkg/workload/bank 0.630s
122932026-03-11T20:45:52.141Zok github.com/cockroachdb/cockroach/pkg/workload/cli 0.019s
122942026-03-11T20:45:52.141Zok github.com/cockroachdb/cockroach/pkg/workload/faker 0.003s
122952026-03-11T20:45:52.141Zok github.com/cockroachdb/cockroach/pkg/workload/histogram 0.004s
122962026-03-11T20:45:52.141Zok github.com/cockroachdb/cockroach/pkg/workload/movr 0.011s
122972026-03-11T20:45:52.141Zok github.com/cockroachdb/cockroach/pkg/workload/tpcc 0.010s
122982026-03-11T20:45:52.141Zok github.com/cockroachdb/cockroach/pkg/workload/tpch 0.007s
122992026-03-11T20:45:52.141Zok github.com/cockroachdb/cockroach/pkg/workload/workloadimpl 0.002s
123002026-03-11T20:45:52.141Zok github.com/cockroachdb/cockroach/pkg/workload/workloadsql 2.553s
123012026-03-11T20:45:52.141Zok github.com/cockroachdb/cockroach/pkg/workload/ycsb 7.378s
123022026-03-11T20:45:52.141ZFAIL
123032026-03-11T20:45:53.476Zgmake: *** [Makefile:949: test] Error 1
123042026-03-11T20:45:53.479Zprocess exited: duration 3856183 ms, exit code 2
 
123052026-03-11T20:45:53.485Zfound 0 output files