Max level shown:
12026-07-29T01:15:03.392ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::16]:32345 remote_addr = [fd00:1122:3344:101::a]:40667
22026-07-29T01:15:03.436ZINFOcrucible-agent (datafile): region 0aca266c-8762-4650-93f8-641f5007c832 state: Requested
32026-07-29T01:15:03.441ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
42026-07-29T01:15:03.444ZINFOcrucible-agent (dropshot): request completed latency_us = 21557 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:40667 req_id = d74f4436-0987-4cfa-a6d4-722bbdb28ce5 response_code = 200 uri = /crucible/0/regions
52026-07-29T01:15:03.452ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_99e704a5-2c98-4cb5-8f68-e584bcbe9876/crucible/regions/0aca266c-8762-4650-93f8-641f5007c832
62026-07-29T01:15:03.452ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_99e704a5-2c98-4cb5-8f68-e584bcbe9876/crucible/regions/0aca266c-8762-4650-93f8-641f5007c832
72026-07-29T01:15:03.556ZINFOcrucible-agent (worker): creating region Region { id: RegionId("0aca266c-8762-4650-93f8-641f5007c832"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/0aca266c-8762-4650-93f8-641f5007c832" region = 0aca266c-8762-4650-93f8-641f5007c832
82026-07-29T01:15:03.559ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "0aca266c-8762-4650-93f8-641f5007c832" "--data" "/data/regions/0aca266c-8762-4650-93f8-641f5007c832" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 0aca266c-8762-4650-93f8-641f5007c832
92026-07-29T01:15:03.644ZINFOcrucible-agent (dropshot): request completed latency_us = 58 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:40667 req_id = bf626776-5b0c-4747-b0cc-8c5ee17d621c response_code = 200 uri = /crucible/0/regions
102026-07-29T01:15:03.897ZINFOcrucible-agent (worker): region files created ok region = 0aca266c-8762-4650-93f8-641f5007c832
112026-07-29T01:15:03.901ZINFOcrucible-agent (datafile): region 0aca266c-8762-4650-93f8-641f5007c832 state: Requested -> Created
122026-07-29T01:15:03.902ZINFOcrucible-agent (worker): applying SMF actions post create...
132026-07-29T01:15:03.904ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-0aca266c-8762-4650-93f8-641f5007c832
142026-07-29T01:15:03.913ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-0aca266c-8762-4650-93f8-641f5007c832
152026-07-29T01:15:03.928ZINFOcrucible-agent (worker): creating config property group
162026-07-29T01:15:03.931ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-0aca266c-8762-4650-93f8-641f5007c832
172026-07-29T01:15:03.932ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/0aca266c-8762-4650-93f8-641f5007c832
182026-07-29T01:15:03.934ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
192026-07-29T01:15:03.934ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::16
202026-07-29T01:15:03.934ZINFOcrucible-agent (worker): commit
212026-07-29T01:15:03.949ZINFOcrucible-agent (worker): ok!
222026-07-29T01:15:03.953ZINFOcrucible-agent (worker): SMF ok!
232026-07-29T01:15:04.105ZINFOcrucible-agent (dropshot): request completed latency_us = 69 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:40667 req_id = ad1cd309-57e8-4181-b320-4d8de62f03d2 response_code = 200 uri = /crucible/0/regions
242026-07-29T01:26:17.513ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::16]:32345 remote_addr = [fd00:1122:3344:101::a]:36173
252026-07-29T01:26:17.513ZINFOcrucible-agent (dropshot): request completed latency_us = 54 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:36173 req_id = d67e3619-a745-48e3-a3dd-87e7cdeb1e73 response_code = 200 uri = /crucible/0/regions/0aca266c-8762-4650-93f8-641f5007c832
262026-07-29T01:26:17.517ZINFOcrucible-agent (datafile): path is "/data/regions/0aca266c-8762-4650-93f8-641f5007c832"
272026-07-29T01:26:17.523ZINFOcrucible-agent (datafile): dataset is oxp_99e704a5-2c98-4cb5-8f68-e584bcbe9876/crucible/regions/0aca266c-8762-4650-93f8-641f5007c832
282026-07-29T01:26:17.529ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_99e704a5-2c98-4cb5-8f68-e584bcbe9876/crucible/regions/0aca266c-8762-4650-93f8-641f5007c832@ed2a0319-8638-478a-b715-d6629ff00ed0
292026-07-29T01:26:17.529ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_99e704a5-2c98-4cb5-8f68-e584bcbe9876/crucible/regions/0aca266c-8762-4650-93f8-641f5007c832", "ed2a0319-8638-478a-b715-d6629ff00ed0"]
302026-07-29T01:26:17.541ZINFOcrucible-agent (dropshot): request completed latency_us = 24649 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:36173 req_id = a2ee9cc3-9bc1-495e-9678-72716b6d0298 response_code = 200 uri = /crucible/0/regions/0aca266c-8762-4650-93f8-641f5007c832/snapshots/ed2a0319-8638-478a-b715-d6629ff00ed0
312026-07-29T01:26:17.545ZINFOcrucible-agent (datafile): path is "/data/regions/0aca266c-8762-4650-93f8-641f5007c832"
322026-07-29T01:26:17.551ZINFOcrucible-agent (datafile): dataset is oxp_99e704a5-2c98-4cb5-8f68-e584bcbe9876/crucible/regions/0aca266c-8762-4650-93f8-641f5007c832
332026-07-29T01:26:17.557ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_99e704a5-2c98-4cb5-8f68-e584bcbe9876/crucible/regions/0aca266c-8762-4650-93f8-641f5007c832@ed2a0319-8638-478a-b715-d6629ff00ed0
342026-07-29T01:26:17.557ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_99e704a5-2c98-4cb5-8f68-e584bcbe9876/crucible/regions/0aca266c-8762-4650-93f8-641f5007c832", "ed2a0319-8638-478a-b715-d6629ff00ed0"]
352026-07-29T01:26:17.563ZINFOcrucible-agent (datafile): requesting running snapshot 0aca266c-8762-4650-93f8-641f5007c832-ed2a0319-8638-478a-b715-d6629ff00ed0 state: Requested
362026-07-29T01:26:17.567ZINFOcrucible-agent (worker): applying SMF actions for region 0aca266c-8762-4650-93f8-641f5007c832 running snapshot ed2a0319-8638-478a-b715-d6629ff00ed0 (state Requested)...
372026-07-29T01:26:17.567ZINFOcrucible-agent (dropshot): request completed latency_us = 22140 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:36173 req_id = eb4b2c00-5bf9-4b85-ab2e-8554e74d8774 response_code = 200 uri = /crucible/0/regions/0aca266c-8762-4650-93f8-641f5007c832/snapshots/ed2a0319-8638-478a-b715-d6629ff00ed0/run
382026-07-29T01:26:17.573ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/0aca266c-8762-4650-93f8-641f5007c832
392026-07-29T01:26:17.573ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
402026-07-29T01:26:17.573ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::16
412026-07-29T01:26:17.573ZINFOcrucible-agent (worker): reconfig required
422026-07-29T01:26:17.573ZINFOcrucible-agent (worker): using existing config property group
432026-07-29T01:26:17.573ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-0aca266c-8762-4650-93f8-641f5007c832
442026-07-29T01:26:17.573ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/0aca266c-8762-4650-93f8-641f5007c832
452026-07-29T01:26:17.573ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
462026-07-29T01:26:17.573ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::16
472026-07-29T01:26:17.573ZINFOcrucible-agent (worker): commit
482026-07-29T01:26:17.581ZINFOcrucible-agent (worker): ok!
492026-07-29T01:26:17.584ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-0aca266c-8762-4650-93f8-641f5007c832-ed2a0319-8638-478a-b715-d6629ff00ed0
502026-07-29T01:26:17.587ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-0aca266c-8762-4650-93f8-641f5007c832-ed2a0319-8638-478a-b715-d6629ff00ed0
512026-07-29T01:26:17.600ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-0aca266c-8762-4650-93f8-641f5007c832-ed2a0319-8638-478a-b715-d6629ff00ed0
522026-07-29T01:26:17.600ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/0aca266c-8762-4650-93f8-641f5007c832/.zfs/snapshot/ed2a0319-8638-478a-b715-d6629ff00ed0
532026-07-29T01:26:17.600ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
542026-07-29T01:26:17.600ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
552026-07-29T01:26:17.600ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::16
562026-07-29T01:26:17.600ZINFOcrucible-agent (worker): commit
572026-07-29T01:26:17.604ZINFOcrucible-agent (worker): ok!
582026-07-29T01:26:17.609ZINFOcrucible-agent (worker): SMF ok!
592026-07-29T01:26:17.609ZINFOcrucible-agent (datafile): region 0aca266c-8762-4650-93f8-641f5007c832 running snapshot ed2a0319-8638-478a-b715-d6629ff00ed0 state: Requested -> Created
602026-07-29T01:26:17.713ZINFOcrucible-agent (datafile): path is "/data/regions/0aca266c-8762-4650-93f8-641f5007c832"
612026-07-29T01:26:17.720ZINFOcrucible-agent (datafile): dataset is oxp_99e704a5-2c98-4cb5-8f68-e584bcbe9876/crucible/regions/0aca266c-8762-4650-93f8-641f5007c832
622026-07-29T01:26:17.726ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_99e704a5-2c98-4cb5-8f68-e584bcbe9876/crucible/regions/0aca266c-8762-4650-93f8-641f5007c832@ed2a0319-8638-478a-b715-d6629ff00ed0
632026-07-29T01:26:17.726ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_99e704a5-2c98-4cb5-8f68-e584bcbe9876/crucible/regions/0aca266c-8762-4650-93f8-641f5007c832", "ed2a0319-8638-478a-b715-d6629ff00ed0"]
642026-07-29T01:26:17.732ZINFOcrucible-agent (dropshot): request completed latency_us = 18908 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:36173 req_id = 624222e0-791b-4c64-a356-aa92fc5c12bb response_code = 200 uri = /crucible/0/regions/0aca266c-8762-4650-93f8-641f5007c832/snapshots/ed2a0319-8638-478a-b715-d6629ff00ed0/run
652026-07-29T01:26:22.531ZINFOcrucible-agent (datafile): region 84d77408-8a03-45a0-8ef7-9bd72caa40a3 state: Requested
662026-07-29T01:26:22.531ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
672026-07-29T01:26:22.531ZINFOcrucible-agent (dropshot): request completed latency_us = 312 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:36173 req_id = fe06e167-f30d-44de-854f-0ba2eaa71a8e response_code = 200 uri = /crucible/0/regions
682026-07-29T01:26:22.537ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_99e704a5-2c98-4cb5-8f68-e584bcbe9876/crucible/regions/84d77408-8a03-45a0-8ef7-9bd72caa40a3
692026-07-29T01:26:22.537ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_99e704a5-2c98-4cb5-8f68-e584bcbe9876/crucible/regions/84d77408-8a03-45a0-8ef7-9bd72caa40a3
702026-07-29T01:26:22.602ZINFOcrucible-agent (worker): creating region Region { id: RegionId("84d77408-8a03-45a0-8ef7-9bd72caa40a3"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19002, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/84d77408-8a03-45a0-8ef7-9bd72caa40a3" region = 84d77408-8a03-45a0-8ef7-9bd72caa40a3
712026-07-29T01:26:22.603ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "84d77408-8a03-45a0-8ef7-9bd72caa40a3" "--data" "/data/regions/84d77408-8a03-45a0-8ef7-9bd72caa40a3" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 84d77408-8a03-45a0-8ef7-9bd72caa40a3
722026-07-29T01:26:22.681ZINFOcrucible-agent (worker): region files created ok region = 84d77408-8a03-45a0-8ef7-9bd72caa40a3
732026-07-29T01:26:22.681ZINFOcrucible-agent (datafile): region 84d77408-8a03-45a0-8ef7-9bd72caa40a3 state: Requested -> Created
742026-07-29T01:26:22.681ZINFOcrucible-agent (worker): applying SMF actions post create...
752026-07-29T01:26:22.682ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/0aca266c-8762-4650-93f8-641f5007c832
762026-07-29T01:26:22.682ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
772026-07-29T01:26:22.682ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::16
782026-07-29T01:26:22.682ZINFOcrucible-agent (worker): reconfig required
792026-07-29T01:26:22.682ZINFOcrucible-agent (worker): using existing config property group
802026-07-29T01:26:22.682ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-0aca266c-8762-4650-93f8-641f5007c832
812026-07-29T01:26:22.682ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/0aca266c-8762-4650-93f8-641f5007c832
822026-07-29T01:26:22.682ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
832026-07-29T01:26:22.682ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::16
842026-07-29T01:26:22.682ZINFOcrucible-agent (worker): commit
852026-07-29T01:26:22.686ZINFOcrucible-agent (worker): ok!
862026-07-29T01:26:22.687ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-84d77408-8a03-45a0-8ef7-9bd72caa40a3
872026-07-29T01:26:22.696ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-84d77408-8a03-45a0-8ef7-9bd72caa40a3
882026-07-29T01:26:22.707ZINFOcrucible-agent (worker): creating config property group
892026-07-29T01:26:22.711ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-84d77408-8a03-45a0-8ef7-9bd72caa40a3
902026-07-29T01:26:22.711ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/84d77408-8a03-45a0-8ef7-9bd72caa40a3
912026-07-29T01:26:22.711ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
922026-07-29T01:26:22.711ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::16
932026-07-29T01:26:22.711ZINFOcrucible-agent (worker): commit
942026-07-29T01:26:22.715ZINFOcrucible-agent (worker): ok!
952026-07-29T01:26:22.721ZINFOcrucible-agent (worker): SMF ok!
962026-07-29T01:26:22.895ZINFOcrucible-agent (dropshot): request completed latency_us = 61 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:36173 req_id = 9a59f252-e451-4246-a2ea-30fa8bfc89e9 response_code = 200 uri = /crucible/0/regions
972026-07-29T01:26:51.456ZINFOcrucible-agent (dropshot): request completed latency_us = 55 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:36173 req_id = 689a6d77-a8aa-40be-9efd-0ec371ff6b64 response_code = 200 uri = /crucible/0/regions/84d77408-8a03-45a0-8ef7-9bd72caa40a3
982026-07-29T01:26:51.460ZINFOcrucible-agent (datafile): path is "/data/regions/84d77408-8a03-45a0-8ef7-9bd72caa40a3"
992026-07-29T01:26:51.478ZINFOcrucible-agent (datafile): dataset is oxp_99e704a5-2c98-4cb5-8f68-e584bcbe9876/crucible/regions/84d77408-8a03-45a0-8ef7-9bd72caa40a3
1002026-07-29T01:26:51.486ZINFOcrucible-agent (datafile): region 84d77408-8a03-45a0-8ef7-9bd72caa40a3 state: Created -> Tombstoned
1012026-07-29T01:26:51.486ZINFOcrucible-agent (worker): applying SMF actions before removal...
1022026-07-29T01:26:51.488ZINFOcrucible-agent (dropshot): request completed latency_us = 28014 local_addr = [fd00:1122:3344:101::16]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:36173 req_id = 27627339-d618-4614-a5f0-93ec5388953b response_code = 204 uri = /crucible/0/regions/84d77408-8a03-45a0-8ef7-9bd72caa40a3
1032026-07-29T01:26:51.490ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-84d77408-8a03-45a0-8ef7-9bd72caa40a3 (instance states: (Some(Online), None))
1042026-07-29T01:26:51.491ZINFOcrucible-agent (dropshot): request completed latency_us = 37 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:36173 req_id = 20e7043a-e0c8-4c65-807c-4c5b8cb0ff21 response_code = 200 uri = /crucible/0/regions/84d77408-8a03-45a0-8ef7-9bd72caa40a3
1052026-07-29T01:26:51.515ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/0aca266c-8762-4650-93f8-641f5007c832
1062026-07-29T01:26:51.515ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
1072026-07-29T01:26:51.515ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::16
1082026-07-29T01:26:51.515ZINFOcrucible-agent (worker): reconfig required
1092026-07-29T01:26:51.515ZINFOcrucible-agent (worker): using existing config property group
1102026-07-29T01:26:51.515ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-0aca266c-8762-4650-93f8-641f5007c832
1112026-07-29T01:26:51.516ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/0aca266c-8762-4650-93f8-641f5007c832
1122026-07-29T01:26:51.516ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
1132026-07-29T01:26:51.516ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::16
1142026-07-29T01:26:51.516ZINFOcrucible-agent (worker): commit
1152026-07-29T01:26:51.531ZINFOcrucible-agent (worker): ok!
1162026-07-29T01:26:51.532ZINFOcrucible-agent (worker): SMF ok!
1172026-07-29T01:26:51.540ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_99e704a5-2c98-4cb5-8f68-e584bcbe9876/crucible/regions/84d77408-8a03-45a0-8ef7-9bd72caa40a3" region = 84d77408-8a03-45a0-8ef7-9bd72caa40a3
1182026-07-29T01:26:51.625ZINFOcrucible-agent (datafile): region 84d77408-8a03-45a0-8ef7-9bd72caa40a3 state: Tombstoned -> Destroyed
1192026-07-29T01:26:51.625ZINFOcrucible-agent (dropshot): request completed latency_us = 116 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:36173 req_id = 14c7de0a-de9d-4456-9c20-3089f5cf3bf7 response_code = 200 uri = /crucible/0/regions/84d77408-8a03-45a0-8ef7-9bd72caa40a3