Max level shown:
12026-07-21T19:15:02.434ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::1a]:32345 remote_addr = [fd00:1122:3344:101::a]:39353
22026-07-21T19:15:02.478ZINFOcrucible-agent (datafile): region 585a3e97-c603-4d2c-ba0b-cebad921a175 state: Requested
32026-07-21T19:15:02.484ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
42026-07-21T19:15:02.487ZINFOcrucible-agent (dropshot): request completed latency_us = 20261 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:39353 req_id = 8056e590-fc78-4dac-be1c-8a8023acf3fb response_code = 200 uri = /crucible/0/regions
52026-07-21T19:15:02.495ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_11da1c25-6699-487c-ab5a-3055928b001d/crucible/regions/585a3e97-c603-4d2c-ba0b-cebad921a175
62026-07-21T19:15:02.495ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_11da1c25-6699-487c-ab5a-3055928b001d/crucible/regions/585a3e97-c603-4d2c-ba0b-cebad921a175
72026-07-21T19:15:02.571ZINFOcrucible-agent (worker): creating region Region { id: RegionId("585a3e97-c603-4d2c-ba0b-cebad921a175"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/585a3e97-c603-4d2c-ba0b-cebad921a175" region = 585a3e97-c603-4d2c-ba0b-cebad921a175
82026-07-21T19:15:02.574ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "585a3e97-c603-4d2c-ba0b-cebad921a175" "--data" "/data/regions/585a3e97-c603-4d2c-ba0b-cebad921a175" "--block-size" "512" "--extent-size" "131072" "--extent-count" "16" "--encrypted" region = 585a3e97-c603-4d2c-ba0b-cebad921a175
92026-07-21T19:15:02.627ZINFOcrucible-agent (dropshot): request completed latency_us = 68 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:39353 req_id = 1906653a-ae50-40e6-a056-bfb869a26b3c response_code = 200 uri = /crucible/0/regions
102026-07-21T19:15:02.897ZINFOcrucible-agent (worker): region files created ok region = 585a3e97-c603-4d2c-ba0b-cebad921a175
112026-07-21T19:15:02.901ZINFOcrucible-agent (datafile): region 585a3e97-c603-4d2c-ba0b-cebad921a175 state: Requested -> Created
122026-07-21T19:15:02.901ZINFOcrucible-agent (worker): applying SMF actions post create...
132026-07-21T19:15:02.904ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-585a3e97-c603-4d2c-ba0b-cebad921a175
142026-07-21T19:15:02.911ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-585a3e97-c603-4d2c-ba0b-cebad921a175
152026-07-21T19:15:02.923ZINFOcrucible-agent (worker): creating config property group
162026-07-21T19:15:02.925ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-585a3e97-c603-4d2c-ba0b-cebad921a175
172026-07-21T19:15:02.926ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/585a3e97-c603-4d2c-ba0b-cebad921a175
182026-07-21T19:15:02.928ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
192026-07-21T19:15:02.928ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
202026-07-21T19:15:02.928ZINFOcrucible-agent (worker): commit
212026-07-21T19:15:02.941ZINFOcrucible-agent (worker): ok!
222026-07-21T19:15:02.944ZINFOcrucible-agent (worker): SMF ok!
232026-07-21T19:15:03.210ZINFOcrucible-agent (dropshot): request completed latency_us = 60 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:39353 req_id = 19d213dc-9034-4b2a-9024-b484b9116d5c response_code = 200 uri = /crucible/0/regions
242026-07-21T19:15:03.894ZINFOcrucible-agent (dropshot): request completed latency_us = 73 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:39353 req_id = 12ba4198-e68d-4fda-92dc-008dda6dd646 response_code = 200 uri = /crucible/0/regions/585a3e97-c603-4d2c-ba0b-cebad921a175
252026-07-21T19:15:03.897ZINFOcrucible-agent (datafile): path is "/data/regions/585a3e97-c603-4d2c-ba0b-cebad921a175"
262026-07-21T19:15:03.906ZINFOcrucible-agent (datafile): dataset is oxp_11da1c25-6699-487c-ab5a-3055928b001d/crucible/regions/585a3e97-c603-4d2c-ba0b-cebad921a175
272026-07-21T19:15:03.914ZINFOcrucible-agent (datafile): region 585a3e97-c603-4d2c-ba0b-cebad921a175 state: Created -> Tombstoned
282026-07-21T19:15:03.914ZINFOcrucible-agent (worker): applying SMF actions before removal...
292026-07-21T19:15:03.914ZINFOcrucible-agent (dropshot): request completed latency_us = 16840 local_addr = [fd00:1122:3344:101::1a]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:39353 req_id = d2348171-bc30-4ab9-84a2-b939129ede04 response_code = 204 uri = /crucible/0/regions/585a3e97-c603-4d2c-ba0b-cebad921a175
302026-07-21T19:15:03.915ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-585a3e97-c603-4d2c-ba0b-cebad921a175 (instance states: (Some(Online), None))
312026-07-21T19:15:03.918ZINFOcrucible-agent (worker): SMF ok!
322026-07-21T19:15:03.919ZINFOcrucible-agent (dropshot): request completed latency_us = 35 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:39353 req_id = 0821a9a6-fb54-4cb6-bfd4-3bc860d1b02a response_code = 200 uri = /crucible/0/regions/585a3e97-c603-4d2c-ba0b-cebad921a175
332026-07-21T19:15:03.927ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_11da1c25-6699-487c-ab5a-3055928b001d/crucible/regions/585a3e97-c603-4d2c-ba0b-cebad921a175" region = 585a3e97-c603-4d2c-ba0b-cebad921a175
342026-07-21T19:15:03.989ZINFOcrucible-agent (datafile): region 585a3e97-c603-4d2c-ba0b-cebad921a175 state: Tombstoned -> Destroyed
352026-07-21T19:15:04.008ZINFOcrucible-agent (dropshot): request completed latency_us = 48 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:39353 req_id = 8528e33c-642d-40f1-a24c-3f121cedb53f response_code = 200 uri = /crucible/0/regions/585a3e97-c603-4d2c-ba0b-cebad921a175
362026-07-21T19:15:06.249ZINFOcrucible-agent (datafile): region 9d940410-e969-4bc3-82ab-e552d532ac25 state: Requested
372026-07-21T19:15:06.249ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
382026-07-21T19:15:06.249ZINFOcrucible-agent (dropshot): request completed latency_us = 263 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:39353 req_id = c86bb6dc-6c23-466f-b265-aed4f1e5259c response_code = 200 uri = /crucible/0/regions
392026-07-21T19:15:06.255ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_11da1c25-6699-487c-ab5a-3055928b001d/crucible/regions/9d940410-e969-4bc3-82ab-e552d532ac25
402026-07-21T19:15:06.255ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_11da1c25-6699-487c-ab5a-3055928b001d/crucible/regions/9d940410-e969-4bc3-82ab-e552d532ac25
412026-07-21T19:15:06.316ZINFOcrucible-agent (worker): creating region Region { id: RegionId("9d940410-e969-4bc3-82ab-e552d532ac25"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/9d940410-e969-4bc3-82ab-e552d532ac25" region = 9d940410-e969-4bc3-82ab-e552d532ac25
422026-07-21T19:15:06.316ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "9d940410-e969-4bc3-82ab-e552d532ac25" "--data" "/data/regions/9d940410-e969-4bc3-82ab-e552d532ac25" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 9d940410-e969-4bc3-82ab-e552d532ac25
432026-07-21T19:15:06.363ZINFOcrucible-agent (worker): region files created ok region = 9d940410-e969-4bc3-82ab-e552d532ac25
442026-07-21T19:15:06.363ZINFOcrucible-agent (datafile): region 9d940410-e969-4bc3-82ab-e552d532ac25 state: Requested -> Created
452026-07-21T19:15:06.363ZINFOcrucible-agent (worker): applying SMF actions post create...
462026-07-21T19:15:06.364ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-9d940410-e969-4bc3-82ab-e552d532ac25
472026-07-21T19:15:06.366ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-9d940410-e969-4bc3-82ab-e552d532ac25
482026-07-21T19:15:06.370ZINFOcrucible-agent (worker): creating config property group
492026-07-21T19:15:06.371ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-9d940410-e969-4bc3-82ab-e552d532ac25
502026-07-21T19:15:06.372ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/9d940410-e969-4bc3-82ab-e552d532ac25
512026-07-21T19:15:06.372ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
522026-07-21T19:15:06.372ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
532026-07-21T19:15:06.372ZINFOcrucible-agent (worker): commit
542026-07-21T19:15:06.374ZINFOcrucible-agent (worker): ok!
552026-07-21T19:15:06.378ZINFOcrucible-agent (worker): SMF ok!
562026-07-21T19:15:06.381ZINFOcrucible-agent (dropshot): request completed latency_us = 53 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:39353 req_id = a9500b93-f3e5-410a-9baf-3706b92fc595 response_code = 200 uri = /crucible/0/regions
572026-07-21T19:26:06.111ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::1a]:32345 remote_addr = [fd00:1122:3344:101::a]:61273
582026-07-21T19:26:06.111ZINFOcrucible-agent (datafile): region 8af2bb1d-873a-463e-8f92-f1cb6a0c83c7 state: Requested
592026-07-21T19:26:06.112ZINFOcrucible-agent (dropshot): request completed latency_us = 821 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:61273 req_id = 85e1d76c-1724-4296-8b3e-1b1e641c3221 response_code = 200 uri = /crucible/0/regions
602026-07-21T19:26:06.112ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
612026-07-21T19:26:06.120ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_11da1c25-6699-487c-ab5a-3055928b001d/crucible/regions/8af2bb1d-873a-463e-8f92-f1cb6a0c83c7
622026-07-21T19:26:06.120ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_11da1c25-6699-487c-ab5a-3055928b001d/crucible/regions/8af2bb1d-873a-463e-8f92-f1cb6a0c83c7
632026-07-21T19:26:06.157ZINFOcrucible-agent (worker): creating region Region { id: RegionId("8af2bb1d-873a-463e-8f92-f1cb6a0c83c7"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/8af2bb1d-873a-463e-8f92-f1cb6a0c83c7" region = 8af2bb1d-873a-463e-8f92-f1cb6a0c83c7
642026-07-21T19:26:06.157ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "8af2bb1d-873a-463e-8f92-f1cb6a0c83c7" "--data" "/data/regions/8af2bb1d-873a-463e-8f92-f1cb6a0c83c7" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 8af2bb1d-873a-463e-8f92-f1cb6a0c83c7
652026-07-21T19:26:06.203ZINFOcrucible-agent (worker): region files created ok region = 8af2bb1d-873a-463e-8f92-f1cb6a0c83c7
662026-07-21T19:26:06.203ZINFOcrucible-agent (datafile): region 8af2bb1d-873a-463e-8f92-f1cb6a0c83c7 state: Requested -> Created
672026-07-21T19:26:06.203ZINFOcrucible-agent (worker): applying SMF actions post create...
682026-07-21T19:26:06.204ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-8af2bb1d-873a-463e-8f92-f1cb6a0c83c7
692026-07-21T19:26:06.214ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-8af2bb1d-873a-463e-8f92-f1cb6a0c83c7
702026-07-21T19:26:06.222ZINFOcrucible-agent (worker): creating config property group
712026-07-21T19:26:06.224ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-8af2bb1d-873a-463e-8f92-f1cb6a0c83c7
722026-07-21T19:26:06.224ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/8af2bb1d-873a-463e-8f92-f1cb6a0c83c7
732026-07-21T19:26:06.224ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
742026-07-21T19:26:06.224ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
752026-07-21T19:26:06.224ZINFOcrucible-agent (worker): commit
762026-07-21T19:26:06.226ZINFOcrucible-agent (worker): ok!
772026-07-21T19:26:06.230ZINFOcrucible-agent (worker): SMF ok!
782026-07-21T19:26:06.475ZINFOcrucible-agent (dropshot): request completed latency_us = 63 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:61273 req_id = a8a9503f-f07b-43d2-bba1-663aa7379ec3 response_code = 200 uri = /crucible/0/regions
792026-07-21T19:26:07.318ZINFOcrucible-agent (dropshot): request completed latency_us = 54 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:61273 req_id = 8b562295-29a6-418e-97a8-f425e07c63b0 response_code = 200 uri = /crucible/0/regions/9d940410-e969-4bc3-82ab-e552d532ac25
802026-07-21T19:26:07.323ZINFOcrucible-agent (datafile): path is "/data/regions/9d940410-e969-4bc3-82ab-e552d532ac25"
812026-07-21T19:26:07.329ZINFOcrucible-agent (datafile): dataset is oxp_11da1c25-6699-487c-ab5a-3055928b001d/crucible/regions/9d940410-e969-4bc3-82ab-e552d532ac25
822026-07-21T19:26:07.335ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_11da1c25-6699-487c-ab5a-3055928b001d/crucible/regions/9d940410-e969-4bc3-82ab-e552d532ac25@9f9ff737-c6d6-4528-a70d-75c746399b42
832026-07-21T19:26:07.335ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_11da1c25-6699-487c-ab5a-3055928b001d/crucible/regions/9d940410-e969-4bc3-82ab-e552d532ac25", "9f9ff737-c6d6-4528-a70d-75c746399b42"]
842026-07-21T19:26:07.347ZINFOcrucible-agent (dropshot): request completed latency_us = 24514 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:61273 req_id = 2e0b2951-5d3a-44af-b73d-c81006941a33 response_code = 200 uri = /crucible/0/regions/9d940410-e969-4bc3-82ab-e552d532ac25/snapshots/9f9ff737-c6d6-4528-a70d-75c746399b42
852026-07-21T19:26:07.359ZINFOcrucible-agent (datafile): path is "/data/regions/9d940410-e969-4bc3-82ab-e552d532ac25"
862026-07-21T19:26:07.365ZINFOcrucible-agent (datafile): dataset is oxp_11da1c25-6699-487c-ab5a-3055928b001d/crucible/regions/9d940410-e969-4bc3-82ab-e552d532ac25
872026-07-21T19:26:07.370ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_11da1c25-6699-487c-ab5a-3055928b001d/crucible/regions/9d940410-e969-4bc3-82ab-e552d532ac25@9f9ff737-c6d6-4528-a70d-75c746399b42
882026-07-21T19:26:07.370ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_11da1c25-6699-487c-ab5a-3055928b001d/crucible/regions/9d940410-e969-4bc3-82ab-e552d532ac25", "9f9ff737-c6d6-4528-a70d-75c746399b42"]
892026-07-21T19:26:07.376ZINFOcrucible-agent (datafile): requesting running snapshot 9d940410-e969-4bc3-82ab-e552d532ac25-9f9ff737-c6d6-4528-a70d-75c746399b42 state: Requested
902026-07-21T19:26:07.378ZINFOcrucible-agent (worker): applying SMF actions for region 9d940410-e969-4bc3-82ab-e552d532ac25 running snapshot 9f9ff737-c6d6-4528-a70d-75c746399b42 (state Requested)...
912026-07-21T19:26:07.378ZINFOcrucible-agent (dropshot): request completed latency_us = 19695 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:61273 req_id = 4da1121f-a2a2-4e0f-a155-ea5178097931 response_code = 200 uri = /crucible/0/regions/9d940410-e969-4bc3-82ab-e552d532ac25/snapshots/9f9ff737-c6d6-4528-a70d-75c746399b42/run
922026-07-21T19:26:07.383ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-9d940410-e969-4bc3-82ab-e552d532ac25-9f9ff737-c6d6-4528-a70d-75c746399b42
932026-07-21T19:26:07.385ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-9d940410-e969-4bc3-82ab-e552d532ac25-9f9ff737-c6d6-4528-a70d-75c746399b42
942026-07-21T19:26:07.390ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-9d940410-e969-4bc3-82ab-e552d532ac25-9f9ff737-c6d6-4528-a70d-75c746399b42
952026-07-21T19:26:07.390ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/9d940410-e969-4bc3-82ab-e552d532ac25/.zfs/snapshot/9f9ff737-c6d6-4528-a70d-75c746399b42
962026-07-21T19:26:07.390ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
972026-07-21T19:26:07.391ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
982026-07-21T19:26:07.391ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
992026-07-21T19:26:07.391ZINFOcrucible-agent (worker): commit
1002026-07-21T19:26:07.393ZINFOcrucible-agent (worker): ok!
1012026-07-21T19:26:07.396ZINFOcrucible-agent (worker): SMF ok!
1022026-07-21T19:26:07.396ZINFOcrucible-agent (datafile): region 9d940410-e969-4bc3-82ab-e552d532ac25 running snapshot 9f9ff737-c6d6-4528-a70d-75c746399b42 state: Requested -> Created
1032026-07-21T19:26:07.588ZINFOcrucible-agent (datafile): path is "/data/regions/9d940410-e969-4bc3-82ab-e552d532ac25"
1042026-07-21T19:26:07.595ZINFOcrucible-agent (datafile): dataset is oxp_11da1c25-6699-487c-ab5a-3055928b001d/crucible/regions/9d940410-e969-4bc3-82ab-e552d532ac25
1052026-07-21T19:26:07.601ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_11da1c25-6699-487c-ab5a-3055928b001d/crucible/regions/9d940410-e969-4bc3-82ab-e552d532ac25@9f9ff737-c6d6-4528-a70d-75c746399b42
1062026-07-21T19:26:07.601ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_11da1c25-6699-487c-ab5a-3055928b001d/crucible/regions/9d940410-e969-4bc3-82ab-e552d532ac25", "9f9ff737-c6d6-4528-a70d-75c746399b42"]
1072026-07-21T19:26:07.608ZINFOcrucible-agent (dropshot): request completed latency_us = 19963 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:61273 req_id = 95dcf40b-06d3-4b8c-824b-80ea128b3cab response_code = 200 uri = /crucible/0/regions/9d940410-e969-4bc3-82ab-e552d532ac25/snapshots/9f9ff737-c6d6-4528-a70d-75c746399b42/run