Max level shown:
12026-07-02T21:46:05.739ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::b]:64965
22026-07-02T21:46:05.771ZINFOcrucible-agent (datafile): region 5409600d-4bbb-4f5e-87f8-b563ce471303 state: Requested
32026-07-02T21:46:05.775ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
42026-07-02T21:46:05.777ZINFOcrucible-agent (dropshot): request completed latency_us = 14937 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:64965 req_id = f3122d8c-4aa8-4327-a104-49ec287fce46 response_code = 200 uri = /crucible/0/regions
52026-07-02T21:46:05.784ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_31d9609e-d3d0-42f6-b828-d7459f4ff787/crucible/regions/5409600d-4bbb-4f5e-87f8-b563ce471303
62026-07-02T21:46:05.784ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_31d9609e-d3d0-42f6-b828-d7459f4ff787/crucible/regions/5409600d-4bbb-4f5e-87f8-b563ce471303
72026-07-02T21:46:05.824ZINFOcrucible-agent (worker): creating region Region { id: RegionId("5409600d-4bbb-4f5e-87f8-b563ce471303"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/5409600d-4bbb-4f5e-87f8-b563ce471303" region = 5409600d-4bbb-4f5e-87f8-b563ce471303
82026-07-02T21:46:05.826ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "5409600d-4bbb-4f5e-87f8-b563ce471303" "--data" "/data/regions/5409600d-4bbb-4f5e-87f8-b563ce471303" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 5409600d-4bbb-4f5e-87f8-b563ce471303
92026-07-02T21:46:05.954ZINFOcrucible-agent (worker): region files created ok region = 5409600d-4bbb-4f5e-87f8-b563ce471303
102026-07-02T21:46:05.956ZINFOcrucible-agent (datafile): region 5409600d-4bbb-4f5e-87f8-b563ce471303 state: Requested -> Created
112026-07-02T21:46:05.956ZINFOcrucible-agent (worker): applying SMF actions post create...
122026-07-02T21:46:05.959ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-5409600d-4bbb-4f5e-87f8-b563ce471303
132026-07-02T21:46:05.965ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-5409600d-4bbb-4f5e-87f8-b563ce471303
142026-07-02T21:46:05.976ZINFOcrucible-agent (worker): creating config property group
152026-07-02T21:46:05.978ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-5409600d-4bbb-4f5e-87f8-b563ce471303
162026-07-02T21:46:05.979ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/5409600d-4bbb-4f5e-87f8-b563ce471303
172026-07-02T21:46:05.981ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
182026-07-02T21:46:05.981ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
192026-07-02T21:46:05.981ZINFOcrucible-agent (worker): commit
202026-07-02T21:46:05.989ZINFOcrucible-agent (worker): ok!
212026-07-02T21:46:05.991ZINFOcrucible-agent (worker): SMF ok!
222026-07-02T21:46:06.084ZINFOcrucible-agent (dropshot): request completed latency_us = 75 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:64965 req_id = dec469fd-1d08-40b3-a65a-0d18e79605d0 response_code = 200 uri = /crucible/0/regions
232026-07-02T21:57:17.238ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::b]:40159
242026-07-02T21:57:17.238ZINFOcrucible-agent (dropshot): request completed latency_us = 52 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:40159 req_id = dbfe8cc9-ce7e-4f2e-a517-c8fa586ab54d response_code = 200 uri = /crucible/0/regions/5409600d-4bbb-4f5e-87f8-b563ce471303
252026-07-02T21:57:17.242ZINFOcrucible-agent (datafile): path is "/data/regions/5409600d-4bbb-4f5e-87f8-b563ce471303"
262026-07-02T21:57:17.248ZINFOcrucible-agent (datafile): dataset is oxp_31d9609e-d3d0-42f6-b828-d7459f4ff787/crucible/regions/5409600d-4bbb-4f5e-87f8-b563ce471303
272026-07-02T21:57:17.253ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_31d9609e-d3d0-42f6-b828-d7459f4ff787/crucible/regions/5409600d-4bbb-4f5e-87f8-b563ce471303@d2f97121-c40c-4d44-ac8d-f835aea2eca9
282026-07-02T21:57:17.253ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_31d9609e-d3d0-42f6-b828-d7459f4ff787/crucible/regions/5409600d-4bbb-4f5e-87f8-b563ce471303", "d2f97121-c40c-4d44-ac8d-f835aea2eca9"]
292026-07-02T21:57:17.266ZINFOcrucible-agent (dropshot): request completed latency_us = 23926 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:40159 req_id = 15175891-6a75-4b39-8d81-d22e7f5f1560 response_code = 200 uri = /crucible/0/regions/5409600d-4bbb-4f5e-87f8-b563ce471303/snapshots/d2f97121-c40c-4d44-ac8d-f835aea2eca9
302026-07-02T21:57:17.269ZINFOcrucible-agent (datafile): path is "/data/regions/5409600d-4bbb-4f5e-87f8-b563ce471303"
312026-07-02T21:57:17.275ZINFOcrucible-agent (datafile): dataset is oxp_31d9609e-d3d0-42f6-b828-d7459f4ff787/crucible/regions/5409600d-4bbb-4f5e-87f8-b563ce471303
322026-07-02T21:57:17.281ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_31d9609e-d3d0-42f6-b828-d7459f4ff787/crucible/regions/5409600d-4bbb-4f5e-87f8-b563ce471303@d2f97121-c40c-4d44-ac8d-f835aea2eca9
332026-07-02T21:57:17.281ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_31d9609e-d3d0-42f6-b828-d7459f4ff787/crucible/regions/5409600d-4bbb-4f5e-87f8-b563ce471303", "d2f97121-c40c-4d44-ac8d-f835aea2eca9"]
342026-07-02T21:57:17.291ZINFOcrucible-agent (datafile): requesting running snapshot 5409600d-4bbb-4f5e-87f8-b563ce471303-d2f97121-c40c-4d44-ac8d-f835aea2eca9 state: Requested
352026-07-02T21:57:17.294ZINFOcrucible-agent (worker): applying SMF actions for region 5409600d-4bbb-4f5e-87f8-b563ce471303 running snapshot d2f97121-c40c-4d44-ac8d-f835aea2eca9 (state Requested)...
362026-07-02T21:57:17.294ZINFOcrucible-agent (dropshot): request completed latency_us = 24966 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:40159 req_id = 128b5c6f-a00f-4d85-bb4c-0724090dc0f7 response_code = 200 uri = /crucible/0/regions/5409600d-4bbb-4f5e-87f8-b563ce471303/snapshots/d2f97121-c40c-4d44-ac8d-f835aea2eca9/run
372026-07-02T21:57:17.308ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/5409600d-4bbb-4f5e-87f8-b563ce471303
382026-07-02T21:57:17.308ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
392026-07-02T21:57:17.308ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::12
402026-07-02T21:57:17.308ZINFOcrucible-agent (worker): reconfig required
412026-07-02T21:57:17.308ZINFOcrucible-agent (worker): using existing config property group
422026-07-02T21:57:17.308ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-5409600d-4bbb-4f5e-87f8-b563ce471303
432026-07-02T21:57:17.309ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/5409600d-4bbb-4f5e-87f8-b563ce471303
442026-07-02T21:57:17.309ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
452026-07-02T21:57:17.309ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
462026-07-02T21:57:17.309ZINFOcrucible-agent (worker): commit
472026-07-02T21:57:17.315ZINFOcrucible-agent (worker): ok!
482026-07-02T21:57:17.319ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-5409600d-4bbb-4f5e-87f8-b563ce471303-d2f97121-c40c-4d44-ac8d-f835aea2eca9
492026-07-02T21:57:17.320ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-5409600d-4bbb-4f5e-87f8-b563ce471303-d2f97121-c40c-4d44-ac8d-f835aea2eca9
502026-07-02T21:57:17.328ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-5409600d-4bbb-4f5e-87f8-b563ce471303-d2f97121-c40c-4d44-ac8d-f835aea2eca9
512026-07-02T21:57:17.328ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/5409600d-4bbb-4f5e-87f8-b563ce471303/.zfs/snapshot/d2f97121-c40c-4d44-ac8d-f835aea2eca9
522026-07-02T21:57:17.328ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
532026-07-02T21:57:17.328ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
542026-07-02T21:57:17.328ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
552026-07-02T21:57:17.328ZINFOcrucible-agent (worker): commit
562026-07-02T21:57:17.330ZINFOcrucible-agent (worker): ok!
572026-07-02T21:57:17.333ZINFOcrucible-agent (worker): SMF ok!
582026-07-02T21:57:17.333ZINFOcrucible-agent (datafile): region 5409600d-4bbb-4f5e-87f8-b563ce471303 running snapshot d2f97121-c40c-4d44-ac8d-f835aea2eca9 state: Requested -> Created
592026-07-02T21:57:17.465ZINFOcrucible-agent (datafile): path is "/data/regions/5409600d-4bbb-4f5e-87f8-b563ce471303"
602026-07-02T21:57:17.471ZINFOcrucible-agent (datafile): dataset is oxp_31d9609e-d3d0-42f6-b828-d7459f4ff787/crucible/regions/5409600d-4bbb-4f5e-87f8-b563ce471303
612026-07-02T21:57:17.478ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_31d9609e-d3d0-42f6-b828-d7459f4ff787/crucible/regions/5409600d-4bbb-4f5e-87f8-b563ce471303@d2f97121-c40c-4d44-ac8d-f835aea2eca9
622026-07-02T21:57:17.478ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_31d9609e-d3d0-42f6-b828-d7459f4ff787/crucible/regions/5409600d-4bbb-4f5e-87f8-b563ce471303", "d2f97121-c40c-4d44-ac8d-f835aea2eca9"]
632026-07-02T21:57:17.483ZINFOcrucible-agent (dropshot): request completed latency_us = 18412 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:40159 req_id = 96f388d4-84e9-4207-8326-78010e6b1249 response_code = 200 uri = /crucible/0/regions/5409600d-4bbb-4f5e-87f8-b563ce471303/snapshots/d2f97121-c40c-4d44-ac8d-f835aea2eca9/run
642026-07-02T21:57:22.151ZINFOcrucible-agent (datafile): region 98f20bee-e460-4f4f-9bc8-3a28fd02b0e3 state: Requested
652026-07-02T21:57:22.152ZINFOcrucible-agent (dropshot): request completed latency_us = 220 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:40159 req_id = 637d4ea5-b006-499b-9689-700e3b54cf49 response_code = 200 uri = /crucible/0/regions
662026-07-02T21:57:22.152ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
672026-07-02T21:57:22.158ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_31d9609e-d3d0-42f6-b828-d7459f4ff787/crucible/regions/98f20bee-e460-4f4f-9bc8-3a28fd02b0e3
682026-07-02T21:57:22.158ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_31d9609e-d3d0-42f6-b828-d7459f4ff787/crucible/regions/98f20bee-e460-4f4f-9bc8-3a28fd02b0e3
692026-07-02T21:57:22.215ZINFOcrucible-agent (worker): creating region Region { id: RegionId("98f20bee-e460-4f4f-9bc8-3a28fd02b0e3"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19002, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/98f20bee-e460-4f4f-9bc8-3a28fd02b0e3" region = 98f20bee-e460-4f4f-9bc8-3a28fd02b0e3
702026-07-02T21:57:22.215ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "98f20bee-e460-4f4f-9bc8-3a28fd02b0e3" "--data" "/data/regions/98f20bee-e460-4f4f-9bc8-3a28fd02b0e3" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 98f20bee-e460-4f4f-9bc8-3a28fd02b0e3
712026-07-02T21:57:22.260ZINFOcrucible-agent (worker): region files created ok region = 98f20bee-e460-4f4f-9bc8-3a28fd02b0e3
722026-07-02T21:57:22.260ZINFOcrucible-agent (datafile): region 98f20bee-e460-4f4f-9bc8-3a28fd02b0e3 state: Requested -> Created
732026-07-02T21:57:22.260ZINFOcrucible-agent (worker): applying SMF actions post create...
742026-07-02T21:57:22.261ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/5409600d-4bbb-4f5e-87f8-b563ce471303
752026-07-02T21:57:22.261ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
762026-07-02T21:57:22.261ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::12
772026-07-02T21:57:22.261ZINFOcrucible-agent (worker): reconfig required
782026-07-02T21:57:22.261ZINFOcrucible-agent (worker): using existing config property group
792026-07-02T21:57:22.261ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-5409600d-4bbb-4f5e-87f8-b563ce471303
802026-07-02T21:57:22.261ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/5409600d-4bbb-4f5e-87f8-b563ce471303
812026-07-02T21:57:22.261ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
822026-07-02T21:57:22.261ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
832026-07-02T21:57:22.261ZINFOcrucible-agent (worker): commit
842026-07-02T21:57:22.263ZINFOcrucible-agent (worker): ok!
852026-07-02T21:57:22.264ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-98f20bee-e460-4f4f-9bc8-3a28fd02b0e3
862026-07-02T21:57:22.272ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-98f20bee-e460-4f4f-9bc8-3a28fd02b0e3
872026-07-02T21:57:22.276ZINFOcrucible-agent (worker): creating config property group
882026-07-02T21:57:22.277ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-98f20bee-e460-4f4f-9bc8-3a28fd02b0e3
892026-07-02T21:57:22.277ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/98f20bee-e460-4f4f-9bc8-3a28fd02b0e3
902026-07-02T21:57:22.277ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
912026-07-02T21:57:22.277ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
922026-07-02T21:57:22.277ZINFOcrucible-agent (worker): commit
932026-07-02T21:57:22.279ZINFOcrucible-agent (worker): ok!
942026-07-02T21:57:22.283ZINFOcrucible-agent (worker): SMF ok!
952026-07-02T21:57:22.343ZINFOcrucible-agent (dropshot): request completed latency_us = 61 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:40159 req_id = a7d8bbef-de50-4eae-9e7a-f1ee9576695b response_code = 200 uri = /crucible/0/regions
962026-07-02T21:57:47.296ZINFOcrucible-agent (dropshot): request completed latency_us = 59 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:40159 req_id = 82f4fd41-1575-483a-9e2d-a0550a6d1ec2 response_code = 200 uri = /crucible/0/regions/98f20bee-e460-4f4f-9bc8-3a28fd02b0e3
972026-07-02T21:57:47.299ZINFOcrucible-agent (datafile): path is "/data/regions/98f20bee-e460-4f4f-9bc8-3a28fd02b0e3"
982026-07-02T21:57:47.309ZINFOcrucible-agent (datafile): dataset is oxp_31d9609e-d3d0-42f6-b828-d7459f4ff787/crucible/regions/98f20bee-e460-4f4f-9bc8-3a28fd02b0e3
992026-07-02T21:57:47.315ZINFOcrucible-agent (datafile): region 98f20bee-e460-4f4f-9bc8-3a28fd02b0e3 state: Created -> Tombstoned
1002026-07-02T21:57:47.316ZINFOcrucible-agent (worker): applying SMF actions before removal...
1012026-07-02T21:57:47.318ZINFOcrucible-agent (dropshot): request completed latency_us = 19803 local_addr = [fd00:1122:3344:101::12]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::b]:40159 req_id = 50057a42-38f7-470b-b679-b7c005bf979a response_code = 204 uri = /crucible/0/regions/98f20bee-e460-4f4f-9bc8-3a28fd02b0e3
1022026-07-02T21:57:47.319ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-98f20bee-e460-4f4f-9bc8-3a28fd02b0e3 (instance states: (Some(Online), None))
1032026-07-02T21:57:47.322ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/5409600d-4bbb-4f5e-87f8-b563ce471303
1042026-07-02T21:57:47.322ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
1052026-07-02T21:57:47.323ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::12
1062026-07-02T21:57:47.323ZINFOcrucible-agent (worker): reconfig required
1072026-07-02T21:57:47.323ZINFOcrucible-agent (worker): using existing config property group
1082026-07-02T21:57:47.323ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-5409600d-4bbb-4f5e-87f8-b563ce471303
1092026-07-02T21:57:47.323ZINFOcrucible-agent (dropshot): request completed latency_us = 41 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:40159 req_id = 8574c18f-b06c-475a-822f-b9e1110ced05 response_code = 200 uri = /crucible/0/regions/98f20bee-e460-4f4f-9bc8-3a28fd02b0e3
1102026-07-02T21:57:47.323ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/5409600d-4bbb-4f5e-87f8-b563ce471303
1112026-07-02T21:57:47.323ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
1122026-07-02T21:57:47.323ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
1132026-07-02T21:57:47.323ZINFOcrucible-agent (worker): commit
1142026-07-02T21:57:47.325ZINFOcrucible-agent (worker): ok!
1152026-07-02T21:57:47.327ZINFOcrucible-agent (worker): SMF ok!
1162026-07-02T21:57:47.334ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_31d9609e-d3d0-42f6-b828-d7459f4ff787/crucible/regions/98f20bee-e460-4f4f-9bc8-3a28fd02b0e3" region = 98f20bee-e460-4f4f-9bc8-3a28fd02b0e3
1172026-07-02T21:57:47.379ZINFOcrucible-agent (datafile): region 98f20bee-e460-4f4f-9bc8-3a28fd02b0e3 state: Tombstoned -> Destroyed
1182026-07-02T21:57:47.400ZINFOcrucible-agent (dropshot): request completed latency_us = 53 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:40159 req_id = cb9c956b-16be-4bcc-a02f-4bd134146df3 response_code = 200 uri = /crucible/0/regions/98f20bee-e460-4f4f-9bc8-3a28fd02b0e3