Max level shown:
12026-08-07T17:33:57.500ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::18]:32345 remote_addr = [fd00:1122:3344:101::c]:62969
22026-08-07T17:33:57.500ZINFOcrucible-agent (datafile): region a92abfc1-0f92-4c17-b774-ca3ae0ef1600 state: Requested
32026-08-07T17:33:57.501ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
42026-08-07T17:33:57.501ZINFOcrucible-agent (dropshot): request completed latency_us = 811 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:62969 req_id = 13e1fa06-48cf-49ad-aa7c-749094c55c01 response_code = 200 uri = /crucible/0/regions
52026-08-07T17:33:57.511ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_3e9f92e1-b5f2-4c05-b3d0-3c468fb47ca6/crucible/regions/a92abfc1-0f92-4c17-b774-ca3ae0ef1600
62026-08-07T17:33:57.511ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_3e9f92e1-b5f2-4c05-b3d0-3c468fb47ca6/crucible/regions/a92abfc1-0f92-4c17-b774-ca3ae0ef1600
72026-08-07T17:33:57.554ZINFOcrucible-agent (worker): creating region Region { id: RegionId("a92abfc1-0f92-4c17-b774-ca3ae0ef1600"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/a92abfc1-0f92-4c17-b774-ca3ae0ef1600" region = a92abfc1-0f92-4c17-b774-ca3ae0ef1600
82026-08-07T17:33:57.554ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "a92abfc1-0f92-4c17-b774-ca3ae0ef1600" "--data" "/data/regions/a92abfc1-0f92-4c17-b774-ca3ae0ef1600" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = a92abfc1-0f92-4c17-b774-ca3ae0ef1600
92026-08-07T17:33:57.605ZINFOcrucible-agent (worker): region files created ok region = a92abfc1-0f92-4c17-b774-ca3ae0ef1600
102026-08-07T17:33:57.605ZINFOcrucible-agent (datafile): region a92abfc1-0f92-4c17-b774-ca3ae0ef1600 state: Requested -> Created
112026-08-07T17:33:57.605ZINFOcrucible-agent (worker): applying SMF actions post create...
122026-08-07T17:33:57.606ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-a92abfc1-0f92-4c17-b774-ca3ae0ef1600
132026-08-07T17:33:57.614ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-a92abfc1-0f92-4c17-b774-ca3ae0ef1600
142026-08-07T17:33:57.637ZINFOcrucible-agent (worker): creating config property group
152026-08-07T17:33:57.639ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-a92abfc1-0f92-4c17-b774-ca3ae0ef1600
162026-08-07T17:33:57.639ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/a92abfc1-0f92-4c17-b774-ca3ae0ef1600
172026-08-07T17:33:57.639ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
182026-08-07T17:33:57.639ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
192026-08-07T17:33:57.639ZINFOcrucible-agent (worker): commit
202026-08-07T17:33:57.641ZINFOcrucible-agent (worker): ok!
212026-08-07T17:33:57.645ZINFOcrucible-agent (worker): SMF ok!
222026-08-07T17:33:57.673ZINFOcrucible-agent (dropshot): request completed latency_us = 60 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:62969 req_id = ff248163-59d1-4f17-bc04-2639b96217d8 response_code = 200 uri = /crucible/0/regions
232026-08-07T17:33:59.023ZINFOcrucible-agent (dropshot): request completed latency_us = 84 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:62969 req_id = 3c5ba1d4-69a2-4a51-a37f-f8c3d6dc8bc6 response_code = 200 uri = /crucible/0/regions/dc91e91e-2e4d-48fb-828a-7991d9677452
242026-08-07T17:33:59.026ZINFOcrucible-agent (datafile): path is "/data/regions/dc91e91e-2e4d-48fb-828a-7991d9677452"
252026-08-07T17:33:59.033ZINFOcrucible-agent (datafile): dataset is oxp_3e9f92e1-b5f2-4c05-b3d0-3c468fb47ca6/crucible/regions/dc91e91e-2e4d-48fb-828a-7991d9677452
262026-08-07T17:33:59.039ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_3e9f92e1-b5f2-4c05-b3d0-3c468fb47ca6/crucible/regions/dc91e91e-2e4d-48fb-828a-7991d9677452@15102f16-d9dc-493d-8b66-87c4742c8e5c
272026-08-07T17:33:59.039ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_3e9f92e1-b5f2-4c05-b3d0-3c468fb47ca6/crucible/regions/dc91e91e-2e4d-48fb-828a-7991d9677452", "15102f16-d9dc-493d-8b66-87c4742c8e5c"]
282026-08-07T17:33:59.051ZINFOcrucible-agent (dropshot): request completed latency_us = 25130 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:62969 req_id = 8f27fadd-1c51-4df3-8b6d-c1970d852c0a response_code = 200 uri = /crucible/0/regions/dc91e91e-2e4d-48fb-828a-7991d9677452/snapshots/15102f16-d9dc-493d-8b66-87c4742c8e5c
292026-08-07T17:33:59.059ZINFOcrucible-agent (datafile): path is "/data/regions/dc91e91e-2e4d-48fb-828a-7991d9677452"
302026-08-07T17:33:59.066ZINFOcrucible-agent (datafile): dataset is oxp_3e9f92e1-b5f2-4c05-b3d0-3c468fb47ca6/crucible/regions/dc91e91e-2e4d-48fb-828a-7991d9677452
312026-08-07T17:33:59.073ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_3e9f92e1-b5f2-4c05-b3d0-3c468fb47ca6/crucible/regions/dc91e91e-2e4d-48fb-828a-7991d9677452@15102f16-d9dc-493d-8b66-87c4742c8e5c
322026-08-07T17:33:59.073ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_3e9f92e1-b5f2-4c05-b3d0-3c468fb47ca6/crucible/regions/dc91e91e-2e4d-48fb-828a-7991d9677452", "15102f16-d9dc-493d-8b66-87c4742c8e5c"]
332026-08-07T17:33:59.081ZINFOcrucible-agent (datafile): requesting running snapshot dc91e91e-2e4d-48fb-828a-7991d9677452-15102f16-d9dc-493d-8b66-87c4742c8e5c state: Requested
342026-08-07T17:33:59.086ZINFOcrucible-agent (worker): applying SMF actions for region dc91e91e-2e4d-48fb-828a-7991d9677452 running snapshot 15102f16-d9dc-493d-8b66-87c4742c8e5c (state Requested)...
352026-08-07T17:33:59.086ZINFOcrucible-agent (dropshot): request completed latency_us = 26969 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:62969 req_id = 558b2cee-b62b-4d1b-a77c-38c38984910c response_code = 200 uri = /crucible/0/regions/dc91e91e-2e4d-48fb-828a-7991d9677452/snapshots/15102f16-d9dc-493d-8b66-87c4742c8e5c/run
362026-08-07T17:33:59.089ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-dc91e91e-2e4d-48fb-828a-7991d9677452-15102f16-d9dc-493d-8b66-87c4742c8e5c
372026-08-07T17:33:59.098ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-dc91e91e-2e4d-48fb-828a-7991d9677452-15102f16-d9dc-493d-8b66-87c4742c8e5c
382026-08-07T17:33:59.103ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-dc91e91e-2e4d-48fb-828a-7991d9677452-15102f16-d9dc-493d-8b66-87c4742c8e5c
392026-08-07T17:33:59.103ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/dc91e91e-2e4d-48fb-828a-7991d9677452/.zfs/snapshot/15102f16-d9dc-493d-8b66-87c4742c8e5c
402026-08-07T17:33:59.103ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
412026-08-07T17:33:59.103ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
422026-08-07T17:33:59.103ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
432026-08-07T17:33:59.103ZINFOcrucible-agent (worker): commit
442026-08-07T17:33:59.105ZINFOcrucible-agent (worker): ok!
452026-08-07T17:33:59.108ZINFOcrucible-agent (worker): SMF ok!
462026-08-07T17:33:59.108ZINFOcrucible-agent (datafile): region dc91e91e-2e4d-48fb-828a-7991d9677452 running snapshot 15102f16-d9dc-493d-8b66-87c4742c8e5c state: Requested -> Created
472026-08-07T17:33:59.292ZINFOcrucible-agent (datafile): path is "/data/regions/dc91e91e-2e4d-48fb-828a-7991d9677452"
482026-08-07T17:33:59.298ZINFOcrucible-agent (datafile): dataset is oxp_3e9f92e1-b5f2-4c05-b3d0-3c468fb47ca6/crucible/regions/dc91e91e-2e4d-48fb-828a-7991d9677452
492026-08-07T17:33:59.304ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_3e9f92e1-b5f2-4c05-b3d0-3c468fb47ca6/crucible/regions/dc91e91e-2e4d-48fb-828a-7991d9677452@15102f16-d9dc-493d-8b66-87c4742c8e5c
502026-08-07T17:33:59.304ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_3e9f92e1-b5f2-4c05-b3d0-3c468fb47ca6/crucible/regions/dc91e91e-2e4d-48fb-828a-7991d9677452", "15102f16-d9dc-493d-8b66-87c4742c8e5c"]
512026-08-07T17:33:59.310ZINFOcrucible-agent (dropshot): request completed latency_us = 18482 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:62969 req_id = 87af82e6-7b13-40d9-bf56-9fb993e5acc5 response_code = 200 uri = /crucible/0/regions/dc91e91e-2e4d-48fb-828a-7991d9677452/snapshots/15102f16-d9dc-493d-8b66-87c4742c8e5c/run
522026-08-07T17:34:04.960ZINFOcrucible-agent (datafile): region 048cdf0c-8d41-40fb-b9ed-91434f09b2d4 state: Requested
532026-08-07T17:34:04.960ZINFOcrucible-agent (dropshot): request completed latency_us = 228 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:62969 req_id = b87982af-007c-454b-857b-192713674f2e response_code = 200 uri = /crucible/0/regions
542026-08-07T17:34:04.960ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
552026-08-07T17:34:04.967ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_3e9f92e1-b5f2-4c05-b3d0-3c468fb47ca6/crucible/regions/048cdf0c-8d41-40fb-b9ed-91434f09b2d4
562026-08-07T17:34:04.967ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_3e9f92e1-b5f2-4c05-b3d0-3c468fb47ca6/crucible/regions/048cdf0c-8d41-40fb-b9ed-91434f09b2d4
572026-08-07T17:34:05.000ZINFOcrucible-agent (worker): creating region Region { id: RegionId("048cdf0c-8d41-40fb-b9ed-91434f09b2d4"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19003, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/048cdf0c-8d41-40fb-b9ed-91434f09b2d4" region = 048cdf0c-8d41-40fb-b9ed-91434f09b2d4
582026-08-07T17:34:05.000ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "048cdf0c-8d41-40fb-b9ed-91434f09b2d4" "--data" "/data/regions/048cdf0c-8d41-40fb-b9ed-91434f09b2d4" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 048cdf0c-8d41-40fb-b9ed-91434f09b2d4
592026-08-07T17:34:05.044ZINFOcrucible-agent (worker): region files created ok region = 048cdf0c-8d41-40fb-b9ed-91434f09b2d4
602026-08-07T17:34:05.044ZINFOcrucible-agent (datafile): region 048cdf0c-8d41-40fb-b9ed-91434f09b2d4 state: Requested -> Created
612026-08-07T17:34:05.044ZINFOcrucible-agent (worker): applying SMF actions post create...
622026-08-07T17:34:05.044ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-048cdf0c-8d41-40fb-b9ed-91434f09b2d4
632026-08-07T17:34:05.046ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-048cdf0c-8d41-40fb-b9ed-91434f09b2d4
642026-08-07T17:34:05.049ZINFOcrucible-agent (worker): creating config property group
652026-08-07T17:34:05.051ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-048cdf0c-8d41-40fb-b9ed-91434f09b2d4
662026-08-07T17:34:05.051ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/048cdf0c-8d41-40fb-b9ed-91434f09b2d4
672026-08-07T17:34:05.051ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19003
682026-08-07T17:34:05.051ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
692026-08-07T17:34:05.051ZINFOcrucible-agent (worker): commit
702026-08-07T17:34:05.053ZINFOcrucible-agent (worker): ok!
712026-08-07T17:34:05.059ZINFOcrucible-agent (worker): SMF ok!
722026-08-07T17:34:05.211ZINFOcrucible-agent (dropshot): request completed latency_us = 56 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:62969 req_id = b4fe5cb1-646d-4d15-bbc1-d45b03ba27cf response_code = 200 uri = /crucible/0/regions
732026-08-07T17:34:31.926ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::18]:32345 remote_addr = [fd00:1122:3344:101::c]:48512
742026-08-07T17:34:31.926ZINFOcrucible-agent (dropshot): request completed latency_us = 48 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:48512 req_id = d09639c5-7e04-471a-a316-ad0138be7179 response_code = 200 uri = /crucible/0/regions/048cdf0c-8d41-40fb-b9ed-91434f09b2d4
752026-08-07T17:34:31.930ZINFOcrucible-agent (datafile): path is "/data/regions/048cdf0c-8d41-40fb-b9ed-91434f09b2d4"
762026-08-07T17:34:31.936ZINFOcrucible-agent (datafile): dataset is oxp_3e9f92e1-b5f2-4c05-b3d0-3c468fb47ca6/crucible/regions/048cdf0c-8d41-40fb-b9ed-91434f09b2d4
772026-08-07T17:34:31.943ZINFOcrucible-agent (datafile): region 048cdf0c-8d41-40fb-b9ed-91434f09b2d4 state: Created -> Tombstoned
782026-08-07T17:34:31.943ZINFOcrucible-agent (worker): applying SMF actions before removal...
792026-08-07T17:34:31.943ZINFOcrucible-agent (dropshot): request completed latency_us = 13435 local_addr = [fd00:1122:3344:101::18]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::c]:48512 req_id = 881e90d9-cdfe-444b-81e9-ec2e32e7a36c response_code = 204 uri = /crucible/0/regions/048cdf0c-8d41-40fb-b9ed-91434f09b2d4
802026-08-07T17:34:31.944ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-048cdf0c-8d41-40fb-b9ed-91434f09b2d4 (instance states: (Some(Online), None))
812026-08-07T17:34:31.946ZINFOcrucible-agent (dropshot): request completed latency_us = 43 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:48512 req_id = 847fdd0a-5e7b-4c11-a98a-996367f9ee1b response_code = 200 uri = /crucible/0/regions/048cdf0c-8d41-40fb-b9ed-91434f09b2d4
822026-08-07T17:34:31.949ZINFOcrucible-agent (worker): SMF ok!
832026-08-07T17:34:31.956ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_3e9f92e1-b5f2-4c05-b3d0-3c468fb47ca6/crucible/regions/048cdf0c-8d41-40fb-b9ed-91434f09b2d4" region = 048cdf0c-8d41-40fb-b9ed-91434f09b2d4
842026-08-07T17:34:31.994ZINFOcrucible-agent (dropshot): request completed latency_us = 65 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:48512 req_id = 4ea996ee-d67e-4a07-af72-495fe6375265 response_code = 200 uri = /crucible/0/regions/048cdf0c-8d41-40fb-b9ed-91434f09b2d4
852026-08-07T17:34:31.997ZINFOcrucible-agent (dropshot): request completed latency_us = 52 local_addr = [fd00:1122:3344:101::18]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::c]:48512 req_id = d27dab30-8656-4629-9f9f-cc0928b6c308 response_code = 204 uri = /crucible/0/regions/048cdf0c-8d41-40fb-b9ed-91434f09b2d4
862026-08-07T17:34:32.001ZINFOcrucible-agent (dropshot): request completed latency_us = 66 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:48512 req_id = 33bbc6aa-174d-43b8-8bbb-f255f920018a response_code = 200 uri = /crucible/0/regions/048cdf0c-8d41-40fb-b9ed-91434f09b2d4
872026-08-07T17:34:32.005ZINFOcrucible-agent (datafile): region 048cdf0c-8d41-40fb-b9ed-91434f09b2d4 state: Tombstoned -> Destroyed
882026-08-07T17:34:57.210ZINFOcrucible-agent (dropshot): request completed latency_us = 73 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:48512 req_id = 60f9f286-f9c1-4b46-8e35-3d6277865853 response_code = 200 uri = /crucible/0/regions/048cdf0c-8d41-40fb-b9ed-91434f09b2d4
892026-08-07T17:34:57.219ZINFOcrucible-agent (dropshot): request completed latency_us = 54 local_addr = [fd00:1122:3344:101::18]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::c]:48512 req_id = 6aa3684a-f6ee-4f3a-990b-9121da7c280e response_code = 204 uri = /crucible/0/regions/048cdf0c-8d41-40fb-b9ed-91434f09b2d4
902026-08-07T17:34:57.223ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::18]:32345 remote_addr = [fd00:1122:3344:101::b]:46596
912026-08-07T17:34:57.223ZINFOcrucible-agent (dropshot): request completed latency_us = 95 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:46596 req_id = f8efc110-ce24-4a68-9831-64da7322a3ab response_code = 200 uri = /crucible/0/regions/048cdf0c-8d41-40fb-b9ed-91434f09b2d4
922026-08-07T17:34:57.223ZINFOcrucible-agent (dropshot): request completed latency_us = 25 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:48512 req_id = 267f0b66-6136-4e8f-8035-cb7632036c10 response_code = 200 uri = /crucible/0/regions/048cdf0c-8d41-40fb-b9ed-91434f09b2d4
932026-08-07T17:34:57.228ZINFOcrucible-agent (dropshot): request completed latency_us = 65 local_addr = [fd00:1122:3344:101::18]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::b]:46596 req_id = 4dcd9724-d816-4a4c-a3ed-1c3e3bae9d56 response_code = 204 uri = /crucible/0/regions/048cdf0c-8d41-40fb-b9ed-91434f09b2d4
942026-08-07T17:34:57.233ZINFOcrucible-agent (dropshot): request completed latency_us = 58 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:46596 req_id = 91d7773f-36e5-466e-bb32-c5844931081c response_code = 200 uri = /crucible/0/regions/048cdf0c-8d41-40fb-b9ed-91434f09b2d4