Max level shown:
12026-09-11T22:24:30.106ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::16]:32345 remote_addr = [fd00:1122:3344:101::b]:58141
22026-09-11T22:24:30.106ZINFOcrucible-agent (datafile): region 0b74a99f-3d0a-4079-bbd9-7c9e85e35fbb state: Requested
32026-09-11T22:24:30.107ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
42026-09-11T22:24:30.108ZINFOcrucible-agent (dropshot): request completed latency_us = 664 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:58141 req_id = b11bdf96-5b64-4b60-afcb-012d704a19d9 response_code = 200 uri = /crucible/0/regions
52026-09-11T22:24:30.116ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_7f1f5994-904b-4bc0-886c-fd953900e72b/crucible/regions/0b74a99f-3d0a-4079-bbd9-7c9e85e35fbb
62026-09-11T22:24:30.116ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_7f1f5994-904b-4bc0-886c-fd953900e72b/crucible/regions/0b74a99f-3d0a-4079-bbd9-7c9e85e35fbb
72026-09-11T22:24:30.164ZINFOcrucible-agent (worker): creating region Region { id: RegionId("0b74a99f-3d0a-4079-bbd9-7c9e85e35fbb"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/0b74a99f-3d0a-4079-bbd9-7c9e85e35fbb" region = 0b74a99f-3d0a-4079-bbd9-7c9e85e35fbb
82026-09-11T22:24:30.164ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "0b74a99f-3d0a-4079-bbd9-7c9e85e35fbb" "--data" "/data/regions/0b74a99f-3d0a-4079-bbd9-7c9e85e35fbb" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 0b74a99f-3d0a-4079-bbd9-7c9e85e35fbb
92026-09-11T22:24:30.244ZINFOcrucible-agent (worker): region files created ok region = 0b74a99f-3d0a-4079-bbd9-7c9e85e35fbb
102026-09-11T22:24:30.244ZINFOcrucible-agent (datafile): region 0b74a99f-3d0a-4079-bbd9-7c9e85e35fbb state: Requested -> Created
112026-09-11T22:24:30.244ZINFOcrucible-agent (worker): applying SMF actions post create...
122026-09-11T22:24:30.245ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-0b74a99f-3d0a-4079-bbd9-7c9e85e35fbb
132026-09-11T22:24:30.263ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-0b74a99f-3d0a-4079-bbd9-7c9e85e35fbb
142026-09-11T22:24:30.279ZINFOcrucible-agent (worker): creating config property group
152026-09-11T22:24:30.282ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-0b74a99f-3d0a-4079-bbd9-7c9e85e35fbb
162026-09-11T22:24:30.283ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/0b74a99f-3d0a-4079-bbd9-7c9e85e35fbb
172026-09-11T22:24:30.283ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
182026-09-11T22:24:30.283ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::16
192026-09-11T22:24:30.283ZINFOcrucible-agent (worker): commit
202026-09-11T22:24:30.287ZINFOcrucible-agent (worker): ok!
212026-09-11T22:24:30.292ZINFOcrucible-agent (worker): SMF ok!
222026-09-11T22:24:30.399ZINFOcrucible-agent (dropshot): request completed latency_us = 64 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:58141 req_id = c0b9f75e-9a39-4f78-bd79-6e0da721f973 response_code = 200 uri = /crucible/0/regions
232026-09-11T22:24:32.374ZINFOcrucible-agent (dropshot): request completed latency_us = 54 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:58141 req_id = acf5e5fd-e9ab-4103-9491-143683499993 response_code = 200 uri = /crucible/0/regions/678e9119-11fe-4b00-b079-03c815e3aa50
242026-09-11T22:24:32.377ZINFOcrucible-agent (datafile): path is "/data/regions/678e9119-11fe-4b00-b079-03c815e3aa50"
252026-09-11T22:24:32.384ZINFOcrucible-agent (datafile): dataset is oxp_7f1f5994-904b-4bc0-886c-fd953900e72b/crucible/regions/678e9119-11fe-4b00-b079-03c815e3aa50
262026-09-11T22:24:32.389ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_7f1f5994-904b-4bc0-886c-fd953900e72b/crucible/regions/678e9119-11fe-4b00-b079-03c815e3aa50@0f7b5f75-6806-4d01-b17e-e458df0e7a85
272026-09-11T22:24:32.389ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_7f1f5994-904b-4bc0-886c-fd953900e72b/crucible/regions/678e9119-11fe-4b00-b079-03c815e3aa50", "0f7b5f75-6806-4d01-b17e-e458df0e7a85"]
282026-09-11T22:24:32.418ZINFOcrucible-agent (dropshot): request completed latency_us = 40450 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:58141 req_id = 85d60a68-2137-4134-9157-ffb829479846 response_code = 200 uri = /crucible/0/regions/678e9119-11fe-4b00-b079-03c815e3aa50/snapshots/0f7b5f75-6806-4d01-b17e-e458df0e7a85
292026-09-11T22:24:32.421ZINFOcrucible-agent (datafile): path is "/data/regions/678e9119-11fe-4b00-b079-03c815e3aa50"
302026-09-11T22:24:32.427ZINFOcrucible-agent (datafile): dataset is oxp_7f1f5994-904b-4bc0-886c-fd953900e72b/crucible/regions/678e9119-11fe-4b00-b079-03c815e3aa50
312026-09-11T22:24:32.434ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_7f1f5994-904b-4bc0-886c-fd953900e72b/crucible/regions/678e9119-11fe-4b00-b079-03c815e3aa50@0f7b5f75-6806-4d01-b17e-e458df0e7a85
322026-09-11T22:24:32.434ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_7f1f5994-904b-4bc0-886c-fd953900e72b/crucible/regions/678e9119-11fe-4b00-b079-03c815e3aa50", "0f7b5f75-6806-4d01-b17e-e458df0e7a85"]
332026-09-11T22:24:32.447ZINFOcrucible-agent (datafile): requesting running snapshot 678e9119-11fe-4b00-b079-03c815e3aa50-0f7b5f75-6806-4d01-b17e-e458df0e7a85 state: Requested
342026-09-11T22:24:32.458ZINFOcrucible-agent (worker): applying SMF actions for region 678e9119-11fe-4b00-b079-03c815e3aa50 running snapshot 0f7b5f75-6806-4d01-b17e-e458df0e7a85 (state Requested)...
352026-09-11T22:24:32.458ZINFOcrucible-agent (dropshot): request completed latency_us = 36928 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:58141 req_id = 8b6955c8-f8fd-4cbf-ac05-4dcb1bfdacbe response_code = 200 uri = /crucible/0/regions/678e9119-11fe-4b00-b079-03c815e3aa50/snapshots/0f7b5f75-6806-4d01-b17e-e458df0e7a85/run
362026-09-11T22:24:32.461ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-678e9119-11fe-4b00-b079-03c815e3aa50-0f7b5f75-6806-4d01-b17e-e458df0e7a85
372026-09-11T22:24:32.561ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-678e9119-11fe-4b00-b079-03c815e3aa50-0f7b5f75-6806-4d01-b17e-e458df0e7a85
382026-09-11T22:24:32.634ZINFOcrucible-agent (datafile): path is "/data/regions/678e9119-11fe-4b00-b079-03c815e3aa50"
392026-09-11T22:24:32.640ZINFOcrucible-agent (datafile): dataset is oxp_7f1f5994-904b-4bc0-886c-fd953900e72b/crucible/regions/678e9119-11fe-4b00-b079-03c815e3aa50
402026-09-11T22:24:32.645ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_7f1f5994-904b-4bc0-886c-fd953900e72b/crucible/regions/678e9119-11fe-4b00-b079-03c815e3aa50@0f7b5f75-6806-4d01-b17e-e458df0e7a85
412026-09-11T22:24:32.645ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_7f1f5994-904b-4bc0-886c-fd953900e72b/crucible/regions/678e9119-11fe-4b00-b079-03c815e3aa50", "0f7b5f75-6806-4d01-b17e-e458df0e7a85"]
422026-09-11T22:24:32.651ZINFOcrucible-agent (dropshot): request completed latency_us = 17034 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:58141 req_id = 7c4e70f5-53d6-4b5c-8acb-3de2fb45c2d4 response_code = 200 uri = /crucible/0/regions/678e9119-11fe-4b00-b079-03c815e3aa50/snapshots/0f7b5f75-6806-4d01-b17e-e458df0e7a85/run
432026-09-11T22:24:32.905ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-678e9119-11fe-4b00-b079-03c815e3aa50-0f7b5f75-6806-4d01-b17e-e458df0e7a85
442026-09-11T22:24:32.905ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/678e9119-11fe-4b00-b079-03c815e3aa50/.zfs/snapshot/0f7b5f75-6806-4d01-b17e-e458df0e7a85
452026-09-11T22:24:32.905ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
462026-09-11T22:24:32.905ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
472026-09-11T22:24:32.905ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::16
482026-09-11T22:24:32.905ZINFOcrucible-agent (worker): commit
492026-09-11T22:24:32.940ZINFOcrucible-agent (datafile): path is "/data/regions/678e9119-11fe-4b00-b079-03c815e3aa50"
502026-09-11T22:24:32.947ZINFOcrucible-agent (datafile): dataset is oxp_7f1f5994-904b-4bc0-886c-fd953900e72b/crucible/regions/678e9119-11fe-4b00-b079-03c815e3aa50
512026-09-11T22:24:32.952ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_7f1f5994-904b-4bc0-886c-fd953900e72b/crucible/regions/678e9119-11fe-4b00-b079-03c815e3aa50@0f7b5f75-6806-4d01-b17e-e458df0e7a85
522026-09-11T22:24:32.952ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_7f1f5994-904b-4bc0-886c-fd953900e72b/crucible/regions/678e9119-11fe-4b00-b079-03c815e3aa50", "0f7b5f75-6806-4d01-b17e-e458df0e7a85"]
532026-09-11T22:24:32.957ZINFOcrucible-agent (dropshot): request completed latency_us = 17039 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:58141 req_id = 2a760a6f-9dbf-4c52-bbdb-41e132827767 response_code = 200 uri = /crucible/0/regions/678e9119-11fe-4b00-b079-03c815e3aa50/snapshots/0f7b5f75-6806-4d01-b17e-e458df0e7a85/run
542026-09-11T22:24:33.076ZINFOcrucible-agent (worker): ok!
552026-09-11T22:24:33.293ZINFOcrucible-agent (worker): SMF ok!
562026-09-11T22:24:33.293ZINFOcrucible-agent (datafile): region 678e9119-11fe-4b00-b079-03c815e3aa50 running snapshot 0f7b5f75-6806-4d01-b17e-e458df0e7a85 state: Requested -> Created
572026-09-11T22:24:34.073ZINFOcrucible-agent (datafile): path is "/data/regions/678e9119-11fe-4b00-b079-03c815e3aa50"
582026-09-11T22:24:34.079ZINFOcrucible-agent (datafile): dataset is oxp_7f1f5994-904b-4bc0-886c-fd953900e72b/crucible/regions/678e9119-11fe-4b00-b079-03c815e3aa50
592026-09-11T22:24:34.086ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_7f1f5994-904b-4bc0-886c-fd953900e72b/crucible/regions/678e9119-11fe-4b00-b079-03c815e3aa50@0f7b5f75-6806-4d01-b17e-e458df0e7a85
602026-09-11T22:24:34.086ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_7f1f5994-904b-4bc0-886c-fd953900e72b/crucible/regions/678e9119-11fe-4b00-b079-03c815e3aa50", "0f7b5f75-6806-4d01-b17e-e458df0e7a85"]
612026-09-11T22:24:34.092ZINFOcrucible-agent (dropshot): request completed latency_us = 18872 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:58141 req_id = fa734150-4f4e-48b5-a777-f454355830f9 response_code = 200 uri = /crucible/0/regions/678e9119-11fe-4b00-b079-03c815e3aa50/snapshots/0f7b5f75-6806-4d01-b17e-e458df0e7a85/run
622026-09-11T22:24:49.295ZINFOcrucible-agent (datafile): region 6a0dbc52-87f8-47ab-9b59-1ceecf7f77c3 state: Requested
632026-09-11T22:24:49.296ZINFOcrucible-agent (dropshot): request completed latency_us = 230 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:58141 req_id = 8bc3434f-8096-49f2-a4ac-c8062f462ade response_code = 200 uri = /crucible/0/regions
642026-09-11T22:24:49.296ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
652026-09-11T22:24:49.303ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_7f1f5994-904b-4bc0-886c-fd953900e72b/crucible/regions/6a0dbc52-87f8-47ab-9b59-1ceecf7f77c3
662026-09-11T22:24:49.303ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_7f1f5994-904b-4bc0-886c-fd953900e72b/crucible/regions/6a0dbc52-87f8-47ab-9b59-1ceecf7f77c3
672026-09-11T22:24:49.623ZINFOcrucible-agent (worker): creating region Region { id: RegionId("6a0dbc52-87f8-47ab-9b59-1ceecf7f77c3"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19003, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/6a0dbc52-87f8-47ab-9b59-1ceecf7f77c3" region = 6a0dbc52-87f8-47ab-9b59-1ceecf7f77c3
682026-09-11T22:24:49.623ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "6a0dbc52-87f8-47ab-9b59-1ceecf7f77c3" "--data" "/data/regions/6a0dbc52-87f8-47ab-9b59-1ceecf7f77c3" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 6a0dbc52-87f8-47ab-9b59-1ceecf7f77c3
692026-09-11T22:24:49.624ZINFOcrucible-agent (dropshot): request completed latency_us = 57 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:58141 req_id = 35d7c866-923d-4ff2-ba8f-308cb2c62466 response_code = 200 uri = /crucible/0/regions
702026-09-11T22:24:49.821ZINFOcrucible-agent (worker): region files created ok region = 6a0dbc52-87f8-47ab-9b59-1ceecf7f77c3
712026-09-11T22:24:49.821ZINFOcrucible-agent (datafile): region 6a0dbc52-87f8-47ab-9b59-1ceecf7f77c3 state: Requested -> Created
722026-09-11T22:24:49.821ZINFOcrucible-agent (worker): applying SMF actions post create...
732026-09-11T22:24:49.823ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-6a0dbc52-87f8-47ab-9b59-1ceecf7f77c3
742026-09-11T22:24:49.836ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-6a0dbc52-87f8-47ab-9b59-1ceecf7f77c3
752026-09-11T22:24:49.866ZINFOcrucible-agent (worker): creating config property group
762026-09-11T22:24:49.884ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-6a0dbc52-87f8-47ab-9b59-1ceecf7f77c3
772026-09-11T22:24:49.885ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/6a0dbc52-87f8-47ab-9b59-1ceecf7f77c3
782026-09-11T22:24:49.885ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19003
792026-09-11T22:24:49.885ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::16
802026-09-11T22:24:49.885ZINFOcrucible-agent (worker): commit
812026-09-11T22:24:49.901ZINFOcrucible-agent (worker): ok!
822026-09-11T22:24:49.916ZINFOcrucible-agent (worker): SMF ok!
832026-09-11T22:24:50.064ZINFOcrucible-agent (dropshot): request completed latency_us = 57 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:58141 req_id = 668b5fb2-84d0-4cad-82ea-123253201059 response_code = 200 uri = /crucible/0/regions
842026-09-11T22:25:17.969ZINFOcrucible-agent (dropshot): request completed latency_us = 57 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:58141 req_id = 2ea655eb-bf8e-4eba-a6f9-cc2d29cb33aa response_code = 200 uri = /crucible/0/regions/6a0dbc52-87f8-47ab-9b59-1ceecf7f77c3
852026-09-11T22:25:17.972ZINFOcrucible-agent (datafile): path is "/data/regions/6a0dbc52-87f8-47ab-9b59-1ceecf7f77c3"
862026-09-11T22:25:17.982ZINFOcrucible-agent (datafile): dataset is oxp_7f1f5994-904b-4bc0-886c-fd953900e72b/crucible/regions/6a0dbc52-87f8-47ab-9b59-1ceecf7f77c3
872026-09-11T22:25:17.989ZINFOcrucible-agent (datafile): region 6a0dbc52-87f8-47ab-9b59-1ceecf7f77c3 state: Created -> Tombstoned
882026-09-11T22:25:17.989ZINFOcrucible-agent (worker): applying SMF actions before removal...
892026-09-11T22:25:17.989ZINFOcrucible-agent (dropshot): request completed latency_us = 17102 local_addr = [fd00:1122:3344:101::16]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::b]:58141 req_id = 58dbdc03-722c-4473-9cf2-8a56cc8b5c77 response_code = 204 uri = /crucible/0/regions/6a0dbc52-87f8-47ab-9b59-1ceecf7f77c3
902026-09-11T22:25:17.990ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-6a0dbc52-87f8-47ab-9b59-1ceecf7f77c3 (instance states: (Some(Online), None))
912026-09-11T22:25:17.993ZINFOcrucible-agent (dropshot): request completed latency_us = 51 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:58141 req_id = 3641e4a9-c941-457f-936f-4f1540510a44 response_code = 200 uri = /crucible/0/regions/6a0dbc52-87f8-47ab-9b59-1ceecf7f77c3
922026-09-11T22:25:18.000ZINFOcrucible-agent (worker): SMF ok!
932026-09-11T22:25:18.008ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_7f1f5994-904b-4bc0-886c-fd953900e72b/crucible/regions/6a0dbc52-87f8-47ab-9b59-1ceecf7f77c3" region = 6a0dbc52-87f8-47ab-9b59-1ceecf7f77c3
942026-09-11T22:25:18.069ZINFOcrucible-agent (datafile): region 6a0dbc52-87f8-47ab-9b59-1ceecf7f77c3 state: Tombstoned -> Destroyed
952026-09-11T22:25:18.109ZINFOcrucible-agent (dropshot): request completed latency_us = 57 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:58141 req_id = 56129000-e7ae-4556-8c96-d7681a50eb8e response_code = 200 uri = /crucible/0/regions/6a0dbc52-87f8-47ab-9b59-1ceecf7f77c3