Max level shown:
12025-01-15T00:45:43.343ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::1a]:32345 remote_addr = [fd00:1122:3344:101::c]:55518
22025-01-15T00:45:43.373ZINFOcrucible-agent (datafile): region 668a94a6-1bb2-4b72-ae50-a51d5c47286b state: Requested
32025-01-15T00:45:43.375ZINFOcrucible-agent (dropshot): request completed latency_us = 12672 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:55518 req_id = 682afec5-6491-43e8-85a8-4fbd2f625274 response_code = 200 uri = /crucible/0/regions
42025-01-15T00:45:43.378ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
52025-01-15T00:45:43.388ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_3131655f-f0d6-4fd3-bf28-70e12f0bf549/crucible/regions/668a94a6-1bb2-4b72-ae50-a51d5c47286b
62025-01-15T00:45:43.388ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_3131655f-f0d6-4fd3-bf28-70e12f0bf549/crucible/regions/668a94a6-1bb2-4b72-ae50-a51d5c47286b
72025-01-15T00:45:43.442ZINFOcrucible-agent (worker): creating region Region { id: RegionId("668a94a6-1bb2-4b72-ae50-a51d5c47286b"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/668a94a6-1bb2-4b72-ae50-a51d5c47286b" region = 668a94a6-1bb2-4b72-ae50-a51d5c47286b
82025-01-15T00:45:43.445ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "668a94a6-1bb2-4b72-ae50-a51d5c47286b" "--data" "/data/regions/668a94a6-1bb2-4b72-ae50-a51d5c47286b" "--block-size" "512" "--extent-size" "131072" "--extent-count" "16" "--encrypted" region = 668a94a6-1bb2-4b72-ae50-a51d5c47286b
92025-01-15T00:45:43.682ZINFOcrucible-agent (worker): region files created ok region = 668a94a6-1bb2-4b72-ae50-a51d5c47286b
102025-01-15T00:45:43.682ZINFOcrucible-agent (datafile): region 668a94a6-1bb2-4b72-ae50-a51d5c47286b state: Requested -> Created
112025-01-15T00:45:43.682ZINFOcrucible-agent (worker): applying SMF actions post create...
122025-01-15T00:45:43.683ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-668a94a6-1bb2-4b72-ae50-a51d5c47286b
132025-01-15T00:45:43.691ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-668a94a6-1bb2-4b72-ae50-a51d5c47286b
142025-01-15T00:45:43.707ZINFOcrucible-agent (worker): creating config property group
152025-01-15T00:45:43.715ZINFOcrucible-agent (dropshot): request completed latency_us = 75 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:55518 req_id = e9cfec1e-7644-4906-8600-526eafe68267 response_code = 200 uri = /crucible/0/regions
162025-01-15T00:45:43.724ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-668a94a6-1bb2-4b72-ae50-a51d5c47286b
172025-01-15T00:45:43.724ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/668a94a6-1bb2-4b72-ae50-a51d5c47286b
182025-01-15T00:45:43.727ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
192025-01-15T00:45:43.727ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
202025-01-15T00:45:43.727ZINFOcrucible-agent (worker): commit
212025-01-15T00:45:43.731ZINFOcrucible-agent (worker): ok!
222025-01-15T00:45:43.736ZINFOcrucible-agent (worker): SMF ok!
232025-01-15T00:45:44.629ZINFOcrucible-agent (dropshot): request completed latency_us = 52 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:55518 req_id = a8855196-ffa6-4ec9-a819-8af869166aa5 response_code = 200 uri = /crucible/0/regions/668a94a6-1bb2-4b72-ae50-a51d5c47286b
242025-01-15T00:45:44.633ZINFOcrucible-agent (datafile): path is "/data/regions/668a94a6-1bb2-4b72-ae50-a51d5c47286b"
252025-01-15T00:45:44.644ZINFOcrucible-agent (datafile): dataset is oxp_3131655f-f0d6-4fd3-bf28-70e12f0bf549/crucible/regions/668a94a6-1bb2-4b72-ae50-a51d5c47286b
262025-01-15T00:45:44.654ZINFOcrucible-agent (datafile): region 668a94a6-1bb2-4b72-ae50-a51d5c47286b state: Created -> Tombstoned
272025-01-15T00:45:44.654ZINFOcrucible-agent (worker): applying SMF actions before removal...
282025-01-15T00:45:44.654ZINFOcrucible-agent (dropshot): request completed latency_us = 21576 local_addr = [fd00:1122:3344:101::1a]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::c]:55518 req_id = faed0309-cf7b-4657-88a3-bccb6ca4b031 response_code = 204 uri = /crucible/0/regions/668a94a6-1bb2-4b72-ae50-a51d5c47286b
292025-01-15T00:45:44.655ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-668a94a6-1bb2-4b72-ae50-a51d5c47286b (instance states: (Some(Online), None))
302025-01-15T00:45:44.658ZINFOcrucible-agent (dropshot): request completed latency_us = 28 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:55518 req_id = e417cec5-54ab-4e45-8af6-4d6dc438a26f response_code = 200 uri = /crucible/0/regions/668a94a6-1bb2-4b72-ae50-a51d5c47286b
312025-01-15T00:45:44.662ZINFOcrucible-agent (worker): SMF ok!
322025-01-15T00:45:44.673ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_3131655f-f0d6-4fd3-bf28-70e12f0bf549/crucible/regions/668a94a6-1bb2-4b72-ae50-a51d5c47286b" region = 668a94a6-1bb2-4b72-ae50-a51d5c47286b
332025-01-15T00:45:44.735ZINFOcrucible-agent (datafile): region 668a94a6-1bb2-4b72-ae50-a51d5c47286b state: Tombstoned -> Destroyed
342025-01-15T00:45:44.800ZINFOcrucible-agent (dropshot): request completed latency_us = 77 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:55518 req_id = e93651ce-aebd-439a-b1cd-60313d883aa0 response_code = 200 uri = /crucible/0/regions/668a94a6-1bb2-4b72-ae50-a51d5c47286b
352025-01-15T00:45:46.527ZINFOcrucible-agent (datafile): region 72d94856-4ac9-481a-bb58-8057983d3458 state: Requested
362025-01-15T00:45:46.527ZINFOcrucible-agent (dropshot): request completed latency_us = 232 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:55518 req_id = d993f1af-75d3-4e59-8863-d4cf94895ab9 response_code = 200 uri = /crucible/0/regions
372025-01-15T00:45:46.527ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
382025-01-15T00:45:46.537ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_3131655f-f0d6-4fd3-bf28-70e12f0bf549/crucible/regions/72d94856-4ac9-481a-bb58-8057983d3458
392025-01-15T00:45:46.537ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_3131655f-f0d6-4fd3-bf28-70e12f0bf549/crucible/regions/72d94856-4ac9-481a-bb58-8057983d3458
402025-01-15T00:45:46.591ZINFOcrucible-agent (worker): creating region Region { id: RegionId("72d94856-4ac9-481a-bb58-8057983d3458"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/72d94856-4ac9-481a-bb58-8057983d3458" region = 72d94856-4ac9-481a-bb58-8057983d3458
412025-01-15T00:45:46.593ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "72d94856-4ac9-481a-bb58-8057983d3458" "--data" "/data/regions/72d94856-4ac9-481a-bb58-8057983d3458" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 72d94856-4ac9-481a-bb58-8057983d3458
422025-01-15T00:45:46.668ZINFOcrucible-agent (worker): region files created ok region = 72d94856-4ac9-481a-bb58-8057983d3458
432025-01-15T00:45:46.668ZINFOcrucible-agent (datafile): region 72d94856-4ac9-481a-bb58-8057983d3458 state: Requested -> Created
442025-01-15T00:45:46.668ZINFOcrucible-agent (worker): applying SMF actions post create...
452025-01-15T00:45:46.670ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-72d94856-4ac9-481a-bb58-8057983d3458
462025-01-15T00:45:46.673ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-72d94856-4ac9-481a-bb58-8057983d3458
472025-01-15T00:45:46.681ZINFOcrucible-agent (worker): creating config property group
482025-01-15T00:45:46.684ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-72d94856-4ac9-481a-bb58-8057983d3458
492025-01-15T00:45:46.685ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/72d94856-4ac9-481a-bb58-8057983d3458
502025-01-15T00:45:46.685ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
512025-01-15T00:45:46.685ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
522025-01-15T00:45:46.685ZINFOcrucible-agent (worker): commit
532025-01-15T00:45:46.689ZINFOcrucible-agent (worker): ok!
542025-01-15T00:45:46.694ZINFOcrucible-agent (worker): SMF ok!
552025-01-15T00:45:46.830ZINFOcrucible-agent (dropshot): request completed latency_us = 79 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:55518 req_id = 93f53e8c-4788-4359-9f3f-fabdfddee9d9 response_code = 200 uri = /crucible/0/regions
562025-01-15T00:56:31.539ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::1a]:32345 remote_addr = [fd00:1122:3344:101::c]:32812
572025-01-15T00:56:31.539ZINFOcrucible-agent (datafile): region 25149b97-9add-4c7b-be69-385a509226d8 state: Requested
582025-01-15T00:56:31.540ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
592025-01-15T00:56:31.540ZINFOcrucible-agent (dropshot): request completed latency_us = 705 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:32812 req_id = 44445e9e-6dcd-41aa-ac23-f5b5421dc329 response_code = 200 uri = /crucible/0/regions
602025-01-15T00:56:31.550ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_3131655f-f0d6-4fd3-bf28-70e12f0bf549/crucible/regions/25149b97-9add-4c7b-be69-385a509226d8
612025-01-15T00:56:31.550ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_3131655f-f0d6-4fd3-bf28-70e12f0bf549/crucible/regions/25149b97-9add-4c7b-be69-385a509226d8
622025-01-15T00:56:31.638ZINFOcrucible-agent (worker): creating region Region { id: RegionId("25149b97-9add-4c7b-be69-385a509226d8"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/25149b97-9add-4c7b-be69-385a509226d8" region = 25149b97-9add-4c7b-be69-385a509226d8
632025-01-15T00:56:31.639ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "25149b97-9add-4c7b-be69-385a509226d8" "--data" "/data/regions/25149b97-9add-4c7b-be69-385a509226d8" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 25149b97-9add-4c7b-be69-385a509226d8
642025-01-15T00:56:31.702ZINFOcrucible-agent (dropshot): request completed latency_us = 88 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:32812 req_id = 48a0decb-cf8c-411c-995c-2b9f3de39a9b response_code = 200 uri = /crucible/0/regions
652025-01-15T00:56:31.763ZINFOcrucible-agent (worker): region files created ok region = 25149b97-9add-4c7b-be69-385a509226d8
662025-01-15T00:56:31.763ZINFOcrucible-agent (datafile): region 25149b97-9add-4c7b-be69-385a509226d8 state: Requested -> Created
672025-01-15T00:56:31.763ZINFOcrucible-agent (worker): applying SMF actions post create...
682025-01-15T00:56:31.764ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-25149b97-9add-4c7b-be69-385a509226d8
692025-01-15T00:56:31.776ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-25149b97-9add-4c7b-be69-385a509226d8
702025-01-15T00:56:31.791ZINFOcrucible-agent (worker): creating config property group
712025-01-15T00:56:31.795ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-25149b97-9add-4c7b-be69-385a509226d8
722025-01-15T00:56:31.795ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/25149b97-9add-4c7b-be69-385a509226d8
732025-01-15T00:56:31.795ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
742025-01-15T00:56:31.795ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
752025-01-15T00:56:31.795ZINFOcrucible-agent (worker): commit
762025-01-15T00:56:31.799ZINFOcrucible-agent (worker): ok!
772025-01-15T00:56:31.805ZINFOcrucible-agent (worker): SMF ok!
782025-01-15T00:56:32.172ZINFOcrucible-agent (dropshot): request completed latency_us = 68 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:32812 req_id = a1d109c8-fdf9-4b26-b304-39690d93b0f4 response_code = 200 uri = /crucible/0/regions
792025-01-15T00:56:32.540ZINFOcrucible-agent (dropshot): request completed latency_us = 52 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:32812 req_id = db38e24d-6364-42e7-9298-c061ab354a21 response_code = 200 uri = /crucible/0/regions/72d94856-4ac9-481a-bb58-8057983d3458
802025-01-15T00:56:32.544ZINFOcrucible-agent (datafile): path is "/data/regions/72d94856-4ac9-481a-bb58-8057983d3458"
812025-01-15T00:56:32.553ZINFOcrucible-agent (datafile): dataset is oxp_3131655f-f0d6-4fd3-bf28-70e12f0bf549/crucible/regions/72d94856-4ac9-481a-bb58-8057983d3458
822025-01-15T00:56:32.560ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_3131655f-f0d6-4fd3-bf28-70e12f0bf549/crucible/regions/72d94856-4ac9-481a-bb58-8057983d3458@a22bc5f7-9abe-4c15-9e50-b1f528d6ffe7
832025-01-15T00:56:32.560ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_3131655f-f0d6-4fd3-bf28-70e12f0bf549/crucible/regions/72d94856-4ac9-481a-bb58-8057983d3458", "a22bc5f7-9abe-4c15-9e50-b1f528d6ffe7"]
842025-01-15T00:56:32.568ZINFOcrucible-agent (dropshot): request completed latency_us = 24703 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:32812 req_id = 7fa22453-0376-427e-957b-9590d7e769a1 response_code = 200 uri = /crucible/0/regions/72d94856-4ac9-481a-bb58-8057983d3458/snapshots/a22bc5f7-9abe-4c15-9e50-b1f528d6ffe7
852025-01-15T00:56:32.571ZINFOcrucible-agent (datafile): path is "/data/regions/72d94856-4ac9-481a-bb58-8057983d3458"
862025-01-15T00:56:32.579ZINFOcrucible-agent (datafile): dataset is oxp_3131655f-f0d6-4fd3-bf28-70e12f0bf549/crucible/regions/72d94856-4ac9-481a-bb58-8057983d3458
872025-01-15T00:56:32.587ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_3131655f-f0d6-4fd3-bf28-70e12f0bf549/crucible/regions/72d94856-4ac9-481a-bb58-8057983d3458@a22bc5f7-9abe-4c15-9e50-b1f528d6ffe7
882025-01-15T00:56:32.587ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_3131655f-f0d6-4fd3-bf28-70e12f0bf549/crucible/regions/72d94856-4ac9-481a-bb58-8057983d3458", "a22bc5f7-9abe-4c15-9e50-b1f528d6ffe7"]
892025-01-15T00:56:32.596ZINFOcrucible-agent (datafile): requesting running snapshot 72d94856-4ac9-481a-bb58-8057983d3458-a22bc5f7-9abe-4c15-9e50-b1f528d6ffe7 state: Requested
902025-01-15T00:56:32.596ZINFOcrucible-agent (worker): applying SMF actions for region 72d94856-4ac9-481a-bb58-8057983d3458 running snapshot a22bc5f7-9abe-4c15-9e50-b1f528d6ffe7 (state Requested)...
912025-01-15T00:56:32.596ZINFOcrucible-agent (dropshot): request completed latency_us = 25107 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:32812 req_id = bebc7890-c085-4bda-81cb-5c6a929734c3 response_code = 200 uri = /crucible/0/regions/72d94856-4ac9-481a-bb58-8057983d3458/snapshots/a22bc5f7-9abe-4c15-9e50-b1f528d6ffe7/run
922025-01-15T00:56:32.599ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-72d94856-4ac9-481a-bb58-8057983d3458-a22bc5f7-9abe-4c15-9e50-b1f528d6ffe7
932025-01-15T00:56:32.602ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-72d94856-4ac9-481a-bb58-8057983d3458-a22bc5f7-9abe-4c15-9e50-b1f528d6ffe7
942025-01-15T00:56:32.612ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-72d94856-4ac9-481a-bb58-8057983d3458-a22bc5f7-9abe-4c15-9e50-b1f528d6ffe7
952025-01-15T00:56:32.613ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/72d94856-4ac9-481a-bb58-8057983d3458/.zfs/snapshot/a22bc5f7-9abe-4c15-9e50-b1f528d6ffe7
962025-01-15T00:56:32.613ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
972025-01-15T00:56:32.613ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
982025-01-15T00:56:32.613ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
992025-01-15T00:56:32.613ZINFOcrucible-agent (worker): commit
1002025-01-15T00:56:32.617ZINFOcrucible-agent (worker): ok!
1012025-01-15T00:56:32.622ZINFOcrucible-agent (worker): SMF ok!
1022025-01-15T00:56:32.622ZINFOcrucible-agent (datafile): region 72d94856-4ac9-481a-bb58-8057983d3458 running snapshot a22bc5f7-9abe-4c15-9e50-b1f528d6ffe7 state: Requested -> Created
1032025-01-15T00:56:32.778ZINFOcrucible-agent (datafile): path is "/data/regions/72d94856-4ac9-481a-bb58-8057983d3458"
1042025-01-15T00:56:32.787ZINFOcrucible-agent (datafile): dataset is oxp_3131655f-f0d6-4fd3-bf28-70e12f0bf549/crucible/regions/72d94856-4ac9-481a-bb58-8057983d3458
1052025-01-15T00:56:32.795ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_3131655f-f0d6-4fd3-bf28-70e12f0bf549/crucible/regions/72d94856-4ac9-481a-bb58-8057983d3458@a22bc5f7-9abe-4c15-9e50-b1f528d6ffe7
1062025-01-15T00:56:32.795ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_3131655f-f0d6-4fd3-bf28-70e12f0bf549/crucible/regions/72d94856-4ac9-481a-bb58-8057983d3458", "a22bc5f7-9abe-4c15-9e50-b1f528d6ffe7"]
1072025-01-15T00:56:32.803ZINFOcrucible-agent (dropshot): request completed latency_us = 25055 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:32812 req_id = 4e25d1a1-6d9d-4c27-9941-6f73729c9c07 response_code = 200 uri = /crucible/0/regions/72d94856-4ac9-481a-bb58-8057983d3458/snapshots/a22bc5f7-9abe-4c15-9e50-b1f528d6ffe7/run