Max level shown:
12026-11-03T06:00:08.540ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:62694
22026-11-03T06:00:08.570ZINFOcrucible-agent (datafile): region edc6fed5-5d72-4e1f-93ca-03f40bd9d743 state: Requested
32026-11-03T06:00:08.578ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
42026-11-03T06:00:08.583ZINFOcrucible-agent (dropshot): request completed latency_us = 17006 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:62694 req_id = e3fd25bb-5ec6-4934-a21b-1eff2853bd3d response_code = 200 uri = /crucible/0/regions
52026-11-03T06:00:08.585ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_21cc73c3-5fdf-432b-9f4d-7516f624c87f/crucible/regions/edc6fed5-5d72-4e1f-93ca-03f40bd9d743
62026-11-03T06:00:08.585ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_21cc73c3-5fdf-432b-9f4d-7516f624c87f/crucible/regions/edc6fed5-5d72-4e1f-93ca-03f40bd9d743
72026-11-03T06:00:08.656ZINFOcrucible-agent (worker): creating region Region { id: RegionId("edc6fed5-5d72-4e1f-93ca-03f40bd9d743"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/edc6fed5-5d72-4e1f-93ca-03f40bd9d743" region = edc6fed5-5d72-4e1f-93ca-03f40bd9d743
82026-11-03T06:00:08.656ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "edc6fed5-5d72-4e1f-93ca-03f40bd9d743" "--data" "/data/regions/edc6fed5-5d72-4e1f-93ca-03f40bd9d743" "--block-size" "512" "--extent-size" "131072" "--extent-count" "16" "--encrypted" region = edc6fed5-5d72-4e1f-93ca-03f40bd9d743
92026-11-03T06:00:08.746ZINFOcrucible-agent (dropshot): request completed latency_us = 158 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:62694 req_id = df4d24e9-b6f9-4370-a8db-5ea85b349484 response_code = 200 uri = /crucible/0/regions
102026-11-03T06:00:08.789ZINFOcrucible-agent (worker): region files created ok region = edc6fed5-5d72-4e1f-93ca-03f40bd9d743
112026-11-03T06:00:08.791ZINFOcrucible-agent (datafile): region edc6fed5-5d72-4e1f-93ca-03f40bd9d743 state: Requested -> Created
122026-11-03T06:00:08.791ZINFOcrucible-agent (worker): applying SMF actions post create...
132026-11-03T06:00:08.791ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-edc6fed5-5d72-4e1f-93ca-03f40bd9d743
142026-11-03T06:00:08.799ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-edc6fed5-5d72-4e1f-93ca-03f40bd9d743
152026-11-03T06:00:08.817ZINFOcrucible-agent (worker): creating config property group
162026-11-03T06:00:08.820ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-edc6fed5-5d72-4e1f-93ca-03f40bd9d743
172026-11-03T06:00:08.821ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/edc6fed5-5d72-4e1f-93ca-03f40bd9d743
182026-11-03T06:00:08.821ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
192026-11-03T06:00:08.821ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
202026-11-03T06:00:08.821ZINFOcrucible-agent (worker): commit
212026-11-03T06:00:08.825ZINFOcrucible-agent (worker): ok!
222026-11-03T06:00:08.829ZINFOcrucible-agent (worker): SMF ok!
232026-11-03T06:00:09.216ZINFOcrucible-agent (dropshot): request completed latency_us = 96 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:62694 req_id = 1c9d72c7-1833-4fae-ad27-a2ac6a72eb02 response_code = 200 uri = /crucible/0/regions
242026-11-03T06:00:09.855ZINFOcrucible-agent (dropshot): request completed latency_us = 102 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:62694 req_id = ecb16dd2-f56a-4b21-9ead-fd4f1af600f2 response_code = 200 uri = /crucible/0/regions/edc6fed5-5d72-4e1f-93ca-03f40bd9d743
252026-11-03T06:00:09.857ZINFOcrucible-agent (datafile): path is "/data/regions/edc6fed5-5d72-4e1f-93ca-03f40bd9d743"
262026-11-03T06:00:09.865ZINFOcrucible-agent (datafile): dataset is oxp_21cc73c3-5fdf-432b-9f4d-7516f624c87f/crucible/regions/edc6fed5-5d72-4e1f-93ca-03f40bd9d743
272026-11-03T06:00:09.871ZINFOcrucible-agent (datafile): region edc6fed5-5d72-4e1f-93ca-03f40bd9d743 state: Created -> Tombstoned
282026-11-03T06:00:09.872ZINFOcrucible-agent (worker): applying SMF actions before removal...
292026-11-03T06:00:09.872ZINFOcrucible-agent (dropshot): request completed latency_us = 14371 local_addr = [fd00:1122:3344:101::12]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:62694 req_id = 119a13b2-2047-45bc-93ab-3387534f9905 response_code = 204 uri = /crucible/0/regions/edc6fed5-5d72-4e1f-93ca-03f40bd9d743
302026-11-03T06:00:09.873ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-edc6fed5-5d72-4e1f-93ca-03f40bd9d743 (instance states: (Some(Online), None))
312026-11-03T06:00:09.875ZINFOcrucible-agent (dropshot): request completed latency_us = 43 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:62694 req_id = 2de71393-7ec5-48d0-a3a4-9a51704f2c5e response_code = 200 uri = /crucible/0/regions/edc6fed5-5d72-4e1f-93ca-03f40bd9d743
322026-11-03T06:00:09.878ZINFOcrucible-agent (worker): SMF ok!
332026-11-03T06:00:09.885ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_21cc73c3-5fdf-432b-9f4d-7516f624c87f/crucible/regions/edc6fed5-5d72-4e1f-93ca-03f40bd9d743" region = edc6fed5-5d72-4e1f-93ca-03f40bd9d743
342026-11-03T06:00:09.959ZINFOcrucible-agent (datafile): region edc6fed5-5d72-4e1f-93ca-03f40bd9d743 state: Tombstoned -> Destroyed
352026-11-03T06:00:10.013ZINFOcrucible-agent (dropshot): request completed latency_us = 52 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:62694 req_id = a19791fc-f8d8-4d4b-ae70-82cbded377a8 response_code = 200 uri = /crucible/0/regions/edc6fed5-5d72-4e1f-93ca-03f40bd9d743
362026-11-03T06:00:12.901ZINFOcrucible-agent (datafile): region aa493090-069f-4555-a672-584352356e48 state: Requested
372026-11-03T06:00:12.901ZINFOcrucible-agent (dropshot): request completed latency_us = 184 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:62694 req_id = a713da95-d25f-49b9-83ce-d4ca778fedbb response_code = 200 uri = /crucible/0/regions
382026-11-03T06:00:12.901ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
392026-11-03T06:00:12.907ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_21cc73c3-5fdf-432b-9f4d-7516f624c87f/crucible/regions/aa493090-069f-4555-a672-584352356e48
402026-11-03T06:00:12.907ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_21cc73c3-5fdf-432b-9f4d-7516f624c87f/crucible/regions/aa493090-069f-4555-a672-584352356e48
412026-11-03T06:00:12.961ZINFOcrucible-agent (worker): creating region Region { id: RegionId("aa493090-069f-4555-a672-584352356e48"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/aa493090-069f-4555-a672-584352356e48" region = aa493090-069f-4555-a672-584352356e48
422026-11-03T06:00:12.961ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "aa493090-069f-4555-a672-584352356e48" "--data" "/data/regions/aa493090-069f-4555-a672-584352356e48" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = aa493090-069f-4555-a672-584352356e48
432026-11-03T06:00:13.028ZINFOcrucible-agent (worker): region files created ok region = aa493090-069f-4555-a672-584352356e48
442026-11-03T06:00:13.028ZINFOcrucible-agent (datafile): region aa493090-069f-4555-a672-584352356e48 state: Requested -> Created
452026-11-03T06:00:13.028ZINFOcrucible-agent (worker): applying SMF actions post create...
462026-11-03T06:00:13.029ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-aa493090-069f-4555-a672-584352356e48
472026-11-03T06:00:13.032ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-aa493090-069f-4555-a672-584352356e48
482026-11-03T06:00:13.039ZINFOcrucible-agent (worker): creating config property group
492026-11-03T06:00:13.042ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-aa493090-069f-4555-a672-584352356e48
502026-11-03T06:00:13.043ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/aa493090-069f-4555-a672-584352356e48
512026-11-03T06:00:13.043ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
522026-11-03T06:00:13.043ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
532026-11-03T06:00:13.043ZINFOcrucible-agent (worker): commit
542026-11-03T06:00:13.052ZINFOcrucible-agent (worker): ok!
552026-11-03T06:00:13.059ZINFOcrucible-agent (worker): SMF ok!
562026-11-03T06:00:13.244ZINFOcrucible-agent (dropshot): request completed latency_us = 65 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:62694 req_id = 227bd841-ce8a-44f8-8439-24d565c8dcbe response_code = 200 uri = /crucible/0/regions
572026-11-03T06:11:18.443ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:47857
582026-11-03T06:11:18.443ZINFOcrucible-agent (datafile): region 20d95f3e-8479-4178-8040-68047088b2e0 state: Requested
592026-11-03T06:11:18.444ZINFOcrucible-agent (dropshot): request completed latency_us = 826 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:47857 req_id = 2b01acb5-1990-4bfe-84e7-3de9da03bbb9 response_code = 200 uri = /crucible/0/regions
602026-11-03T06:11:18.444ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
612026-11-03T06:11:18.453ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_21cc73c3-5fdf-432b-9f4d-7516f624c87f/crucible/regions/20d95f3e-8479-4178-8040-68047088b2e0
622026-11-03T06:11:18.453ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_21cc73c3-5fdf-432b-9f4d-7516f624c87f/crucible/regions/20d95f3e-8479-4178-8040-68047088b2e0
632026-11-03T06:11:18.503ZINFOcrucible-agent (worker): creating region Region { id: RegionId("20d95f3e-8479-4178-8040-68047088b2e0"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/20d95f3e-8479-4178-8040-68047088b2e0" region = 20d95f3e-8479-4178-8040-68047088b2e0
642026-11-03T06:11:18.503ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "20d95f3e-8479-4178-8040-68047088b2e0" "--data" "/data/regions/20d95f3e-8479-4178-8040-68047088b2e0" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 20d95f3e-8479-4178-8040-68047088b2e0
652026-11-03T06:11:18.581ZINFOcrucible-agent (worker): region files created ok region = 20d95f3e-8479-4178-8040-68047088b2e0
662026-11-03T06:11:18.581ZINFOcrucible-agent (datafile): region 20d95f3e-8479-4178-8040-68047088b2e0 state: Requested -> Created
672026-11-03T06:11:18.581ZINFOcrucible-agent (worker): applying SMF actions post create...
682026-11-03T06:11:18.582ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-20d95f3e-8479-4178-8040-68047088b2e0
692026-11-03T06:11:18.597ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-20d95f3e-8479-4178-8040-68047088b2e0
702026-11-03T06:11:18.612ZINFOcrucible-agent (worker): creating config property group
712026-11-03T06:11:18.612ZINFOcrucible-agent (dropshot): request completed latency_us = 56 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:47857 req_id = a0201940-8407-400f-abb0-d1b9e3d42f43 response_code = 200 uri = /crucible/0/regions
722026-11-03T06:11:18.617ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-20d95f3e-8479-4178-8040-68047088b2e0
732026-11-03T06:11:18.617ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/20d95f3e-8479-4178-8040-68047088b2e0
742026-11-03T06:11:18.617ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
752026-11-03T06:11:18.618ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
762026-11-03T06:11:18.618ZINFOcrucible-agent (worker): commit
772026-11-03T06:11:18.622ZINFOcrucible-agent (worker): ok!
782026-11-03T06:11:18.628ZINFOcrucible-agent (worker): SMF ok!
792026-11-03T06:11:20.732ZINFOcrucible-agent (dropshot): request completed latency_us = 58 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:47857 req_id = 0d8613ca-f0f4-4d08-bdf3-22a815685205 response_code = 200 uri = /crucible/0/regions/aa493090-069f-4555-a672-584352356e48
802026-11-03T06:11:20.739ZINFOcrucible-agent (datafile): path is "/data/regions/aa493090-069f-4555-a672-584352356e48"
812026-11-03T06:11:20.744ZINFOcrucible-agent (datafile): dataset is oxp_21cc73c3-5fdf-432b-9f4d-7516f624c87f/crucible/regions/aa493090-069f-4555-a672-584352356e48
822026-11-03T06:11:20.749ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_21cc73c3-5fdf-432b-9f4d-7516f624c87f/crucible/regions/aa493090-069f-4555-a672-584352356e48@7c333487-93a5-4da4-833a-ca4f161640f2
832026-11-03T06:11:20.749ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_21cc73c3-5fdf-432b-9f4d-7516f624c87f/crucible/regions/aa493090-069f-4555-a672-584352356e48", "7c333487-93a5-4da4-833a-ca4f161640f2"]
842026-11-03T06:11:20.758ZINFOcrucible-agent (dropshot): request completed latency_us = 22412 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:47857 req_id = 26b15017-b484-49ab-ba54-649dcc42303c response_code = 200 uri = /crucible/0/regions/aa493090-069f-4555-a672-584352356e48/snapshots/7c333487-93a5-4da4-833a-ca4f161640f2
852026-11-03T06:11:20.762ZINFOcrucible-agent (datafile): path is "/data/regions/aa493090-069f-4555-a672-584352356e48"
862026-11-03T06:11:20.767ZINFOcrucible-agent (datafile): dataset is oxp_21cc73c3-5fdf-432b-9f4d-7516f624c87f/crucible/regions/aa493090-069f-4555-a672-584352356e48
872026-11-03T06:11:20.773ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_21cc73c3-5fdf-432b-9f4d-7516f624c87f/crucible/regions/aa493090-069f-4555-a672-584352356e48@7c333487-93a5-4da4-833a-ca4f161640f2
882026-11-03T06:11:20.773ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_21cc73c3-5fdf-432b-9f4d-7516f624c87f/crucible/regions/aa493090-069f-4555-a672-584352356e48", "7c333487-93a5-4da4-833a-ca4f161640f2"]
892026-11-03T06:11:20.780ZINFOcrucible-agent (datafile): requesting running snapshot aa493090-069f-4555-a672-584352356e48-7c333487-93a5-4da4-833a-ca4f161640f2 state: Requested
902026-11-03T06:11:20.780ZINFOcrucible-agent (worker): applying SMF actions for region aa493090-069f-4555-a672-584352356e48 running snapshot 7c333487-93a5-4da4-833a-ca4f161640f2 (state Requested)...
912026-11-03T06:11:20.780ZINFOcrucible-agent (dropshot): request completed latency_us = 18343 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:47857 req_id = de6e0c6e-7955-4e9d-8b99-3fb116d69928 response_code = 200 uri = /crucible/0/regions/aa493090-069f-4555-a672-584352356e48/snapshots/7c333487-93a5-4da4-833a-ca4f161640f2/run
922026-11-03T06:11:20.783ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-aa493090-069f-4555-a672-584352356e48-7c333487-93a5-4da4-833a-ca4f161640f2
932026-11-03T06:11:20.785ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-aa493090-069f-4555-a672-584352356e48-7c333487-93a5-4da4-833a-ca4f161640f2
942026-11-03T06:11:20.794ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-aa493090-069f-4555-a672-584352356e48-7c333487-93a5-4da4-833a-ca4f161640f2
952026-11-03T06:11:20.795ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/aa493090-069f-4555-a672-584352356e48/.zfs/snapshot/7c333487-93a5-4da4-833a-ca4f161640f2
962026-11-03T06:11:20.795ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
972026-11-03T06:11:20.795ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
982026-11-03T06:11:20.795ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
992026-11-03T06:11:20.795ZINFOcrucible-agent (worker): commit
1002026-11-03T06:11:20.798ZINFOcrucible-agent (worker): ok!
1012026-11-03T06:11:20.802ZINFOcrucible-agent (worker): SMF ok!
1022026-11-03T06:11:20.802ZINFOcrucible-agent (datafile): region aa493090-069f-4555-a672-584352356e48 running snapshot 7c333487-93a5-4da4-833a-ca4f161640f2 state: Requested -> Created
1032026-11-03T06:11:21.041ZINFOcrucible-agent (datafile): path is "/data/regions/aa493090-069f-4555-a672-584352356e48"
1042026-11-03T06:11:21.046ZINFOcrucible-agent (datafile): dataset is oxp_21cc73c3-5fdf-432b-9f4d-7516f624c87f/crucible/regions/aa493090-069f-4555-a672-584352356e48
1052026-11-03T06:11:21.053ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_21cc73c3-5fdf-432b-9f4d-7516f624c87f/crucible/regions/aa493090-069f-4555-a672-584352356e48@7c333487-93a5-4da4-833a-ca4f161640f2
1062026-11-03T06:11:21.053ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_21cc73c3-5fdf-432b-9f4d-7516f624c87f/crucible/regions/aa493090-069f-4555-a672-584352356e48", "7c333487-93a5-4da4-833a-ca4f161640f2"]
1072026-11-03T06:11:21.059ZINFOcrucible-agent (dropshot): request completed latency_us = 18515 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:47857 req_id = 17ff1de8-e22d-4ff0-a15b-4bb4ac374a71 response_code = 200 uri = /crucible/0/regions/aa493090-069f-4555-a672-584352356e48/snapshots/7c333487-93a5-4da4-833a-ca4f161640f2/run