Max level shown:
12026-07-02T06:15:40.856ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::13]:32345 remote_addr = [fd00:1122:3344:101::a]:45634
22026-07-02T06:15:40.888ZINFOcrucible-agent (datafile): region 13516634-b634-4fcb-859c-bed98d20e6d0 state: Requested
32026-07-02T06:15:40.893ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
42026-07-02T06:15:40.895ZINFOcrucible-agent (dropshot): request completed latency_us = 15465 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:45634 req_id = 39dc535f-db11-4817-9f7f-7f3234e4c41a response_code = 200 uri = /crucible/0/regions
52026-07-02T06:15:40.901ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_9c018086-746f-432a-a1f3-6615a464521a/crucible/regions/13516634-b634-4fcb-859c-bed98d20e6d0
62026-07-02T06:15:40.901ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_9c018086-746f-432a-a1f3-6615a464521a/crucible/regions/13516634-b634-4fcb-859c-bed98d20e6d0
72026-07-02T06:15:40.947ZINFOcrucible-agent (worker): creating region Region { id: RegionId("13516634-b634-4fcb-859c-bed98d20e6d0"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/13516634-b634-4fcb-859c-bed98d20e6d0" region = 13516634-b634-4fcb-859c-bed98d20e6d0
82026-07-02T06:15:40.949ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "13516634-b634-4fcb-859c-bed98d20e6d0" "--data" "/data/regions/13516634-b634-4fcb-859c-bed98d20e6d0" "--block-size" "512" "--extent-size" "131072" "--extent-count" "16" "--encrypted" region = 13516634-b634-4fcb-859c-bed98d20e6d0
92026-07-02T06:15:41.205ZINFOcrucible-agent (worker): region files created ok region = 13516634-b634-4fcb-859c-bed98d20e6d0
102026-07-02T06:15:41.207ZINFOcrucible-agent (datafile): region 13516634-b634-4fcb-859c-bed98d20e6d0 state: Requested -> Created
112026-07-02T06:15:41.207ZINFOcrucible-agent (worker): applying SMF actions post create...
122026-07-02T06:15:41.209ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-13516634-b634-4fcb-859c-bed98d20e6d0
132026-07-02T06:15:41.214ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-13516634-b634-4fcb-859c-bed98d20e6d0
142026-07-02T06:15:41.225ZINFOcrucible-agent (worker): creating config property group
152026-07-02T06:15:41.226ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-13516634-b634-4fcb-859c-bed98d20e6d0
162026-07-02T06:15:41.227ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/13516634-b634-4fcb-859c-bed98d20e6d0
172026-07-02T06:15:41.229ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
182026-07-02T06:15:41.229ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::13
192026-07-02T06:15:41.229ZINFOcrucible-agent (worker): commit
202026-07-02T06:15:41.237ZINFOcrucible-agent (worker): ok!
212026-07-02T06:15:41.240ZINFOcrucible-agent (worker): SMF ok!
222026-07-02T06:15:41.250ZINFOcrucible-agent (dropshot): request completed latency_us = 60 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:45634 req_id = 4c7d5b23-06a0-47a7-a7ff-560f5501a773 response_code = 200 uri = /crucible/0/regions
232026-07-02T06:15:41.567ZINFOcrucible-agent (dropshot): request completed latency_us = 67 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:45634 req_id = fbb3c6f2-9f10-430e-a4fb-b877bc33f3b2 response_code = 200 uri = /crucible/0/regions/13516634-b634-4fcb-859c-bed98d20e6d0
242026-07-02T06:15:41.570ZINFOcrucible-agent (datafile): path is "/data/regions/13516634-b634-4fcb-859c-bed98d20e6d0"
252026-07-02T06:15:41.578ZINFOcrucible-agent (datafile): dataset is oxp_9c018086-746f-432a-a1f3-6615a464521a/crucible/regions/13516634-b634-4fcb-859c-bed98d20e6d0
262026-07-02T06:15:41.586ZINFOcrucible-agent (datafile): region 13516634-b634-4fcb-859c-bed98d20e6d0 state: Created -> Tombstoned
272026-07-02T06:15:41.586ZINFOcrucible-agent (worker): applying SMF actions before removal...
282026-07-02T06:15:41.586ZINFOcrucible-agent (dropshot): request completed latency_us = 15794 local_addr = [fd00:1122:3344:101::13]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:45634 req_id = e1b114d1-237e-4b04-9103-96b57b993a72 response_code = 204 uri = /crucible/0/regions/13516634-b634-4fcb-859c-bed98d20e6d0
292026-07-02T06:15:41.587ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-13516634-b634-4fcb-859c-bed98d20e6d0 (instance states: (Some(Online), None))
302026-07-02T06:15:41.590ZINFOcrucible-agent (worker): SMF ok!
312026-07-02T06:15:41.590ZINFOcrucible-agent (dropshot): request completed latency_us = 47 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:45634 req_id = afa184b0-5459-440a-b230-68c15192d202 response_code = 200 uri = /crucible/0/regions/13516634-b634-4fcb-859c-bed98d20e6d0
322026-07-02T06:15:41.598ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_9c018086-746f-432a-a1f3-6615a464521a/crucible/regions/13516634-b634-4fcb-859c-bed98d20e6d0" region = 13516634-b634-4fcb-859c-bed98d20e6d0
332026-07-02T06:15:41.648ZINFOcrucible-agent (datafile): region 13516634-b634-4fcb-859c-bed98d20e6d0 state: Tombstoned -> Destroyed
342026-07-02T06:15:41.704ZINFOcrucible-agent (dropshot): request completed latency_us = 50 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:45634 req_id = 626ea375-6248-4ae8-ab04-b347e47cf4eb response_code = 200 uri = /crucible/0/regions/13516634-b634-4fcb-859c-bed98d20e6d0
352026-07-02T06:15:43.629ZINFOcrucible-agent (datafile): region 061326e8-cbdc-4a25-a3b9-145f32d555c0 state: Requested
362026-07-02T06:15:43.629ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
372026-07-02T06:15:43.629ZINFOcrucible-agent (dropshot): request completed latency_us = 284 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:45634 req_id = 927dd3d8-eb04-42bb-860f-6dccd3f46fcd response_code = 200 uri = /crucible/0/regions
382026-07-02T06:15:43.638ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_9c018086-746f-432a-a1f3-6615a464521a/crucible/regions/061326e8-cbdc-4a25-a3b9-145f32d555c0
392026-07-02T06:15:43.638ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_9c018086-746f-432a-a1f3-6615a464521a/crucible/regions/061326e8-cbdc-4a25-a3b9-145f32d555c0
402026-07-02T06:15:43.703ZINFOcrucible-agent (worker): creating region Region { id: RegionId("061326e8-cbdc-4a25-a3b9-145f32d555c0"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/061326e8-cbdc-4a25-a3b9-145f32d555c0" region = 061326e8-cbdc-4a25-a3b9-145f32d555c0
412026-07-02T06:15:43.703ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "061326e8-cbdc-4a25-a3b9-145f32d555c0" "--data" "/data/regions/061326e8-cbdc-4a25-a3b9-145f32d555c0" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 061326e8-cbdc-4a25-a3b9-145f32d555c0
422026-07-02T06:15:43.755ZINFOcrucible-agent (worker): region files created ok region = 061326e8-cbdc-4a25-a3b9-145f32d555c0
432026-07-02T06:15:43.755ZINFOcrucible-agent (datafile): region 061326e8-cbdc-4a25-a3b9-145f32d555c0 state: Requested -> Created
442026-07-02T06:15:43.755ZINFOcrucible-agent (worker): applying SMF actions post create...
452026-07-02T06:15:43.756ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-061326e8-cbdc-4a25-a3b9-145f32d555c0
462026-07-02T06:15:43.758ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-061326e8-cbdc-4a25-a3b9-145f32d555c0
472026-07-02T06:15:43.762ZINFOcrucible-agent (worker): creating config property group
482026-07-02T06:15:43.763ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-061326e8-cbdc-4a25-a3b9-145f32d555c0
492026-07-02T06:15:43.764ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/061326e8-cbdc-4a25-a3b9-145f32d555c0
502026-07-02T06:15:43.764ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
512026-07-02T06:15:43.764ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::13
522026-07-02T06:15:43.764ZINFOcrucible-agent (worker): commit
532026-07-02T06:15:43.766ZINFOcrucible-agent (worker): ok!
542026-07-02T06:15:43.769ZINFOcrucible-agent (worker): SMF ok!
552026-07-02T06:15:43.821ZINFOcrucible-agent (dropshot): request completed latency_us = 75 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:45634 req_id = 9eb936e3-b51a-4c4d-a343-264baeb07a5e response_code = 200 uri = /crucible/0/regions
562026-07-02T06:26:47.573ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::13]:32345 remote_addr = [fd00:1122:3344:101::a]:64703
572026-07-02T06:26:47.573ZINFOcrucible-agent (datafile): region 66cbb576-e564-4eaf-9aad-e35e83c917a6 state: Requested
582026-07-02T06:26:47.574ZINFOcrucible-agent (dropshot): request completed latency_us = 816 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:64703 req_id = d6d0e1fa-f7d1-453d-a6f2-0c582a3c0e81 response_code = 200 uri = /crucible/0/regions
592026-07-02T06:26:47.574ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
602026-07-02T06:26:47.582ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_9c018086-746f-432a-a1f3-6615a464521a/crucible/regions/66cbb576-e564-4eaf-9aad-e35e83c917a6
612026-07-02T06:26:47.582ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_9c018086-746f-432a-a1f3-6615a464521a/crucible/regions/66cbb576-e564-4eaf-9aad-e35e83c917a6
622026-07-02T06:26:47.637ZINFOcrucible-agent (worker): creating region Region { id: RegionId("66cbb576-e564-4eaf-9aad-e35e83c917a6"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/66cbb576-e564-4eaf-9aad-e35e83c917a6" region = 66cbb576-e564-4eaf-9aad-e35e83c917a6
632026-07-02T06:26:47.637ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "66cbb576-e564-4eaf-9aad-e35e83c917a6" "--data" "/data/regions/66cbb576-e564-4eaf-9aad-e35e83c917a6" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 66cbb576-e564-4eaf-9aad-e35e83c917a6
642026-07-02T06:26:47.685ZINFOcrucible-agent (worker): region files created ok region = 66cbb576-e564-4eaf-9aad-e35e83c917a6
652026-07-02T06:26:47.685ZINFOcrucible-agent (datafile): region 66cbb576-e564-4eaf-9aad-e35e83c917a6 state: Requested -> Created
662026-07-02T06:26:47.685ZINFOcrucible-agent (worker): applying SMF actions post create...
672026-07-02T06:26:47.697ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-66cbb576-e564-4eaf-9aad-e35e83c917a6
682026-07-02T06:26:47.707ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-66cbb576-e564-4eaf-9aad-e35e83c917a6
692026-07-02T06:26:47.713ZINFOcrucible-agent (worker): creating config property group
702026-07-02T06:26:47.715ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-66cbb576-e564-4eaf-9aad-e35e83c917a6
712026-07-02T06:26:47.715ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/66cbb576-e564-4eaf-9aad-e35e83c917a6
722026-07-02T06:26:47.715ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
732026-07-02T06:26:47.715ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::13
742026-07-02T06:26:47.715ZINFOcrucible-agent (worker): commit
752026-07-02T06:26:47.717ZINFOcrucible-agent (worker): ok!
762026-07-02T06:26:47.720ZINFOcrucible-agent (worker): SMF ok!
772026-07-02T06:26:47.933ZINFOcrucible-agent (dropshot): request completed latency_us = 58 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:64703 req_id = e615519e-7f16-455d-9622-57661c89cdea response_code = 200 uri = /crucible/0/regions
782026-07-02T06:26:48.734ZINFOcrucible-agent (dropshot): request completed latency_us = 53 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:64703 req_id = 988575c3-cb95-4440-a2b2-e0650e0cda57 response_code = 200 uri = /crucible/0/regions/061326e8-cbdc-4a25-a3b9-145f32d555c0
792026-07-02T06:26:48.740ZINFOcrucible-agent (datafile): path is "/data/regions/061326e8-cbdc-4a25-a3b9-145f32d555c0"
802026-07-02T06:26:48.746ZINFOcrucible-agent (datafile): dataset is oxp_9c018086-746f-432a-a1f3-6615a464521a/crucible/regions/061326e8-cbdc-4a25-a3b9-145f32d555c0
812026-07-02T06:26:48.751ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_9c018086-746f-432a-a1f3-6615a464521a/crucible/regions/061326e8-cbdc-4a25-a3b9-145f32d555c0@af3d4ec6-57f2-496e-a717-465035220dc5
822026-07-02T06:26:48.751ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_9c018086-746f-432a-a1f3-6615a464521a/crucible/regions/061326e8-cbdc-4a25-a3b9-145f32d555c0", "af3d4ec6-57f2-496e-a717-465035220dc5"]
832026-07-02T06:26:48.762ZINFOcrucible-agent (dropshot): request completed latency_us = 22376 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:64703 req_id = 0b71d7ad-e7f8-4f20-88c8-7946e791b838 response_code = 200 uri = /crucible/0/regions/061326e8-cbdc-4a25-a3b9-145f32d555c0/snapshots/af3d4ec6-57f2-496e-a717-465035220dc5
842026-07-02T06:26:48.765ZINFOcrucible-agent (datafile): path is "/data/regions/061326e8-cbdc-4a25-a3b9-145f32d555c0"
852026-07-02T06:26:48.772ZINFOcrucible-agent (datafile): dataset is oxp_9c018086-746f-432a-a1f3-6615a464521a/crucible/regions/061326e8-cbdc-4a25-a3b9-145f32d555c0
862026-07-02T06:26:48.778ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_9c018086-746f-432a-a1f3-6615a464521a/crucible/regions/061326e8-cbdc-4a25-a3b9-145f32d555c0@af3d4ec6-57f2-496e-a717-465035220dc5
872026-07-02T06:26:48.778ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_9c018086-746f-432a-a1f3-6615a464521a/crucible/regions/061326e8-cbdc-4a25-a3b9-145f32d555c0", "af3d4ec6-57f2-496e-a717-465035220dc5"]
882026-07-02T06:26:48.786ZINFOcrucible-agent (datafile): requesting running snapshot 061326e8-cbdc-4a25-a3b9-145f32d555c0-af3d4ec6-57f2-496e-a717-465035220dc5 state: Requested
892026-07-02T06:26:48.789ZINFOcrucible-agent (worker): applying SMF actions for region 061326e8-cbdc-4a25-a3b9-145f32d555c0 running snapshot af3d4ec6-57f2-496e-a717-465035220dc5 (state Requested)...
902026-07-02T06:26:48.789ZINFOcrucible-agent (dropshot): request completed latency_us = 23420 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:64703 req_id = fda124bd-c740-4b8e-80e9-d46804633e33 response_code = 200 uri = /crucible/0/regions/061326e8-cbdc-4a25-a3b9-145f32d555c0/snapshots/af3d4ec6-57f2-496e-a717-465035220dc5/run
912026-07-02T06:26:48.794ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-061326e8-cbdc-4a25-a3b9-145f32d555c0-af3d4ec6-57f2-496e-a717-465035220dc5
922026-07-02T06:26:48.795ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-061326e8-cbdc-4a25-a3b9-145f32d555c0-af3d4ec6-57f2-496e-a717-465035220dc5
932026-07-02T06:26:48.800ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-061326e8-cbdc-4a25-a3b9-145f32d555c0-af3d4ec6-57f2-496e-a717-465035220dc5
942026-07-02T06:26:48.800ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/061326e8-cbdc-4a25-a3b9-145f32d555c0/.zfs/snapshot/af3d4ec6-57f2-496e-a717-465035220dc5
952026-07-02T06:26:48.800ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
962026-07-02T06:26:48.801ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
972026-07-02T06:26:48.801ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::13
982026-07-02T06:26:48.801ZINFOcrucible-agent (worker): commit
992026-07-02T06:26:48.803ZINFOcrucible-agent (worker): ok!
1002026-07-02T06:26:48.806ZINFOcrucible-agent (worker): SMF ok!
1012026-07-02T06:26:48.806ZINFOcrucible-agent (datafile): region 061326e8-cbdc-4a25-a3b9-145f32d555c0 running snapshot af3d4ec6-57f2-496e-a717-465035220dc5 state: Requested -> Created
1022026-07-02T06:26:48.991ZINFOcrucible-agent (datafile): path is "/data/regions/061326e8-cbdc-4a25-a3b9-145f32d555c0"
1032026-07-02T06:26:48.997ZINFOcrucible-agent (datafile): dataset is oxp_9c018086-746f-432a-a1f3-6615a464521a/crucible/regions/061326e8-cbdc-4a25-a3b9-145f32d555c0
1042026-07-02T06:26:49.002ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_9c018086-746f-432a-a1f3-6615a464521a/crucible/regions/061326e8-cbdc-4a25-a3b9-145f32d555c0@af3d4ec6-57f2-496e-a717-465035220dc5
1052026-07-02T06:26:49.002ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_9c018086-746f-432a-a1f3-6615a464521a/crucible/regions/061326e8-cbdc-4a25-a3b9-145f32d555c0", "af3d4ec6-57f2-496e-a717-465035220dc5"]
1062026-07-02T06:26:49.008ZINFOcrucible-agent (dropshot): request completed latency_us = 17003 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:64703 req_id = c64353bf-e68e-4f2b-abeb-54da3cf52e63 response_code = 200 uri = /crucible/0/regions/061326e8-cbdc-4a25-a3b9-145f32d555c0/snapshots/af3d4ec6-57f2-496e-a717-465035220dc5/run