Max level shown:
12025-12-23T23:15:26.960ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::1a]:32345 remote_addr = [fd00:1122:3344:101::c]:54796
22025-12-23T23:15:27.005ZINFOcrucible-agent (datafile): region 24d96f81-ec71-4966-8f93-c86cccc2196a state: Requested
32025-12-23T23:15:27.008ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
42025-12-23T23:15:27.008ZINFOcrucible-agent (dropshot): request completed latency_us = 18971 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:54796 req_id = 98f04854-3a88-46fd-9e44-e08045a5d821 response_code = 200 uri = /crucible/0/regions
52025-12-23T23:15:27.016ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_dc108509-15c0-4169-bcca-e107aa2e215d/crucible/regions/24d96f81-ec71-4966-8f93-c86cccc2196a
62025-12-23T23:15:27.016ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_dc108509-15c0-4169-bcca-e107aa2e215d/crucible/regions/24d96f81-ec71-4966-8f93-c86cccc2196a
72025-12-23T23:15:27.064ZINFOcrucible-agent (worker): creating region Region { id: RegionId("24d96f81-ec71-4966-8f93-c86cccc2196a"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/24d96f81-ec71-4966-8f93-c86cccc2196a" region = 24d96f81-ec71-4966-8f93-c86cccc2196a
82025-12-23T23:15:27.066ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "24d96f81-ec71-4966-8f93-c86cccc2196a" "--data" "/data/regions/24d96f81-ec71-4966-8f93-c86cccc2196a" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 24d96f81-ec71-4966-8f93-c86cccc2196a
92025-12-23T23:15:27.270ZINFOcrucible-agent (dropshot): request completed latency_us = 63 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:54796 req_id = a3b705aa-6514-4d71-a129-a011cd29117b response_code = 200 uri = /crucible/0/regions
102025-12-23T23:15:27.416ZINFOcrucible-agent (worker): region files created ok region = 24d96f81-ec71-4966-8f93-c86cccc2196a
112025-12-23T23:15:27.416ZINFOcrucible-agent (datafile): region 24d96f81-ec71-4966-8f93-c86cccc2196a state: Requested -> Created
122025-12-23T23:15:27.416ZINFOcrucible-agent (worker): applying SMF actions post create...
132025-12-23T23:15:27.417ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-24d96f81-ec71-4966-8f93-c86cccc2196a
142025-12-23T23:15:27.428ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-24d96f81-ec71-4966-8f93-c86cccc2196a
152025-12-23T23:15:27.449ZINFOcrucible-agent (worker): creating config property group
162025-12-23T23:15:27.452ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-24d96f81-ec71-4966-8f93-c86cccc2196a
172025-12-23T23:15:27.453ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/24d96f81-ec71-4966-8f93-c86cccc2196a
182025-12-23T23:15:27.453ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
192025-12-23T23:15:27.453ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
202025-12-23T23:15:27.453ZINFOcrucible-agent (worker): commit
212025-12-23T23:15:27.457ZINFOcrucible-agent (worker): ok!
222025-12-23T23:15:27.460ZINFOcrucible-agent (worker): SMF ok!
232025-12-23T23:15:27.802ZINFOcrucible-agent (dropshot): request completed latency_us = 69 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:54796 req_id = f1e6921f-9015-4389-a4ca-a1ab48d941bf response_code = 200 uri = /crucible/0/regions
242025-12-23T23:26:47.467ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::1a]:32345 remote_addr = [fd00:1122:3344:101::c]:35707
252025-12-23T23:26:47.471ZINFOcrucible-agent (datafile): region 7a133568-86ce-4eed-bc3c-168b0b30908c state: Requested
262025-12-23T23:26:47.472ZINFOcrucible-agent (dropshot): request completed latency_us = 698 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:35707 req_id = 276345c4-0bcc-47dd-8e66-137302a2c399 response_code = 200 uri = /crucible/0/regions
272025-12-23T23:26:47.472ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
282025-12-23T23:26:47.478ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_dc108509-15c0-4169-bcca-e107aa2e215d/crucible/regions/7a133568-86ce-4eed-bc3c-168b0b30908c
292025-12-23T23:26:47.478ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_dc108509-15c0-4169-bcca-e107aa2e215d/crucible/regions/7a133568-86ce-4eed-bc3c-168b0b30908c
302025-12-23T23:26:47.527ZINFOcrucible-agent (worker): creating region Region { id: RegionId("7a133568-86ce-4eed-bc3c-168b0b30908c"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/7a133568-86ce-4eed-bc3c-168b0b30908c" region = 7a133568-86ce-4eed-bc3c-168b0b30908c
312025-12-23T23:26:47.527ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "7a133568-86ce-4eed-bc3c-168b0b30908c" "--data" "/data/regions/7a133568-86ce-4eed-bc3c-168b0b30908c" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 7a133568-86ce-4eed-bc3c-168b0b30908c
322025-12-23T23:26:47.649ZINFOcrucible-agent (worker): region files created ok region = 7a133568-86ce-4eed-bc3c-168b0b30908c
332025-12-23T23:26:47.649ZINFOcrucible-agent (datafile): region 7a133568-86ce-4eed-bc3c-168b0b30908c state: Requested -> Created
342025-12-23T23:26:47.649ZINFOcrucible-agent (worker): applying SMF actions post create...
352025-12-23T23:26:47.667ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-7a133568-86ce-4eed-bc3c-168b0b30908c
362025-12-23T23:26:47.673ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-7a133568-86ce-4eed-bc3c-168b0b30908c
372025-12-23T23:26:47.685ZINFOcrucible-agent (worker): creating config property group
382025-12-23T23:26:47.691ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-7a133568-86ce-4eed-bc3c-168b0b30908c
392025-12-23T23:26:47.692ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/7a133568-86ce-4eed-bc3c-168b0b30908c
402025-12-23T23:26:47.692ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
412025-12-23T23:26:47.692ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
422025-12-23T23:26:47.692ZINFOcrucible-agent (worker): commit
432025-12-23T23:26:47.696ZINFOcrucible-agent (worker): ok!
442025-12-23T23:26:47.703ZINFOcrucible-agent (worker): SMF ok!
452025-12-23T23:26:47.717ZINFOcrucible-agent (dropshot): request completed latency_us = 60 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:35707 req_id = 982061ec-ac07-4d04-9895-5e569663b7d0 response_code = 200 uri = /crucible/0/regions
462025-12-23T23:26:48.775ZINFOcrucible-agent (dropshot): request completed latency_us = 73 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:35707 req_id = f8dc487a-bbae-43c1-a7de-df1ac763a9c1 response_code = 200 uri = /crucible/0/regions/24d96f81-ec71-4966-8f93-c86cccc2196a
472025-12-23T23:26:48.788ZINFOcrucible-agent (datafile): path is "/data/regions/24d96f81-ec71-4966-8f93-c86cccc2196a"
482025-12-23T23:26:48.794ZINFOcrucible-agent (datafile): dataset is oxp_dc108509-15c0-4169-bcca-e107aa2e215d/crucible/regions/24d96f81-ec71-4966-8f93-c86cccc2196a
492025-12-23T23:26:48.801ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_dc108509-15c0-4169-bcca-e107aa2e215d/crucible/regions/24d96f81-ec71-4966-8f93-c86cccc2196a@bba2483e-1625-4ff5-8329-39885b179aa8
502025-12-23T23:26:48.801ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_dc108509-15c0-4169-bcca-e107aa2e215d/crucible/regions/24d96f81-ec71-4966-8f93-c86cccc2196a", "bba2483e-1625-4ff5-8329-39885b179aa8"]
512025-12-23T23:26:48.809ZINFOcrucible-agent (dropshot): request completed latency_us = 20658 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:35707 req_id = 4659f97c-b31d-4a8b-9cfb-adfa85f313c6 response_code = 200 uri = /crucible/0/regions/24d96f81-ec71-4966-8f93-c86cccc2196a/snapshots/bba2483e-1625-4ff5-8329-39885b179aa8
522025-12-23T23:26:48.818ZINFOcrucible-agent (datafile): path is "/data/regions/24d96f81-ec71-4966-8f93-c86cccc2196a"
532025-12-23T23:26:48.826ZINFOcrucible-agent (datafile): dataset is oxp_dc108509-15c0-4169-bcca-e107aa2e215d/crucible/regions/24d96f81-ec71-4966-8f93-c86cccc2196a
542025-12-23T23:26:48.833ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_dc108509-15c0-4169-bcca-e107aa2e215d/crucible/regions/24d96f81-ec71-4966-8f93-c86cccc2196a@bba2483e-1625-4ff5-8329-39885b179aa8
552025-12-23T23:26:48.833ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_dc108509-15c0-4169-bcca-e107aa2e215d/crucible/regions/24d96f81-ec71-4966-8f93-c86cccc2196a", "bba2483e-1625-4ff5-8329-39885b179aa8"]
562025-12-23T23:26:48.841ZINFOcrucible-agent (datafile): requesting running snapshot 24d96f81-ec71-4966-8f93-c86cccc2196a-bba2483e-1625-4ff5-8329-39885b179aa8 state: Requested
572025-12-23T23:26:48.841ZINFOcrucible-agent (dropshot): request completed latency_us = 22764 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:35707 req_id = efeb74d3-9c63-4d62-ac8d-514776544333 response_code = 200 uri = /crucible/0/regions/24d96f81-ec71-4966-8f93-c86cccc2196a/snapshots/bba2483e-1625-4ff5-8329-39885b179aa8/run
582025-12-23T23:26:48.841ZINFOcrucible-agent (worker): applying SMF actions for region 24d96f81-ec71-4966-8f93-c86cccc2196a running snapshot bba2483e-1625-4ff5-8329-39885b179aa8 (state Requested)...
592025-12-23T23:26:48.853ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-24d96f81-ec71-4966-8f93-c86cccc2196a-bba2483e-1625-4ff5-8329-39885b179aa8
602025-12-23T23:26:48.863ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-24d96f81-ec71-4966-8f93-c86cccc2196a-bba2483e-1625-4ff5-8329-39885b179aa8
612025-12-23T23:26:48.884ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-24d96f81-ec71-4966-8f93-c86cccc2196a-bba2483e-1625-4ff5-8329-39885b179aa8
622025-12-23T23:26:48.884ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/24d96f81-ec71-4966-8f93-c86cccc2196a/.zfs/snapshot/bba2483e-1625-4ff5-8329-39885b179aa8
632025-12-23T23:26:48.884ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
642025-12-23T23:26:48.884ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
652025-12-23T23:26:48.884ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
662025-12-23T23:26:48.884ZINFOcrucible-agent (worker): commit
672025-12-23T23:26:48.891ZINFOcrucible-agent (worker): ok!
682025-12-23T23:26:48.900ZINFOcrucible-agent (worker): SMF ok!
692025-12-23T23:26:48.900ZINFOcrucible-agent (datafile): region 24d96f81-ec71-4966-8f93-c86cccc2196a running snapshot bba2483e-1625-4ff5-8329-39885b179aa8 state: Requested -> Created
702025-12-23T23:26:49.073ZINFOcrucible-agent (datafile): path is "/data/regions/24d96f81-ec71-4966-8f93-c86cccc2196a"
712025-12-23T23:26:49.079ZINFOcrucible-agent (datafile): dataset is oxp_dc108509-15c0-4169-bcca-e107aa2e215d/crucible/regions/24d96f81-ec71-4966-8f93-c86cccc2196a
722025-12-23T23:26:49.086ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_dc108509-15c0-4169-bcca-e107aa2e215d/crucible/regions/24d96f81-ec71-4966-8f93-c86cccc2196a@bba2483e-1625-4ff5-8329-39885b179aa8
732025-12-23T23:26:49.086ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_dc108509-15c0-4169-bcca-e107aa2e215d/crucible/regions/24d96f81-ec71-4966-8f93-c86cccc2196a", "bba2483e-1625-4ff5-8329-39885b179aa8"]
742025-12-23T23:26:49.091ZINFOcrucible-agent (dropshot): request completed latency_us = 17611 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:35707 req_id = eeb105d4-6c4f-4654-9dcb-0f799150c983 response_code = 200 uri = /crucible/0/regions/24d96f81-ec71-4966-8f93-c86cccc2196a/snapshots/bba2483e-1625-4ff5-8329-39885b179aa8/run
752025-12-23T23:26:54.750ZINFOcrucible-agent (datafile): region d6c6fb81-175e-4115-84c8-b169936efdfb state: Requested
762025-12-23T23:26:54.750ZINFOcrucible-agent (dropshot): request completed latency_us = 218 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:35707 req_id = 995b6fdd-69ef-4fb4-8918-3d7ee4c3755c response_code = 200 uri = /crucible/0/regions
772025-12-23T23:26:54.750ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
782025-12-23T23:26:54.757ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_dc108509-15c0-4169-bcca-e107aa2e215d/crucible/regions/d6c6fb81-175e-4115-84c8-b169936efdfb
792025-12-23T23:26:54.757ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_dc108509-15c0-4169-bcca-e107aa2e215d/crucible/regions/d6c6fb81-175e-4115-84c8-b169936efdfb
802025-12-23T23:26:54.806ZINFOcrucible-agent (worker): creating region Region { id: RegionId("d6c6fb81-175e-4115-84c8-b169936efdfb"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19003, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/d6c6fb81-175e-4115-84c8-b169936efdfb" region = d6c6fb81-175e-4115-84c8-b169936efdfb
812025-12-23T23:26:54.806ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "d6c6fb81-175e-4115-84c8-b169936efdfb" "--data" "/data/regions/d6c6fb81-175e-4115-84c8-b169936efdfb" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = d6c6fb81-175e-4115-84c8-b169936efdfb
822025-12-23T23:26:54.938ZINFOcrucible-agent (dropshot): request completed latency_us = 61 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:35707 req_id = d0397fce-a367-43b8-8626-bc9e6e024699 response_code = 200 uri = /crucible/0/regions
832025-12-23T23:26:54.952ZINFOcrucible-agent (worker): region files created ok region = d6c6fb81-175e-4115-84c8-b169936efdfb
842025-12-23T23:26:54.952ZINFOcrucible-agent (datafile): region d6c6fb81-175e-4115-84c8-b169936efdfb state: Requested -> Created
852025-12-23T23:26:54.953ZINFOcrucible-agent (worker): applying SMF actions post create...
862025-12-23T23:26:54.954ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-d6c6fb81-175e-4115-84c8-b169936efdfb
872025-12-23T23:26:54.960ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-d6c6fb81-175e-4115-84c8-b169936efdfb
882025-12-23T23:26:54.972ZINFOcrucible-agent (worker): creating config property group
892025-12-23T23:26:54.981ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-d6c6fb81-175e-4115-84c8-b169936efdfb
902025-12-23T23:26:54.981ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/d6c6fb81-175e-4115-84c8-b169936efdfb
912025-12-23T23:26:54.981ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19003
922025-12-23T23:26:54.981ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
932025-12-23T23:26:54.981ZINFOcrucible-agent (worker): commit
942025-12-23T23:26:54.987ZINFOcrucible-agent (worker): ok!
952025-12-23T23:26:54.995ZINFOcrucible-agent (worker): SMF ok!
962025-12-23T23:26:55.613ZINFOcrucible-agent (dropshot): request completed latency_us = 90 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:35707 req_id = bf22e74b-79be-4218-a259-a79d0d760ed4 response_code = 200 uri = /crucible/0/regions
972025-12-23T23:27:22.249ZINFOcrucible-agent (dropshot): request completed latency_us = 74 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:35707 req_id = 4b909608-d75d-4719-8d5b-d7ce4825fe89 response_code = 200 uri = /crucible/0/regions/d6c6fb81-175e-4115-84c8-b169936efdfb
982025-12-23T23:27:22.252ZINFOcrucible-agent (datafile): path is "/data/regions/d6c6fb81-175e-4115-84c8-b169936efdfb"
992025-12-23T23:27:22.263ZINFOcrucible-agent (datafile): dataset is oxp_dc108509-15c0-4169-bcca-e107aa2e215d/crucible/regions/d6c6fb81-175e-4115-84c8-b169936efdfb
1002025-12-23T23:27:22.270ZINFOcrucible-agent (datafile): region d6c6fb81-175e-4115-84c8-b169936efdfb state: Created -> Tombstoned
1012025-12-23T23:27:22.270ZINFOcrucible-agent (dropshot): request completed latency_us = 17705 local_addr = [fd00:1122:3344:101::1a]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::c]:35707 req_id = d2ab80d7-b160-4b1a-add8-482910f5cbd3 response_code = 204 uri = /crucible/0/regions/d6c6fb81-175e-4115-84c8-b169936efdfb
1022025-12-23T23:27:22.270ZINFOcrucible-agent (worker): applying SMF actions before removal...
1032025-12-23T23:27:22.271ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-d6c6fb81-175e-4115-84c8-b169936efdfb (instance states: (Some(Online), None))
1042025-12-23T23:27:22.274ZINFOcrucible-agent (dropshot): request completed latency_us = 29 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:35707 req_id = 023f6703-1e69-467a-8b1c-6f135d60a7f2 response_code = 200 uri = /crucible/0/regions/d6c6fb81-175e-4115-84c8-b169936efdfb
1052025-12-23T23:27:22.282ZINFOcrucible-agent (worker): SMF ok!
1062025-12-23T23:27:22.290ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_dc108509-15c0-4169-bcca-e107aa2e215d/crucible/regions/d6c6fb81-175e-4115-84c8-b169936efdfb" region = d6c6fb81-175e-4115-84c8-b169936efdfb
1072025-12-23T23:27:22.338ZINFOcrucible-agent (datafile): region d6c6fb81-175e-4115-84c8-b169936efdfb state: Tombstoned -> Destroyed
1082025-12-23T23:27:22.424ZINFOcrucible-agent (dropshot): request completed latency_us = 55 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:35707 req_id = f419d2f2-8b69-4190-85f3-bce858eda96b response_code = 200 uri = /crucible/0/regions/d6c6fb81-175e-4115-84c8-b169936efdfb