Max level shown:
12024-08-16T21:30:40.054ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:36523
22024-08-16T21:30:40.073ZINFOcrucible-agent (datafile): region b3fca08d-07d5-496d-a156-a76af4a1223e state: Requested
32024-08-16T21:30:40.078ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
42024-08-16T21:30:40.082ZINFOcrucible-agent (dropshot): request completed latency_us = 15300 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:36523 req_id = e6d01495-5c67-4585-9129-44ccde388e2f response_code = 200 uri = /crucible/0/regions
52024-08-16T21:30:40.090ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_76a1d6b7-c075-4bb0-b840-132e2416251b/crucible/regions/b3fca08d-07d5-496d-a156-a76af4a1223e
62024-08-16T21:30:40.090ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_76a1d6b7-c075-4bb0-b840-132e2416251b/crucible/regions/b3fca08d-07d5-496d-a156-a76af4a1223e
72024-08-16T21:30:40.157ZINFOcrucible-agent (worker): creating region Region { id: RegionId("b3fca08d-07d5-496d-a156-a76af4a1223e"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/b3fca08d-07d5-496d-a156-a76af4a1223e" region = b3fca08d-07d5-496d-a156-a76af4a1223e
82024-08-16T21:30:40.159ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "b3fca08d-07d5-496d-a156-a76af4a1223e" "--data" "/data/regions/b3fca08d-07d5-496d-a156-a76af4a1223e" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = b3fca08d-07d5-496d-a156-a76af4a1223e
92024-08-16T21:30:40.304ZINFOcrucible-agent (worker): region files created ok region = b3fca08d-07d5-496d-a156-a76af4a1223e
102024-08-16T21:30:40.304ZINFOcrucible-agent (datafile): region b3fca08d-07d5-496d-a156-a76af4a1223e state: Requested -> Created
112024-08-16T21:30:40.304ZINFOcrucible-agent (worker): applying SMF actions post create...
122024-08-16T21:30:40.305ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-b3fca08d-07d5-496d-a156-a76af4a1223e
132024-08-16T21:30:40.309ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-b3fca08d-07d5-496d-a156-a76af4a1223e
142024-08-16T21:30:40.322ZINFOcrucible-agent (worker): creating config property group
152024-08-16T21:30:40.325ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-b3fca08d-07d5-496d-a156-a76af4a1223e
162024-08-16T21:30:40.331ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/b3fca08d-07d5-496d-a156-a76af4a1223e
172024-08-16T21:30:40.331ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
182024-08-16T21:30:40.331ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
192024-08-16T21:30:40.332ZINFOcrucible-agent (worker): commit
202024-08-16T21:30:40.335ZINFOcrucible-agent (worker): ok!
212024-08-16T21:30:40.339ZINFOcrucible-agent (worker): SMF ok!
222024-08-16T21:30:40.439ZINFOcrucible-agent (dropshot): request completed latency_us = 72 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:36523 req_id = 16bfc0d4-79c8-458c-b622-83ab62ed29b0 response_code = 200 uri = /crucible/0/regions
232024-08-16T21:35:52.704ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:36803
242024-08-16T21:35:52.704ZINFOcrucible-agent (dropshot): request completed latency_us = 59 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:36803 req_id = a4a27275-e7f1-49fe-94b8-b81369dc7679 response_code = 200 uri = /crucible/0/regions/b3fca08d-07d5-496d-a156-a76af4a1223e
252024-08-16T21:35:52.710ZINFOcrucible-agent (datafile): path is "/data/regions/b3fca08d-07d5-496d-a156-a76af4a1223e"
262024-08-16T21:35:52.720ZINFOcrucible-agent (datafile): dataset is oxp_76a1d6b7-c075-4bb0-b840-132e2416251b/crucible/regions/b3fca08d-07d5-496d-a156-a76af4a1223e
272024-08-16T21:35:52.729ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_76a1d6b7-c075-4bb0-b840-132e2416251b/crucible/regions/b3fca08d-07d5-496d-a156-a76af4a1223e@e10522a9-60d2-4910-96af-df41fd7d982a
282024-08-16T21:35:52.729ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_76a1d6b7-c075-4bb0-b840-132e2416251b/crucible/regions/b3fca08d-07d5-496d-a156-a76af4a1223e", "e10522a9-60d2-4910-96af-df41fd7d982a"]
292024-08-16T21:35:52.740ZINFOcrucible-agent (dropshot): request completed latency_us = 33783 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:36803 req_id = 5f46911a-5287-47ca-9ed3-b97ad43d6bc9 response_code = 200 uri = /crucible/0/regions/b3fca08d-07d5-496d-a156-a76af4a1223e/snapshots/e10522a9-60d2-4910-96af-df41fd7d982a
302024-08-16T21:35:52.747ZINFOcrucible-agent (datafile): path is "/data/regions/b3fca08d-07d5-496d-a156-a76af4a1223e"
312024-08-16T21:35:52.756ZINFOcrucible-agent (datafile): dataset is oxp_76a1d6b7-c075-4bb0-b840-132e2416251b/crucible/regions/b3fca08d-07d5-496d-a156-a76af4a1223e
322024-08-16T21:35:52.764ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_76a1d6b7-c075-4bb0-b840-132e2416251b/crucible/regions/b3fca08d-07d5-496d-a156-a76af4a1223e@e10522a9-60d2-4910-96af-df41fd7d982a
332024-08-16T21:35:52.764ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_76a1d6b7-c075-4bb0-b840-132e2416251b/crucible/regions/b3fca08d-07d5-496d-a156-a76af4a1223e", "e10522a9-60d2-4910-96af-df41fd7d982a"]
342024-08-16T21:35:52.773ZINFOcrucible-agent (datafile): requesting running snapshot b3fca08d-07d5-496d-a156-a76af4a1223e-e10522a9-60d2-4910-96af-df41fd7d982a state: Requested
352024-08-16T21:35:52.774ZINFOcrucible-agent (worker): applying SMF actions for region b3fca08d-07d5-496d-a156-a76af4a1223e running snapshot e10522a9-60d2-4910-96af-df41fd7d982a (state Requested)...
362024-08-16T21:35:52.774ZINFOcrucible-agent (dropshot): request completed latency_us = 26579 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:36803 req_id = c740611a-ddfa-42d6-be9a-1975b7fb1f09 response_code = 200 uri = /crucible/0/regions/b3fca08d-07d5-496d-a156-a76af4a1223e/snapshots/e10522a9-60d2-4910-96af-df41fd7d982a/run
372024-08-16T21:35:52.780ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/b3fca08d-07d5-496d-a156-a76af4a1223e
382024-08-16T21:35:52.780ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
392024-08-16T21:35:52.780ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::12
402024-08-16T21:35:52.780ZINFOcrucible-agent (worker): reconfig required
412024-08-16T21:35:52.780ZINFOcrucible-agent (worker): using existing config property group
422024-08-16T21:35:52.780ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-b3fca08d-07d5-496d-a156-a76af4a1223e
432024-08-16T21:35:52.780ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/b3fca08d-07d5-496d-a156-a76af4a1223e
442024-08-16T21:35:52.780ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
452024-08-16T21:35:52.780ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
462024-08-16T21:35:52.780ZINFOcrucible-agent (worker): commit
472024-08-16T21:35:52.789ZINFOcrucible-agent (worker): ok!
482024-08-16T21:35:52.790ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-b3fca08d-07d5-496d-a156-a76af4a1223e-e10522a9-60d2-4910-96af-df41fd7d982a
492024-08-16T21:35:52.796ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-b3fca08d-07d5-496d-a156-a76af4a1223e-e10522a9-60d2-4910-96af-df41fd7d982a
502024-08-16T21:35:52.812ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-b3fca08d-07d5-496d-a156-a76af4a1223e-e10522a9-60d2-4910-96af-df41fd7d982a
512024-08-16T21:35:52.813ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/b3fca08d-07d5-496d-a156-a76af4a1223e/.zfs/snapshot/e10522a9-60d2-4910-96af-df41fd7d982a
522024-08-16T21:35:52.813ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
532024-08-16T21:35:52.813ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
542024-08-16T21:35:52.813ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
552024-08-16T21:35:52.813ZINFOcrucible-agent (worker): commit
562024-08-16T21:35:52.818ZINFOcrucible-agent (worker): ok!
572024-08-16T21:35:52.824ZINFOcrucible-agent (worker): SMF ok!
582024-08-16T21:35:52.824ZINFOcrucible-agent (datafile): region b3fca08d-07d5-496d-a156-a76af4a1223e running snapshot e10522a9-60d2-4910-96af-df41fd7d982a state: Requested -> Created
592024-08-16T21:35:53.150ZINFOcrucible-agent (datafile): path is "/data/regions/b3fca08d-07d5-496d-a156-a76af4a1223e"
602024-08-16T21:35:53.159ZINFOcrucible-agent (datafile): dataset is oxp_76a1d6b7-c075-4bb0-b840-132e2416251b/crucible/regions/b3fca08d-07d5-496d-a156-a76af4a1223e
612024-08-16T21:35:53.166ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_76a1d6b7-c075-4bb0-b840-132e2416251b/crucible/regions/b3fca08d-07d5-496d-a156-a76af4a1223e@e10522a9-60d2-4910-96af-df41fd7d982a
622024-08-16T21:35:53.166ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_76a1d6b7-c075-4bb0-b840-132e2416251b/crucible/regions/b3fca08d-07d5-496d-a156-a76af4a1223e", "e10522a9-60d2-4910-96af-df41fd7d982a"]
632024-08-16T21:35:53.174ZINFOcrucible-agent (dropshot): request completed latency_us = 23883 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:36803 req_id = 22bd22e1-bc91-44fe-8d2d-701431570a20 response_code = 200 uri = /crucible/0/regions/b3fca08d-07d5-496d-a156-a76af4a1223e/snapshots/e10522a9-60d2-4910-96af-df41fd7d982a/run
642024-08-16T21:35:56.980ZINFOcrucible-agent (datafile): region 2ff43273-3866-4e92-8fb0-7b42a30c9213 state: Requested
652024-08-16T21:35:56.980ZINFOcrucible-agent (dropshot): request completed latency_us = 242 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:36803 req_id = b6af7670-f082-4419-8348-648769c7bf4f response_code = 200 uri = /crucible/0/regions
662024-08-16T21:35:56.980ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
672024-08-16T21:35:56.990ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_76a1d6b7-c075-4bb0-b840-132e2416251b/crucible/regions/2ff43273-3866-4e92-8fb0-7b42a30c9213
682024-08-16T21:35:56.990ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_76a1d6b7-c075-4bb0-b840-132e2416251b/crucible/regions/2ff43273-3866-4e92-8fb0-7b42a30c9213
692024-08-16T21:35:57.066ZINFOcrucible-agent (worker): creating region Region { id: RegionId("2ff43273-3866-4e92-8fb0-7b42a30c9213"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19002, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/2ff43273-3866-4e92-8fb0-7b42a30c9213" region = 2ff43273-3866-4e92-8fb0-7b42a30c9213
702024-08-16T21:35:57.066ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "2ff43273-3866-4e92-8fb0-7b42a30c9213" "--data" "/data/regions/2ff43273-3866-4e92-8fb0-7b42a30c9213" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 2ff43273-3866-4e92-8fb0-7b42a30c9213
712024-08-16T21:35:57.166ZINFOcrucible-agent (worker): region files created ok region = 2ff43273-3866-4e92-8fb0-7b42a30c9213
722024-08-16T21:35:57.166ZINFOcrucible-agent (datafile): region 2ff43273-3866-4e92-8fb0-7b42a30c9213 state: Requested -> Created
732024-08-16T21:35:57.166ZINFOcrucible-agent (worker): applying SMF actions post create...
742024-08-16T21:35:57.167ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-2ff43273-3866-4e92-8fb0-7b42a30c9213
752024-08-16T21:35:57.174ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-2ff43273-3866-4e92-8fb0-7b42a30c9213
762024-08-16T21:35:57.186ZINFOcrucible-agent (worker): creating config property group
772024-08-16T21:35:57.194ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-2ff43273-3866-4e92-8fb0-7b42a30c9213
782024-08-16T21:35:57.194ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/2ff43273-3866-4e92-8fb0-7b42a30c9213
792024-08-16T21:35:57.194ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
802024-08-16T21:35:57.194ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
812024-08-16T21:35:57.194ZINFOcrucible-agent (worker): commit
822024-08-16T21:35:57.201ZINFOcrucible-agent (worker): ok!
832024-08-16T21:35:57.209ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/b3fca08d-07d5-496d-a156-a76af4a1223e
842024-08-16T21:35:57.209ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
852024-08-16T21:35:57.209ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::12
862024-08-16T21:35:57.209ZINFOcrucible-agent (worker): reconfig required
872024-08-16T21:35:57.209ZINFOcrucible-agent (worker): using existing config property group
882024-08-16T21:35:57.209ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-b3fca08d-07d5-496d-a156-a76af4a1223e
892024-08-16T21:35:57.216ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/b3fca08d-07d5-496d-a156-a76af4a1223e
902024-08-16T21:35:57.216ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
912024-08-16T21:35:57.216ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
922024-08-16T21:35:57.216ZINFOcrucible-agent (worker): commit
932024-08-16T21:35:57.223ZINFOcrucible-agent (worker): ok!
942024-08-16T21:35:57.225ZINFOcrucible-agent (worker): SMF ok!
952024-08-16T21:35:57.242ZINFOcrucible-agent (dropshot): request completed latency_us = 82 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:36803 req_id = af4849c7-82f1-4fc0-8e48-2459b08d003d response_code = 200 uri = /crucible/0/regions
962024-08-16T21:36:22.287ZINFOcrucible-agent (dropshot): request completed latency_us = 49 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:36803 req_id = 6c432413-08ba-4008-ae89-339723fbea4d response_code = 200 uri = /crucible/0/regions/2ff43273-3866-4e92-8fb0-7b42a30c9213
972024-08-16T21:36:22.291ZINFOcrucible-agent (datafile): path is "/data/regions/2ff43273-3866-4e92-8fb0-7b42a30c9213"
982024-08-16T21:36:22.303ZINFOcrucible-agent (datafile): dataset is oxp_76a1d6b7-c075-4bb0-b840-132e2416251b/crucible/regions/2ff43273-3866-4e92-8fb0-7b42a30c9213
992024-08-16T21:36:22.312ZINFOcrucible-agent (datafile): region 2ff43273-3866-4e92-8fb0-7b42a30c9213 state: Created -> Tombstoned
1002024-08-16T21:36:22.312ZINFOcrucible-agent (worker): applying SMF actions before removal...
1012024-08-16T21:36:22.312ZINFOcrucible-agent (dropshot): request completed latency_us = 21299 local_addr = [fd00:1122:3344:101::12]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:36803 req_id = 67c41f0c-d74a-472d-941d-45b34b9f6268 response_code = 204 uri = /crucible/0/regions/2ff43273-3866-4e92-8fb0-7b42a30c9213
1022024-08-16T21:36:22.313ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-2ff43273-3866-4e92-8fb0-7b42a30c9213 (instance states: (Some(Online), None))
1032024-08-16T21:36:22.315ZINFOcrucible-agent (dropshot): request completed latency_us = 27 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:36803 req_id = c7d4dd7c-2e03-48e3-abb9-c8f9ebf664d9 response_code = 200 uri = /crucible/0/regions/2ff43273-3866-4e92-8fb0-7b42a30c9213
1042024-08-16T21:36:22.318ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/b3fca08d-07d5-496d-a156-a76af4a1223e
1052024-08-16T21:36:22.318ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
1062024-08-16T21:36:22.318ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::12
1072024-08-16T21:36:22.318ZINFOcrucible-agent (worker): reconfig required
1082024-08-16T21:36:22.318ZINFOcrucible-agent (worker): using existing config property group
1092024-08-16T21:36:22.318ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-b3fca08d-07d5-496d-a156-a76af4a1223e
1102024-08-16T21:36:22.318ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/b3fca08d-07d5-496d-a156-a76af4a1223e
1112024-08-16T21:36:22.318ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
1122024-08-16T21:36:22.318ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
1132024-08-16T21:36:22.318ZINFOcrucible-agent (worker): commit
1142024-08-16T21:36:22.322ZINFOcrucible-agent (worker): ok!
1152024-08-16T21:36:22.323ZINFOcrucible-agent (worker): SMF ok!
1162024-08-16T21:36:22.333ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_76a1d6b7-c075-4bb0-b840-132e2416251b/crucible/regions/2ff43273-3866-4e92-8fb0-7b42a30c9213" region = 2ff43273-3866-4e92-8fb0-7b42a30c9213
1172024-08-16T21:36:22.390ZINFOcrucible-agent (datafile): region 2ff43273-3866-4e92-8fb0-7b42a30c9213 state: Tombstoned -> Destroyed
1182024-08-16T21:36:22.401ZINFOcrucible-agent (dropshot): request completed latency_us = 73 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:36803 req_id = d2994cd5-a703-4e55-aebd-c15a7ec32874 response_code = 200 uri = /crucible/0/regions/2ff43273-3866-4e92-8fb0-7b42a30c9213