Max level shown:
1 [ Oct 18 21:17:37 Enabled. ]
2 [ Oct 18 21:17:37 Rereading configuration. ]
3 [ Oct 18 21:17:42 Rereading configuration. ]
4 [ Oct 18 21:17:49 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_a37fa713-e408-4c30-9412-5e8373254f8d/crucible -l [fd00:1122:3344:101::14]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62024-10-18T21:17:49.110ZINFOcrucible-agent: dataset: "oxp_a37fa713-e408-4c30-9412-5e8373254f8d/crucible"
72024-10-18T21:17:49.111ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::14]:32345
82024-10-18T21:17:49.111ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92024-10-18T21:17:49.125ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102024-10-18T21:17:49.294ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::14]:32345
112024-10-18T21:18:13.856ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::14]:32345 remote_addr = [fd00:1122:3344:101::c]:49034
122024-10-18T21:18:13.865ZINFOcrucible-agent (datafile): region 1c343c62-3be2-438d-a7cd-d2aeb93884db state: Requested
132024-10-18T21:18:13.867ZINFOcrucible-agent (dropshot): request completed latency_us = 2534 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:49034 req_id = f74c6447-46c1-4c7c-8bf1-3458dde28c9e response_code = 200 uri = /crucible/0/regions
142024-10-18T21:18:13.870ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
152024-10-18T21:18:13.878ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_a37fa713-e408-4c30-9412-5e8373254f8d/crucible/regions/1c343c62-3be2-438d-a7cd-d2aeb93884db
162024-10-18T21:18:13.878ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_a37fa713-e408-4c30-9412-5e8373254f8d/crucible/regions/1c343c62-3be2-438d-a7cd-d2aeb93884db
172024-10-18T21:18:13.955ZINFOcrucible-agent (worker): creating region Region { id: RegionId("1c343c62-3be2-438d-a7cd-d2aeb93884db"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/1c343c62-3be2-438d-a7cd-d2aeb93884db" region = 1c343c62-3be2-438d-a7cd-d2aeb93884db
182024-10-18T21:18:13.957ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "1c343c62-3be2-438d-a7cd-d2aeb93884db" "--data" "/data/regions/1c343c62-3be2-438d-a7cd-d2aeb93884db" "--block-size" "512" "--extent-size" "131072" "--extent-count" "16" "--encrypted" region = 1c343c62-3be2-438d-a7cd-d2aeb93884db
192024-10-18T21:18:14.132ZINFOcrucible-agent (worker): region files created ok region = 1c343c62-3be2-438d-a7cd-d2aeb93884db
202024-10-18T21:18:14.132ZINFOcrucible-agent (datafile): region 1c343c62-3be2-438d-a7cd-d2aeb93884db state: Requested -> Created
212024-10-18T21:18:14.132ZINFOcrucible-agent (worker): applying SMF actions post create...
222024-10-18T21:18:14.135ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-1c343c62-3be2-438d-a7cd-d2aeb93884db
232024-10-18T21:18:14.139ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-1c343c62-3be2-438d-a7cd-d2aeb93884db
242024-10-18T21:18:14.153ZINFOcrucible-agent (worker): creating config property group
252024-10-18T21:18:14.157ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-1c343c62-3be2-438d-a7cd-d2aeb93884db
262024-10-18T21:18:14.164ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/1c343c62-3be2-438d-a7cd-d2aeb93884db
272024-10-18T21:18:14.164ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
282024-10-18T21:18:14.164ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
292024-10-18T21:18:14.164ZINFOcrucible-agent (worker): commit
302024-10-18T21:18:14.168ZINFOcrucible-agent (worker): ok!
312024-10-18T21:18:14.173ZINFOcrucible-agent (worker): SMF ok!
322024-10-18T21:18:14.240ZINFOcrucible-agent (dropshot): request completed latency_us = 59 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:49034 req_id = ac78b8fe-a256-4193-b7c3-7318526bb29e response_code = 200 uri = /crucible/0/regions
332024-10-18T21:18:14.808ZINFOcrucible-agent (dropshot): request completed latency_us = 81 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:49034 req_id = 93d4d4fb-e579-4a09-a35b-4ef06c72d9ab response_code = 200 uri = /crucible/0/regions/1c343c62-3be2-438d-a7cd-d2aeb93884db
342024-10-18T21:18:14.810ZINFOcrucible-agent (datafile): path is "/data/regions/1c343c62-3be2-438d-a7cd-d2aeb93884db"
352024-10-18T21:18:14.821ZINFOcrucible-agent (datafile): dataset is oxp_a37fa713-e408-4c30-9412-5e8373254f8d/crucible/regions/1c343c62-3be2-438d-a7cd-d2aeb93884db
362024-10-18T21:18:14.831ZINFOcrucible-agent (datafile): region 1c343c62-3be2-438d-a7cd-d2aeb93884db state: Created -> Tombstoned
372024-10-18T21:18:14.831ZINFOcrucible-agent (worker): applying SMF actions before removal...
382024-10-18T21:18:14.831ZINFOcrucible-agent (dropshot): request completed latency_us = 20882 local_addr = [fd00:1122:3344:101::14]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::c]:49034 req_id = 5faa619c-8723-4ed8-83ae-2ab1ef05e402 response_code = 204 uri = /crucible/0/regions/1c343c62-3be2-438d-a7cd-d2aeb93884db
392024-10-18T21:18:14.832ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-1c343c62-3be2-438d-a7cd-d2aeb93884db (instance states: (Some(Online), None))
402024-10-18T21:18:14.835ZINFOcrucible-agent (dropshot): request completed latency_us = 45 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:49034 req_id = d100cb91-3c93-4cce-ae75-54e6e53a87f4 response_code = 200 uri = /crucible/0/regions/1c343c62-3be2-438d-a7cd-d2aeb93884db
412024-10-18T21:18:14.838ZINFOcrucible-agent (worker): SMF ok!
422024-10-18T21:18:14.849ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_a37fa713-e408-4c30-9412-5e8373254f8d/crucible/regions/1c343c62-3be2-438d-a7cd-d2aeb93884db" region = 1c343c62-3be2-438d-a7cd-d2aeb93884db
432024-10-18T21:18:14.923ZINFOcrucible-agent (dropshot): request completed latency_us = 86 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:49034 req_id = 2fe3a64a-5fd9-408e-98c0-ff7d54722708 response_code = 200 uri = /crucible/0/regions/1c343c62-3be2-438d-a7cd-d2aeb93884db
442024-10-18T21:18:14.930ZINFOcrucible-agent (datafile): region 1c343c62-3be2-438d-a7cd-d2aeb93884db state: Tombstoned -> Destroyed
452024-10-18T21:18:15.016ZINFOcrucible-agent (dropshot): request completed latency_us = 49 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:49034 req_id = 16b7b819-a8a4-42fb-b89f-65e30af98a45 response_code = 200 uri = /crucible/0/regions/1c343c62-3be2-438d-a7cd-d2aeb93884db
462024-10-18T21:19:08.725ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::14]:32345 remote_addr = [fd00:1122:3344:101::a]:36207
472024-10-18T21:19:08.726ZINFOcrucible-agent (datafile): region 41764de2-27b1-492a-a3bc-f3cdfde58d02 state: Requested
482024-10-18T21:19:08.726ZINFOcrucible-agent (dropshot): request completed latency_us = 208 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:36207 req_id = 72ce6a24-90df-48f7-8cfe-ce4509e591de response_code = 200 uri = /crucible/0/regions
492024-10-18T21:19:08.726ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
502024-10-18T21:19:08.735ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_a37fa713-e408-4c30-9412-5e8373254f8d/crucible/regions/41764de2-27b1-492a-a3bc-f3cdfde58d02
512024-10-18T21:19:08.735ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_a37fa713-e408-4c30-9412-5e8373254f8d/crucible/regions/41764de2-27b1-492a-a3bc-f3cdfde58d02
522024-10-18T21:19:08.790ZINFOcrucible-agent (worker): creating region Region { id: RegionId("41764de2-27b1-492a-a3bc-f3cdfde58d02"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/41764de2-27b1-492a-a3bc-f3cdfde58d02" region = 41764de2-27b1-492a-a3bc-f3cdfde58d02
532024-10-18T21:19:08.792ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "41764de2-27b1-492a-a3bc-f3cdfde58d02" "--data" "/data/regions/41764de2-27b1-492a-a3bc-f3cdfde58d02" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 41764de2-27b1-492a-a3bc-f3cdfde58d02
542024-10-18T21:19:08.860ZINFOcrucible-agent (worker): region files created ok region = 41764de2-27b1-492a-a3bc-f3cdfde58d02
552024-10-18T21:19:08.860ZINFOcrucible-agent (datafile): region 41764de2-27b1-492a-a3bc-f3cdfde58d02 state: Requested -> Created
562024-10-18T21:19:08.860ZINFOcrucible-agent (worker): applying SMF actions post create...
572024-10-18T21:19:08.861ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-41764de2-27b1-492a-a3bc-f3cdfde58d02
582024-10-18T21:19:08.864ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-41764de2-27b1-492a-a3bc-f3cdfde58d02
592024-10-18T21:19:08.872ZINFOcrucible-agent (worker): creating config property group
602024-10-18T21:19:08.876ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-41764de2-27b1-492a-a3bc-f3cdfde58d02
612024-10-18T21:19:08.877ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/41764de2-27b1-492a-a3bc-f3cdfde58d02
622024-10-18T21:19:08.877ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
632024-10-18T21:19:08.877ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
642024-10-18T21:19:08.877ZINFOcrucible-agent (worker): commit
652024-10-18T21:19:08.880ZINFOcrucible-agent (worker): ok!
662024-10-18T21:19:08.885ZINFOcrucible-agent (worker): SMF ok!
672024-10-18T21:19:09.099ZINFOcrucible-agent (dropshot): request completed latency_us = 57 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:36207 req_id = b9335fe4-2c72-4b60-8f79-434063d5338e response_code = 200 uri = /crucible/0/regions
682024-10-18T21:22:17.849ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::14]:32345 remote_addr = [fd00:1122:3344:101::a]:44085
692024-10-18T21:22:17.849ZINFOcrucible-agent (dropshot): request completed latency_us = 56 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:44085 req_id = e1f3ab01-feaa-4ae9-bf2d-0bc886fbd6ad response_code = 200 uri = /crucible/0/regions/41764de2-27b1-492a-a3bc-f3cdfde58d02
702024-10-18T21:22:17.852ZINFOcrucible-agent (datafile): path is "/data/regions/41764de2-27b1-492a-a3bc-f3cdfde58d02"
712024-10-18T21:22:17.861ZINFOcrucible-agent (datafile): dataset is oxp_a37fa713-e408-4c30-9412-5e8373254f8d/crucible/regions/41764de2-27b1-492a-a3bc-f3cdfde58d02
722024-10-18T21:22:17.871ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_a37fa713-e408-4c30-9412-5e8373254f8d/crucible/regions/41764de2-27b1-492a-a3bc-f3cdfde58d02@ff9fdf84-4f7e-408d-893e-771b2220716d
732024-10-18T21:22:17.871ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_a37fa713-e408-4c30-9412-5e8373254f8d/crucible/regions/41764de2-27b1-492a-a3bc-f3cdfde58d02", "ff9fdf84-4f7e-408d-893e-771b2220716d"]
742024-10-18T21:22:17.884ZINFOcrucible-agent (dropshot): request completed latency_us = 32769 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:44085 req_id = 1ffd1485-0442-4980-83ad-774a10dee861 response_code = 200 uri = /crucible/0/regions/41764de2-27b1-492a-a3bc-f3cdfde58d02/snapshots/ff9fdf84-4f7e-408d-893e-771b2220716d
752024-10-18T21:22:17.888ZINFOcrucible-agent (datafile): path is "/data/regions/41764de2-27b1-492a-a3bc-f3cdfde58d02"
762024-10-18T21:22:17.896ZINFOcrucible-agent (datafile): dataset is oxp_a37fa713-e408-4c30-9412-5e8373254f8d/crucible/regions/41764de2-27b1-492a-a3bc-f3cdfde58d02
772024-10-18T21:22:17.905ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_a37fa713-e408-4c30-9412-5e8373254f8d/crucible/regions/41764de2-27b1-492a-a3bc-f3cdfde58d02@ff9fdf84-4f7e-408d-893e-771b2220716d
782024-10-18T21:22:17.905ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_a37fa713-e408-4c30-9412-5e8373254f8d/crucible/regions/41764de2-27b1-492a-a3bc-f3cdfde58d02", "ff9fdf84-4f7e-408d-893e-771b2220716d"]
792024-10-18T21:22:17.913ZINFOcrucible-agent (datafile): requesting running snapshot 41764de2-27b1-492a-a3bc-f3cdfde58d02-ff9fdf84-4f7e-408d-893e-771b2220716d state: Requested
802024-10-18T21:22:17.914ZINFOcrucible-agent (worker): applying SMF actions for region 41764de2-27b1-492a-a3bc-f3cdfde58d02 running snapshot ff9fdf84-4f7e-408d-893e-771b2220716d (state Requested)...
812024-10-18T21:22:17.914ZINFOcrucible-agent (dropshot): request completed latency_us = 26640 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:44085 req_id = bc387008-db68-42c3-8c35-96d5f24f8379 response_code = 200 uri = /crucible/0/regions/41764de2-27b1-492a-a3bc-f3cdfde58d02/snapshots/ff9fdf84-4f7e-408d-893e-771b2220716d/run
822024-10-18T21:22:17.923ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-41764de2-27b1-492a-a3bc-f3cdfde58d02-ff9fdf84-4f7e-408d-893e-771b2220716d
832024-10-18T21:22:17.929ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-41764de2-27b1-492a-a3bc-f3cdfde58d02-ff9fdf84-4f7e-408d-893e-771b2220716d
842024-10-18T21:22:17.942ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-41764de2-27b1-492a-a3bc-f3cdfde58d02-ff9fdf84-4f7e-408d-893e-771b2220716d
852024-10-18T21:22:17.942ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/41764de2-27b1-492a-a3bc-f3cdfde58d02/.zfs/snapshot/ff9fdf84-4f7e-408d-893e-771b2220716d
862024-10-18T21:22:17.942ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
872024-10-18T21:22:17.942ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
882024-10-18T21:22:17.942ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
892024-10-18T21:22:17.942ZINFOcrucible-agent (worker): commit
902024-10-18T21:22:17.947ZINFOcrucible-agent (worker): ok!
912024-10-18T21:22:17.952ZINFOcrucible-agent (worker): SMF ok!
922024-10-18T21:22:17.952ZINFOcrucible-agent (datafile): region 41764de2-27b1-492a-a3bc-f3cdfde58d02 running snapshot ff9fdf84-4f7e-408d-893e-771b2220716d state: Requested -> Created
932024-10-18T21:22:18.092ZINFOcrucible-agent (datafile): path is "/data/regions/41764de2-27b1-492a-a3bc-f3cdfde58d02"
942024-10-18T21:22:18.102ZINFOcrucible-agent (datafile): dataset is oxp_a37fa713-e408-4c30-9412-5e8373254f8d/crucible/regions/41764de2-27b1-492a-a3bc-f3cdfde58d02
952024-10-18T21:22:18.109ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_a37fa713-e408-4c30-9412-5e8373254f8d/crucible/regions/41764de2-27b1-492a-a3bc-f3cdfde58d02@ff9fdf84-4f7e-408d-893e-771b2220716d
962024-10-18T21:22:18.110ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_a37fa713-e408-4c30-9412-5e8373254f8d/crucible/regions/41764de2-27b1-492a-a3bc-f3cdfde58d02", "ff9fdf84-4f7e-408d-893e-771b2220716d"]
972024-10-18T21:22:18.117ZINFOcrucible-agent (dropshot): request completed latency_us = 24282 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:44085 req_id = 0ec9e03f-5077-463e-aa9a-f5921a90a335 response_code = 200 uri = /crucible/0/regions/41764de2-27b1-492a-a3bc-f3cdfde58d02/snapshots/ff9fdf84-4f7e-408d-893e-771b2220716d/run
982024-10-18T21:22:23.526ZINFOcrucible-agent (datafile): region c3116493-1fe8-42f0-8835-e4ae99aec403 state: Requested
992024-10-18T21:22:23.526ZINFOcrucible-agent (dropshot): request completed latency_us = 267 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:44085 req_id = e084318f-7ed7-40db-918d-bb2bb679c3a5 response_code = 200 uri = /crucible/0/regions
1002024-10-18T21:22:23.526ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
1012024-10-18T21:22:23.536ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_a37fa713-e408-4c30-9412-5e8373254f8d/crucible/regions/c3116493-1fe8-42f0-8835-e4ae99aec403
1022024-10-18T21:22:23.536ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_a37fa713-e408-4c30-9412-5e8373254f8d/crucible/regions/c3116493-1fe8-42f0-8835-e4ae99aec403
1032024-10-18T21:22:23.723ZINFOcrucible-agent (worker): creating region Region { id: RegionId("c3116493-1fe8-42f0-8835-e4ae99aec403"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19002, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/c3116493-1fe8-42f0-8835-e4ae99aec403" region = c3116493-1fe8-42f0-8835-e4ae99aec403
1042024-10-18T21:22:23.725ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "c3116493-1fe8-42f0-8835-e4ae99aec403" "--data" "/data/regions/c3116493-1fe8-42f0-8835-e4ae99aec403" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = c3116493-1fe8-42f0-8835-e4ae99aec403
1052024-10-18T21:22:23.824ZINFOcrucible-agent (dropshot): request completed latency_us = 78 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:44085 req_id = feec7065-e993-402a-992b-0c27ba2cde29 response_code = 200 uri = /crucible/0/regions
1062024-10-18T21:22:24.021ZINFOcrucible-agent (worker): region files created ok region = c3116493-1fe8-42f0-8835-e4ae99aec403
1072024-10-18T21:22:24.021ZINFOcrucible-agent (datafile): region c3116493-1fe8-42f0-8835-e4ae99aec403 state: Requested -> Created
1082024-10-18T21:22:24.021ZINFOcrucible-agent (worker): applying SMF actions post create...
1092024-10-18T21:22:24.023ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-c3116493-1fe8-42f0-8835-e4ae99aec403
1102024-10-18T21:22:24.077ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-c3116493-1fe8-42f0-8835-e4ae99aec403
1112024-10-18T21:22:24.122ZINFOcrucible-agent (worker): creating config property group
1122024-10-18T21:22:24.148ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-c3116493-1fe8-42f0-8835-e4ae99aec403
1132024-10-18T21:22:24.148ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/c3116493-1fe8-42f0-8835-e4ae99aec403
1142024-10-18T21:22:24.148ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
1152024-10-18T21:22:24.148ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
1162024-10-18T21:22:24.148ZINFOcrucible-agent (worker): commit
1172024-10-18T21:22:24.197ZINFOcrucible-agent (worker): ok!
1182024-10-18T21:22:24.249ZINFOcrucible-agent (worker): SMF ok!
1192024-10-18T21:22:24.327ZINFOcrucible-agent (dropshot): request completed latency_us = 596 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:44085 req_id = ccdaf97e-5999-4b61-ac5b-2463c5fee8ab response_code = 200 uri = /crucible/0/regions
1202024-10-18T21:22:54.492ZINFOcrucible-agent (dropshot): request completed latency_us = 51 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:44085 req_id = 0673f626-e82d-4b75-9ab9-0f26919f6cfe response_code = 200 uri = /crucible/0/regions/c3116493-1fe8-42f0-8835-e4ae99aec403
1212024-10-18T21:22:54.496ZINFOcrucible-agent (datafile): path is "/data/regions/c3116493-1fe8-42f0-8835-e4ae99aec403"
1222024-10-18T21:22:54.509ZINFOcrucible-agent (datafile): dataset is oxp_a37fa713-e408-4c30-9412-5e8373254f8d/crucible/regions/c3116493-1fe8-42f0-8835-e4ae99aec403
1232024-10-18T21:22:54.520ZINFOcrucible-agent (datafile): region c3116493-1fe8-42f0-8835-e4ae99aec403 state: Created -> Tombstoned
1242024-10-18T21:22:54.520ZINFOcrucible-agent (worker): applying SMF actions before removal...
1252024-10-18T21:22:54.520ZINFOcrucible-agent (dropshot): request completed latency_us = 24249 local_addr = [fd00:1122:3344:101::14]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:44085 req_id = 15f99c65-471c-417d-9343-8fe1fc7505fe response_code = 204 uri = /crucible/0/regions/c3116493-1fe8-42f0-8835-e4ae99aec403
1262024-10-18T21:22:54.522ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-c3116493-1fe8-42f0-8835-e4ae99aec403 (instance states: (Some(Online), None))
1272024-10-18T21:22:54.524ZINFOcrucible-agent (dropshot): request completed latency_us = 33 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:44085 req_id = a694be23-354e-48c4-915e-e331c10eee4a response_code = 200 uri = /crucible/0/regions/c3116493-1fe8-42f0-8835-e4ae99aec403
1282024-10-18T21:22:54.535ZINFOcrucible-agent (worker): SMF ok!
1292024-10-18T21:22:54.546ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_a37fa713-e408-4c30-9412-5e8373254f8d/crucible/regions/c3116493-1fe8-42f0-8835-e4ae99aec403" region = c3116493-1fe8-42f0-8835-e4ae99aec403
1302024-10-18T21:22:54.592ZINFOcrucible-agent (dropshot): request completed latency_us = 84 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:44085 req_id = 8b3de9b5-8452-4d98-8625-ff99ccd889f1 response_code = 200 uri = /crucible/0/regions/c3116493-1fe8-42f0-8835-e4ae99aec403
1312024-10-18T21:22:54.619ZINFOcrucible-agent (datafile): region c3116493-1fe8-42f0-8835-e4ae99aec403 state: Tombstoned -> Destroyed
1322024-10-18T21:22:54.761ZINFOcrucible-agent (dropshot): request completed latency_us = 44 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:44085 req_id = 7212faae-5956-4f14-8852-d91b3446cc55 response_code = 200 uri = /crucible/0/regions/c3116493-1fe8-42f0-8835-e4ae99aec403