Max level shown:
1 [ Feb 22 18:31:16 Enabled. ]
2 [ Feb 22 18:31:16 Rereading configuration. ]
3 [ Feb 22 18:31:18 Rereading configuration. ]
4 [ Feb 22 18:31:28 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7/crucible -l [fd00:1122:3344:101::15]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62024-02-22T18:31:28.731ZINFOcrucible-agent: dataset: "oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7/crucible"
72024-02-22T18:31:28.735ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::15]:32345
82024-02-22T18:31:28.735ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92024-02-22T18:31:28.981ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::15]:32345
102024-02-22T18:32:29.955ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::15]:32345 remote_addr = [fd00:1122:3344:101::a]:56574
112024-02-22T18:32:30.024ZINFOcrucible-agent (datafile): region 8626f0ee-549f-4fad-9c87-482b61b90830 state: Requested
122024-02-22T18:32:30.026ZINFOcrucible-agent (dropshot): request completed latency_us = 5155 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:56574 req_id = c02e885f-c7ac-423f-a754-d255f92ee4bc response_code = 200 uri = /crucible/0/regions
132024-02-22T18:32:30.026ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
142024-02-22T18:32:30.035ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7/crucible/regions/8626f0ee-549f-4fad-9c87-482b61b90830
152024-02-22T18:32:30.035ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7/crucible/regions/8626f0ee-549f-4fad-9c87-482b61b90830
162024-02-22T18:32:30.081ZINFOcrucible-agent (worker): creating region Region { id: RegionId("8626f0ee-549f-4fad-9c87-482b61b90830"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None } at "/data/regions/8626f0ee-549f-4fad-9c87-482b61b90830" region = 8626f0ee-549f-4fad-9c87-482b61b90830
172024-02-22T18:32:30.206ZINFOcrucible-agent (worker): region files created ok region = 8626f0ee-549f-4fad-9c87-482b61b90830
182024-02-22T18:32:30.206ZINFOcrucible-agent (datafile): region 8626f0ee-549f-4fad-9c87-482b61b90830 state: Requested -> Created
192024-02-22T18:32:30.207ZINFOcrucible-agent (worker): applying SMF actions post create...
202024-02-22T18:32:30.207ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-8626f0ee-549f-4fad-9c87-482b61b90830
212024-02-22T18:32:30.213ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-8626f0ee-549f-4fad-9c87-482b61b90830
222024-02-22T18:32:30.223ZINFOcrucible-agent (worker): creating config property group
232024-02-22T18:32:30.226ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-8626f0ee-549f-4fad-9c87-482b61b90830
242024-02-22T18:32:30.227ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/8626f0ee-549f-4fad-9c87-482b61b90830
252024-02-22T18:32:30.227ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
262024-02-22T18:32:30.227ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
272024-02-22T18:32:30.227ZINFOcrucible-agent (worker): commit
282024-02-22T18:32:30.231ZINFOcrucible-agent (worker): ok!
292024-02-22T18:32:30.236ZINFOcrucible-agent (worker): SMF ok!
302024-02-22T18:32:30.240ZINFOcrucible-agent (dropshot): request completed latency_us = 70 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:56574 req_id = 8c7b769e-587d-4df8-9174-740441ea34a4 response_code = 200 uri = /crucible/0/regions
312024-02-22T18:37:50.664ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::15]:32345 remote_addr = [fd00:1122:3344:101::a]:38742
322024-02-22T18:37:50.664ZINFOcrucible-agent (datafile): region 3d7b769e-479a-4386-96ce-0f08d2587518 state: Requested
332024-02-22T18:37:50.665ZINFOcrucible-agent (dropshot): request completed latency_us = 1280 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:38742 req_id = b7499113-a1c3-477a-985e-baa11502bfca response_code = 200 uri = /crucible/0/regions
342024-02-22T18:37:50.665ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
352024-02-22T18:37:50.674ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7/crucible/regions/3d7b769e-479a-4386-96ce-0f08d2587518
362024-02-22T18:37:50.674ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7/crucible/regions/3d7b769e-479a-4386-96ce-0f08d2587518
372024-02-22T18:37:50.727ZINFOcrucible-agent (worker): creating region Region { id: RegionId("3d7b769e-479a-4386-96ce-0f08d2587518"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None } at "/data/regions/3d7b769e-479a-4386-96ce-0f08d2587518" region = 3d7b769e-479a-4386-96ce-0f08d2587518
382024-02-22T18:37:50.796ZINFOcrucible-agent (worker): region files created ok region = 3d7b769e-479a-4386-96ce-0f08d2587518
392024-02-22T18:37:50.796ZINFOcrucible-agent (datafile): region 3d7b769e-479a-4386-96ce-0f08d2587518 state: Requested -> Created
402024-02-22T18:37:50.797ZINFOcrucible-agent (worker): applying SMF actions post create...
412024-02-22T18:37:50.797ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-3d7b769e-479a-4386-96ce-0f08d2587518
422024-02-22T18:37:50.803ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-3d7b769e-479a-4386-96ce-0f08d2587518
432024-02-22T18:37:50.803ZINFOcrucible-agent (dropshot): request completed latency_us = 66 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:38742 req_id = 3d427879-492b-419d-ae84-0476cb6f23a3 response_code = 200 uri = /crucible/0/regions
442024-02-22T18:37:50.815ZINFOcrucible-agent (worker): creating config property group
452024-02-22T18:37:50.818ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-3d7b769e-479a-4386-96ce-0f08d2587518
462024-02-22T18:37:50.818ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/3d7b769e-479a-4386-96ce-0f08d2587518
472024-02-22T18:37:50.818ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
482024-02-22T18:37:50.818ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
492024-02-22T18:37:50.819ZINFOcrucible-agent (worker): commit
502024-02-22T18:37:50.822ZINFOcrucible-agent (worker): ok!
512024-02-22T18:37:50.827ZINFOcrucible-agent (worker): SMF ok!
522024-02-22T18:37:52.018ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::15]:32345 remote_addr = [fd00:1122:3344:101::a]:61977
532024-02-22T18:37:52.020ZINFOcrucible-agent (dropshot): request completed latency_us = 2570 local_addr = [fd00:1122:3344:101::15]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:61977 req_id = 9b0ff923-82f9-41d4-9ff0-85b164a66c95 response_code = 200 uri = /crucible/0/regions/8626f0ee-549f-4fad-9c87-482b61b90830
542024-02-22T18:37:52.021ZINFOcrucible-agent (datafile): path is "/data/regions/8626f0ee-549f-4fad-9c87-482b61b90830"
552024-02-22T18:37:52.030ZINFOcrucible-agent (datafile): dataset is oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7/crucible/regions/8626f0ee-549f-4fad-9c87-482b61b90830
562024-02-22T18:37:52.039ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7/crucible/regions/8626f0ee-549f-4fad-9c87-482b61b90830@5d9df203-ce89-4f7f-9ce1-08b8316a4558
572024-02-22T18:37:52.039ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7/crucible/regions/8626f0ee-549f-4fad-9c87-482b61b90830", "5d9df203-ce89-4f7f-9ce1-08b8316a4558"]
582024-02-22T18:37:52.049ZINFOcrucible-agent (dropshot): request completed latency_us = 28736 local_addr = [fd00:1122:3344:101::15]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:61977 req_id = eb5572d1-3d71-46ff-9fae-b616f63abe24 response_code = 200 uri = /crucible/0/regions/8626f0ee-549f-4fad-9c87-482b61b90830/snapshots/5d9df203-ce89-4f7f-9ce1-08b8316a4558
592024-02-22T18:37:52.049ZINFOcrucible-agent (datafile): path is "/data/regions/8626f0ee-549f-4fad-9c87-482b61b90830"
602024-02-22T18:37:52.058ZINFOcrucible-agent (datafile): dataset is oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7/crucible/regions/8626f0ee-549f-4fad-9c87-482b61b90830
612024-02-22T18:37:52.067ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7/crucible/regions/8626f0ee-549f-4fad-9c87-482b61b90830@5d9df203-ce89-4f7f-9ce1-08b8316a4558
622024-02-22T18:37:52.067ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7/crucible/regions/8626f0ee-549f-4fad-9c87-482b61b90830", "5d9df203-ce89-4f7f-9ce1-08b8316a4558"]
632024-02-22T18:37:52.075ZINFOcrucible-agent (datafile): requesting running snapshot 8626f0ee-549f-4fad-9c87-482b61b90830-5d9df203-ce89-4f7f-9ce1-08b8316a4558 state: Requested
642024-02-22T18:37:52.076ZINFOcrucible-agent (worker): applying SMF actions for region 8626f0ee-549f-4fad-9c87-482b61b90830 running snapshot 5d9df203-ce89-4f7f-9ce1-08b8316a4558 (state Requested)...
652024-02-22T18:37:52.076ZINFOcrucible-agent (dropshot): request completed latency_us = 26225 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:61977 req_id = 2b59c7fa-c2da-43fa-a46d-fa83a1893fbb response_code = 200 uri = /crucible/0/regions/8626f0ee-549f-4fad-9c87-482b61b90830/snapshots/5d9df203-ce89-4f7f-9ce1-08b8316a4558/run
662024-02-22T18:37:52.078ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-8626f0ee-549f-4fad-9c87-482b61b90830-5d9df203-ce89-4f7f-9ce1-08b8316a4558
672024-02-22T18:37:52.086ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-8626f0ee-549f-4fad-9c87-482b61b90830-5d9df203-ce89-4f7f-9ce1-08b8316a4558
682024-02-22T18:37:52.098ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-8626f0ee-549f-4fad-9c87-482b61b90830-5d9df203-ce89-4f7f-9ce1-08b8316a4558
692024-02-22T18:37:52.098ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/8626f0ee-549f-4fad-9c87-482b61b90830/.zfs/snapshot/5d9df203-ce89-4f7f-9ce1-08b8316a4558
702024-02-22T18:37:52.098ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
712024-02-22T18:37:52.098ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
722024-02-22T18:37:52.098ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
732024-02-22T18:37:52.098ZINFOcrucible-agent (worker): commit
742024-02-22T18:37:52.102ZINFOcrucible-agent (worker): ok!
752024-02-22T18:37:52.106ZINFOcrucible-agent (worker): SMF ok!
762024-02-22T18:37:52.106ZINFOcrucible-agent (datafile): region 8626f0ee-549f-4fad-9c87-482b61b90830 running snapshot 5d9df203-ce89-4f7f-9ce1-08b8316a4558 state: Requested -> Created
772024-02-22T18:37:54.774ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::15]:32345 remote_addr = [fd00:1122:3344:101::a]:42212
782024-02-22T18:37:54.774ZINFOcrucible-agent (datafile): region c94843fe-a7b0-4b90-883d-d36016f76c26 state: Requested
792024-02-22T18:37:54.774ZINFOcrucible-agent (dropshot): request completed latency_us = 348 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:42212 req_id = f959d025-7cb3-44e6-ac65-6b412417a546 response_code = 200 uri = /crucible/0/regions
802024-02-22T18:37:54.774ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
812024-02-22T18:37:54.783ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7/crucible/regions/c94843fe-a7b0-4b90-883d-d36016f76c26
822024-02-22T18:37:54.783ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7/crucible/regions/c94843fe-a7b0-4b90-883d-d36016f76c26
832024-02-22T18:37:54.849ZINFOcrucible-agent (worker): creating region Region { id: RegionId("c94843fe-a7b0-4b90-883d-d36016f76c26"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19003, cert_pem: None, key_pem: None, root_pem: None } at "/data/regions/c94843fe-a7b0-4b90-883d-d36016f76c26" region = c94843fe-a7b0-4b90-883d-d36016f76c26
842024-02-22T18:37:54.913ZINFOcrucible-agent (worker): region files created ok region = c94843fe-a7b0-4b90-883d-d36016f76c26
852024-02-22T18:37:54.913ZINFOcrucible-agent (datafile): region c94843fe-a7b0-4b90-883d-d36016f76c26 state: Requested -> Created
862024-02-22T18:37:54.913ZINFOcrucible-agent (worker): applying SMF actions post create...
872024-02-22T18:37:54.915ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-c94843fe-a7b0-4b90-883d-d36016f76c26
882024-02-22T18:37:54.918ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-c94843fe-a7b0-4b90-883d-d36016f76c26
892024-02-22T18:37:54.926ZINFOcrucible-agent (worker): creating config property group
902024-02-22T18:37:54.930ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-c94843fe-a7b0-4b90-883d-d36016f76c26
912024-02-22T18:37:54.930ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/c94843fe-a7b0-4b90-883d-d36016f76c26
922024-02-22T18:37:54.930ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19003
932024-02-22T18:37:54.930ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
942024-02-22T18:37:54.930ZINFOcrucible-agent (worker): commit
952024-02-22T18:37:54.934ZINFOcrucible-agent (worker): ok!
962024-02-22T18:37:54.939ZINFOcrucible-agent (worker): SMF ok!
972024-02-22T18:37:55.002ZINFOcrucible-agent (dropshot): request completed latency_us = 65 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:42212 req_id = bf840491-f12d-4160-b2e6-e46b41b4c653 response_code = 200 uri = /crucible/0/regions
982024-02-22T18:38:25.437ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::15]:32345 remote_addr = [fd00:1122:3344:101::a]:37510
992024-02-22T18:38:25.437ZINFOcrucible-agent (dropshot): request completed latency_us = 51 local_addr = [fd00:1122:3344:101::15]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:37510 req_id = 3f7a9082-4e47-435f-ac84-0a05adccc3de response_code = 200 uri = /crucible/0/regions/c94843fe-a7b0-4b90-883d-d36016f76c26
1002024-02-22T18:38:25.438ZINFOcrucible-agent (datafile): path is "/data/regions/c94843fe-a7b0-4b90-883d-d36016f76c26"
1012024-02-22T18:38:25.449ZINFOcrucible-agent (datafile): dataset is oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7/crucible/regions/c94843fe-a7b0-4b90-883d-d36016f76c26
1022024-02-22T18:38:25.460ZINFOcrucible-agent (datafile): region c94843fe-a7b0-4b90-883d-d36016f76c26 state: Created -> Tombstoned
1032024-02-22T18:38:25.461ZINFOcrucible-agent (worker): applying SMF actions before removal...
1042024-02-22T18:38:25.461ZINFOcrucible-agent (dropshot): request completed latency_us = 23102 local_addr = [fd00:1122:3344:101::15]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:37510 req_id = 09caddc3-fc24-4706-af5b-4cdbd7282193 response_code = 204 uri = /crucible/0/regions/c94843fe-a7b0-4b90-883d-d36016f76c26
1052024-02-22T18:38:25.461ZINFOcrucible-agent (dropshot): request completed latency_us = 582 local_addr = [fd00:1122:3344:101::15]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:37510 req_id = 56b7a7bb-12d6-4bff-af7f-b657ca7ce1bc response_code = 200 uri = /crucible/0/regions/c94843fe-a7b0-4b90-883d-d36016f76c26
1062024-02-22T18:38:25.462ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-c94843fe-a7b0-4b90-883d-d36016f76c26 (instance states: (Some(Online), None))
1072024-02-22T18:38:25.469ZINFOcrucible-agent (worker): SMF ok!
1082024-02-22T18:38:25.479ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7/crucible/regions/c94843fe-a7b0-4b90-883d-d36016f76c26" region = c94843fe-a7b0-4b90-883d-d36016f76c26
1092024-02-22T18:38:25.526ZINFOcrucible-agent (datafile): region c94843fe-a7b0-4b90-883d-d36016f76c26 state: Tombstoned -> Destroyed
1102024-02-22T18:38:25.563ZINFOcrucible-agent (dropshot): request completed latency_us = 60 local_addr = [fd00:1122:3344:101::15]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:37510 req_id = 0ab57dc5-0a1a-47ec-814b-6ca1115f513c response_code = 200 uri = /crucible/0/regions/c94843fe-a7b0-4b90-883d-d36016f76c26