Max level shown:
1 [ Jul 15 22:20:10 Enabled. ]
2 [ Jul 15 22:20:10 Rereading configuration. ]
3 [ Jul 15 22:20:12 Rereading configuration. ]
4 [ Jul 15 22:20:14 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_6c05cd98-a794-48e4-b841-27ad4302be3f/crucible -l [fd00:1122:3344:101::18]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62024-07-15T22:20:14.425ZINFOcrucible-agent: dataset: "oxp_6c05cd98-a794-48e4-b841-27ad4302be3f/crucible"
72024-07-15T22:20:14.430ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::18]:32345
82024-07-15T22:20:14.430ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92024-07-15T22:20:14.453ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102024-07-15T22:20:14.563ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::18]:32345
112024-07-15T22:21:14.136ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::18]:32345 remote_addr = [fd00:1122:3344:101::a]:43768
122024-07-15T22:21:14.155ZINFOcrucible-agent (datafile): region 440ade73-fb5f-4226-a738-286077f80dd1 state: Requested
132024-07-15T22:21:14.162ZINFOcrucible-agent (dropshot): request completed latency_us = 14071 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:43768 req_id = b6f710e1-99d9-4947-b5ad-bf0293631b63 response_code = 200 uri = /crucible/0/regions
142024-07-15T22:21:14.164ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
152024-07-15T22:21:14.176ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_6c05cd98-a794-48e4-b841-27ad4302be3f/crucible/regions/440ade73-fb5f-4226-a738-286077f80dd1
162024-07-15T22:21:14.176ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_6c05cd98-a794-48e4-b841-27ad4302be3f/crucible/regions/440ade73-fb5f-4226-a738-286077f80dd1
172024-07-15T22:21:14.229ZINFOcrucible-agent (worker): creating region Region { id: RegionId("440ade73-fb5f-4226-a738-286077f80dd1"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/440ade73-fb5f-4226-a738-286077f80dd1" region = 440ade73-fb5f-4226-a738-286077f80dd1
182024-07-15T22:21:14.231ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "440ade73-fb5f-4226-a738-286077f80dd1" "--data" "/data/regions/440ade73-fb5f-4226-a738-286077f80dd1" "--block-size" "512" "--extent-size" "131072" "--extent-count" "16" "--encrypted" region = 440ade73-fb5f-4226-a738-286077f80dd1
192024-07-15T22:21:14.345ZINFOcrucible-agent (dropshot): request completed latency_us = 81 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:43768 req_id = c7a2cbbc-dc52-476e-aaa2-1b4a2e66fac5 response_code = 200 uri = /crucible/0/regions
202024-07-15T22:21:14.381ZINFOcrucible-agent (worker): region files created ok region = 440ade73-fb5f-4226-a738-286077f80dd1
212024-07-15T22:21:14.381ZINFOcrucible-agent (datafile): region 440ade73-fb5f-4226-a738-286077f80dd1 state: Requested -> Created
222024-07-15T22:21:14.381ZINFOcrucible-agent (worker): applying SMF actions post create...
232024-07-15T22:21:14.382ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-440ade73-fb5f-4226-a738-286077f80dd1
242024-07-15T22:21:14.392ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-440ade73-fb5f-4226-a738-286077f80dd1
252024-07-15T22:21:14.411ZINFOcrucible-agent (worker): creating config property group
262024-07-15T22:21:14.413ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-440ade73-fb5f-4226-a738-286077f80dd1
272024-07-15T22:21:14.414ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/440ade73-fb5f-4226-a738-286077f80dd1
282024-07-15T22:21:14.418ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
292024-07-15T22:21:14.418ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
302024-07-15T22:21:14.418ZINFOcrucible-agent (worker): commit
312024-07-15T22:21:14.426ZINFOcrucible-agent (worker): ok!
322024-07-15T22:21:14.433ZINFOcrucible-agent (worker): SMF ok!
332024-07-15T22:21:14.920ZINFOcrucible-agent (dropshot): request completed latency_us = 90 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:43768 req_id = bc4ddec2-08ab-4558-8802-595b8ceae3a0 response_code = 200 uri = /crucible/0/regions
342024-07-15T22:21:15.411ZINFOcrucible-agent (dropshot): request completed latency_us = 757 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:43768 req_id = e9c9af74-af27-43aa-90dc-333bf4464f6d response_code = 200 uri = /crucible/0/regions/440ade73-fb5f-4226-a738-286077f80dd1
352024-07-15T22:21:15.414ZINFOcrucible-agent (datafile): path is "/data/regions/440ade73-fb5f-4226-a738-286077f80dd1"
362024-07-15T22:21:15.424ZINFOcrucible-agent (datafile): dataset is oxp_6c05cd98-a794-48e4-b841-27ad4302be3f/crucible/regions/440ade73-fb5f-4226-a738-286077f80dd1
372024-07-15T22:21:15.433ZINFOcrucible-agent (datafile): region 440ade73-fb5f-4226-a738-286077f80dd1 state: Created -> Tombstoned
382024-07-15T22:21:15.433ZINFOcrucible-agent (worker): applying SMF actions before removal...
392024-07-15T22:21:15.433ZINFOcrucible-agent (dropshot): request completed latency_us = 19427 local_addr = [fd00:1122:3344:101::18]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:43768 req_id = 1db36fb1-c176-4f94-a815-ddf52c753d48 response_code = 204 uri = /crucible/0/regions/440ade73-fb5f-4226-a738-286077f80dd1
402024-07-15T22:21:15.434ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-440ade73-fb5f-4226-a738-286077f80dd1 (instance states: (Some(Online), None))
412024-07-15T22:21:15.436ZINFOcrucible-agent (dropshot): request completed latency_us = 22 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:43768 req_id = be39c9aa-d4dd-4e34-afea-80262f706ed1 response_code = 200 uri = /crucible/0/regions/440ade73-fb5f-4226-a738-286077f80dd1
422024-07-15T22:21:15.439ZINFOcrucible-agent (worker): SMF ok!
432024-07-15T22:21:15.449ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_6c05cd98-a794-48e4-b841-27ad4302be3f/crucible/regions/440ade73-fb5f-4226-a738-286077f80dd1" region = 440ade73-fb5f-4226-a738-286077f80dd1
442024-07-15T22:21:15.495ZINFOcrucible-agent (datafile): region 440ade73-fb5f-4226-a738-286077f80dd1 state: Tombstoned -> Destroyed
452024-07-15T22:21:15.550ZINFOcrucible-agent (dropshot): request completed latency_us = 50 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:43768 req_id = f037439b-a5f6-40df-a77a-b95cd7b5d201 response_code = 200 uri = /crucible/0/regions/440ade73-fb5f-4226-a738-286077f80dd1
462024-07-15T22:21:16.916ZINFOcrucible-agent (datafile): region 15111146-33e0-47c2-814e-5e1ef661095e state: Requested
472024-07-15T22:21:16.918ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
482024-07-15T22:21:16.918ZINFOcrucible-agent (dropshot): request completed latency_us = 2106 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:43768 req_id = a25117f1-a09e-4002-8fc5-e567ca897888 response_code = 200 uri = /crucible/0/regions
492024-07-15T22:21:16.926ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_6c05cd98-a794-48e4-b841-27ad4302be3f/crucible/regions/15111146-33e0-47c2-814e-5e1ef661095e
502024-07-15T22:21:16.926ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_6c05cd98-a794-48e4-b841-27ad4302be3f/crucible/regions/15111146-33e0-47c2-814e-5e1ef661095e
512024-07-15T22:21:16.975ZINFOcrucible-agent (worker): creating region Region { id: RegionId("15111146-33e0-47c2-814e-5e1ef661095e"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/15111146-33e0-47c2-814e-5e1ef661095e" region = 15111146-33e0-47c2-814e-5e1ef661095e
522024-07-15T22:21:16.978ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "15111146-33e0-47c2-814e-5e1ef661095e" "--data" "/data/regions/15111146-33e0-47c2-814e-5e1ef661095e" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 15111146-33e0-47c2-814e-5e1ef661095e
532024-07-15T22:21:17.037ZINFOcrucible-agent (worker): region files created ok region = 15111146-33e0-47c2-814e-5e1ef661095e
542024-07-15T22:21:17.037ZINFOcrucible-agent (datafile): region 15111146-33e0-47c2-814e-5e1ef661095e state: Requested -> Created
552024-07-15T22:21:17.037ZINFOcrucible-agent (worker): applying SMF actions post create...
562024-07-15T22:21:17.039ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-15111146-33e0-47c2-814e-5e1ef661095e
572024-07-15T22:21:17.042ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-15111146-33e0-47c2-814e-5e1ef661095e
582024-07-15T22:21:17.049ZINFOcrucible-agent (worker): creating config property group
592024-07-15T22:21:17.052ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-15111146-33e0-47c2-814e-5e1ef661095e
602024-07-15T22:21:17.052ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/15111146-33e0-47c2-814e-5e1ef661095e
612024-07-15T22:21:17.052ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
622024-07-15T22:21:17.052ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
632024-07-15T22:21:17.052ZINFOcrucible-agent (worker): commit
642024-07-15T22:21:17.055ZINFOcrucible-agent (worker): ok!
652024-07-15T22:21:17.059ZINFOcrucible-agent (worker): SMF ok!
662024-07-15T22:21:17.157ZINFOcrucible-agent (dropshot): request completed latency_us = 69 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:43768 req_id = 249a4327-c8c3-4567-8b97-be34b0b6f1bb response_code = 200 uri = /crucible/0/regions
672024-07-15T22:26:33.929ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::18]:32345 remote_addr = [fd00:1122:3344:101::a]:36742
682024-07-15T22:26:33.929ZINFOcrucible-agent (datafile): region 5c445ff2-3db8-436e-a43d-5fcfe3711a22 state: Requested
692024-07-15T22:26:33.930ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
702024-07-15T22:26:33.930ZINFOcrucible-agent (dropshot): request completed latency_us = 774 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:36742 req_id = 44db837f-8c81-4005-b3db-560bd5b1c562 response_code = 200 uri = /crucible/0/regions
712024-07-15T22:26:33.940ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_6c05cd98-a794-48e4-b841-27ad4302be3f/crucible/regions/5c445ff2-3db8-436e-a43d-5fcfe3711a22
722024-07-15T22:26:33.940ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_6c05cd98-a794-48e4-b841-27ad4302be3f/crucible/regions/5c445ff2-3db8-436e-a43d-5fcfe3711a22
732024-07-15T22:26:33.997ZINFOcrucible-agent (worker): creating region Region { id: RegionId("5c445ff2-3db8-436e-a43d-5fcfe3711a22"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/5c445ff2-3db8-436e-a43d-5fcfe3711a22" region = 5c445ff2-3db8-436e-a43d-5fcfe3711a22
742024-07-15T22:26:33.999ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "5c445ff2-3db8-436e-a43d-5fcfe3711a22" "--data" "/data/regions/5c445ff2-3db8-436e-a43d-5fcfe3711a22" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 5c445ff2-3db8-436e-a43d-5fcfe3711a22
752024-07-15T22:26:34.118ZINFOcrucible-agent (worker): region files created ok region = 5c445ff2-3db8-436e-a43d-5fcfe3711a22
762024-07-15T22:26:34.118ZINFOcrucible-agent (datafile): region 5c445ff2-3db8-436e-a43d-5fcfe3711a22 state: Requested -> Created
772024-07-15T22:26:34.118ZINFOcrucible-agent (worker): applying SMF actions post create...
782024-07-15T22:26:34.126ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-5c445ff2-3db8-436e-a43d-5fcfe3711a22
792024-07-15T22:26:34.136ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-5c445ff2-3db8-436e-a43d-5fcfe3711a22
802024-07-15T22:26:34.150ZINFOcrucible-agent (worker): creating config property group
812024-07-15T22:26:34.157ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-5c445ff2-3db8-436e-a43d-5fcfe3711a22
822024-07-15T22:26:34.157ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/5c445ff2-3db8-436e-a43d-5fcfe3711a22
832024-07-15T22:26:34.158ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
842024-07-15T22:26:34.158ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
852024-07-15T22:26:34.158ZINFOcrucible-agent (worker): commit
862024-07-15T22:26:34.166ZINFOcrucible-agent (worker): ok!
872024-07-15T22:26:34.172ZINFOcrucible-agent (worker): SMF ok!
882024-07-15T22:26:34.287ZINFOcrucible-agent (dropshot): request completed latency_us = 76 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:36742 req_id = 931bf0fa-70dc-4199-a5cf-1084186e8f8f response_code = 200 uri = /crucible/0/regions
892024-07-15T22:26:35.217ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::18]:32345 remote_addr = [fd00:1122:3344:101::a]:51550
902024-07-15T22:26:35.218ZINFOcrucible-agent (dropshot): request completed latency_us = 61 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:51550 req_id = 04dbba38-07a4-4a74-8432-ebe98b7bd7ff response_code = 200 uri = /crucible/0/regions/15111146-33e0-47c2-814e-5e1ef661095e
912024-07-15T22:26:35.220ZINFOcrucible-agent (datafile): path is "/data/regions/15111146-33e0-47c2-814e-5e1ef661095e"
922024-07-15T22:26:35.229ZINFOcrucible-agent (datafile): dataset is oxp_6c05cd98-a794-48e4-b841-27ad4302be3f/crucible/regions/15111146-33e0-47c2-814e-5e1ef661095e
932024-07-15T22:26:35.238ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_6c05cd98-a794-48e4-b841-27ad4302be3f/crucible/regions/15111146-33e0-47c2-814e-5e1ef661095e@bdb257f9-ff1c-4eb6-85a9-a642ac5979b1
942024-07-15T22:26:35.238ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_6c05cd98-a794-48e4-b841-27ad4302be3f/crucible/regions/15111146-33e0-47c2-814e-5e1ef661095e", "bdb257f9-ff1c-4eb6-85a9-a642ac5979b1"]
952024-07-15T22:26:35.247ZINFOcrucible-agent (dropshot): request completed latency_us = 26806 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:51550 req_id = 8f620c83-16f6-43ab-ad5f-0d413ace36a2 response_code = 200 uri = /crucible/0/regions/15111146-33e0-47c2-814e-5e1ef661095e/snapshots/bdb257f9-ff1c-4eb6-85a9-a642ac5979b1
962024-07-15T22:26:35.247ZINFOcrucible-agent (datafile): path is "/data/regions/15111146-33e0-47c2-814e-5e1ef661095e"
972024-07-15T22:26:35.255ZINFOcrucible-agent (datafile): dataset is oxp_6c05cd98-a794-48e4-b841-27ad4302be3f/crucible/regions/15111146-33e0-47c2-814e-5e1ef661095e
982024-07-15T22:26:35.263ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_6c05cd98-a794-48e4-b841-27ad4302be3f/crucible/regions/15111146-33e0-47c2-814e-5e1ef661095e@bdb257f9-ff1c-4eb6-85a9-a642ac5979b1
992024-07-15T22:26:35.263ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_6c05cd98-a794-48e4-b841-27ad4302be3f/crucible/regions/15111146-33e0-47c2-814e-5e1ef661095e", "bdb257f9-ff1c-4eb6-85a9-a642ac5979b1"]
1002024-07-15T22:26:35.274ZINFOcrucible-agent (datafile): requesting running snapshot 15111146-33e0-47c2-814e-5e1ef661095e-bdb257f9-ff1c-4eb6-85a9-a642ac5979b1 state: Requested
1012024-07-15T22:26:35.274ZINFOcrucible-agent (worker): applying SMF actions for region 15111146-33e0-47c2-814e-5e1ef661095e running snapshot bdb257f9-ff1c-4eb6-85a9-a642ac5979b1 (state Requested)...
1022024-07-15T22:26:35.274ZINFOcrucible-agent (dropshot): request completed latency_us = 27279 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:51550 req_id = 431e9201-95de-40c5-ae1c-0d1b3cb04d1f response_code = 200 uri = /crucible/0/regions/15111146-33e0-47c2-814e-5e1ef661095e/snapshots/bdb257f9-ff1c-4eb6-85a9-a642ac5979b1/run
1032024-07-15T22:26:35.277ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-15111146-33e0-47c2-814e-5e1ef661095e-bdb257f9-ff1c-4eb6-85a9-a642ac5979b1
1042024-07-15T22:26:35.284ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-15111146-33e0-47c2-814e-5e1ef661095e-bdb257f9-ff1c-4eb6-85a9-a642ac5979b1
1052024-07-15T22:26:35.304ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-15111146-33e0-47c2-814e-5e1ef661095e-bdb257f9-ff1c-4eb6-85a9-a642ac5979b1
1062024-07-15T22:26:35.304ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/15111146-33e0-47c2-814e-5e1ef661095e/.zfs/snapshot/bdb257f9-ff1c-4eb6-85a9-a642ac5979b1
1072024-07-15T22:26:35.304ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
1082024-07-15T22:26:35.304ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
1092024-07-15T22:26:35.304ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
1102024-07-15T22:26:35.304ZINFOcrucible-agent (worker): commit
1112024-07-15T22:26:35.310ZINFOcrucible-agent (worker): ok!
1122024-07-15T22:26:35.317ZINFOcrucible-agent (worker): SMF ok!
1132024-07-15T22:26:35.317ZINFOcrucible-agent (datafile): region 15111146-33e0-47c2-814e-5e1ef661095e running snapshot bdb257f9-ff1c-4eb6-85a9-a642ac5979b1 state: Requested -> Created