Max level shown:
1 [ Dec 14 01:03:05 Enabled. ]
2 [ Dec 14 01:03:05 Rereading configuration. ]
3 [ Dec 14 01:03:07 Rereading configuration. ]
4 [ Dec 14 01:03:13 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_452732a4-6182-4108-9b68-a3447fd2d0a0/crucible -l [fd00:1122:3344:101::19]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62024-12-14T01:03:13.252ZINFOcrucible-agent: dataset: "oxp_452732a4-6182-4108-9b68-a3447fd2d0a0/crucible"
72024-12-14T01:03:13.257ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::19]:32345
82024-12-14T01:03:13.257ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92024-12-14T01:03:13.270ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102024-12-14T01:03:13.340ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::19]:32345
112024-12-14T01:04:33.420ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::19]:32345 remote_addr = [fd00:1122:3344:101::a]:36052
122024-12-14T01:04:33.464ZINFOcrucible-agent (datafile): region d7e2f9f8-47fd-4f8e-8a66-19a382443a57 state: Requested
132024-12-14T01:04:33.470ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
142024-12-14T01:04:33.474ZINFOcrucible-agent (dropshot): request completed latency_us = 17079 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:36052 req_id = 4232c4ed-7a0c-49cf-812c-664462efa0c5 response_code = 200 uri = /crucible/0/regions
152024-12-14T01:04:33.483ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_452732a4-6182-4108-9b68-a3447fd2d0a0/crucible/regions/d7e2f9f8-47fd-4f8e-8a66-19a382443a57
162024-12-14T01:04:33.483ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_452732a4-6182-4108-9b68-a3447fd2d0a0/crucible/regions/d7e2f9f8-47fd-4f8e-8a66-19a382443a57
172024-12-14T01:04:33.533ZINFOcrucible-agent (worker): creating region Region { id: RegionId("d7e2f9f8-47fd-4f8e-8a66-19a382443a57"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/d7e2f9f8-47fd-4f8e-8a66-19a382443a57" region = d7e2f9f8-47fd-4f8e-8a66-19a382443a57
182024-12-14T01:04:33.535ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "d7e2f9f8-47fd-4f8e-8a66-19a382443a57" "--data" "/data/regions/d7e2f9f8-47fd-4f8e-8a66-19a382443a57" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = d7e2f9f8-47fd-4f8e-8a66-19a382443a57
192024-12-14T01:04:33.800ZINFOcrucible-agent (dropshot): request completed latency_us = 82 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:36052 req_id = f7355423-2f46-47c2-b1a0-aa06e02ba3f8 response_code = 200 uri = /crucible/0/regions
202024-12-14T01:04:33.826ZINFOcrucible-agent (worker): region files created ok region = d7e2f9f8-47fd-4f8e-8a66-19a382443a57
212024-12-14T01:04:33.826ZINFOcrucible-agent (datafile): region d7e2f9f8-47fd-4f8e-8a66-19a382443a57 state: Requested -> Created
222024-12-14T01:04:33.826ZINFOcrucible-agent (worker): applying SMF actions post create...
232024-12-14T01:04:33.826ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-d7e2f9f8-47fd-4f8e-8a66-19a382443a57
242024-12-14T01:04:33.832ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-d7e2f9f8-47fd-4f8e-8a66-19a382443a57
252024-12-14T01:04:33.843ZINFOcrucible-agent (worker): creating config property group
262024-12-14T01:04:33.845ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-d7e2f9f8-47fd-4f8e-8a66-19a382443a57
272024-12-14T01:04:33.846ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/d7e2f9f8-47fd-4f8e-8a66-19a382443a57
282024-12-14T01:04:33.847ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
292024-12-14T01:04:33.847ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
302024-12-14T01:04:33.847ZINFOcrucible-agent (worker): commit
312024-12-14T01:04:33.850ZINFOcrucible-agent (worker): ok!
322024-12-14T01:04:33.854ZINFOcrucible-agent (worker): SMF ok!
332024-12-14T01:04:34.167ZINFOcrucible-agent (dropshot): request completed latency_us = 75 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:36052 req_id = 5c9748d4-3482-49a3-9817-7bc119a45f4e response_code = 200 uri = /crucible/0/regions
342024-12-14T01:14:49.169ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::19]:32345 remote_addr = [fd00:1122:3344:101::a]:50686
352024-12-14T01:14:49.169ZINFOcrucible-agent (datafile): region 05b679a0-126f-427d-98cf-bb23b0fb0faf state: Requested
362024-12-14T01:14:49.170ZINFOcrucible-agent (dropshot): request completed latency_us = 584 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:50686 req_id = 7bb58602-f68c-473d-bb15-1e0f8d3da0a0 response_code = 200 uri = /crucible/0/regions
372024-12-14T01:14:49.170ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
382024-12-14T01:14:49.180ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_452732a4-6182-4108-9b68-a3447fd2d0a0/crucible/regions/05b679a0-126f-427d-98cf-bb23b0fb0faf
392024-12-14T01:14:49.180ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_452732a4-6182-4108-9b68-a3447fd2d0a0/crucible/regions/05b679a0-126f-427d-98cf-bb23b0fb0faf
402024-12-14T01:14:49.229ZINFOcrucible-agent (worker): creating region Region { id: RegionId("05b679a0-126f-427d-98cf-bb23b0fb0faf"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/05b679a0-126f-427d-98cf-bb23b0fb0faf" region = 05b679a0-126f-427d-98cf-bb23b0fb0faf
412024-12-14T01:14:49.229ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "05b679a0-126f-427d-98cf-bb23b0fb0faf" "--data" "/data/regions/05b679a0-126f-427d-98cf-bb23b0fb0faf" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 05b679a0-126f-427d-98cf-bb23b0fb0faf
422024-12-14T01:14:49.343ZINFOcrucible-agent (worker): region files created ok region = 05b679a0-126f-427d-98cf-bb23b0fb0faf
432024-12-14T01:14:49.343ZINFOcrucible-agent (datafile): region 05b679a0-126f-427d-98cf-bb23b0fb0faf state: Requested -> Created
442024-12-14T01:14:49.343ZINFOcrucible-agent (worker): applying SMF actions post create...
452024-12-14T01:14:49.344ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-05b679a0-126f-427d-98cf-bb23b0fb0faf
462024-12-14T01:14:49.349ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-05b679a0-126f-427d-98cf-bb23b0fb0faf
472024-12-14T01:14:49.362ZINFOcrucible-agent (worker): creating config property group
482024-12-14T01:14:49.364ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-05b679a0-126f-427d-98cf-bb23b0fb0faf
492024-12-14T01:14:49.365ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/05b679a0-126f-427d-98cf-bb23b0fb0faf
502024-12-14T01:14:49.365ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
512024-12-14T01:14:49.365ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
522024-12-14T01:14:49.365ZINFOcrucible-agent (worker): commit
532024-12-14T01:14:49.368ZINFOcrucible-agent (worker): ok!
542024-12-14T01:14:49.376ZINFOcrucible-agent (worker): SMF ok!
552024-12-14T01:14:49.452ZINFOcrucible-agent (dropshot): request completed latency_us = 71 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:50686 req_id = ba369262-ea33-48c5-acf7-0b28bf998cd5 response_code = 200 uri = /crucible/0/regions
562024-12-14T01:14:49.891ZINFOcrucible-agent (dropshot): request completed latency_us = 2389 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:50686 req_id = 9858e2b7-6c50-4b9c-b229-b8b0266e26ee response_code = 200 uri = /crucible/0/regions/d7e2f9f8-47fd-4f8e-8a66-19a382443a57
572024-12-14T01:14:49.899ZINFOcrucible-agent (datafile): path is "/data/regions/d7e2f9f8-47fd-4f8e-8a66-19a382443a57"
582024-12-14T01:14:49.908ZINFOcrucible-agent (datafile): dataset is oxp_452732a4-6182-4108-9b68-a3447fd2d0a0/crucible/regions/d7e2f9f8-47fd-4f8e-8a66-19a382443a57
592024-12-14T01:14:49.918ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_452732a4-6182-4108-9b68-a3447fd2d0a0/crucible/regions/d7e2f9f8-47fd-4f8e-8a66-19a382443a57@2dfb33fe-d532-48ad-b085-3f8cd422800d
602024-12-14T01:14:49.918ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_452732a4-6182-4108-9b68-a3447fd2d0a0/crucible/regions/d7e2f9f8-47fd-4f8e-8a66-19a382443a57", "2dfb33fe-d532-48ad-b085-3f8cd422800d"]
612024-12-14T01:14:49.929ZINFOcrucible-agent (dropshot): request completed latency_us = 30898 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:50686 req_id = e230e591-27db-400b-97ff-f73457c7501e response_code = 200 uri = /crucible/0/regions/d7e2f9f8-47fd-4f8e-8a66-19a382443a57/snapshots/2dfb33fe-d532-48ad-b085-3f8cd422800d
622024-12-14T01:14:49.936ZINFOcrucible-agent (datafile): path is "/data/regions/d7e2f9f8-47fd-4f8e-8a66-19a382443a57"
632024-12-14T01:14:49.945ZINFOcrucible-agent (datafile): dataset is oxp_452732a4-6182-4108-9b68-a3447fd2d0a0/crucible/regions/d7e2f9f8-47fd-4f8e-8a66-19a382443a57
642024-12-14T01:14:49.953ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_452732a4-6182-4108-9b68-a3447fd2d0a0/crucible/regions/d7e2f9f8-47fd-4f8e-8a66-19a382443a57@2dfb33fe-d532-48ad-b085-3f8cd422800d
652024-12-14T01:14:49.953ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_452732a4-6182-4108-9b68-a3447fd2d0a0/crucible/regions/d7e2f9f8-47fd-4f8e-8a66-19a382443a57", "2dfb33fe-d532-48ad-b085-3f8cd422800d"]
662024-12-14T01:14:49.962ZINFOcrucible-agent (datafile): requesting running snapshot d7e2f9f8-47fd-4f8e-8a66-19a382443a57-2dfb33fe-d532-48ad-b085-3f8cd422800d state: Requested
672024-12-14T01:14:49.962ZINFOcrucible-agent (worker): applying SMF actions for region d7e2f9f8-47fd-4f8e-8a66-19a382443a57 running snapshot 2dfb33fe-d532-48ad-b085-3f8cd422800d (state Requested)...
682024-12-14T01:14:49.962ZINFOcrucible-agent (dropshot): request completed latency_us = 27045 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:50686 req_id = 13f67d46-5123-4097-856d-4beffce067c6 response_code = 200 uri = /crucible/0/regions/d7e2f9f8-47fd-4f8e-8a66-19a382443a57/snapshots/2dfb33fe-d532-48ad-b085-3f8cd422800d/run
692024-12-14T01:14:49.965ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-d7e2f9f8-47fd-4f8e-8a66-19a382443a57-2dfb33fe-d532-48ad-b085-3f8cd422800d
702024-12-14T01:14:49.973ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-d7e2f9f8-47fd-4f8e-8a66-19a382443a57-2dfb33fe-d532-48ad-b085-3f8cd422800d
712024-12-14T01:14:49.981ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-d7e2f9f8-47fd-4f8e-8a66-19a382443a57-2dfb33fe-d532-48ad-b085-3f8cd422800d
722024-12-14T01:14:49.982ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/d7e2f9f8-47fd-4f8e-8a66-19a382443a57/.zfs/snapshot/2dfb33fe-d532-48ad-b085-3f8cd422800d
732024-12-14T01:14:49.982ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
742024-12-14T01:14:49.982ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
752024-12-14T01:14:49.982ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
762024-12-14T01:14:49.982ZINFOcrucible-agent (worker): commit
772024-12-14T01:14:49.985ZINFOcrucible-agent (worker): ok!
782024-12-14T01:14:49.989ZINFOcrucible-agent (worker): SMF ok!
792024-12-14T01:14:49.989ZINFOcrucible-agent (datafile): region d7e2f9f8-47fd-4f8e-8a66-19a382443a57 running snapshot 2dfb33fe-d532-48ad-b085-3f8cd422800d state: Requested -> Created
802024-12-14T01:14:50.298ZINFOcrucible-agent (datafile): path is "/data/regions/d7e2f9f8-47fd-4f8e-8a66-19a382443a57"
812024-12-14T01:14:50.306ZINFOcrucible-agent (datafile): dataset is oxp_452732a4-6182-4108-9b68-a3447fd2d0a0/crucible/regions/d7e2f9f8-47fd-4f8e-8a66-19a382443a57
822024-12-14T01:14:50.315ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_452732a4-6182-4108-9b68-a3447fd2d0a0/crucible/regions/d7e2f9f8-47fd-4f8e-8a66-19a382443a57@2dfb33fe-d532-48ad-b085-3f8cd422800d
832024-12-14T01:14:50.315ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_452732a4-6182-4108-9b68-a3447fd2d0a0/crucible/regions/d7e2f9f8-47fd-4f8e-8a66-19a382443a57", "2dfb33fe-d532-48ad-b085-3f8cd422800d"]
842024-12-14T01:14:50.323ZINFOcrucible-agent (dropshot): request completed latency_us = 24717 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:50686 req_id = 23ecc485-ae14-437f-8ab1-e2df4043bb58 response_code = 200 uri = /crucible/0/regions/d7e2f9f8-47fd-4f8e-8a66-19a382443a57/snapshots/2dfb33fe-d532-48ad-b085-3f8cd422800d/run
852024-12-14T01:14:54.199ZINFOcrucible-agent (datafile): region 6d9e3caf-eaa9-4ef0-acbe-4e90df20205b state: Requested
862024-12-14T01:14:54.199ZINFOcrucible-agent (dropshot): request completed latency_us = 244 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:50686 req_id = 4f525deb-a71f-4e5c-a281-248e6ff4310d response_code = 200 uri = /crucible/0/regions
872024-12-14T01:14:54.199ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
882024-12-14T01:14:54.213ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_452732a4-6182-4108-9b68-a3447fd2d0a0/crucible/regions/6d9e3caf-eaa9-4ef0-acbe-4e90df20205b
892024-12-14T01:14:54.213ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_452732a4-6182-4108-9b68-a3447fd2d0a0/crucible/regions/6d9e3caf-eaa9-4ef0-acbe-4e90df20205b
902024-12-14T01:14:54.288ZINFOcrucible-agent (worker): creating region Region { id: RegionId("6d9e3caf-eaa9-4ef0-acbe-4e90df20205b"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19003, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/6d9e3caf-eaa9-4ef0-acbe-4e90df20205b" region = 6d9e3caf-eaa9-4ef0-acbe-4e90df20205b
912024-12-14T01:14:54.290ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "6d9e3caf-eaa9-4ef0-acbe-4e90df20205b" "--data" "/data/regions/6d9e3caf-eaa9-4ef0-acbe-4e90df20205b" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 6d9e3caf-eaa9-4ef0-acbe-4e90df20205b
922024-12-14T01:14:54.358ZINFOcrucible-agent (worker): region files created ok region = 6d9e3caf-eaa9-4ef0-acbe-4e90df20205b
932024-12-14T01:14:54.358ZINFOcrucible-agent (datafile): region 6d9e3caf-eaa9-4ef0-acbe-4e90df20205b state: Requested -> Created
942024-12-14T01:14:54.359ZINFOcrucible-agent (worker): applying SMF actions post create...
952024-12-14T01:14:54.360ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-6d9e3caf-eaa9-4ef0-acbe-4e90df20205b
962024-12-14T01:14:54.363ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-6d9e3caf-eaa9-4ef0-acbe-4e90df20205b
972024-12-14T01:14:54.381ZINFOcrucible-agent (worker): creating config property group
982024-12-14T01:14:54.384ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-6d9e3caf-eaa9-4ef0-acbe-4e90df20205b
992024-12-14T01:14:54.385ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/6d9e3caf-eaa9-4ef0-acbe-4e90df20205b
1002024-12-14T01:14:54.385ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19003
1012024-12-14T01:14:54.385ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
1022024-12-14T01:14:54.385ZINFOcrucible-agent (worker): commit
1032024-12-14T01:14:54.391ZINFOcrucible-agent (worker): ok!
1042024-12-14T01:14:54.403ZINFOcrucible-agent (worker): SMF ok!
1052024-12-14T01:14:54.523ZINFOcrucible-agent (dropshot): request completed latency_us = 108 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:50686 req_id = 0eb82e00-c216-446b-9914-f5f304c847d7 response_code = 200 uri = /crucible/0/regions