Max level shown:
1 [ Nov 21 19:33:04 Enabled. ]
2 [ Nov 21 19:33:04 Rereading configuration. ]
3 [ Nov 21 19:33:06 Rereading configuration. ]
4 [ Nov 21 19:33:10 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_695b40ec-ab62-486b-99a5-161c372e3934/crucible -l [fd00:1122:3344:101::19]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62024-11-21T19:33:10.716ZINFOcrucible-agent: dataset: "oxp_695b40ec-ab62-486b-99a5-161c372e3934/crucible"
72024-11-21T19:33:10.718ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::19]:32345
82024-11-21T19:33:10.718ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92024-11-21T19:33:10.736ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102024-11-21T19:33:10.801ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::19]:32345
112024-11-21T19:34:19.926ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::19]:32345 remote_addr = [fd00:1122:3344:101::a]:34635
122024-11-21T19:34:19.962ZINFOcrucible-agent (datafile): region b618511d-7fe5-4df6-8c94-9f261c0cf414 state: Requested
132024-11-21T19:34:19.967ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
142024-11-21T19:34:19.972ZINFOcrucible-agent (dropshot): request completed latency_us = 15915 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:34635 req_id = 237fc298-9d7d-4827-a8d3-ea3b228df02c response_code = 200 uri = /crucible/0/regions
152024-11-21T19:34:19.981ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_695b40ec-ab62-486b-99a5-161c372e3934/crucible/regions/b618511d-7fe5-4df6-8c94-9f261c0cf414
162024-11-21T19:34:19.981ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_695b40ec-ab62-486b-99a5-161c372e3934/crucible/regions/b618511d-7fe5-4df6-8c94-9f261c0cf414
172024-11-21T19:34:20.047ZINFOcrucible-agent (worker): creating region Region { id: RegionId("b618511d-7fe5-4df6-8c94-9f261c0cf414"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/b618511d-7fe5-4df6-8c94-9f261c0cf414" region = b618511d-7fe5-4df6-8c94-9f261c0cf414
182024-11-21T19:34:20.049ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "b618511d-7fe5-4df6-8c94-9f261c0cf414" "--data" "/data/regions/b618511d-7fe5-4df6-8c94-9f261c0cf414" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = b618511d-7fe5-4df6-8c94-9f261c0cf414
192024-11-21T19:34:20.115ZINFOcrucible-agent (dropshot): request completed latency_us = 70 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:34635 req_id = 1d55bb9b-025e-4a36-9fe6-165b335a9ff0 response_code = 200 uri = /crucible/0/regions
202024-11-21T19:34:20.347ZINFOcrucible-agent (worker): region files created ok region = b618511d-7fe5-4df6-8c94-9f261c0cf414
212024-11-21T19:34:20.347ZINFOcrucible-agent (datafile): region b618511d-7fe5-4df6-8c94-9f261c0cf414 state: Requested -> Created
222024-11-21T19:34:20.347ZINFOcrucible-agent (worker): applying SMF actions post create...
232024-11-21T19:34:20.347ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-b618511d-7fe5-4df6-8c94-9f261c0cf414
242024-11-21T19:34:20.358ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-b618511d-7fe5-4df6-8c94-9f261c0cf414
252024-11-21T19:34:20.377ZINFOcrucible-agent (worker): creating config property group
262024-11-21T19:34:20.388ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-b618511d-7fe5-4df6-8c94-9f261c0cf414
272024-11-21T19:34:20.389ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/b618511d-7fe5-4df6-8c94-9f261c0cf414
282024-11-21T19:34:20.389ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
292024-11-21T19:34:20.389ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
302024-11-21T19:34:20.389ZINFOcrucible-agent (worker): commit
312024-11-21T19:34:20.395ZINFOcrucible-agent (dropshot): request completed latency_us = 82 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:34635 req_id = eb22b472-26ec-47c1-ae07-44c0d6283c49 response_code = 200 uri = /crucible/0/regions
322024-11-21T19:34:20.398ZINFOcrucible-agent (worker): ok!
332024-11-21T19:34:20.412ZINFOcrucible-agent (worker): SMF ok!
342024-11-21T19:44:50.283ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::19]:32345 remote_addr = [fd00:1122:3344:101::a]:53020
352024-11-21T19:44:50.283ZINFOcrucible-agent (datafile): region f2bb78f3-0d93-443c-a7da-f965710be4c2 state: Requested
362024-11-21T19:44:50.284ZINFOcrucible-agent (dropshot): request completed latency_us = 781 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:53020 req_id = fb058669-3363-47e1-8b3c-31e72e429f98 response_code = 200 uri = /crucible/0/regions
372024-11-21T19:44:50.284ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
382024-11-21T19:44:50.295ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_695b40ec-ab62-486b-99a5-161c372e3934/crucible/regions/f2bb78f3-0d93-443c-a7da-f965710be4c2
392024-11-21T19:44:50.295ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_695b40ec-ab62-486b-99a5-161c372e3934/crucible/regions/f2bb78f3-0d93-443c-a7da-f965710be4c2
402024-11-21T19:44:50.350ZINFOcrucible-agent (worker): creating region Region { id: RegionId("f2bb78f3-0d93-443c-a7da-f965710be4c2"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/f2bb78f3-0d93-443c-a7da-f965710be4c2" region = f2bb78f3-0d93-443c-a7da-f965710be4c2
412024-11-21T19:44:50.350ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "f2bb78f3-0d93-443c-a7da-f965710be4c2" "--data" "/data/regions/f2bb78f3-0d93-443c-a7da-f965710be4c2" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = f2bb78f3-0d93-443c-a7da-f965710be4c2
422024-11-21T19:44:50.469ZINFOcrucible-agent (worker): region files created ok region = f2bb78f3-0d93-443c-a7da-f965710be4c2
432024-11-21T19:44:50.469ZINFOcrucible-agent (datafile): region f2bb78f3-0d93-443c-a7da-f965710be4c2 state: Requested -> Created
442024-11-21T19:44:50.469ZINFOcrucible-agent (worker): applying SMF actions post create...
452024-11-21T19:44:50.479ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-f2bb78f3-0d93-443c-a7da-f965710be4c2
462024-11-21T19:44:50.485ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-f2bb78f3-0d93-443c-a7da-f965710be4c2
472024-11-21T19:44:50.502ZINFOcrucible-agent (worker): creating config property group
482024-11-21T19:44:50.508ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-f2bb78f3-0d93-443c-a7da-f965710be4c2
492024-11-21T19:44:50.509ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/f2bb78f3-0d93-443c-a7da-f965710be4c2
502024-11-21T19:44:50.509ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
512024-11-21T19:44:50.509ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
522024-11-21T19:44:50.509ZINFOcrucible-agent (worker): commit
532024-11-21T19:44:50.518ZINFOcrucible-agent (worker): ok!
542024-11-21T19:44:50.531ZINFOcrucible-agent (worker): SMF ok!
552024-11-21T19:44:50.539ZINFOcrucible-agent (dropshot): request completed latency_us = 71 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:53020 req_id = 03ecb4cc-255b-4214-8e4f-39d30791eafa response_code = 200 uri = /crucible/0/regions
562024-11-21T19:44:52.448ZINFOcrucible-agent (dropshot): request completed latency_us = 2360 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:53020 req_id = df8dd52f-ab87-427b-ac33-e5b1f3809bc5 response_code = 200 uri = /crucible/0/regions/b618511d-7fe5-4df6-8c94-9f261c0cf414
572024-11-21T19:44:52.452ZINFOcrucible-agent (datafile): path is "/data/regions/b618511d-7fe5-4df6-8c94-9f261c0cf414"
582024-11-21T19:44:52.461ZINFOcrucible-agent (datafile): dataset is oxp_695b40ec-ab62-486b-99a5-161c372e3934/crucible/regions/b618511d-7fe5-4df6-8c94-9f261c0cf414
592024-11-21T19:44:52.470ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_695b40ec-ab62-486b-99a5-161c372e3934/crucible/regions/b618511d-7fe5-4df6-8c94-9f261c0cf414@728cd9b1-e3ec-4e7d-b786-e0e10f0569d2
602024-11-21T19:44:52.470ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_695b40ec-ab62-486b-99a5-161c372e3934/crucible/regions/b618511d-7fe5-4df6-8c94-9f261c0cf414", "728cd9b1-e3ec-4e7d-b786-e0e10f0569d2"]
612024-11-21T19:44:52.481ZINFOcrucible-agent (dropshot): request completed latency_us = 29380 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:53020 req_id = 8e17fc9d-be17-4a86-8f9a-aef2ff38209b response_code = 200 uri = /crucible/0/regions/b618511d-7fe5-4df6-8c94-9f261c0cf414/snapshots/728cd9b1-e3ec-4e7d-b786-e0e10f0569d2
622024-11-21T19:44:52.485ZINFOcrucible-agent (datafile): path is "/data/regions/b618511d-7fe5-4df6-8c94-9f261c0cf414"
632024-11-21T19:44:52.493ZINFOcrucible-agent (datafile): dataset is oxp_695b40ec-ab62-486b-99a5-161c372e3934/crucible/regions/b618511d-7fe5-4df6-8c94-9f261c0cf414
642024-11-21T19:44:52.501ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_695b40ec-ab62-486b-99a5-161c372e3934/crucible/regions/b618511d-7fe5-4df6-8c94-9f261c0cf414@728cd9b1-e3ec-4e7d-b786-e0e10f0569d2
652024-11-21T19:44:52.501ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_695b40ec-ab62-486b-99a5-161c372e3934/crucible/regions/b618511d-7fe5-4df6-8c94-9f261c0cf414", "728cd9b1-e3ec-4e7d-b786-e0e10f0569d2"]
662024-11-21T19:44:52.511ZINFOcrucible-agent (datafile): requesting running snapshot b618511d-7fe5-4df6-8c94-9f261c0cf414-728cd9b1-e3ec-4e7d-b786-e0e10f0569d2 state: Requested
672024-11-21T19:44:52.511ZINFOcrucible-agent (worker): applying SMF actions for region b618511d-7fe5-4df6-8c94-9f261c0cf414 running snapshot 728cd9b1-e3ec-4e7d-b786-e0e10f0569d2 (state Requested)...
682024-11-21T19:44:52.511ZINFOcrucible-agent (dropshot): request completed latency_us = 26073 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:53020 req_id = d9be5ead-66cf-4c62-9d0e-169739628261 response_code = 200 uri = /crucible/0/regions/b618511d-7fe5-4df6-8c94-9f261c0cf414/snapshots/728cd9b1-e3ec-4e7d-b786-e0e10f0569d2/run
692024-11-21T19:44:52.514ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-b618511d-7fe5-4df6-8c94-9f261c0cf414-728cd9b1-e3ec-4e7d-b786-e0e10f0569d2
702024-11-21T19:44:52.525ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-b618511d-7fe5-4df6-8c94-9f261c0cf414-728cd9b1-e3ec-4e7d-b786-e0e10f0569d2
712024-11-21T19:44:52.546ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-b618511d-7fe5-4df6-8c94-9f261c0cf414-728cd9b1-e3ec-4e7d-b786-e0e10f0569d2
722024-11-21T19:44:52.546ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/b618511d-7fe5-4df6-8c94-9f261c0cf414/.zfs/snapshot/728cd9b1-e3ec-4e7d-b786-e0e10f0569d2
732024-11-21T19:44:52.547ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
742024-11-21T19:44:52.547ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
752024-11-21T19:44:52.547ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
762024-11-21T19:44:52.547ZINFOcrucible-agent (worker): commit
772024-11-21T19:44:52.552ZINFOcrucible-agent (worker): ok!
782024-11-21T19:44:52.560ZINFOcrucible-agent (worker): SMF ok!
792024-11-21T19:44:52.560ZINFOcrucible-agent (datafile): region b618511d-7fe5-4df6-8c94-9f261c0cf414 running snapshot 728cd9b1-e3ec-4e7d-b786-e0e10f0569d2 state: Requested -> Created
802024-11-21T19:44:52.649ZINFOcrucible-agent (datafile): path is "/data/regions/b618511d-7fe5-4df6-8c94-9f261c0cf414"
812024-11-21T19:44:52.658ZINFOcrucible-agent (datafile): dataset is oxp_695b40ec-ab62-486b-99a5-161c372e3934/crucible/regions/b618511d-7fe5-4df6-8c94-9f261c0cf414
822024-11-21T19:44:52.667ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_695b40ec-ab62-486b-99a5-161c372e3934/crucible/regions/b618511d-7fe5-4df6-8c94-9f261c0cf414@728cd9b1-e3ec-4e7d-b786-e0e10f0569d2
832024-11-21T19:44:52.667ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_695b40ec-ab62-486b-99a5-161c372e3934/crucible/regions/b618511d-7fe5-4df6-8c94-9f261c0cf414", "728cd9b1-e3ec-4e7d-b786-e0e10f0569d2"]
842024-11-21T19:44:52.675ZINFOcrucible-agent (dropshot): request completed latency_us = 26627 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:53020 req_id = 56139982-4d07-4e2e-aad1-35147dd23d5e response_code = 200 uri = /crucible/0/regions/b618511d-7fe5-4df6-8c94-9f261c0cf414/snapshots/728cd9b1-e3ec-4e7d-b786-e0e10f0569d2/run
852024-11-21T19:44:56.442ZINFOcrucible-agent (datafile): region 2999f0f4-62f2-4adf-b474-d963c68541d8 state: Requested
862024-11-21T19:44:56.442ZINFOcrucible-agent (dropshot): request completed latency_us = 271 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:53020 req_id = f46d22bd-139b-4ab9-99f9-701e57d8f0f6 response_code = 200 uri = /crucible/0/regions
872024-11-21T19:44:56.442ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
882024-11-21T19:44:56.452ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_695b40ec-ab62-486b-99a5-161c372e3934/crucible/regions/2999f0f4-62f2-4adf-b474-d963c68541d8
892024-11-21T19:44:56.452ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_695b40ec-ab62-486b-99a5-161c372e3934/crucible/regions/2999f0f4-62f2-4adf-b474-d963c68541d8
902024-11-21T19:44:56.514ZINFOcrucible-agent (worker): creating region Region { id: RegionId("2999f0f4-62f2-4adf-b474-d963c68541d8"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19003, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/2999f0f4-62f2-4adf-b474-d963c68541d8" region = 2999f0f4-62f2-4adf-b474-d963c68541d8
912024-11-21T19:44:56.517ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "2999f0f4-62f2-4adf-b474-d963c68541d8" "--data" "/data/regions/2999f0f4-62f2-4adf-b474-d963c68541d8" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 2999f0f4-62f2-4adf-b474-d963c68541d8
922024-11-21T19:44:56.590ZINFOcrucible-agent (worker): region files created ok region = 2999f0f4-62f2-4adf-b474-d963c68541d8
932024-11-21T19:44:56.590ZINFOcrucible-agent (datafile): region 2999f0f4-62f2-4adf-b474-d963c68541d8 state: Requested -> Created
942024-11-21T19:44:56.590ZINFOcrucible-agent (worker): applying SMF actions post create...
952024-11-21T19:44:56.591ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-2999f0f4-62f2-4adf-b474-d963c68541d8
962024-11-21T19:44:56.594ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-2999f0f4-62f2-4adf-b474-d963c68541d8
972024-11-21T19:44:56.604ZINFOcrucible-agent (worker): creating config property group
982024-11-21T19:44:56.608ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-2999f0f4-62f2-4adf-b474-d963c68541d8
992024-11-21T19:44:56.609ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/2999f0f4-62f2-4adf-b474-d963c68541d8
1002024-11-21T19:44:56.609ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19003
1012024-11-21T19:44:56.609ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
1022024-11-21T19:44:56.609ZINFOcrucible-agent (worker): commit
1032024-11-21T19:44:56.612ZINFOcrucible-agent (worker): ok!
1042024-11-21T19:44:56.625ZINFOcrucible-agent (worker): SMF ok!
1052024-11-21T19:44:56.631ZINFOcrucible-agent (dropshot): request completed latency_us = 129 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:53020 req_id = 04c771d1-fc6c-4c9e-9649-6c5464e994e1 response_code = 200 uri = /crucible/0/regions