Max level shown:
1 [ Oct 1 18:23:32 Enabled. ]
2 [ Oct 1 18:23:32 Rereading configuration. ]
3 [ Oct 1 18:23:33 Rereading configuration. ]
4 [ Oct 1 18:23:35 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_4562b4bc-fd1e-41e9-b26e-5a5229354600/crucible -l [fd00:1122:3344:101::14]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62024-10-01T18:23:35.506ZINFOcrucible-agent: dataset: "oxp_4562b4bc-fd1e-41e9-b26e-5a5229354600/crucible"
72024-10-01T18:23:35.508ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::14]:32345
82024-10-01T18:23:35.508ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92024-10-01T18:23:35.529ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102024-10-01T18:23:35.609ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::14]:32345
112024-10-01T18:24:37.564ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::14]:32345 remote_addr = [fd00:1122:3344:101::a]:53311
122024-10-01T18:24:37.589ZINFOcrucible-agent (datafile): region bbcdf273-b537-4f3a-b5d8-89e63981dbed state: Requested
132024-10-01T18:24:37.591ZINFOcrucible-agent (dropshot): request completed latency_us = 6728 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:53311 req_id = 662b66ba-5412-4dc7-8b1e-19e080efa587 response_code = 200 uri = /crucible/0/regions
142024-10-01T18:24:37.592ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
152024-10-01T18:24:37.602ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_4562b4bc-fd1e-41e9-b26e-5a5229354600/crucible/regions/bbcdf273-b537-4f3a-b5d8-89e63981dbed
162024-10-01T18:24:37.602ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_4562b4bc-fd1e-41e9-b26e-5a5229354600/crucible/regions/bbcdf273-b537-4f3a-b5d8-89e63981dbed
172024-10-01T18:24:37.670ZINFOcrucible-agent (worker): creating region Region { id: RegionId("bbcdf273-b537-4f3a-b5d8-89e63981dbed"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/bbcdf273-b537-4f3a-b5d8-89e63981dbed" region = bbcdf273-b537-4f3a-b5d8-89e63981dbed
182024-10-01T18:24:37.675ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "bbcdf273-b537-4f3a-b5d8-89e63981dbed" "--data" "/data/regions/bbcdf273-b537-4f3a-b5d8-89e63981dbed" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = bbcdf273-b537-4f3a-b5d8-89e63981dbed
192024-10-01T18:24:37.831ZINFOcrucible-agent (worker): region files created ok region = bbcdf273-b537-4f3a-b5d8-89e63981dbed
202024-10-01T18:24:37.831ZINFOcrucible-agent (datafile): region bbcdf273-b537-4f3a-b5d8-89e63981dbed state: Requested -> Created
212024-10-01T18:24:37.831ZINFOcrucible-agent (worker): applying SMF actions post create...
222024-10-01T18:24:37.833ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-bbcdf273-b537-4f3a-b5d8-89e63981dbed
232024-10-01T18:24:37.838ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-bbcdf273-b537-4f3a-b5d8-89e63981dbed
242024-10-01T18:24:37.855ZINFOcrucible-agent (worker): creating config property group
252024-10-01T18:24:37.858ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-bbcdf273-b537-4f3a-b5d8-89e63981dbed
262024-10-01T18:24:37.859ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/bbcdf273-b537-4f3a-b5d8-89e63981dbed
272024-10-01T18:24:37.859ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
282024-10-01T18:24:37.859ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
292024-10-01T18:24:37.859ZINFOcrucible-agent (worker): commit
302024-10-01T18:24:37.863ZINFOcrucible-agent (worker): ok!
312024-10-01T18:24:37.868ZINFOcrucible-agent (worker): SMF ok!
322024-10-01T18:24:37.870ZINFOcrucible-agent (dropshot): request completed latency_us = 1127 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:53311 req_id = 2dfcd538-53ad-43f5-ac9f-2fe85b63a21a response_code = 200 uri = /crucible/0/regions
332024-10-01T18:29:45.110ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::14]:32345 remote_addr = [fd00:1122:3344:101::a]:61530
342024-10-01T18:29:45.110ZINFOcrucible-agent (datafile): region 000c3aea-859f-4227-8a39-2d4bbecac67e state: Requested
352024-10-01T18:29:45.111ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
362024-10-01T18:29:45.111ZINFOcrucible-agent (dropshot): request completed latency_us = 572 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:61530 req_id = 9ee2a9ad-e104-4b05-9222-2ef54365a37c response_code = 200 uri = /crucible/0/regions
372024-10-01T18:29:45.121ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_4562b4bc-fd1e-41e9-b26e-5a5229354600/crucible/regions/000c3aea-859f-4227-8a39-2d4bbecac67e
382024-10-01T18:29:45.121ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_4562b4bc-fd1e-41e9-b26e-5a5229354600/crucible/regions/000c3aea-859f-4227-8a39-2d4bbecac67e
392024-10-01T18:29:45.190ZINFOcrucible-agent (worker): creating region Region { id: RegionId("000c3aea-859f-4227-8a39-2d4bbecac67e"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/000c3aea-859f-4227-8a39-2d4bbecac67e" region = 000c3aea-859f-4227-8a39-2d4bbecac67e
402024-10-01T18:29:45.190ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "000c3aea-859f-4227-8a39-2d4bbecac67e" "--data" "/data/regions/000c3aea-859f-4227-8a39-2d4bbecac67e" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 000c3aea-859f-4227-8a39-2d4bbecac67e
412024-10-01T18:29:45.265ZINFOcrucible-agent (worker): region files created ok region = 000c3aea-859f-4227-8a39-2d4bbecac67e
422024-10-01T18:29:45.265ZINFOcrucible-agent (datafile): region 000c3aea-859f-4227-8a39-2d4bbecac67e state: Requested -> Created
432024-10-01T18:29:45.265ZINFOcrucible-agent (worker): applying SMF actions post create...
442024-10-01T18:29:45.265ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-000c3aea-859f-4227-8a39-2d4bbecac67e
452024-10-01T18:29:45.270ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-000c3aea-859f-4227-8a39-2d4bbecac67e
462024-10-01T18:29:45.283ZINFOcrucible-agent (worker): creating config property group
472024-10-01T18:29:45.287ZINFOcrucible-agent (dropshot): request completed latency_us = 64 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:61530 req_id = 4216f41d-3ea4-48b0-9742-cfd0239a260e response_code = 200 uri = /crucible/0/regions
482024-10-01T18:29:45.287ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-000c3aea-859f-4227-8a39-2d4bbecac67e
492024-10-01T18:29:45.287ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/000c3aea-859f-4227-8a39-2d4bbecac67e
502024-10-01T18:29:45.287ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
512024-10-01T18:29:45.287ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
522024-10-01T18:29:45.287ZINFOcrucible-agent (worker): commit
532024-10-01T18:29:45.291ZINFOcrucible-agent (worker): ok!
542024-10-01T18:29:45.298ZINFOcrucible-agent (worker): SMF ok!
552024-10-01T18:29:46.782ZINFOcrucible-agent (dropshot): request completed latency_us = 3091 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:61530 req_id = 53c53d51-4ed6-46cb-99da-883af65510ac response_code = 200 uri = /crucible/0/regions/bbcdf273-b537-4f3a-b5d8-89e63981dbed
562024-10-01T18:29:46.786ZINFOcrucible-agent (datafile): path is "/data/regions/bbcdf273-b537-4f3a-b5d8-89e63981dbed"
572024-10-01T18:29:46.795ZINFOcrucible-agent (datafile): dataset is oxp_4562b4bc-fd1e-41e9-b26e-5a5229354600/crucible/regions/bbcdf273-b537-4f3a-b5d8-89e63981dbed
582024-10-01T18:29:46.804ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_4562b4bc-fd1e-41e9-b26e-5a5229354600/crucible/regions/bbcdf273-b537-4f3a-b5d8-89e63981dbed@c69b7335-b7d0-4dc0-94a1-2792b4d75d33
592024-10-01T18:29:46.804ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_4562b4bc-fd1e-41e9-b26e-5a5229354600/crucible/regions/bbcdf273-b537-4f3a-b5d8-89e63981dbed", "c69b7335-b7d0-4dc0-94a1-2792b4d75d33"]
602024-10-01T18:29:46.817ZINFOcrucible-agent (dropshot): request completed latency_us = 31090 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:61530 req_id = c56dc9b7-eb04-478f-8cd9-befb95629b82 response_code = 200 uri = /crucible/0/regions/bbcdf273-b537-4f3a-b5d8-89e63981dbed/snapshots/c69b7335-b7d0-4dc0-94a1-2792b4d75d33
612024-10-01T18:29:46.821ZINFOcrucible-agent (datafile): path is "/data/regions/bbcdf273-b537-4f3a-b5d8-89e63981dbed"
622024-10-01T18:29:46.829ZINFOcrucible-agent (datafile): dataset is oxp_4562b4bc-fd1e-41e9-b26e-5a5229354600/crucible/regions/bbcdf273-b537-4f3a-b5d8-89e63981dbed
632024-10-01T18:29:46.836ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_4562b4bc-fd1e-41e9-b26e-5a5229354600/crucible/regions/bbcdf273-b537-4f3a-b5d8-89e63981dbed@c69b7335-b7d0-4dc0-94a1-2792b4d75d33
642024-10-01T18:29:46.836ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_4562b4bc-fd1e-41e9-b26e-5a5229354600/crucible/regions/bbcdf273-b537-4f3a-b5d8-89e63981dbed", "c69b7335-b7d0-4dc0-94a1-2792b4d75d33"]
652024-10-01T18:29:46.844ZINFOcrucible-agent (datafile): requesting running snapshot bbcdf273-b537-4f3a-b5d8-89e63981dbed-c69b7335-b7d0-4dc0-94a1-2792b4d75d33 state: Requested
662024-10-01T18:29:46.844ZINFOcrucible-agent (worker): applying SMF actions for region bbcdf273-b537-4f3a-b5d8-89e63981dbed running snapshot c69b7335-b7d0-4dc0-94a1-2792b4d75d33 (state Requested)...
672024-10-01T18:29:46.844ZINFOcrucible-agent (dropshot): request completed latency_us = 24257 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:61530 req_id = c52ab0a7-ee91-4a1f-9cdd-da6b6d408fc4 response_code = 200 uri = /crucible/0/regions/bbcdf273-b537-4f3a-b5d8-89e63981dbed/snapshots/c69b7335-b7d0-4dc0-94a1-2792b4d75d33/run
682024-10-01T18:29:46.846ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-bbcdf273-b537-4f3a-b5d8-89e63981dbed-c69b7335-b7d0-4dc0-94a1-2792b4d75d33
692024-10-01T18:29:46.851ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-bbcdf273-b537-4f3a-b5d8-89e63981dbed-c69b7335-b7d0-4dc0-94a1-2792b4d75d33
702024-10-01T18:29:46.860ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-bbcdf273-b537-4f3a-b5d8-89e63981dbed-c69b7335-b7d0-4dc0-94a1-2792b4d75d33
712024-10-01T18:29:46.860ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/bbcdf273-b537-4f3a-b5d8-89e63981dbed/.zfs/snapshot/c69b7335-b7d0-4dc0-94a1-2792b4d75d33
722024-10-01T18:29:46.860ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
732024-10-01T18:29:46.860ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
742024-10-01T18:29:46.861ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
752024-10-01T18:29:46.861ZINFOcrucible-agent (worker): commit
762024-10-01T18:29:46.864ZINFOcrucible-agent (worker): ok!
772024-10-01T18:29:46.869ZINFOcrucible-agent (worker): SMF ok!
782024-10-01T18:29:46.869ZINFOcrucible-agent (datafile): region bbcdf273-b537-4f3a-b5d8-89e63981dbed running snapshot c69b7335-b7d0-4dc0-94a1-2792b4d75d33 state: Requested -> Created
792024-10-01T18:29:47.050ZINFOcrucible-agent (datafile): path is "/data/regions/bbcdf273-b537-4f3a-b5d8-89e63981dbed"
802024-10-01T18:29:47.058ZINFOcrucible-agent (datafile): dataset is oxp_4562b4bc-fd1e-41e9-b26e-5a5229354600/crucible/regions/bbcdf273-b537-4f3a-b5d8-89e63981dbed
812024-10-01T18:29:47.066ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_4562b4bc-fd1e-41e9-b26e-5a5229354600/crucible/regions/bbcdf273-b537-4f3a-b5d8-89e63981dbed@c69b7335-b7d0-4dc0-94a1-2792b4d75d33
822024-10-01T18:29:47.066ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_4562b4bc-fd1e-41e9-b26e-5a5229354600/crucible/regions/bbcdf273-b537-4f3a-b5d8-89e63981dbed", "c69b7335-b7d0-4dc0-94a1-2792b4d75d33"]
832024-10-01T18:29:47.075ZINFOcrucible-agent (dropshot): request completed latency_us = 24846 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:61530 req_id = c2b415b4-7f44-47f6-8dd8-4cf6c25b3174 response_code = 200 uri = /crucible/0/regions/bbcdf273-b537-4f3a-b5d8-89e63981dbed/snapshots/c69b7335-b7d0-4dc0-94a1-2792b4d75d33/run
842024-10-01T18:29:50.573ZINFOcrucible-agent (datafile): region 91ee6808-2ff3-46f7-81df-5223d3ab8541 state: Requested
852024-10-01T18:29:50.573ZINFOcrucible-agent (dropshot): request completed latency_us = 263 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:61530 req_id = 4629f050-1e9e-4935-811c-0b1f7cbdf170 response_code = 200 uri = /crucible/0/regions
862024-10-01T18:29:50.573ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
872024-10-01T18:29:50.583ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_4562b4bc-fd1e-41e9-b26e-5a5229354600/crucible/regions/91ee6808-2ff3-46f7-81df-5223d3ab8541
882024-10-01T18:29:50.583ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_4562b4bc-fd1e-41e9-b26e-5a5229354600/crucible/regions/91ee6808-2ff3-46f7-81df-5223d3ab8541
892024-10-01T18:29:50.646ZINFOcrucible-agent (worker): creating region Region { id: RegionId("91ee6808-2ff3-46f7-81df-5223d3ab8541"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19003, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/91ee6808-2ff3-46f7-81df-5223d3ab8541" region = 91ee6808-2ff3-46f7-81df-5223d3ab8541
902024-10-01T18:29:50.646ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "91ee6808-2ff3-46f7-81df-5223d3ab8541" "--data" "/data/regions/91ee6808-2ff3-46f7-81df-5223d3ab8541" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 91ee6808-2ff3-46f7-81df-5223d3ab8541
912024-10-01T18:29:50.715ZINFOcrucible-agent (worker): region files created ok region = 91ee6808-2ff3-46f7-81df-5223d3ab8541
922024-10-01T18:29:50.715ZINFOcrucible-agent (datafile): region 91ee6808-2ff3-46f7-81df-5223d3ab8541 state: Requested -> Created
932024-10-01T18:29:50.715ZINFOcrucible-agent (worker): applying SMF actions post create...
942024-10-01T18:29:50.716ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-91ee6808-2ff3-46f7-81df-5223d3ab8541
952024-10-01T18:29:50.719ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-91ee6808-2ff3-46f7-81df-5223d3ab8541
962024-10-01T18:29:50.726ZINFOcrucible-agent (worker): creating config property group
972024-10-01T18:29:50.729ZINFOcrucible-agent (dropshot): request completed latency_us = 64 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:61530 req_id = ed9c9980-1214-43eb-9e86-b838a32f3f8a response_code = 200 uri = /crucible/0/regions
982024-10-01T18:29:50.730ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-91ee6808-2ff3-46f7-81df-5223d3ab8541
992024-10-01T18:29:50.730ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/91ee6808-2ff3-46f7-81df-5223d3ab8541
1002024-10-01T18:29:50.730ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19003
1012024-10-01T18:29:50.730ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
1022024-10-01T18:29:50.730ZINFOcrucible-agent (worker): commit
1032024-10-01T18:29:50.734ZINFOcrucible-agent (worker): ok!
1042024-10-01T18:29:50.745ZINFOcrucible-agent (worker): SMF ok!