Max level shown:
1 [ Jul 3 17:22:24 Enabled. ]
2 [ Jul 3 17:22:24 Rereading configuration. ]
3 [ Jul 3 17:22:26 Rereading configuration. ]
4 [ Jul 3 17:22:31 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_da517756-cb4c-4c7d-801d-76be4f2c6b99/crucible -l [fd00:1122:3344:101::17]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62024-07-03T17:22:31.485ZINFOcrucible-agent: dataset: "oxp_da517756-cb4c-4c7d-801d-76be4f2c6b99/crucible"
72024-07-03T17:22:31.490ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::17]:32345
82024-07-03T17:22:31.490ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92024-07-03T17:22:31.505ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102024-07-03T17:22:31.707ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::17]:32345
112024-07-03T17:23:43.418ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::17]:32345 remote_addr = [fd00:1122:3344:101::a]:63074
122024-07-03T17:23:43.427ZINFOcrucible-agent (datafile): region dad1bc4e-caa2-435f-924d-78418179ab56 state: Requested
132024-07-03T17:23:43.433ZINFOcrucible-agent (dropshot): request completed latency_us = 12836 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:63074 req_id = ae5d93c6-c914-48b7-bb31-2e8b324efcf6 response_code = 200 uri = /crucible/0/regions
142024-07-03T17:23:43.436ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
152024-07-03T17:23:43.444ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_da517756-cb4c-4c7d-801d-76be4f2c6b99/crucible/regions/dad1bc4e-caa2-435f-924d-78418179ab56
162024-07-03T17:23:43.444ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_da517756-cb4c-4c7d-801d-76be4f2c6b99/crucible/regions/dad1bc4e-caa2-435f-924d-78418179ab56
172024-07-03T17:23:43.504ZINFOcrucible-agent (worker): creating region Region { id: RegionId("dad1bc4e-caa2-435f-924d-78418179ab56"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/dad1bc4e-caa2-435f-924d-78418179ab56" region = dad1bc4e-caa2-435f-924d-78418179ab56
182024-07-03T17:23:43.507ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "dad1bc4e-caa2-435f-924d-78418179ab56" "--data" "/data/regions/dad1bc4e-caa2-435f-924d-78418179ab56" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = dad1bc4e-caa2-435f-924d-78418179ab56
192024-07-03T17:23:43.715ZINFOcrucible-agent (dropshot): request completed latency_us = 78 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:63074 req_id = 3c370683-fae2-4c7a-a383-7256c9f83392 response_code = 200 uri = /crucible/0/regions
202024-07-03T17:23:43.811ZINFOcrucible-agent (worker): region files created ok region = dad1bc4e-caa2-435f-924d-78418179ab56
212024-07-03T17:23:43.811ZINFOcrucible-agent (datafile): region dad1bc4e-caa2-435f-924d-78418179ab56 state: Requested -> Created
222024-07-03T17:23:43.811ZINFOcrucible-agent (worker): applying SMF actions post create...
232024-07-03T17:23:43.812ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-dad1bc4e-caa2-435f-924d-78418179ab56
242024-07-03T17:23:43.820ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-dad1bc4e-caa2-435f-924d-78418179ab56
252024-07-03T17:23:43.836ZINFOcrucible-agent (worker): creating config property group
262024-07-03T17:23:43.842ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-dad1bc4e-caa2-435f-924d-78418179ab56
272024-07-03T17:23:43.843ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/dad1bc4e-caa2-435f-924d-78418179ab56
282024-07-03T17:23:43.843ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
292024-07-03T17:23:43.843ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
302024-07-03T17:23:43.843ZINFOcrucible-agent (worker): commit
312024-07-03T17:23:43.851ZINFOcrucible-agent (worker): ok!
322024-07-03T17:23:43.863ZINFOcrucible-agent (worker): SMF ok!
332024-07-03T17:23:44.264ZINFOcrucible-agent (dropshot): request completed latency_us = 90 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:63074 req_id = 65883376-6968-4a0a-8679-49cb533168f2 response_code = 200 uri = /crucible/0/regions
342024-07-03T17:29:07.327ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::17]:32345 remote_addr = [fd00:1122:3344:101::a]:49327
352024-07-03T17:29:07.327ZINFOcrucible-agent (datafile): region 0ac17eed-4e75-4332-b410-6ca9052c0c55 state: Requested
362024-07-03T17:29:07.328ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
372024-07-03T17:29:07.328ZINFOcrucible-agent (dropshot): request completed latency_us = 834 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:49327 req_id = 8ec11f60-eaba-44c1-b359-1bc3c81d6e0b response_code = 200 uri = /crucible/0/regions
382024-07-03T17:29:07.339ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_da517756-cb4c-4c7d-801d-76be4f2c6b99/crucible/regions/0ac17eed-4e75-4332-b410-6ca9052c0c55
392024-07-03T17:29:07.339ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_da517756-cb4c-4c7d-801d-76be4f2c6b99/crucible/regions/0ac17eed-4e75-4332-b410-6ca9052c0c55
402024-07-03T17:29:07.399ZINFOcrucible-agent (worker): creating region Region { id: RegionId("0ac17eed-4e75-4332-b410-6ca9052c0c55"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/0ac17eed-4e75-4332-b410-6ca9052c0c55" region = 0ac17eed-4e75-4332-b410-6ca9052c0c55
412024-07-03T17:29:07.399ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "0ac17eed-4e75-4332-b410-6ca9052c0c55" "--data" "/data/regions/0ac17eed-4e75-4332-b410-6ca9052c0c55" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 0ac17eed-4e75-4332-b410-6ca9052c0c55
422024-07-03T17:29:07.485ZINFOcrucible-agent (dropshot): request completed latency_us = 681 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:49327 req_id = af91afa1-91a6-4982-9c40-b5bc57549d6b response_code = 200 uri = /crucible/0/regions
432024-07-03T17:29:07.497ZINFOcrucible-agent (worker): region files created ok region = 0ac17eed-4e75-4332-b410-6ca9052c0c55
442024-07-03T17:29:07.497ZINFOcrucible-agent (datafile): region 0ac17eed-4e75-4332-b410-6ca9052c0c55 state: Requested -> Created
452024-07-03T17:29:07.497ZINFOcrucible-agent (worker): applying SMF actions post create...
462024-07-03T17:29:07.497ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-0ac17eed-4e75-4332-b410-6ca9052c0c55
472024-07-03T17:29:07.502ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-0ac17eed-4e75-4332-b410-6ca9052c0c55
482024-07-03T17:29:07.518ZINFOcrucible-agent (worker): creating config property group
492024-07-03T17:29:07.521ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-0ac17eed-4e75-4332-b410-6ca9052c0c55
502024-07-03T17:29:07.522ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/0ac17eed-4e75-4332-b410-6ca9052c0c55
512024-07-03T17:29:07.522ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
522024-07-03T17:29:07.522ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
532024-07-03T17:29:07.522ZINFOcrucible-agent (worker): commit
542024-07-03T17:29:07.525ZINFOcrucible-agent (worker): ok!
552024-07-03T17:29:07.534ZINFOcrucible-agent (worker): SMF ok!
562024-07-03T17:29:08.010ZINFOcrucible-agent (dropshot): request completed latency_us = 53 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:49327 req_id = 19e88d96-8847-42dc-bbf7-60bd057f6dbf response_code = 200 uri = /crucible/0/regions
572024-07-03T17:29:08.620ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::17]:32345 remote_addr = [fd00:1122:3344:101::a]:43469
582024-07-03T17:29:08.620ZINFOcrucible-agent (dropshot): request completed latency_us = 89 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:43469 req_id = 2d5bfa41-2cb3-4ede-aa32-a88c4ba341b6 response_code = 200 uri = /crucible/0/regions/dad1bc4e-caa2-435f-924d-78418179ab56
592024-07-03T17:29:08.625ZINFOcrucible-agent (datafile): path is "/data/regions/dad1bc4e-caa2-435f-924d-78418179ab56"
602024-07-03T17:29:08.639ZINFOcrucible-agent (datafile): dataset is oxp_da517756-cb4c-4c7d-801d-76be4f2c6b99/crucible/regions/dad1bc4e-caa2-435f-924d-78418179ab56
612024-07-03T17:29:08.651ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_da517756-cb4c-4c7d-801d-76be4f2c6b99/crucible/regions/dad1bc4e-caa2-435f-924d-78418179ab56@f21c53cc-7f60-4c98-85eb-88fe670d1630
622024-07-03T17:29:08.651ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_da517756-cb4c-4c7d-801d-76be4f2c6b99/crucible/regions/dad1bc4e-caa2-435f-924d-78418179ab56", "f21c53cc-7f60-4c98-85eb-88fe670d1630"]
632024-07-03T17:29:08.667ZINFOcrucible-agent (dropshot): request completed latency_us = 46472 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:43469 req_id = f613d527-28ef-41fd-8055-e67a84b1b7e7 response_code = 200 uri = /crucible/0/regions/dad1bc4e-caa2-435f-924d-78418179ab56/snapshots/f21c53cc-7f60-4c98-85eb-88fe670d1630
642024-07-03T17:29:08.667ZINFOcrucible-agent (datafile): path is "/data/regions/dad1bc4e-caa2-435f-924d-78418179ab56"
652024-07-03T17:29:08.680ZINFOcrucible-agent (datafile): dataset is oxp_da517756-cb4c-4c7d-801d-76be4f2c6b99/crucible/regions/dad1bc4e-caa2-435f-924d-78418179ab56
662024-07-03T17:29:08.692ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_da517756-cb4c-4c7d-801d-76be4f2c6b99/crucible/regions/dad1bc4e-caa2-435f-924d-78418179ab56@f21c53cc-7f60-4c98-85eb-88fe670d1630
672024-07-03T17:29:08.692ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_da517756-cb4c-4c7d-801d-76be4f2c6b99/crucible/regions/dad1bc4e-caa2-435f-924d-78418179ab56", "f21c53cc-7f60-4c98-85eb-88fe670d1630"]
682024-07-03T17:29:08.705ZINFOcrucible-agent (datafile): requesting running snapshot dad1bc4e-caa2-435f-924d-78418179ab56-f21c53cc-7f60-4c98-85eb-88fe670d1630 state: Requested
692024-07-03T17:29:08.705ZINFOcrucible-agent (worker): applying SMF actions for region dad1bc4e-caa2-435f-924d-78418179ab56 running snapshot f21c53cc-7f60-4c98-85eb-88fe670d1630 (state Requested)...
702024-07-03T17:29:08.705ZINFOcrucible-agent (dropshot): request completed latency_us = 37819 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:43469 req_id = 5f9ab526-ade9-4334-a067-609918ecbff9 response_code = 200 uri = /crucible/0/regions/dad1bc4e-caa2-435f-924d-78418179ab56/snapshots/f21c53cc-7f60-4c98-85eb-88fe670d1630/run
712024-07-03T17:29:08.709ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-dad1bc4e-caa2-435f-924d-78418179ab56-f21c53cc-7f60-4c98-85eb-88fe670d1630
722024-07-03T17:29:08.717ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-dad1bc4e-caa2-435f-924d-78418179ab56-f21c53cc-7f60-4c98-85eb-88fe670d1630
732024-07-03T17:29:08.730ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-dad1bc4e-caa2-435f-924d-78418179ab56-f21c53cc-7f60-4c98-85eb-88fe670d1630
742024-07-03T17:29:08.730ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/dad1bc4e-caa2-435f-924d-78418179ab56/.zfs/snapshot/f21c53cc-7f60-4c98-85eb-88fe670d1630
752024-07-03T17:29:08.730ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
762024-07-03T17:29:08.730ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
772024-07-03T17:29:08.730ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
782024-07-03T17:29:08.730ZINFOcrucible-agent (worker): commit
792024-07-03T17:29:08.735ZINFOcrucible-agent (worker): ok!
802024-07-03T17:29:08.739ZINFOcrucible-agent (worker): SMF ok!
812024-07-03T17:29:08.739ZINFOcrucible-agent (datafile): region dad1bc4e-caa2-435f-924d-78418179ab56 running snapshot f21c53cc-7f60-4c98-85eb-88fe670d1630 state: Requested -> Created
822024-07-03T17:29:11.565ZINFOcrucible-agent (datafile): region 7e565daa-43db-424c-b412-571c00cfa39f state: Requested
832024-07-03T17:29:11.565ZINFOcrucible-agent (dropshot): request completed latency_us = 208 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:49327 req_id = 7b26fed3-ed70-4aa4-bf73-2cfd41a80a0d response_code = 200 uri = /crucible/0/regions
842024-07-03T17:29:11.565ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
852024-07-03T17:29:11.574ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_da517756-cb4c-4c7d-801d-76be4f2c6b99/crucible/regions/7e565daa-43db-424c-b412-571c00cfa39f
862024-07-03T17:29:11.574ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_da517756-cb4c-4c7d-801d-76be4f2c6b99/crucible/regions/7e565daa-43db-424c-b412-571c00cfa39f
872024-07-03T17:29:11.623ZINFOcrucible-agent (worker): creating region Region { id: RegionId("7e565daa-43db-424c-b412-571c00cfa39f"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19003, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/7e565daa-43db-424c-b412-571c00cfa39f" region = 7e565daa-43db-424c-b412-571c00cfa39f
882024-07-03T17:29:11.625ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "7e565daa-43db-424c-b412-571c00cfa39f" "--data" "/data/regions/7e565daa-43db-424c-b412-571c00cfa39f" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 7e565daa-43db-424c-b412-571c00cfa39f
892024-07-03T17:29:11.692ZINFOcrucible-agent (worker): region files created ok region = 7e565daa-43db-424c-b412-571c00cfa39f
902024-07-03T17:29:11.692ZINFOcrucible-agent (datafile): region 7e565daa-43db-424c-b412-571c00cfa39f state: Requested -> Created
912024-07-03T17:29:11.692ZINFOcrucible-agent (worker): applying SMF actions post create...
922024-07-03T17:29:11.693ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-7e565daa-43db-424c-b412-571c00cfa39f
932024-07-03T17:29:11.697ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-7e565daa-43db-424c-b412-571c00cfa39f
942024-07-03T17:29:11.705ZINFOcrucible-agent (worker): creating config property group
952024-07-03T17:29:11.708ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-7e565daa-43db-424c-b412-571c00cfa39f
962024-07-03T17:29:11.708ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/7e565daa-43db-424c-b412-571c00cfa39f
972024-07-03T17:29:11.708ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19003
982024-07-03T17:29:11.708ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
992024-07-03T17:29:11.708ZINFOcrucible-agent (worker): commit
1002024-07-03T17:29:11.712ZINFOcrucible-agent (worker): ok!
1012024-07-03T17:29:11.723ZINFOcrucible-agent (worker): SMF ok!
1022024-07-03T17:29:11.890ZINFOcrucible-agent (dropshot): request completed latency_us = 590 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:49327 req_id = 2cc8c93d-e54a-459d-82c5-a31521c3724e response_code = 200 uri = /crucible/0/regions
1032024-07-03T17:29:36.670ZINFOcrucible-agent (dropshot): request completed latency_us = 54 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:49327 req_id = 65a10db8-0dd4-4117-9c00-3609037702be response_code = 200 uri = /crucible/0/regions/7e565daa-43db-424c-b412-571c00cfa39f
1042024-07-03T17:29:36.673ZINFOcrucible-agent (datafile): path is "/data/regions/7e565daa-43db-424c-b412-571c00cfa39f"
1052024-07-03T17:29:36.683ZINFOcrucible-agent (datafile): dataset is oxp_da517756-cb4c-4c7d-801d-76be4f2c6b99/crucible/regions/7e565daa-43db-424c-b412-571c00cfa39f
1062024-07-03T17:29:36.691ZINFOcrucible-agent (datafile): region 7e565daa-43db-424c-b412-571c00cfa39f state: Created -> Tombstoned
1072024-07-03T17:29:36.691ZINFOcrucible-agent (worker): applying SMF actions before removal...
1082024-07-03T17:29:36.691ZINFOcrucible-agent (dropshot): request completed latency_us = 18899 local_addr = [fd00:1122:3344:101::17]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:49327 req_id = 80d3d58a-1071-44da-9587-8c5b2837bc31 response_code = 204 uri = /crucible/0/regions/7e565daa-43db-424c-b412-571c00cfa39f
1092024-07-03T17:29:36.692ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-7e565daa-43db-424c-b412-571c00cfa39f (instance states: (Some(Online), None))
1102024-07-03T17:29:36.694ZINFOcrucible-agent (dropshot): request completed latency_us = 27 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:49327 req_id = 1f294bb9-a161-4cd8-abe9-813e5513fb16 response_code = 200 uri = /crucible/0/regions/7e565daa-43db-424c-b412-571c00cfa39f
1112024-07-03T17:29:36.699ZINFOcrucible-agent (worker): SMF ok!
1122024-07-03T17:29:36.709ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_da517756-cb4c-4c7d-801d-76be4f2c6b99/crucible/regions/7e565daa-43db-424c-b412-571c00cfa39f" region = 7e565daa-43db-424c-b412-571c00cfa39f
1132024-07-03T17:29:36.753ZINFOcrucible-agent (datafile): region 7e565daa-43db-424c-b412-571c00cfa39f state: Tombstoned -> Destroyed
1142024-07-03T17:29:36.785ZINFOcrucible-agent (dropshot): request completed latency_us = 505 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:49327 req_id = 9858b5f0-850d-4e91-87da-1a0fd88a07a1 response_code = 200 uri = /crucible/0/regions/7e565daa-43db-424c-b412-571c00cfa39f