Max level shown:
1 [ Jun 27 21:49:54 Enabled. ]
2 [ Jun 27 21:49:54 Rereading configuration. ]
3 [ Jun 27 21:49:57 Rereading configuration. ]
4 [ Jun 27 21:50:03 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_f8b5491e-78ae-4725-a4e0-5d2f23407448/crucible -l [fd00:1122:3344:101::1a]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62024-06-27T21:50:03.873ZINFOcrucible-agent: dataset: "oxp_f8b5491e-78ae-4725-a4e0-5d2f23407448/crucible"
72024-06-27T21:50:03.876ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::1a]:32345
82024-06-27T21:50:03.876ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92024-06-27T21:50:03.890ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102024-06-27T21:50:04.053ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::1a]:32345
112024-06-27T21:51:07.707ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::1a]:32345 remote_addr = [fd00:1122:3344:101::a]:49139
122024-06-27T21:51:07.729ZINFOcrucible-agent (datafile): region 2ab3593d-c760-43b8-a461-f585a7b2dfed state: Requested
132024-06-27T21:51:07.737ZINFOcrucible-agent (dropshot): request completed latency_us = 18114 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:49139 req_id = c55f5d5a-cdc7-4189-bf20-0bf40ce356c3 response_code = 200 uri = /crucible/0/regions
142024-06-27T21:51:07.739ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
152024-06-27T21:51:07.750ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_f8b5491e-78ae-4725-a4e0-5d2f23407448/crucible/regions/2ab3593d-c760-43b8-a461-f585a7b2dfed
162024-06-27T21:51:07.750ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_f8b5491e-78ae-4725-a4e0-5d2f23407448/crucible/regions/2ab3593d-c760-43b8-a461-f585a7b2dfed
172024-06-27T21:51:07.803ZINFOcrucible-agent (worker): creating region Region { id: RegionId("2ab3593d-c760-43b8-a461-f585a7b2dfed"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/2ab3593d-c760-43b8-a461-f585a7b2dfed" region = 2ab3593d-c760-43b8-a461-f585a7b2dfed
182024-06-27T21:51:07.806ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "2ab3593d-c760-43b8-a461-f585a7b2dfed" "--data" "/data/regions/2ab3593d-c760-43b8-a461-f585a7b2dfed" "--block-size" "512" "--extent-size" "131072" "--extent-count" "16" "--encrypted" region = 2ab3593d-c760-43b8-a461-f585a7b2dfed
192024-06-27T21:51:07.999ZINFOcrucible-agent (dropshot): request completed latency_us = 87 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:49139 req_id = 89423b8b-699d-4797-b7f8-fc836ff70a4a response_code = 200 uri = /crucible/0/regions
202024-06-27T21:51:08.042ZINFOcrucible-agent (worker): region files created ok region = 2ab3593d-c760-43b8-a461-f585a7b2dfed
212024-06-27T21:51:08.042ZINFOcrucible-agent (datafile): region 2ab3593d-c760-43b8-a461-f585a7b2dfed state: Requested -> Created
222024-06-27T21:51:08.042ZINFOcrucible-agent (worker): applying SMF actions post create...
232024-06-27T21:51:08.043ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-2ab3593d-c760-43b8-a461-f585a7b2dfed
242024-06-27T21:51:08.050ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-2ab3593d-c760-43b8-a461-f585a7b2dfed
252024-06-27T21:51:08.069ZINFOcrucible-agent (worker): creating config property group
262024-06-27T21:51:08.073ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-2ab3593d-c760-43b8-a461-f585a7b2dfed
272024-06-27T21:51:08.074ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/2ab3593d-c760-43b8-a461-f585a7b2dfed
282024-06-27T21:51:08.076ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
292024-06-27T21:51:08.076ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
302024-06-27T21:51:08.076ZINFOcrucible-agent (worker): commit
312024-06-27T21:51:08.086ZINFOcrucible-agent (worker): ok!
322024-06-27T21:51:08.093ZINFOcrucible-agent (worker): SMF ok!
332024-06-27T21:51:08.499ZINFOcrucible-agent (dropshot): request completed latency_us = 81 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:49139 req_id = 514ff51a-48d1-4a89-bc91-af1bc3e84307 response_code = 200 uri = /crucible/0/regions
342024-06-27T21:51:08.787ZINFOcrucible-agent (dropshot): request completed latency_us = 72 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:49139 req_id = 3a211638-6008-459f-a198-73edd9df1b5e response_code = 200 uri = /crucible/0/regions/2ab3593d-c760-43b8-a461-f585a7b2dfed
352024-06-27T21:51:08.789ZINFOcrucible-agent (datafile): path is "/data/regions/2ab3593d-c760-43b8-a461-f585a7b2dfed"
362024-06-27T21:51:08.798ZINFOcrucible-agent (datafile): dataset is oxp_f8b5491e-78ae-4725-a4e0-5d2f23407448/crucible/regions/2ab3593d-c760-43b8-a461-f585a7b2dfed
372024-06-27T21:51:08.808ZINFOcrucible-agent (datafile): region 2ab3593d-c760-43b8-a461-f585a7b2dfed state: Created -> Tombstoned
382024-06-27T21:51:08.808ZINFOcrucible-agent (worker): applying SMF actions before removal...
392024-06-27T21:51:08.808ZINFOcrucible-agent (dropshot): request completed latency_us = 19221 local_addr = [fd00:1122:3344:101::1a]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:49139 req_id = d4a9d0f8-ac93-4e84-be18-99f3070e777a response_code = 204 uri = /crucible/0/regions/2ab3593d-c760-43b8-a461-f585a7b2dfed
402024-06-27T21:51:08.809ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-2ab3593d-c760-43b8-a461-f585a7b2dfed (instance states: (Some(Online), None))
412024-06-27T21:51:08.810ZINFOcrucible-agent (dropshot): request completed latency_us = 23 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:49139 req_id = b28b0958-3917-485f-9c5e-9560229bab65 response_code = 200 uri = /crucible/0/regions/2ab3593d-c760-43b8-a461-f585a7b2dfed
422024-06-27T21:51:08.813ZINFOcrucible-agent (worker): SMF ok!
432024-06-27T21:51:08.823ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_f8b5491e-78ae-4725-a4e0-5d2f23407448/crucible/regions/2ab3593d-c760-43b8-a461-f585a7b2dfed" region = 2ab3593d-c760-43b8-a461-f585a7b2dfed
442024-06-27T21:51:08.872ZINFOcrucible-agent (datafile): region 2ab3593d-c760-43b8-a461-f585a7b2dfed state: Tombstoned -> Destroyed
452024-06-27T21:51:08.910ZINFOcrucible-agent (dropshot): request completed latency_us = 48 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:49139 req_id = 39a11ada-a5f1-4da3-aa5a-c0d276be7be0 response_code = 200 uri = /crucible/0/regions/2ab3593d-c760-43b8-a461-f585a7b2dfed
462024-06-27T21:51:10.394ZINFOcrucible-agent (datafile): region a165be3c-2dc9-4e51-a247-96660f2a91ef state: Requested
472024-06-27T21:51:10.394ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
482024-06-27T21:51:10.394ZINFOcrucible-agent (dropshot): request completed latency_us = 250 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:49139 req_id = 88518fee-5051-4bc7-ab73-1fd9d975d329 response_code = 200 uri = /crucible/0/regions
492024-06-27T21:51:10.402ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_f8b5491e-78ae-4725-a4e0-5d2f23407448/crucible/regions/a165be3c-2dc9-4e51-a247-96660f2a91ef
502024-06-27T21:51:10.402ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_f8b5491e-78ae-4725-a4e0-5d2f23407448/crucible/regions/a165be3c-2dc9-4e51-a247-96660f2a91ef
512024-06-27T21:51:10.453ZINFOcrucible-agent (worker): creating region Region { id: RegionId("a165be3c-2dc9-4e51-a247-96660f2a91ef"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/a165be3c-2dc9-4e51-a247-96660f2a91ef" region = a165be3c-2dc9-4e51-a247-96660f2a91ef
522024-06-27T21:51:10.456ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "a165be3c-2dc9-4e51-a247-96660f2a91ef" "--data" "/data/regions/a165be3c-2dc9-4e51-a247-96660f2a91ef" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = a165be3c-2dc9-4e51-a247-96660f2a91ef
532024-06-27T21:51:10.529ZINFOcrucible-agent (worker): region files created ok region = a165be3c-2dc9-4e51-a247-96660f2a91ef
542024-06-27T21:51:10.529ZINFOcrucible-agent (datafile): region a165be3c-2dc9-4e51-a247-96660f2a91ef state: Requested -> Created
552024-06-27T21:51:10.529ZINFOcrucible-agent (worker): applying SMF actions post create...
562024-06-27T21:51:10.531ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-a165be3c-2dc9-4e51-a247-96660f2a91ef
572024-06-27T21:51:10.534ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-a165be3c-2dc9-4e51-a247-96660f2a91ef
582024-06-27T21:51:10.542ZINFOcrucible-agent (worker): creating config property group
592024-06-27T21:51:10.545ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-a165be3c-2dc9-4e51-a247-96660f2a91ef
602024-06-27T21:51:10.546ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/a165be3c-2dc9-4e51-a247-96660f2a91ef
612024-06-27T21:51:10.546ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
622024-06-27T21:51:10.546ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
632024-06-27T21:51:10.546ZINFOcrucible-agent (worker): commit
642024-06-27T21:51:10.550ZINFOcrucible-agent (worker): ok!
652024-06-27T21:51:10.555ZINFOcrucible-agent (worker): SMF ok!
662024-06-27T21:51:10.747ZINFOcrucible-agent (dropshot): request completed latency_us = 74 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:49139 req_id = 0f8d470e-286d-45db-b8a4-9fd3cb4bc78e response_code = 200 uri = /crucible/0/regions
672024-06-27T21:56:37.696ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::1a]:32345 remote_addr = [fd00:1122:3344:101::a]:33626
682024-06-27T21:56:37.697ZINFOcrucible-agent (datafile): region f311d1d5-5bbc-40aa-9816-920626e6637e state: Requested
692024-06-27T21:56:37.698ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
702024-06-27T21:56:37.698ZINFOcrucible-agent (dropshot): request completed latency_us = 949 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:33626 req_id = cd960255-5adb-49d9-82b3-befaf2a18157 response_code = 200 uri = /crucible/0/regions
712024-06-27T21:56:37.708ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_f8b5491e-78ae-4725-a4e0-5d2f23407448/crucible/regions/f311d1d5-5bbc-40aa-9816-920626e6637e
722024-06-27T21:56:37.708ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_f8b5491e-78ae-4725-a4e0-5d2f23407448/crucible/regions/f311d1d5-5bbc-40aa-9816-920626e6637e
732024-06-27T21:56:37.764ZINFOcrucible-agent (worker): creating region Region { id: RegionId("f311d1d5-5bbc-40aa-9816-920626e6637e"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/f311d1d5-5bbc-40aa-9816-920626e6637e" region = f311d1d5-5bbc-40aa-9816-920626e6637e
742024-06-27T21:56:37.764ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "f311d1d5-5bbc-40aa-9816-920626e6637e" "--data" "/data/regions/f311d1d5-5bbc-40aa-9816-920626e6637e" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = f311d1d5-5bbc-40aa-9816-920626e6637e
752024-06-27T21:56:37.837ZINFOcrucible-agent (worker): region files created ok region = f311d1d5-5bbc-40aa-9816-920626e6637e
762024-06-27T21:56:37.837ZINFOcrucible-agent (datafile): region f311d1d5-5bbc-40aa-9816-920626e6637e state: Requested -> Created
772024-06-27T21:56:37.837ZINFOcrucible-agent (worker): applying SMF actions post create...
782024-06-27T21:56:37.846ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-f311d1d5-5bbc-40aa-9816-920626e6637e
792024-06-27T21:56:37.853ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-f311d1d5-5bbc-40aa-9816-920626e6637e
802024-06-27T21:56:37.864ZINFOcrucible-agent (worker): creating config property group
812024-06-27T21:56:37.867ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-f311d1d5-5bbc-40aa-9816-920626e6637e
822024-06-27T21:56:37.867ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/f311d1d5-5bbc-40aa-9816-920626e6637e
832024-06-27T21:56:37.868ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
842024-06-27T21:56:37.868ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
852024-06-27T21:56:37.868ZINFOcrucible-agent (worker): commit
862024-06-27T21:56:37.871ZINFOcrucible-agent (worker): ok!
872024-06-27T21:56:37.878ZINFOcrucible-agent (worker): SMF ok!
882024-06-27T21:56:37.946ZINFOcrucible-agent (dropshot): request completed latency_us = 81 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:33626 req_id = 1f02b48c-2aff-43b5-915d-d31ce3eccbb1 response_code = 200 uri = /crucible/0/regions
892024-06-27T21:56:38.414ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::1a]:32345 remote_addr = [fd00:1122:3344:101::a]:40440
902024-06-27T21:56:38.414ZINFOcrucible-agent (dropshot): request completed latency_us = 52 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:40440 req_id = 0c817aa2-cb4c-4e06-b4b4-aff096ce2050 response_code = 200 uri = /crucible/0/regions/a165be3c-2dc9-4e51-a247-96660f2a91ef
912024-06-27T21:56:38.414ZINFOcrucible-agent (datafile): path is "/data/regions/a165be3c-2dc9-4e51-a247-96660f2a91ef"
922024-06-27T21:56:38.425ZINFOcrucible-agent (datafile): dataset is oxp_f8b5491e-78ae-4725-a4e0-5d2f23407448/crucible/regions/a165be3c-2dc9-4e51-a247-96660f2a91ef
932024-06-27T21:56:38.434ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_f8b5491e-78ae-4725-a4e0-5d2f23407448/crucible/regions/a165be3c-2dc9-4e51-a247-96660f2a91ef@bd357839-a321-4a4a-b806-b5f3cf68741c
942024-06-27T21:56:38.434ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_f8b5491e-78ae-4725-a4e0-5d2f23407448/crucible/regions/a165be3c-2dc9-4e51-a247-96660f2a91ef", "bd357839-a321-4a4a-b806-b5f3cf68741c"]
952024-06-27T21:56:38.442ZINFOcrucible-agent (dropshot): request completed latency_us = 27740 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:40440 req_id = 36bea492-1401-4465-806b-76f8c424b467 response_code = 200 uri = /crucible/0/regions/a165be3c-2dc9-4e51-a247-96660f2a91ef/snapshots/bd357839-a321-4a4a-b806-b5f3cf68741c
962024-06-27T21:56:38.442ZINFOcrucible-agent (datafile): path is "/data/regions/a165be3c-2dc9-4e51-a247-96660f2a91ef"
972024-06-27T21:56:38.452ZINFOcrucible-agent (datafile): dataset is oxp_f8b5491e-78ae-4725-a4e0-5d2f23407448/crucible/regions/a165be3c-2dc9-4e51-a247-96660f2a91ef
982024-06-27T21:56:38.460ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_f8b5491e-78ae-4725-a4e0-5d2f23407448/crucible/regions/a165be3c-2dc9-4e51-a247-96660f2a91ef@bd357839-a321-4a4a-b806-b5f3cf68741c
992024-06-27T21:56:38.460ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_f8b5491e-78ae-4725-a4e0-5d2f23407448/crucible/regions/a165be3c-2dc9-4e51-a247-96660f2a91ef", "bd357839-a321-4a4a-b806-b5f3cf68741c"]
1002024-06-27T21:56:38.473ZINFOcrucible-agent (datafile): requesting running snapshot a165be3c-2dc9-4e51-a247-96660f2a91ef-bd357839-a321-4a4a-b806-b5f3cf68741c state: Requested
1012024-06-27T21:56:38.474ZINFOcrucible-agent (worker): applying SMF actions for region a165be3c-2dc9-4e51-a247-96660f2a91ef running snapshot bd357839-a321-4a4a-b806-b5f3cf68741c (state Requested)...
1022024-06-27T21:56:38.474ZINFOcrucible-agent (dropshot): request completed latency_us = 31415 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:40440 req_id = 0cf50ea0-8c40-4fbd-9c1e-e584306b0efa response_code = 200 uri = /crucible/0/regions/a165be3c-2dc9-4e51-a247-96660f2a91ef/snapshots/bd357839-a321-4a4a-b806-b5f3cf68741c/run
1032024-06-27T21:56:38.476ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-a165be3c-2dc9-4e51-a247-96660f2a91ef-bd357839-a321-4a4a-b806-b5f3cf68741c
1042024-06-27T21:56:38.483ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-a165be3c-2dc9-4e51-a247-96660f2a91ef-bd357839-a321-4a4a-b806-b5f3cf68741c
1052024-06-27T21:56:38.495ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-a165be3c-2dc9-4e51-a247-96660f2a91ef-bd357839-a321-4a4a-b806-b5f3cf68741c
1062024-06-27T21:56:38.496ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/a165be3c-2dc9-4e51-a247-96660f2a91ef/.zfs/snapshot/bd357839-a321-4a4a-b806-b5f3cf68741c
1072024-06-27T21:56:38.496ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
1082024-06-27T21:56:38.496ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
1092024-06-27T21:56:38.496ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
1102024-06-27T21:56:38.496ZINFOcrucible-agent (worker): commit
1112024-06-27T21:56:38.500ZINFOcrucible-agent (worker): ok!
1122024-06-27T21:56:38.504ZINFOcrucible-agent (worker): SMF ok!
1132024-06-27T21:56:38.504ZINFOcrucible-agent (datafile): region a165be3c-2dc9-4e51-a247-96660f2a91ef running snapshot bd357839-a321-4a4a-b806-b5f3cf68741c state: Requested -> Created