Max level shown:
1 [ Jul 25 05:48:05 Enabled. ]
2 [ Jul 25 05:48:06 Rereading configuration. ]
3 [ Jul 25 05:48:10 Rereading configuration. ]
4 [ Jul 25 05:48:13 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_50e63493-6b9c-42de-846b-797a312b5fbc/crucible -l [fd00:1122:3344:101::17]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62024-07-25T05:48:13.794ZINFOcrucible-agent: dataset: "oxp_50e63493-6b9c-42de-846b-797a312b5fbc/crucible"
72024-07-25T05:48:13.801ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::17]:32345
82024-07-25T05:48:13.801ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92024-07-25T05:48:13.822ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102024-07-25T05:48:13.979ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::17]:32345
112024-07-25T05:49:26.257ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::17]:32345 remote_addr = [fd00:1122:3344:101::a]:48751
122024-07-25T05:49:26.282ZINFOcrucible-agent (datafile): region 473e6bbf-37d3-4fca-9547-a231d4e332b4 state: Requested
132024-07-25T05:49:26.290ZINFOcrucible-agent (dropshot): request completed latency_us = 18596 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:48751 req_id = 9601a5f2-a6bc-493b-ac4a-d83db88ee03b response_code = 200 uri = /crucible/0/regions
142024-07-25T05:49:26.292ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
152024-07-25T05:49:26.304ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_50e63493-6b9c-42de-846b-797a312b5fbc/crucible/regions/473e6bbf-37d3-4fca-9547-a231d4e332b4
162024-07-25T05:49:26.304ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_50e63493-6b9c-42de-846b-797a312b5fbc/crucible/regions/473e6bbf-37d3-4fca-9547-a231d4e332b4
172024-07-25T05:49:26.359ZINFOcrucible-agent (worker): creating region Region { id: RegionId("473e6bbf-37d3-4fca-9547-a231d4e332b4"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/473e6bbf-37d3-4fca-9547-a231d4e332b4" region = 473e6bbf-37d3-4fca-9547-a231d4e332b4
182024-07-25T05:49:26.361ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "473e6bbf-37d3-4fca-9547-a231d4e332b4" "--data" "/data/regions/473e6bbf-37d3-4fca-9547-a231d4e332b4" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 473e6bbf-37d3-4fca-9547-a231d4e332b4
192024-07-25T05:49:26.603ZINFOcrucible-agent (worker): region files created ok region = 473e6bbf-37d3-4fca-9547-a231d4e332b4
202024-07-25T05:49:26.603ZINFOcrucible-agent (datafile): region 473e6bbf-37d3-4fca-9547-a231d4e332b4 state: Requested -> Created
212024-07-25T05:49:26.603ZINFOcrucible-agent (worker): applying SMF actions post create...
222024-07-25T05:49:26.604ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-473e6bbf-37d3-4fca-9547-a231d4e332b4
232024-07-25T05:49:26.611ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-473e6bbf-37d3-4fca-9547-a231d4e332b4
242024-07-25T05:49:26.615ZINFOcrucible-agent (dropshot): request completed latency_us = 75 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:48751 req_id = addd7711-914d-4ab8-bb92-2e733fe64bf6 response_code = 200 uri = /crucible/0/regions
252024-07-25T05:49:26.627ZINFOcrucible-agent (worker): creating config property group
262024-07-25T05:49:26.630ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-473e6bbf-37d3-4fca-9547-a231d4e332b4
272024-07-25T05:49:26.631ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/473e6bbf-37d3-4fca-9547-a231d4e332b4
282024-07-25T05:49:26.636ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
292024-07-25T05:49:26.636ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
302024-07-25T05:49:26.636ZINFOcrucible-agent (worker): commit
312024-07-25T05:49:26.641ZINFOcrucible-agent (worker): ok!
322024-07-25T05:49:26.649ZINFOcrucible-agent (worker): SMF ok!
332024-07-25T05:54:54.546ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::17]:32345 remote_addr = [fd00:1122:3344:101::a]:34554
342024-07-25T05:54:54.554ZINFOcrucible-agent (dropshot): request completed latency_us = 7963 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:34554 req_id = cd48796d-da01-4b2d-8bd2-45f7ed024495 response_code = 200 uri = /crucible/0/regions/473e6bbf-37d3-4fca-9547-a231d4e332b4
352024-07-25T05:54:54.556ZINFOcrucible-agent (datafile): path is "/data/regions/473e6bbf-37d3-4fca-9547-a231d4e332b4"
362024-07-25T05:54:54.566ZINFOcrucible-agent (datafile): dataset is oxp_50e63493-6b9c-42de-846b-797a312b5fbc/crucible/regions/473e6bbf-37d3-4fca-9547-a231d4e332b4
372024-07-25T05:54:54.575ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_50e63493-6b9c-42de-846b-797a312b5fbc/crucible/regions/473e6bbf-37d3-4fca-9547-a231d4e332b4@02c17e09-5830-4ee8-a416-a66468634fe2
382024-07-25T05:54:54.575ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_50e63493-6b9c-42de-846b-797a312b5fbc/crucible/regions/473e6bbf-37d3-4fca-9547-a231d4e332b4", "02c17e09-5830-4ee8-a416-a66468634fe2"]
392024-07-25T05:54:54.591ZINFOcrucible-agent (dropshot): request completed latency_us = 36915 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:34554 req_id = d508e34e-3ae6-46ef-85ca-18adfceee3fa response_code = 200 uri = /crucible/0/regions/473e6bbf-37d3-4fca-9547-a231d4e332b4/snapshots/02c17e09-5830-4ee8-a416-a66468634fe2
402024-07-25T05:54:54.591ZINFOcrucible-agent (datafile): path is "/data/regions/473e6bbf-37d3-4fca-9547-a231d4e332b4"
412024-07-25T05:54:54.599ZINFOcrucible-agent (datafile): dataset is oxp_50e63493-6b9c-42de-846b-797a312b5fbc/crucible/regions/473e6bbf-37d3-4fca-9547-a231d4e332b4
422024-07-25T05:54:54.607ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_50e63493-6b9c-42de-846b-797a312b5fbc/crucible/regions/473e6bbf-37d3-4fca-9547-a231d4e332b4@02c17e09-5830-4ee8-a416-a66468634fe2
432024-07-25T05:54:54.607ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_50e63493-6b9c-42de-846b-797a312b5fbc/crucible/regions/473e6bbf-37d3-4fca-9547-a231d4e332b4", "02c17e09-5830-4ee8-a416-a66468634fe2"]
442024-07-25T05:54:54.616ZINFOcrucible-agent (datafile): requesting running snapshot 473e6bbf-37d3-4fca-9547-a231d4e332b4-02c17e09-5830-4ee8-a416-a66468634fe2 state: Requested
452024-07-25T05:54:54.621ZINFOcrucible-agent (worker): applying SMF actions for region 473e6bbf-37d3-4fca-9547-a231d4e332b4 running snapshot 02c17e09-5830-4ee8-a416-a66468634fe2 (state Requested)...
462024-07-25T05:54:54.621ZINFOcrucible-agent (dropshot): request completed latency_us = 29883 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:34554 req_id = 97de1e72-2653-4270-b148-00a2b62be883 response_code = 200 uri = /crucible/0/regions/473e6bbf-37d3-4fca-9547-a231d4e332b4/snapshots/02c17e09-5830-4ee8-a416-a66468634fe2/run
472024-07-25T05:54:54.632ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/473e6bbf-37d3-4fca-9547-a231d4e332b4
482024-07-25T05:54:54.632ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
492024-07-25T05:54:54.632ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::17
502024-07-25T05:54:54.632ZINFOcrucible-agent (worker): reconfig required
512024-07-25T05:54:54.632ZINFOcrucible-agent (worker): using existing config property group
522024-07-25T05:54:54.632ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-473e6bbf-37d3-4fca-9547-a231d4e332b4
532024-07-25T05:54:54.632ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/473e6bbf-37d3-4fca-9547-a231d4e332b4
542024-07-25T05:54:54.632ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
552024-07-25T05:54:54.632ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
562024-07-25T05:54:54.633ZINFOcrucible-agent (worker): commit
572024-07-25T05:54:54.660ZINFOcrucible-agent (worker): ok!
582024-07-25T05:54:54.661ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-473e6bbf-37d3-4fca-9547-a231d4e332b4-02c17e09-5830-4ee8-a416-a66468634fe2
592024-07-25T05:54:54.688ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-473e6bbf-37d3-4fca-9547-a231d4e332b4-02c17e09-5830-4ee8-a416-a66468634fe2
602024-07-25T05:54:54.759ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-473e6bbf-37d3-4fca-9547-a231d4e332b4-02c17e09-5830-4ee8-a416-a66468634fe2
612024-07-25T05:54:54.759ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/473e6bbf-37d3-4fca-9547-a231d4e332b4/.zfs/snapshot/02c17e09-5830-4ee8-a416-a66468634fe2
622024-07-25T05:54:54.759ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
632024-07-25T05:54:54.759ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
642024-07-25T05:54:54.759ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
652024-07-25T05:54:54.759ZINFOcrucible-agent (worker): commit
662024-07-25T05:54:54.786ZINFOcrucible-agent (worker): ok!
672024-07-25T05:54:54.801ZINFOcrucible-agent (worker): SMF ok!
682024-07-25T05:54:54.801ZINFOcrucible-agent (datafile): region 473e6bbf-37d3-4fca-9547-a231d4e332b4 running snapshot 02c17e09-5830-4ee8-a416-a66468634fe2 state: Requested -> Created
692024-07-25T05:54:58.617ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::17]:32345 remote_addr = [fd00:1122:3344:101::a]:57253
702024-07-25T05:54:58.617ZINFOcrucible-agent (datafile): region 497052a2-77a0-47df-a927-ed5dda7e9049 state: Requested
712024-07-25T05:54:58.621ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
722024-07-25T05:54:58.621ZINFOcrucible-agent (dropshot): request completed latency_us = 4073 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:57253 req_id = f99ed8cd-53ac-45e8-8088-2bb84d03355e response_code = 200 uri = /crucible/0/regions
732024-07-25T05:54:58.630ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_50e63493-6b9c-42de-846b-797a312b5fbc/crucible/regions/497052a2-77a0-47df-a927-ed5dda7e9049
742024-07-25T05:54:58.630ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_50e63493-6b9c-42de-846b-797a312b5fbc/crucible/regions/497052a2-77a0-47df-a927-ed5dda7e9049
752024-07-25T05:54:58.705ZINFOcrucible-agent (worker): creating region Region { id: RegionId("497052a2-77a0-47df-a927-ed5dda7e9049"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19002, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/497052a2-77a0-47df-a927-ed5dda7e9049" region = 497052a2-77a0-47df-a927-ed5dda7e9049
762024-07-25T05:54:58.705ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "497052a2-77a0-47df-a927-ed5dda7e9049" "--data" "/data/regions/497052a2-77a0-47df-a927-ed5dda7e9049" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 497052a2-77a0-47df-a927-ed5dda7e9049
772024-07-25T05:54:58.838ZINFOcrucible-agent (dropshot): request completed latency_us = 83 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:57253 req_id = 5267a516-5972-4fcd-98bc-0b63874e012e response_code = 200 uri = /crucible/0/regions
782024-07-25T05:54:58.914ZINFOcrucible-agent (worker): region files created ok region = 497052a2-77a0-47df-a927-ed5dda7e9049
792024-07-25T05:54:58.914ZINFOcrucible-agent (datafile): region 497052a2-77a0-47df-a927-ed5dda7e9049 state: Requested -> Created
802024-07-25T05:54:58.914ZINFOcrucible-agent (worker): applying SMF actions post create...
812024-07-25T05:54:58.915ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/473e6bbf-37d3-4fca-9547-a231d4e332b4
822024-07-25T05:54:58.915ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
832024-07-25T05:54:58.915ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::17
842024-07-25T05:54:58.915ZINFOcrucible-agent (worker): reconfig required
852024-07-25T05:54:58.915ZINFOcrucible-agent (worker): using existing config property group
862024-07-25T05:54:58.915ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-473e6bbf-37d3-4fca-9547-a231d4e332b4
872024-07-25T05:54:58.915ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/473e6bbf-37d3-4fca-9547-a231d4e332b4
882024-07-25T05:54:58.915ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
892024-07-25T05:54:58.915ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
902024-07-25T05:54:58.915ZINFOcrucible-agent (worker): commit
912024-07-25T05:54:58.928ZINFOcrucible-agent (worker): ok!
922024-07-25T05:54:58.929ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-497052a2-77a0-47df-a927-ed5dda7e9049
932024-07-25T05:54:58.941ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-497052a2-77a0-47df-a927-ed5dda7e9049
942024-07-25T05:54:58.970ZINFOcrucible-agent (worker): creating config property group
952024-07-25T05:54:58.976ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-497052a2-77a0-47df-a927-ed5dda7e9049
962024-07-25T05:54:58.977ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/497052a2-77a0-47df-a927-ed5dda7e9049
972024-07-25T05:54:58.977ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
982024-07-25T05:54:58.977ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
992024-07-25T05:54:58.977ZINFOcrucible-agent (worker): commit
1002024-07-25T05:54:58.994ZINFOcrucible-agent (worker): ok!
1012024-07-25T05:54:59.013ZINFOcrucible-agent (worker): SMF ok!
1022024-07-25T05:54:59.244ZINFOcrucible-agent (dropshot): request completed latency_us = 632 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:57253 req_id = 700819e2-8055-4443-bf56-386a2f55c871 response_code = 200 uri = /crucible/0/regions
1032024-07-25T05:55:26.803ZINFOcrucible-agent (dropshot): request completed latency_us = 55 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:57253 req_id = 6d23dbc8-80e7-4868-b7a3-89ac06e063cf response_code = 200 uri = /crucible/0/regions/497052a2-77a0-47df-a927-ed5dda7e9049
1042024-07-25T05:55:26.806ZINFOcrucible-agent (datafile): path is "/data/regions/497052a2-77a0-47df-a927-ed5dda7e9049"
1052024-07-25T05:55:26.832ZINFOcrucible-agent (datafile): dataset is oxp_50e63493-6b9c-42de-846b-797a312b5fbc/crucible/regions/497052a2-77a0-47df-a927-ed5dda7e9049
1062024-07-25T05:55:26.841ZINFOcrucible-agent (datafile): region 497052a2-77a0-47df-a927-ed5dda7e9049 state: Created -> Tombstoned
1072024-07-25T05:55:26.841ZINFOcrucible-agent (worker): applying SMF actions before removal...
1082024-07-25T05:55:26.841ZINFOcrucible-agent (dropshot): request completed latency_us = 34785 local_addr = [fd00:1122:3344:101::17]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:57253 req_id = f1652f12-eb84-4971-997d-90835dfdb079 response_code = 204 uri = /crucible/0/regions/497052a2-77a0-47df-a927-ed5dda7e9049
1092024-07-25T05:55:26.842ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-497052a2-77a0-47df-a927-ed5dda7e9049 (instance states: (Some(Online), None))
1102024-07-25T05:55:26.844ZINFOcrucible-agent (dropshot): request completed latency_us = 543 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:57253 req_id = 8fde695f-b240-429f-9349-ff8696403c8b response_code = 200 uri = /crucible/0/regions/497052a2-77a0-47df-a927-ed5dda7e9049
1112024-07-25T05:55:26.858ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/473e6bbf-37d3-4fca-9547-a231d4e332b4
1122024-07-25T05:55:26.858ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
1132024-07-25T05:55:26.858ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::17
1142024-07-25T05:55:26.858ZINFOcrucible-agent (worker): reconfig required
1152024-07-25T05:55:26.858ZINFOcrucible-agent (worker): using existing config property group
1162024-07-25T05:55:26.859ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-473e6bbf-37d3-4fca-9547-a231d4e332b4
1172024-07-25T05:55:26.859ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/473e6bbf-37d3-4fca-9547-a231d4e332b4
1182024-07-25T05:55:26.859ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
1192024-07-25T05:55:26.859ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
1202024-07-25T05:55:26.859ZINFOcrucible-agent (worker): commit
1212024-07-25T05:55:26.892ZINFOcrucible-agent (worker): ok!
1222024-07-25T05:55:26.893ZINFOcrucible-agent (worker): SMF ok!
1232024-07-25T05:55:26.902ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_50e63493-6b9c-42de-846b-797a312b5fbc/crucible/regions/497052a2-77a0-47df-a927-ed5dda7e9049" region = 497052a2-77a0-47df-a927-ed5dda7e9049
1242024-07-25T05:55:26.973ZINFOcrucible-agent (datafile): region 497052a2-77a0-47df-a927-ed5dda7e9049 state: Tombstoned -> Destroyed
1252024-07-25T05:55:26.995ZINFOcrucible-agent (dropshot): request completed latency_us = 87 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:57253 req_id = c08a3dec-4ee5-4e9f-88b5-4110db5b044f response_code = 200 uri = /crucible/0/regions/497052a2-77a0-47df-a927-ed5dda7e9049