Max level shown:
1 [ Jul 13 05:35:51 Enabled. ]
2 [ Jul 13 05:35:51 Rereading configuration. ]
3 [ Jul 13 05:35:53 Rereading configuration. ]
4 [ Jul 13 05:35:56 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_59a1b65b-0e8e-4af6-9fcd-02021a3973de/crucible -l [fd00:1122:3344:101::18]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62024-07-13T05:35:56.223ZINFOcrucible-agent: dataset: "oxp_59a1b65b-0e8e-4af6-9fcd-02021a3973de/crucible"
72024-07-13T05:35:56.227ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::18]:32345
82024-07-13T05:35:56.227ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92024-07-13T05:35:56.247ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102024-07-13T05:35:56.583ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::18]:32345
112024-07-13T05:37:10.762ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::18]:32345 remote_addr = [fd00:1122:3344:101::a]:63889
122024-07-13T05:37:10.786ZINFOcrucible-agent (datafile): region d70635da-c833-4c73-87b2-27c0905f6896 state: Requested
132024-07-13T05:37:10.794ZINFOcrucible-agent (dropshot): request completed latency_us = 17598 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:63889 req_id = 6ac075b1-7252-4df9-a161-b7b9cb16d4a8 response_code = 200 uri = /crucible/0/regions
142024-07-13T05:37:10.796ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
152024-07-13T05:37:10.809ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_59a1b65b-0e8e-4af6-9fcd-02021a3973de/crucible/regions/d70635da-c833-4c73-87b2-27c0905f6896
162024-07-13T05:37:10.809ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_59a1b65b-0e8e-4af6-9fcd-02021a3973de/crucible/regions/d70635da-c833-4c73-87b2-27c0905f6896
172024-07-13T05:37:10.874ZINFOcrucible-agent (worker): creating region Region { id: RegionId("d70635da-c833-4c73-87b2-27c0905f6896"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/d70635da-c833-4c73-87b2-27c0905f6896" region = d70635da-c833-4c73-87b2-27c0905f6896
182024-07-13T05:37:10.877ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "d70635da-c833-4c73-87b2-27c0905f6896" "--data" "/data/regions/d70635da-c833-4c73-87b2-27c0905f6896" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = d70635da-c833-4c73-87b2-27c0905f6896
192024-07-13T05:37:11.055ZINFOcrucible-agent (worker): region files created ok region = d70635da-c833-4c73-87b2-27c0905f6896
202024-07-13T05:37:11.055ZINFOcrucible-agent (datafile): region d70635da-c833-4c73-87b2-27c0905f6896 state: Requested -> Created
212024-07-13T05:37:11.055ZINFOcrucible-agent (worker): applying SMF actions post create...
222024-07-13T05:37:11.056ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-d70635da-c833-4c73-87b2-27c0905f6896
232024-07-13T05:37:11.063ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-d70635da-c833-4c73-87b2-27c0905f6896
242024-07-13T05:37:11.079ZINFOcrucible-agent (worker): creating config property group
252024-07-13T05:37:11.082ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-d70635da-c833-4c73-87b2-27c0905f6896
262024-07-13T05:37:11.083ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/d70635da-c833-4c73-87b2-27c0905f6896
272024-07-13T05:37:11.088ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
282024-07-13T05:37:11.088ZINFOcrucible-agent (dropshot): request completed latency_us = 2333 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:63889 req_id = 06a792eb-a9be-46f7-b699-207a6f1d9eac response_code = 200 uri = /crucible/0/regions
292024-07-13T05:37:11.088ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
302024-07-13T05:37:11.088ZINFOcrucible-agent (worker): commit
312024-07-13T05:37:11.094ZINFOcrucible-agent (worker): ok!
322024-07-13T05:37:11.101ZINFOcrucible-agent (worker): SMF ok!
332024-07-13T05:42:29.040ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::18]:32345 remote_addr = [fd00:1122:3344:101::a]:37684
342024-07-13T05:42:29.043ZINFOcrucible-agent (dropshot): request completed latency_us = 3397 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:37684 req_id = 1f3322d8-a660-4bf4-b3da-8bb1fe40f492 response_code = 200 uri = /crucible/0/regions/d70635da-c833-4c73-87b2-27c0905f6896
352024-07-13T05:42:29.046ZINFOcrucible-agent (datafile): path is "/data/regions/d70635da-c833-4c73-87b2-27c0905f6896"
362024-07-13T05:42:29.055ZINFOcrucible-agent (datafile): dataset is oxp_59a1b65b-0e8e-4af6-9fcd-02021a3973de/crucible/regions/d70635da-c833-4c73-87b2-27c0905f6896
372024-07-13T05:42:29.065ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_59a1b65b-0e8e-4af6-9fcd-02021a3973de/crucible/regions/d70635da-c833-4c73-87b2-27c0905f6896@5d2d3acf-ae8e-42b2-be21-a0630b25a89a
382024-07-13T05:42:29.065ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_59a1b65b-0e8e-4af6-9fcd-02021a3973de/crucible/regions/d70635da-c833-4c73-87b2-27c0905f6896", "5d2d3acf-ae8e-42b2-be21-a0630b25a89a"]
392024-07-13T05:42:29.077ZINFOcrucible-agent (dropshot): request completed latency_us = 33217 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:37684 req_id = 8452d6b1-8f20-48a3-b2f5-6595a5feac86 response_code = 200 uri = /crucible/0/regions/d70635da-c833-4c73-87b2-27c0905f6896/snapshots/5d2d3acf-ae8e-42b2-be21-a0630b25a89a
402024-07-13T05:42:29.077ZINFOcrucible-agent (datafile): path is "/data/regions/d70635da-c833-4c73-87b2-27c0905f6896"
412024-07-13T05:42:29.086ZINFOcrucible-agent (datafile): dataset is oxp_59a1b65b-0e8e-4af6-9fcd-02021a3973de/crucible/regions/d70635da-c833-4c73-87b2-27c0905f6896
422024-07-13T05:42:29.094ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_59a1b65b-0e8e-4af6-9fcd-02021a3973de/crucible/regions/d70635da-c833-4c73-87b2-27c0905f6896@5d2d3acf-ae8e-42b2-be21-a0630b25a89a
432024-07-13T05:42:29.094ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_59a1b65b-0e8e-4af6-9fcd-02021a3973de/crucible/regions/d70635da-c833-4c73-87b2-27c0905f6896", "5d2d3acf-ae8e-42b2-be21-a0630b25a89a"]
442024-07-13T05:42:29.104ZINFOcrucible-agent (datafile): requesting running snapshot d70635da-c833-4c73-87b2-27c0905f6896-5d2d3acf-ae8e-42b2-be21-a0630b25a89a state: Requested
452024-07-13T05:42:29.105ZINFOcrucible-agent (worker): applying SMF actions for region d70635da-c833-4c73-87b2-27c0905f6896 running snapshot 5d2d3acf-ae8e-42b2-be21-a0630b25a89a (state Requested)...
462024-07-13T05:42:29.105ZINFOcrucible-agent (dropshot): request completed latency_us = 27828 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:37684 req_id = 23ad6054-da02-41f5-9817-9dad20828802 response_code = 200 uri = /crucible/0/regions/d70635da-c833-4c73-87b2-27c0905f6896/snapshots/5d2d3acf-ae8e-42b2-be21-a0630b25a89a/run
472024-07-13T05:42:29.117ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/d70635da-c833-4c73-87b2-27c0905f6896
482024-07-13T05:42:29.117ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
492024-07-13T05:42:29.117ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::18
502024-07-13T05:42:29.117ZINFOcrucible-agent (worker): reconfig required
512024-07-13T05:42:29.117ZINFOcrucible-agent (worker): using existing config property group
522024-07-13T05:42:29.117ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-d70635da-c833-4c73-87b2-27c0905f6896
532024-07-13T05:42:29.117ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/d70635da-c833-4c73-87b2-27c0905f6896
542024-07-13T05:42:29.117ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
552024-07-13T05:42:29.117ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
562024-07-13T05:42:29.117ZINFOcrucible-agent (worker): commit
572024-07-13T05:42:29.125ZINFOcrucible-agent (worker): ok!
582024-07-13T05:42:29.126ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-d70635da-c833-4c73-87b2-27c0905f6896-5d2d3acf-ae8e-42b2-be21-a0630b25a89a
592024-07-13T05:42:29.131ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-d70635da-c833-4c73-87b2-27c0905f6896-5d2d3acf-ae8e-42b2-be21-a0630b25a89a
602024-07-13T05:42:29.152ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-d70635da-c833-4c73-87b2-27c0905f6896-5d2d3acf-ae8e-42b2-be21-a0630b25a89a
612024-07-13T05:42:29.153ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/d70635da-c833-4c73-87b2-27c0905f6896/.zfs/snapshot/5d2d3acf-ae8e-42b2-be21-a0630b25a89a
622024-07-13T05:42:29.153ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
632024-07-13T05:42:29.153ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
642024-07-13T05:42:29.153ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
652024-07-13T05:42:29.153ZINFOcrucible-agent (worker): commit
662024-07-13T05:42:29.160ZINFOcrucible-agent (worker): ok!
672024-07-13T05:42:29.168ZINFOcrucible-agent (worker): SMF ok!
682024-07-13T05:42:29.168ZINFOcrucible-agent (datafile): region d70635da-c833-4c73-87b2-27c0905f6896 running snapshot 5d2d3acf-ae8e-42b2-be21-a0630b25a89a state: Requested -> Created
692024-07-13T05:42:32.126ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::18]:32345 remote_addr = [fd00:1122:3344:101::a]:63439
702024-07-13T05:42:32.126ZINFOcrucible-agent (datafile): region 73a48ee4-d45a-4bef-bdd1-5c3ffc7fc931 state: Requested
712024-07-13T05:42:32.129ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
722024-07-13T05:42:32.129ZINFOcrucible-agent (dropshot): request completed latency_us = 2553 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:63439 req_id = a48f8ed0-128f-4dab-88d8-18f2c3fe6f2c response_code = 200 uri = /crucible/0/regions
732024-07-13T05:42:32.138ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_59a1b65b-0e8e-4af6-9fcd-02021a3973de/crucible/regions/73a48ee4-d45a-4bef-bdd1-5c3ffc7fc931
742024-07-13T05:42:32.139ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_59a1b65b-0e8e-4af6-9fcd-02021a3973de/crucible/regions/73a48ee4-d45a-4bef-bdd1-5c3ffc7fc931
752024-07-13T05:42:32.206ZINFOcrucible-agent (worker): creating region Region { id: RegionId("73a48ee4-d45a-4bef-bdd1-5c3ffc7fc931"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19002, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/73a48ee4-d45a-4bef-bdd1-5c3ffc7fc931" region = 73a48ee4-d45a-4bef-bdd1-5c3ffc7fc931
762024-07-13T05:42:32.206ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "73a48ee4-d45a-4bef-bdd1-5c3ffc7fc931" "--data" "/data/regions/73a48ee4-d45a-4bef-bdd1-5c3ffc7fc931" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 73a48ee4-d45a-4bef-bdd1-5c3ffc7fc931
772024-07-13T05:42:32.332ZINFOcrucible-agent (worker): region files created ok region = 73a48ee4-d45a-4bef-bdd1-5c3ffc7fc931
782024-07-13T05:42:32.332ZINFOcrucible-agent (datafile): region 73a48ee4-d45a-4bef-bdd1-5c3ffc7fc931 state: Requested -> Created
792024-07-13T05:42:32.332ZINFOcrucible-agent (worker): applying SMF actions post create...
802024-07-13T05:42:32.332ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-73a48ee4-d45a-4bef-bdd1-5c3ffc7fc931
812024-07-13T05:42:32.342ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-73a48ee4-d45a-4bef-bdd1-5c3ffc7fc931
822024-07-13T05:42:32.347ZINFOcrucible-agent (dropshot): request completed latency_us = 78 local_addr = [fd00:1122:3344:101::18]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:63439 req_id = bcb33452-591d-4497-89c4-433dc6c8675c response_code = 200 uri = /crucible/0/regions
832024-07-13T05:42:32.359ZINFOcrucible-agent (worker): creating config property group
842024-07-13T05:42:32.366ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-73a48ee4-d45a-4bef-bdd1-5c3ffc7fc931
852024-07-13T05:42:32.366ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/73a48ee4-d45a-4bef-bdd1-5c3ffc7fc931
862024-07-13T05:42:32.366ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
872024-07-13T05:42:32.366ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
882024-07-13T05:42:32.366ZINFOcrucible-agent (worker): commit
892024-07-13T05:42:32.374ZINFOcrucible-agent (worker): ok!
902024-07-13T05:42:32.381ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/d70635da-c833-4c73-87b2-27c0905f6896
912024-07-13T05:42:32.381ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
922024-07-13T05:42:32.381ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::18
932024-07-13T05:42:32.381ZINFOcrucible-agent (worker): reconfig required
942024-07-13T05:42:32.381ZINFOcrucible-agent (worker): using existing config property group
952024-07-13T05:42:32.381ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-d70635da-c833-4c73-87b2-27c0905f6896
962024-07-13T05:42:32.387ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/d70635da-c833-4c73-87b2-27c0905f6896
972024-07-13T05:42:32.387ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
982024-07-13T05:42:32.387ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
992024-07-13T05:42:32.387ZINFOcrucible-agent (worker): commit
1002024-07-13T05:42:32.398ZINFOcrucible-agent (worker): ok!
1012024-07-13T05:42:32.400ZINFOcrucible-agent (worker): SMF ok!
1022024-07-13T05:42:58.514ZINFOcrucible-agent (dropshot): request completed latency_us = 61 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:63439 req_id = cee27305-23e2-477f-9585-128de0bc9d91 response_code = 200 uri = /crucible/0/regions/73a48ee4-d45a-4bef-bdd1-5c3ffc7fc931
1032024-07-13T05:42:58.516ZINFOcrucible-agent (datafile): path is "/data/regions/73a48ee4-d45a-4bef-bdd1-5c3ffc7fc931"
1042024-07-13T05:42:58.527ZINFOcrucible-agent (datafile): dataset is oxp_59a1b65b-0e8e-4af6-9fcd-02021a3973de/crucible/regions/73a48ee4-d45a-4bef-bdd1-5c3ffc7fc931
1052024-07-13T05:42:58.536ZINFOcrucible-agent (datafile): region 73a48ee4-d45a-4bef-bdd1-5c3ffc7fc931 state: Created -> Tombstoned
1062024-07-13T05:42:58.536ZINFOcrucible-agent (worker): applying SMF actions before removal...
1072024-07-13T05:42:58.536ZINFOcrucible-agent (dropshot): request completed latency_us = 20094 local_addr = [fd00:1122:3344:101::18]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:63439 req_id = 9f55570a-d8a2-4249-9d3c-8b3872307634 response_code = 204 uri = /crucible/0/regions/73a48ee4-d45a-4bef-bdd1-5c3ffc7fc931
1082024-07-13T05:42:58.537ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-73a48ee4-d45a-4bef-bdd1-5c3ffc7fc931 (instance states: (Some(Online), None))
1092024-07-13T05:42:58.539ZINFOcrucible-agent (dropshot): request completed latency_us = 26 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:63439 req_id = 90b289bb-91ce-4c6b-8754-d6329d030c55 response_code = 200 uri = /crucible/0/regions/73a48ee4-d45a-4bef-bdd1-5c3ffc7fc931
1102024-07-13T05:42:58.542ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/d70635da-c833-4c73-87b2-27c0905f6896
1112024-07-13T05:42:58.542ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
1122024-07-13T05:42:58.542ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::18
1132024-07-13T05:42:58.542ZINFOcrucible-agent (worker): reconfig required
1142024-07-13T05:42:58.542ZINFOcrucible-agent (worker): using existing config property group
1152024-07-13T05:42:58.542ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-d70635da-c833-4c73-87b2-27c0905f6896
1162024-07-13T05:42:58.542ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/d70635da-c833-4c73-87b2-27c0905f6896
1172024-07-13T05:42:58.542ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
1182024-07-13T05:42:58.542ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::18
1192024-07-13T05:42:58.542ZINFOcrucible-agent (worker): commit
1202024-07-13T05:42:58.547ZINFOcrucible-agent (worker): ok!
1212024-07-13T05:42:58.549ZINFOcrucible-agent (worker): SMF ok!
1222024-07-13T05:42:58.559ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_59a1b65b-0e8e-4af6-9fcd-02021a3973de/crucible/regions/73a48ee4-d45a-4bef-bdd1-5c3ffc7fc931" region = 73a48ee4-d45a-4bef-bdd1-5c3ffc7fc931
1232024-07-13T05:42:58.605ZINFOcrucible-agent (datafile): region 73a48ee4-d45a-4bef-bdd1-5c3ffc7fc931 state: Tombstoned -> Destroyed
1242024-07-13T05:42:58.690ZINFOcrucible-agent (dropshot): request completed latency_us = 59 local_addr = [fd00:1122:3344:101::18]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:63439 req_id = bffd117a-3d94-4456-9b9f-905aa0785ca7 response_code = 200 uri = /crucible/0/regions/73a48ee4-d45a-4bef-bdd1-5c3ffc7fc931