Max level shown:
1 [ Jun 28 20:20:19 Enabled. ]
2 [ Jun 28 20:20:19 Rereading configuration. ]
3 [ Jun 28 20:20:20 Rereading configuration. ]
4 [ Jun 28 20:20:23 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_5ac4a919-8b24-423d-ad25-5b564847ba3f/crucible -l [fd00:1122:3344:101::12]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62024-06-28T20:20:23.568ZINFOcrucible-agent: dataset: "oxp_5ac4a919-8b24-423d-ad25-5b564847ba3f/crucible"
72024-06-28T20:20:23.572ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::12]:32345
82024-06-28T20:20:23.572ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92024-06-28T20:20:23.614ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102024-06-28T20:20:23.933ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::12]:32345
112024-06-28T20:21:35.330ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:54851
122024-06-28T20:21:35.339ZINFOcrucible-agent (datafile): region 5b9b7096-9b6d-4865-b16f-5b5429b3f257 state: Requested
132024-06-28T20:21:35.347ZINFOcrucible-agent (dropshot): request completed latency_us = 9578 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:54851 req_id = 82dcc047-93d5-4687-81be-03b20fc67854 response_code = 200 uri = /crucible/0/regions
142024-06-28T20:21:35.347ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
152024-06-28T20:21:35.357ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_5ac4a919-8b24-423d-ad25-5b564847ba3f/crucible/regions/5b9b7096-9b6d-4865-b16f-5b5429b3f257
162024-06-28T20:21:35.358ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_5ac4a919-8b24-423d-ad25-5b564847ba3f/crucible/regions/5b9b7096-9b6d-4865-b16f-5b5429b3f257
172024-06-28T20:21:35.423ZINFOcrucible-agent (worker): creating region Region { id: RegionId("5b9b7096-9b6d-4865-b16f-5b5429b3f257"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/5b9b7096-9b6d-4865-b16f-5b5429b3f257" region = 5b9b7096-9b6d-4865-b16f-5b5429b3f257
182024-06-28T20:21:35.425ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "5b9b7096-9b6d-4865-b16f-5b5429b3f257" "--data" "/data/regions/5b9b7096-9b6d-4865-b16f-5b5429b3f257" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 5b9b7096-9b6d-4865-b16f-5b5429b3f257
192024-06-28T20:21:35.518ZINFOcrucible-agent (dropshot): request completed latency_us = 85 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:54851 req_id = e4b0ec79-b2b4-40a2-9a1c-eb74a5edd183 response_code = 200 uri = /crucible/0/regions
202024-06-28T20:21:35.570ZINFOcrucible-agent (worker): region files created ok region = 5b9b7096-9b6d-4865-b16f-5b5429b3f257
212024-06-28T20:21:35.571ZINFOcrucible-agent (datafile): region 5b9b7096-9b6d-4865-b16f-5b5429b3f257 state: Requested -> Created
222024-06-28T20:21:35.571ZINFOcrucible-agent (worker): applying SMF actions post create...
232024-06-28T20:21:35.571ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-5b9b7096-9b6d-4865-b16f-5b5429b3f257
242024-06-28T20:21:35.577ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-5b9b7096-9b6d-4865-b16f-5b5429b3f257
252024-06-28T20:21:35.590ZINFOcrucible-agent (worker): creating config property group
262024-06-28T20:21:35.593ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-5b9b7096-9b6d-4865-b16f-5b5429b3f257
272024-06-28T20:21:35.594ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/5b9b7096-9b6d-4865-b16f-5b5429b3f257
282024-06-28T20:21:35.596ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
292024-06-28T20:21:35.596ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
302024-06-28T20:21:35.596ZINFOcrucible-agent (worker): commit
312024-06-28T20:21:35.603ZINFOcrucible-agent (worker): ok!
322024-06-28T20:21:35.609ZINFOcrucible-agent (worker): SMF ok!
332024-06-28T20:21:36.224ZINFOcrucible-agent (dropshot): request completed latency_us = 94 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:54851 req_id = 4c61cc0d-8773-40bc-a036-b001481571c4 response_code = 200 uri = /crucible/0/regions
342024-06-28T20:26:57.916ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:59956
352024-06-28T20:26:57.917ZINFOcrucible-agent (dropshot): request completed latency_us = 49 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:59956 req_id = 023ac951-d9ae-44fa-aff9-24b74c08c250 response_code = 200 uri = /crucible/0/regions/5b9b7096-9b6d-4865-b16f-5b5429b3f257
362024-06-28T20:26:57.920ZINFOcrucible-agent (datafile): path is "/data/regions/5b9b7096-9b6d-4865-b16f-5b5429b3f257"
372024-06-28T20:26:57.929ZINFOcrucible-agent (datafile): dataset is oxp_5ac4a919-8b24-423d-ad25-5b564847ba3f/crucible/regions/5b9b7096-9b6d-4865-b16f-5b5429b3f257
382024-06-28T20:26:57.939ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_5ac4a919-8b24-423d-ad25-5b564847ba3f/crucible/regions/5b9b7096-9b6d-4865-b16f-5b5429b3f257@19891c95-ff5a-4815-a720-829e6cd5a4f6
392024-06-28T20:26:57.939ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_5ac4a919-8b24-423d-ad25-5b564847ba3f/crucible/regions/5b9b7096-9b6d-4865-b16f-5b5429b3f257", "19891c95-ff5a-4815-a720-829e6cd5a4f6"]
402024-06-28T20:26:57.951ZINFOcrucible-agent (dropshot): request completed latency_us = 34412 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:59956 req_id = de01cc98-1338-4464-9975-b9663c0d65ee response_code = 200 uri = /crucible/0/regions/5b9b7096-9b6d-4865-b16f-5b5429b3f257/snapshots/19891c95-ff5a-4815-a720-829e6cd5a4f6
412024-06-28T20:26:57.951ZINFOcrucible-agent (datafile): path is "/data/regions/5b9b7096-9b6d-4865-b16f-5b5429b3f257"
422024-06-28T20:26:57.960ZINFOcrucible-agent (datafile): dataset is oxp_5ac4a919-8b24-423d-ad25-5b564847ba3f/crucible/regions/5b9b7096-9b6d-4865-b16f-5b5429b3f257
432024-06-28T20:26:57.968ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_5ac4a919-8b24-423d-ad25-5b564847ba3f/crucible/regions/5b9b7096-9b6d-4865-b16f-5b5429b3f257@19891c95-ff5a-4815-a720-829e6cd5a4f6
442024-06-28T20:26:57.969ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_5ac4a919-8b24-423d-ad25-5b564847ba3f/crucible/regions/5b9b7096-9b6d-4865-b16f-5b5429b3f257", "19891c95-ff5a-4815-a720-829e6cd5a4f6"]
452024-06-28T20:26:57.979ZINFOcrucible-agent (datafile): requesting running snapshot 5b9b7096-9b6d-4865-b16f-5b5429b3f257-19891c95-ff5a-4815-a720-829e6cd5a4f6 state: Requested
462024-06-28T20:26:57.979ZINFOcrucible-agent (worker): applying SMF actions for region 5b9b7096-9b6d-4865-b16f-5b5429b3f257 running snapshot 19891c95-ff5a-4815-a720-829e6cd5a4f6 (state Requested)...
472024-06-28T20:26:57.979ZINFOcrucible-agent (dropshot): request completed latency_us = 28050 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:59956 req_id = e0eb1283-c941-4ada-89f3-12b170495526 response_code = 200 uri = /crucible/0/regions/5b9b7096-9b6d-4865-b16f-5b5429b3f257/snapshots/19891c95-ff5a-4815-a720-829e6cd5a4f6/run
482024-06-28T20:26:57.991ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/5b9b7096-9b6d-4865-b16f-5b5429b3f257
492024-06-28T20:26:57.991ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
502024-06-28T20:26:57.991ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::12
512024-06-28T20:26:57.991ZINFOcrucible-agent (worker): reconfig required
522024-06-28T20:26:57.991ZINFOcrucible-agent (worker): using existing config property group
532024-06-28T20:26:57.991ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-5b9b7096-9b6d-4865-b16f-5b5429b3f257
542024-06-28T20:26:57.991ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/5b9b7096-9b6d-4865-b16f-5b5429b3f257
552024-06-28T20:26:57.991ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
562024-06-28T20:26:57.991ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
572024-06-28T20:26:57.991ZINFOcrucible-agent (worker): commit
582024-06-28T20:26:57.997ZINFOcrucible-agent (worker): ok!
592024-06-28T20:26:57.998ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-5b9b7096-9b6d-4865-b16f-5b5429b3f257-19891c95-ff5a-4815-a720-829e6cd5a4f6
602024-06-28T20:26:58.001ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-5b9b7096-9b6d-4865-b16f-5b5429b3f257-19891c95-ff5a-4815-a720-829e6cd5a4f6
612024-06-28T20:26:58.015ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-5b9b7096-9b6d-4865-b16f-5b5429b3f257-19891c95-ff5a-4815-a720-829e6cd5a4f6
622024-06-28T20:26:58.015ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/5b9b7096-9b6d-4865-b16f-5b5429b3f257/.zfs/snapshot/19891c95-ff5a-4815-a720-829e6cd5a4f6
632024-06-28T20:26:58.016ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
642024-06-28T20:26:58.016ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
652024-06-28T20:26:58.016ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
662024-06-28T20:26:58.016ZINFOcrucible-agent (worker): commit
672024-06-28T20:26:58.020ZINFOcrucible-agent (worker): ok!
682024-06-28T20:26:58.024ZINFOcrucible-agent (worker): SMF ok!
692024-06-28T20:26:58.024ZINFOcrucible-agent (datafile): region 5b9b7096-9b6d-4865-b16f-5b5429b3f257 running snapshot 19891c95-ff5a-4815-a720-829e6cd5a4f6 state: Requested -> Created
702024-06-28T20:27:00.853ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:44974
712024-06-28T20:27:00.854ZINFOcrucible-agent (datafile): region 9296fd53-220b-4817-9198-7cecfe4cef11 state: Requested
722024-06-28T20:27:00.854ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
732024-06-28T20:27:00.854ZINFOcrucible-agent (dropshot): request completed latency_us = 267 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:44974 req_id = ac6d0839-f335-46e3-abe3-7b6f42e106a8 response_code = 200 uri = /crucible/0/regions
742024-06-28T20:27:00.864ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_5ac4a919-8b24-423d-ad25-5b564847ba3f/crucible/regions/9296fd53-220b-4817-9198-7cecfe4cef11
752024-06-28T20:27:00.864ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_5ac4a919-8b24-423d-ad25-5b564847ba3f/crucible/regions/9296fd53-220b-4817-9198-7cecfe4cef11
762024-06-28T20:27:00.933ZINFOcrucible-agent (worker): creating region Region { id: RegionId("9296fd53-220b-4817-9198-7cecfe4cef11"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19002, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/9296fd53-220b-4817-9198-7cecfe4cef11" region = 9296fd53-220b-4817-9198-7cecfe4cef11
772024-06-28T20:27:00.933ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "9296fd53-220b-4817-9198-7cecfe4cef11" "--data" "/data/regions/9296fd53-220b-4817-9198-7cecfe4cef11" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 9296fd53-220b-4817-9198-7cecfe4cef11
782024-06-28T20:27:01.016ZINFOcrucible-agent (worker): region files created ok region = 9296fd53-220b-4817-9198-7cecfe4cef11
792024-06-28T20:27:01.016ZINFOcrucible-agent (datafile): region 9296fd53-220b-4817-9198-7cecfe4cef11 state: Requested -> Created
802024-06-28T20:27:01.017ZINFOcrucible-agent (worker): applying SMF actions post create...
812024-06-28T20:27:01.017ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/5b9b7096-9b6d-4865-b16f-5b5429b3f257
822024-06-28T20:27:01.017ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
832024-06-28T20:27:01.017ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::12
842024-06-28T20:27:01.017ZINFOcrucible-agent (worker): reconfig required
852024-06-28T20:27:01.017ZINFOcrucible-agent (worker): using existing config property group
862024-06-28T20:27:01.017ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-5b9b7096-9b6d-4865-b16f-5b5429b3f257
872024-06-28T20:27:01.018ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/5b9b7096-9b6d-4865-b16f-5b5429b3f257
882024-06-28T20:27:01.018ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
892024-06-28T20:27:01.018ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
902024-06-28T20:27:01.018ZINFOcrucible-agent (worker): commit
912024-06-28T20:27:01.022ZINFOcrucible-agent (worker): ok!
922024-06-28T20:27:01.022ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-9296fd53-220b-4817-9198-7cecfe4cef11
932024-06-28T20:27:01.027ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-9296fd53-220b-4817-9198-7cecfe4cef11
942024-06-28T20:27:01.034ZINFOcrucible-agent (worker): creating config property group
952024-06-28T20:27:01.037ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-9296fd53-220b-4817-9198-7cecfe4cef11
962024-06-28T20:27:01.037ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/9296fd53-220b-4817-9198-7cecfe4cef11
972024-06-28T20:27:01.037ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
982024-06-28T20:27:01.037ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
992024-06-28T20:27:01.037ZINFOcrucible-agent (worker): commit
1002024-06-28T20:27:01.040ZINFOcrucible-agent (worker): ok!
1012024-06-28T20:27:01.046ZINFOcrucible-agent (worker): SMF ok!
1022024-06-28T20:27:01.126ZINFOcrucible-agent (dropshot): request completed latency_us = 79 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:44974 req_id = 1848317e-3398-445d-b386-cf5352a37957 response_code = 200 uri = /crucible/0/regions
1032024-06-28T20:27:26.357ZINFOcrucible-agent (dropshot): request completed latency_us = 53 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:44974 req_id = 9f91f4fa-ab1a-40cf-9c2d-fddb34708390 response_code = 200 uri = /crucible/0/regions/9296fd53-220b-4817-9198-7cecfe4cef11
1042024-06-28T20:27:26.359ZINFOcrucible-agent (datafile): path is "/data/regions/9296fd53-220b-4817-9198-7cecfe4cef11"
1052024-06-28T20:27:26.372ZINFOcrucible-agent (datafile): dataset is oxp_5ac4a919-8b24-423d-ad25-5b564847ba3f/crucible/regions/9296fd53-220b-4817-9198-7cecfe4cef11
1062024-06-28T20:27:26.381ZINFOcrucible-agent (datafile): region 9296fd53-220b-4817-9198-7cecfe4cef11 state: Created -> Tombstoned
1072024-06-28T20:27:26.381ZINFOcrucible-agent (worker): applying SMF actions before removal...
1082024-06-28T20:27:26.381ZINFOcrucible-agent (dropshot): request completed latency_us = 21642 local_addr = [fd00:1122:3344:101::12]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:44974 req_id = 3da6f315-8db2-4288-8858-37c288b55404 response_code = 204 uri = /crucible/0/regions/9296fd53-220b-4817-9198-7cecfe4cef11
1092024-06-28T20:27:26.382ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-9296fd53-220b-4817-9198-7cecfe4cef11 (instance states: (Some(Online), None))
1102024-06-28T20:27:26.384ZINFOcrucible-agent (dropshot): request completed latency_us = 34 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:44974 req_id = bdfaf795-6b25-4f73-be5c-64b3a3fe67a8 response_code = 200 uri = /crucible/0/regions/9296fd53-220b-4817-9198-7cecfe4cef11
1112024-06-28T20:27:26.387ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/5b9b7096-9b6d-4865-b16f-5b5429b3f257
1122024-06-28T20:27:26.387ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
1132024-06-28T20:27:26.387ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::12
1142024-06-28T20:27:26.387ZINFOcrucible-agent (worker): reconfig required
1152024-06-28T20:27:26.387ZINFOcrucible-agent (worker): using existing config property group
1162024-06-28T20:27:26.387ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-5b9b7096-9b6d-4865-b16f-5b5429b3f257
1172024-06-28T20:27:26.387ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/5b9b7096-9b6d-4865-b16f-5b5429b3f257
1182024-06-28T20:27:26.387ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
1192024-06-28T20:27:26.387ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
1202024-06-28T20:27:26.387ZINFOcrucible-agent (worker): commit
1212024-06-28T20:27:26.391ZINFOcrucible-agent (worker): ok!
1222024-06-28T20:27:26.392ZINFOcrucible-agent (worker): SMF ok!
1232024-06-28T20:27:26.402ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_5ac4a919-8b24-423d-ad25-5b564847ba3f/crucible/regions/9296fd53-220b-4817-9198-7cecfe4cef11" region = 9296fd53-220b-4817-9198-7cecfe4cef11
1242024-06-28T20:27:26.456ZINFOcrucible-agent (datafile): region 9296fd53-220b-4817-9198-7cecfe4cef11 state: Tombstoned -> Destroyed
1252024-06-28T20:27:26.465ZINFOcrucible-agent (dropshot): request completed latency_us = 67 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:44974 req_id = d390ecbe-ee61-4ae2-bb2b-4f167e647662 response_code = 200 uri = /crucible/0/regions/9296fd53-220b-4817-9198-7cecfe4cef11