Max level shown:
1 [ May 22 21:17:24 Enabled. ]
2 [ May 22 21:17:24 Rereading configuration. ]
3 [ May 22 21:17:26 Rereading configuration. ]
4 [ May 22 21:17:29 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_d3f1a944-1600-4eae-a191-e371b04a8674/crucible -l [fd00:1122:3344:101::17]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62025-05-22T21:17:29.822ZINFOcrucible-agent: dataset: "oxp_d3f1a944-1600-4eae-a191-e371b04a8674/crucible"
72025-05-22T21:17:29.826ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::17]:32345
82025-05-22T21:17:29.826ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92025-05-22T21:17:29.843ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102025-05-22T21:17:29.940ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::17]:32345
112025-05-22T21:17:54.913ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::17]:32345 remote_addr = [fd00:1122:3344:101::c]:41579
122025-05-22T21:17:54.931ZINFOcrucible-agent (datafile): region 265c9bd3-fd38-4927-8cb0-8004352d4b6b state: Requested
132025-05-22T21:17:54.933ZINFOcrucible-agent (dropshot): request completed latency_us = 15485 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:41579 req_id = aed62c36-68ef-454a-b5de-206e1d0e6c29 response_code = 200 uri = /crucible/0/regions
142025-05-22T21:17:54.936ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
152025-05-22T21:17:54.945ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_d3f1a944-1600-4eae-a191-e371b04a8674/crucible/regions/265c9bd3-fd38-4927-8cb0-8004352d4b6b
162025-05-22T21:17:54.945ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_d3f1a944-1600-4eae-a191-e371b04a8674/crucible/regions/265c9bd3-fd38-4927-8cb0-8004352d4b6b
172025-05-22T21:17:55.015ZINFOcrucible-agent (worker): creating region Region { id: RegionId("265c9bd3-fd38-4927-8cb0-8004352d4b6b"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/265c9bd3-fd38-4927-8cb0-8004352d4b6b" region = 265c9bd3-fd38-4927-8cb0-8004352d4b6b
182025-05-22T21:17:55.017ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "265c9bd3-fd38-4927-8cb0-8004352d4b6b" "--data" "/data/regions/265c9bd3-fd38-4927-8cb0-8004352d4b6b" "--block-size" "512" "--extent-size" "131072" "--extent-count" "16" "--encrypted" region = 265c9bd3-fd38-4927-8cb0-8004352d4b6b
192025-05-22T21:17:55.071ZINFOcrucible-agent (dropshot): request completed latency_us = 70 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:41579 req_id = d4fd1d7e-4061-477b-9d3a-fea5a1585350 response_code = 200 uri = /crucible/0/regions
202025-05-22T21:17:55.319ZINFOcrucible-agent (worker): region files created ok region = 265c9bd3-fd38-4927-8cb0-8004352d4b6b
212025-05-22T21:17:55.319ZINFOcrucible-agent (datafile): region 265c9bd3-fd38-4927-8cb0-8004352d4b6b state: Requested -> Created
222025-05-22T21:17:55.319ZINFOcrucible-agent (worker): applying SMF actions post create...
232025-05-22T21:17:55.319ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-265c9bd3-fd38-4927-8cb0-8004352d4b6b
242025-05-22T21:17:55.324ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-265c9bd3-fd38-4927-8cb0-8004352d4b6b
252025-05-22T21:17:55.340ZINFOcrucible-agent (worker): creating config property group
262025-05-22T21:17:55.345ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-265c9bd3-fd38-4927-8cb0-8004352d4b6b
272025-05-22T21:17:55.345ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/265c9bd3-fd38-4927-8cb0-8004352d4b6b
282025-05-22T21:17:55.345ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
292025-05-22T21:17:55.345ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
302025-05-22T21:17:55.345ZINFOcrucible-agent (worker): commit
312025-05-22T21:17:55.351ZINFOcrucible-agent (worker): ok!
322025-05-22T21:17:55.357ZINFOcrucible-agent (worker): SMF ok!
332025-05-22T21:17:55.363ZINFOcrucible-agent (dropshot): request completed latency_us = 55 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:41579 req_id = d1b3c1ef-8ca0-42f1-9f61-2c951bcf0e44 response_code = 200 uri = /crucible/0/regions
342025-05-22T21:17:56.172ZINFOcrucible-agent (dropshot): request completed latency_us = 87 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:41579 req_id = e2b3b5f9-4a62-403f-996e-c899b5a0ab42 response_code = 200 uri = /crucible/0/regions/265c9bd3-fd38-4927-8cb0-8004352d4b6b
352025-05-22T21:17:56.179ZINFOcrucible-agent (datafile): path is "/data/regions/265c9bd3-fd38-4927-8cb0-8004352d4b6b"
362025-05-22T21:17:56.192ZINFOcrucible-agent (datafile): dataset is oxp_d3f1a944-1600-4eae-a191-e371b04a8674/crucible/regions/265c9bd3-fd38-4927-8cb0-8004352d4b6b
372025-05-22T21:17:56.203ZINFOcrucible-agent (datafile): region 265c9bd3-fd38-4927-8cb0-8004352d4b6b state: Created -> Tombstoned
382025-05-22T21:17:56.203ZINFOcrucible-agent (worker): applying SMF actions before removal...
392025-05-22T21:17:56.203ZINFOcrucible-agent (dropshot): request completed latency_us = 24209 local_addr = [fd00:1122:3344:101::17]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::c]:41579 req_id = 09ffc37b-c382-458e-b8e2-eb90e63e4a17 response_code = 204 uri = /crucible/0/regions/265c9bd3-fd38-4927-8cb0-8004352d4b6b
402025-05-22T21:17:56.204ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-265c9bd3-fd38-4927-8cb0-8004352d4b6b (instance states: (Some(Online), None))
412025-05-22T21:17:56.207ZINFOcrucible-agent (dropshot): request completed latency_us = 32 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:41579 req_id = a99056fa-e0c9-47ff-b1b4-d4682e99ce98 response_code = 200 uri = /crucible/0/regions/265c9bd3-fd38-4927-8cb0-8004352d4b6b
422025-05-22T21:17:56.210ZINFOcrucible-agent (worker): SMF ok!
432025-05-22T21:17:56.221ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_d3f1a944-1600-4eae-a191-e371b04a8674/crucible/regions/265c9bd3-fd38-4927-8cb0-8004352d4b6b" region = 265c9bd3-fd38-4927-8cb0-8004352d4b6b
442025-05-22T21:17:56.294ZINFOcrucible-agent (datafile): region 265c9bd3-fd38-4927-8cb0-8004352d4b6b state: Tombstoned -> Destroyed
452025-05-22T21:17:56.348ZINFOcrucible-agent (dropshot): request completed latency_us = 60 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:41579 req_id = c08b1c11-d72d-43b0-9d2c-3aba0ed1ca02 response_code = 200 uri = /crucible/0/regions/265c9bd3-fd38-4927-8cb0-8004352d4b6b
462025-05-22T21:28:23.634ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::17]:32345 remote_addr = [fd00:1122:3344:101::c]:62386
472025-05-22T21:28:23.634ZINFOcrucible-agent (datafile): region 5e1c0c6e-de4c-4608-9fe0-7bc2b43e0800 state: Requested
482025-05-22T21:28:23.635ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
492025-05-22T21:28:23.635ZINFOcrucible-agent (dropshot): request completed latency_us = 955 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:62386 req_id = 3fc3ae2e-06f2-42e2-b954-e1e8954cf687 response_code = 200 uri = /crucible/0/regions
502025-05-22T21:28:23.647ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_d3f1a944-1600-4eae-a191-e371b04a8674/crucible/regions/5e1c0c6e-de4c-4608-9fe0-7bc2b43e0800
512025-05-22T21:28:23.647ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_d3f1a944-1600-4eae-a191-e371b04a8674/crucible/regions/5e1c0c6e-de4c-4608-9fe0-7bc2b43e0800
522025-05-22T21:28:23.694ZINFOcrucible-agent (worker): creating region Region { id: RegionId("5e1c0c6e-de4c-4608-9fe0-7bc2b43e0800"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/5e1c0c6e-de4c-4608-9fe0-7bc2b43e0800" region = 5e1c0c6e-de4c-4608-9fe0-7bc2b43e0800
532025-05-22T21:28:23.694ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "5e1c0c6e-de4c-4608-9fe0-7bc2b43e0800" "--data" "/data/regions/5e1c0c6e-de4c-4608-9fe0-7bc2b43e0800" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 5e1c0c6e-de4c-4608-9fe0-7bc2b43e0800
542025-05-22T21:28:23.780ZINFOcrucible-agent (worker): region files created ok region = 5e1c0c6e-de4c-4608-9fe0-7bc2b43e0800
552025-05-22T21:28:23.780ZINFOcrucible-agent (datafile): region 5e1c0c6e-de4c-4608-9fe0-7bc2b43e0800 state: Requested -> Created
562025-05-22T21:28:23.780ZINFOcrucible-agent (worker): applying SMF actions post create...
572025-05-22T21:28:23.782ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-5e1c0c6e-de4c-4608-9fe0-7bc2b43e0800
582025-05-22T21:28:23.792ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-5e1c0c6e-de4c-4608-9fe0-7bc2b43e0800
592025-05-22T21:28:23.811ZINFOcrucible-agent (worker): creating config property group
602025-05-22T21:28:23.814ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-5e1c0c6e-de4c-4608-9fe0-7bc2b43e0800
612025-05-22T21:28:23.814ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/5e1c0c6e-de4c-4608-9fe0-7bc2b43e0800
622025-05-22T21:28:23.814ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
632025-05-22T21:28:23.814ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
642025-05-22T21:28:23.814ZINFOcrucible-agent (worker): commit
652025-05-22T21:28:23.818ZINFOcrucible-agent (worker): ok!
662025-05-22T21:28:23.822ZINFOcrucible-agent (worker): SMF ok!
672025-05-22T21:28:23.959ZINFOcrucible-agent (dropshot): request completed latency_us = 90 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:62386 req_id = 2ac0b683-35ac-4a3f-b534-3df542a74cc7 response_code = 200 uri = /crucible/0/regions