Max level shown:
1 [ Aug 1 15:57:28 Enabled. ]
2 [ Aug 1 15:57:28 Rereading configuration. ]
3 [ Aug 1 15:57:30 Rereading configuration. ]
4 [ Aug 1 15:57:37 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_6a104170-7e1f-43ee-b4ef-55167ed557e6/crucible -l [fd00:1122:3344:101::17]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62025-08-01T15:57:38.293ZINFOcrucible-agent: dataset: "oxp_6a104170-7e1f-43ee-b4ef-55167ed557e6/crucible"
72025-08-01T15:57:38.296ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::17]:32345
82025-08-01T15:57:38.297ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92025-08-01T15:57:38.314ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102025-08-01T15:57:38.417ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::17]:32345
112025-08-01T15:58:41.028ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::17]:32345 remote_addr = [fd00:1122:3344:101::c]:49154
122025-08-01T15:58:41.053ZINFOcrucible-agent (datafile): region 68d7b555-6222-4016-aada-e9fc7213186a state: Requested
132025-08-01T15:58:41.058ZINFOcrucible-agent (dropshot): request completed latency_us = 17937 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:49154 req_id = 3e10a1bc-03bb-4e31-82f1-9c10d15f94e3 response_code = 200 uri = /crucible/0/regions
142025-08-01T15:58:41.060ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
152025-08-01T15:58:41.071ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_6a104170-7e1f-43ee-b4ef-55167ed557e6/crucible/regions/68d7b555-6222-4016-aada-e9fc7213186a
162025-08-01T15:58:41.072ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_6a104170-7e1f-43ee-b4ef-55167ed557e6/crucible/regions/68d7b555-6222-4016-aada-e9fc7213186a
172025-08-01T15:58:41.146ZINFOcrucible-agent (worker): creating region Region { id: RegionId("68d7b555-6222-4016-aada-e9fc7213186a"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/68d7b555-6222-4016-aada-e9fc7213186a" region = 68d7b555-6222-4016-aada-e9fc7213186a
182025-08-01T15:58:41.148ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "68d7b555-6222-4016-aada-e9fc7213186a" "--data" "/data/regions/68d7b555-6222-4016-aada-e9fc7213186a" "--block-size" "512" "--extent-size" "131072" "--extent-count" "16" "--encrypted" region = 68d7b555-6222-4016-aada-e9fc7213186a
192025-08-01T15:58:41.214ZINFOcrucible-agent (dropshot): request completed latency_us = 76 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:49154 req_id = c5d3dc1e-f6fe-44d9-a59c-0223cfea2073 response_code = 200 uri = /crucible/0/regions
202025-08-01T15:58:41.442ZINFOcrucible-agent (worker): region files created ok region = 68d7b555-6222-4016-aada-e9fc7213186a
212025-08-01T15:58:41.442ZINFOcrucible-agent (datafile): region 68d7b555-6222-4016-aada-e9fc7213186a state: Requested -> Created
222025-08-01T15:58:41.442ZINFOcrucible-agent (worker): applying SMF actions post create...
232025-08-01T15:58:41.442ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-68d7b555-6222-4016-aada-e9fc7213186a
242025-08-01T15:58:41.448ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-68d7b555-6222-4016-aada-e9fc7213186a
252025-08-01T15:58:41.462ZINFOcrucible-agent (worker): creating config property group
262025-08-01T15:58:41.465ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-68d7b555-6222-4016-aada-e9fc7213186a
272025-08-01T15:58:41.466ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/68d7b555-6222-4016-aada-e9fc7213186a
282025-08-01T15:58:41.466ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
292025-08-01T15:58:41.466ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
302025-08-01T15:58:41.466ZINFOcrucible-agent (worker): commit
312025-08-01T15:58:41.470ZINFOcrucible-agent (worker): ok!
322025-08-01T15:58:41.481ZINFOcrucible-agent (worker): SMF ok!
332025-08-01T15:58:41.573ZINFOcrucible-agent (dropshot): request completed latency_us = 55 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:49154 req_id = 32601445-8b17-45ea-bf39-a8fa5f64f290 response_code = 200 uri = /crucible/0/regions
342025-08-01T15:58:42.202ZINFOcrucible-agent (dropshot): request completed latency_us = 88 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:49154 req_id = 32630c4e-4993-47d8-949a-fe3a30a56e03 response_code = 200 uri = /crucible/0/regions/68d7b555-6222-4016-aada-e9fc7213186a
352025-08-01T15:58:42.204ZINFOcrucible-agent (datafile): path is "/data/regions/68d7b555-6222-4016-aada-e9fc7213186a"
362025-08-01T15:58:42.216ZINFOcrucible-agent (datafile): dataset is oxp_6a104170-7e1f-43ee-b4ef-55167ed557e6/crucible/regions/68d7b555-6222-4016-aada-e9fc7213186a
372025-08-01T15:58:42.227ZINFOcrucible-agent (datafile): region 68d7b555-6222-4016-aada-e9fc7213186a state: Created -> Tombstoned
382025-08-01T15:58:42.227ZINFOcrucible-agent (worker): applying SMF actions before removal...
392025-08-01T15:58:42.227ZINFOcrucible-agent (dropshot): request completed latency_us = 22368 local_addr = [fd00:1122:3344:101::17]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::c]:49154 req_id = 90d12d76-5b81-47d6-9f11-fe5b0fdd9323 response_code = 204 uri = /crucible/0/regions/68d7b555-6222-4016-aada-e9fc7213186a
402025-08-01T15:58:42.228ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-68d7b555-6222-4016-aada-e9fc7213186a (instance states: (Some(Online), None))
412025-08-01T15:58:42.230ZINFOcrucible-agent (dropshot): request completed latency_us = 27 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:49154 req_id = 165e3fa0-5b43-4568-b77e-58598b62714d response_code = 200 uri = /crucible/0/regions/68d7b555-6222-4016-aada-e9fc7213186a
422025-08-01T15:58:42.233ZINFOcrucible-agent (worker): SMF ok!
432025-08-01T15:58:42.245ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_6a104170-7e1f-43ee-b4ef-55167ed557e6/crucible/regions/68d7b555-6222-4016-aada-e9fc7213186a" region = 68d7b555-6222-4016-aada-e9fc7213186a
442025-08-01T15:58:42.307ZINFOcrucible-agent (dropshot): request completed latency_us = 77 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:49154 req_id = 36e0a457-d398-4afd-9842-79fe0366a9b6 response_code = 200 uri = /crucible/0/regions/68d7b555-6222-4016-aada-e9fc7213186a
452025-08-01T15:58:42.317ZINFOcrucible-agent (datafile): region 68d7b555-6222-4016-aada-e9fc7213186a state: Tombstoned -> Destroyed
462025-08-01T15:58:42.384ZINFOcrucible-agent (dropshot): request completed latency_us = 35 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:49154 req_id = 11661748-b1d4-4839-8426-2976849355c9 response_code = 200 uri = /crucible/0/regions/68d7b555-6222-4016-aada-e9fc7213186a
472025-08-01T15:58:44.256ZINFOcrucible-agent (datafile): region a1787730-7ff1-4075-814b-abf2e5ba57ff state: Requested
482025-08-01T15:58:44.257ZINFOcrucible-agent (dropshot): request completed latency_us = 314 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:49154 req_id = e06fdbea-0c6d-4f55-9845-3d026bc00f81 response_code = 200 uri = /crucible/0/regions
492025-08-01T15:58:44.257ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
502025-08-01T15:58:44.267ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_6a104170-7e1f-43ee-b4ef-55167ed557e6/crucible/regions/a1787730-7ff1-4075-814b-abf2e5ba57ff
512025-08-01T15:58:44.267ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_6a104170-7e1f-43ee-b4ef-55167ed557e6/crucible/regions/a1787730-7ff1-4075-814b-abf2e5ba57ff
522025-08-01T15:58:44.334ZINFOcrucible-agent (worker): creating region Region { id: RegionId("a1787730-7ff1-4075-814b-abf2e5ba57ff"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/a1787730-7ff1-4075-814b-abf2e5ba57ff" region = a1787730-7ff1-4075-814b-abf2e5ba57ff
532025-08-01T15:58:44.336ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "a1787730-7ff1-4075-814b-abf2e5ba57ff" "--data" "/data/regions/a1787730-7ff1-4075-814b-abf2e5ba57ff" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = a1787730-7ff1-4075-814b-abf2e5ba57ff
542025-08-01T15:58:44.412ZINFOcrucible-agent (worker): region files created ok region = a1787730-7ff1-4075-814b-abf2e5ba57ff
552025-08-01T15:58:44.412ZINFOcrucible-agent (datafile): region a1787730-7ff1-4075-814b-abf2e5ba57ff state: Requested -> Created
562025-08-01T15:58:44.412ZINFOcrucible-agent (worker): applying SMF actions post create...
572025-08-01T15:58:44.413ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-a1787730-7ff1-4075-814b-abf2e5ba57ff
582025-08-01T15:58:44.416ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-a1787730-7ff1-4075-814b-abf2e5ba57ff
592025-08-01T15:58:44.424ZINFOcrucible-agent (worker): creating config property group
602025-08-01T15:58:44.428ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-a1787730-7ff1-4075-814b-abf2e5ba57ff
612025-08-01T15:58:44.428ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/a1787730-7ff1-4075-814b-abf2e5ba57ff
622025-08-01T15:58:44.429ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
632025-08-01T15:58:44.429ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
642025-08-01T15:58:44.429ZINFOcrucible-agent (worker): commit
652025-08-01T15:58:44.433ZINFOcrucible-agent (worker): ok!
662025-08-01T15:58:44.438ZINFOcrucible-agent (worker): SMF ok!
672025-08-01T15:58:44.506ZINFOcrucible-agent (dropshot): request completed latency_us = 56 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:49154 req_id = 099e8c90-e5db-4b7b-abfd-9994e6b141a6 response_code = 200 uri = /crucible/0/regions