Max level shown:
1 [ May 22 21:17:24 Enabled. ]
2 [ May 22 21:17:24 Rereading configuration. ]
3 [ May 22 21:17:26 Rereading configuration. ]
4 [ May 22 21:17:29 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_c0db3660-af9b-47a3-b4e0-f5b172db3f55/crucible -l [fd00:1122:3344:101::19]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62025-05-22T21:17:29.940ZINFOcrucible-agent: dataset: "oxp_c0db3660-af9b-47a3-b4e0-f5b172db3f55/crucible"
72025-05-22T21:17:29.943ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::19]:32345
82025-05-22T21:17:29.943ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92025-05-22T21:17:29.956ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102025-05-22T21:17:30.034ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::19]:32345
112025-05-22T21:17:58.202ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::19]:32345 remote_addr = [fd00:1122:3344:101::c]:61398
122025-05-22T21:17:58.218ZINFOcrucible-agent (datafile): region a8ffce74-77d3-49af-ba5c-e99858a2dfe3 state: Requested
132025-05-22T21:17:58.221ZINFOcrucible-agent (dropshot): request completed latency_us = 14290 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:61398 req_id = 47436e58-b617-4bc6-b20b-bdb83b4b43c1 response_code = 200 uri = /crucible/0/regions
142025-05-22T21:17:58.224ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
152025-05-22T21:17:58.234ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_c0db3660-af9b-47a3-b4e0-f5b172db3f55/crucible/regions/a8ffce74-77d3-49af-ba5c-e99858a2dfe3
162025-05-22T21:17:58.234ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_c0db3660-af9b-47a3-b4e0-f5b172db3f55/crucible/regions/a8ffce74-77d3-49af-ba5c-e99858a2dfe3
172025-05-22T21:17:58.295ZINFOcrucible-agent (worker): creating region Region { id: RegionId("a8ffce74-77d3-49af-ba5c-e99858a2dfe3"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/a8ffce74-77d3-49af-ba5c-e99858a2dfe3" region = a8ffce74-77d3-49af-ba5c-e99858a2dfe3
182025-05-22T21:17:58.295ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "a8ffce74-77d3-49af-ba5c-e99858a2dfe3" "--data" "/data/regions/a8ffce74-77d3-49af-ba5c-e99858a2dfe3" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = a8ffce74-77d3-49af-ba5c-e99858a2dfe3
192025-05-22T21:17:58.439ZINFOcrucible-agent (dropshot): request completed latency_us = 72 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:61398 req_id = 7cb872fd-475f-4cfd-aab9-9a9b2b899ee1 response_code = 200 uri = /crucible/0/regions
202025-05-22T21:17:58.602ZINFOcrucible-agent (worker): region files created ok region = a8ffce74-77d3-49af-ba5c-e99858a2dfe3
212025-05-22T21:17:58.602ZINFOcrucible-agent (datafile): region a8ffce74-77d3-49af-ba5c-e99858a2dfe3 state: Requested -> Created
222025-05-22T21:17:58.602ZINFOcrucible-agent (worker): applying SMF actions post create...
232025-05-22T21:17:58.603ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-a8ffce74-77d3-49af-ba5c-e99858a2dfe3
242025-05-22T21:17:58.609ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-a8ffce74-77d3-49af-ba5c-e99858a2dfe3
252025-05-22T21:17:58.627ZINFOcrucible-agent (worker): creating config property group
262025-05-22T21:17:58.632ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-a8ffce74-77d3-49af-ba5c-e99858a2dfe3
272025-05-22T21:17:58.632ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/a8ffce74-77d3-49af-ba5c-e99858a2dfe3
282025-05-22T21:17:58.632ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
292025-05-22T21:17:58.632ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
302025-05-22T21:17:58.632ZINFOcrucible-agent (worker): commit
312025-05-22T21:17:58.637ZINFOcrucible-agent (worker): ok!
322025-05-22T21:17:58.643ZINFOcrucible-agent (worker): SMF ok!
332025-05-22T21:17:58.797ZINFOcrucible-agent (dropshot): request completed latency_us = 46 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:61398 req_id = ce17d726-9d0e-4f0e-b6ea-0a10704a3d5e response_code = 200 uri = /crucible/0/regions
342025-05-22T21:28:24.654ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::19]:32345 remote_addr = [fd00:1122:3344:101::c]:42072
352025-05-22T21:28:24.658ZINFOcrucible-agent (dropshot): request completed latency_us = 3479 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:42072 req_id = 247b61fb-df09-4226-904a-35af45d6bdf5 response_code = 200 uri = /crucible/0/regions/a8ffce74-77d3-49af-ba5c-e99858a2dfe3
362025-05-22T21:28:24.660ZINFOcrucible-agent (datafile): path is "/data/regions/a8ffce74-77d3-49af-ba5c-e99858a2dfe3"
372025-05-22T21:28:24.670ZINFOcrucible-agent (datafile): dataset is oxp_c0db3660-af9b-47a3-b4e0-f5b172db3f55/crucible/regions/a8ffce74-77d3-49af-ba5c-e99858a2dfe3
382025-05-22T21:28:24.678ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_c0db3660-af9b-47a3-b4e0-f5b172db3f55/crucible/regions/a8ffce74-77d3-49af-ba5c-e99858a2dfe3@da1a9ac6-2a61-425b-b61f-ad2b80183749
392025-05-22T21:28:24.678ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_c0db3660-af9b-47a3-b4e0-f5b172db3f55/crucible/regions/a8ffce74-77d3-49af-ba5c-e99858a2dfe3", "da1a9ac6-2a61-425b-b61f-ad2b80183749"]
402025-05-22T21:28:24.688ZINFOcrucible-agent (dropshot): request completed latency_us = 27874 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:42072 req_id = 8668d903-144b-4337-a60d-0d307c17b5bd response_code = 200 uri = /crucible/0/regions/a8ffce74-77d3-49af-ba5c-e99858a2dfe3/snapshots/da1a9ac6-2a61-425b-b61f-ad2b80183749
412025-05-22T21:28:24.694ZINFOcrucible-agent (datafile): path is "/data/regions/a8ffce74-77d3-49af-ba5c-e99858a2dfe3"
422025-05-22T21:28:24.703ZINFOcrucible-agent (datafile): dataset is oxp_c0db3660-af9b-47a3-b4e0-f5b172db3f55/crucible/regions/a8ffce74-77d3-49af-ba5c-e99858a2dfe3
432025-05-22T21:28:24.712ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_c0db3660-af9b-47a3-b4e0-f5b172db3f55/crucible/regions/a8ffce74-77d3-49af-ba5c-e99858a2dfe3@da1a9ac6-2a61-425b-b61f-ad2b80183749
442025-05-22T21:28:24.712ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_c0db3660-af9b-47a3-b4e0-f5b172db3f55/crucible/regions/a8ffce74-77d3-49af-ba5c-e99858a2dfe3", "da1a9ac6-2a61-425b-b61f-ad2b80183749"]
452025-05-22T21:28:24.721ZINFOcrucible-agent (datafile): requesting running snapshot a8ffce74-77d3-49af-ba5c-e99858a2dfe3-da1a9ac6-2a61-425b-b61f-ad2b80183749 state: Requested
462025-05-22T21:28:24.721ZINFOcrucible-agent (worker): applying SMF actions for region a8ffce74-77d3-49af-ba5c-e99858a2dfe3 running snapshot da1a9ac6-2a61-425b-b61f-ad2b80183749 (state Requested)...
472025-05-22T21:28:24.721ZINFOcrucible-agent (dropshot): request completed latency_us = 27843 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:42072 req_id = a29acaf7-b6fd-4b2e-b842-eaedba0b7179 response_code = 200 uri = /crucible/0/regions/a8ffce74-77d3-49af-ba5c-e99858a2dfe3/snapshots/da1a9ac6-2a61-425b-b61f-ad2b80183749/run
482025-05-22T21:28:24.737ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-a8ffce74-77d3-49af-ba5c-e99858a2dfe3-da1a9ac6-2a61-425b-b61f-ad2b80183749
492025-05-22T21:28:24.745ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-a8ffce74-77d3-49af-ba5c-e99858a2dfe3-da1a9ac6-2a61-425b-b61f-ad2b80183749
502025-05-22T21:28:24.757ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-a8ffce74-77d3-49af-ba5c-e99858a2dfe3-da1a9ac6-2a61-425b-b61f-ad2b80183749
512025-05-22T21:28:24.757ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/a8ffce74-77d3-49af-ba5c-e99858a2dfe3/.zfs/snapshot/da1a9ac6-2a61-425b-b61f-ad2b80183749
522025-05-22T21:28:24.757ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
532025-05-22T21:28:24.757ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
542025-05-22T21:28:24.757ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
552025-05-22T21:28:24.757ZINFOcrucible-agent (worker): commit
562025-05-22T21:28:24.761ZINFOcrucible-agent (worker): ok!
572025-05-22T21:28:24.765ZINFOcrucible-agent (worker): SMF ok!
582025-05-22T21:28:24.765ZINFOcrucible-agent (datafile): region a8ffce74-77d3-49af-ba5c-e99858a2dfe3 running snapshot da1a9ac6-2a61-425b-b61f-ad2b80183749 state: Requested -> Created
592025-05-22T21:28:24.955ZINFOcrucible-agent (datafile): path is "/data/regions/a8ffce74-77d3-49af-ba5c-e99858a2dfe3"
602025-05-22T21:28:24.964ZINFOcrucible-agent (datafile): dataset is oxp_c0db3660-af9b-47a3-b4e0-f5b172db3f55/crucible/regions/a8ffce74-77d3-49af-ba5c-e99858a2dfe3
612025-05-22T21:28:24.972ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_c0db3660-af9b-47a3-b4e0-f5b172db3f55/crucible/regions/a8ffce74-77d3-49af-ba5c-e99858a2dfe3@da1a9ac6-2a61-425b-b61f-ad2b80183749
622025-05-22T21:28:24.972ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_c0db3660-af9b-47a3-b4e0-f5b172db3f55/crucible/regions/a8ffce74-77d3-49af-ba5c-e99858a2dfe3", "da1a9ac6-2a61-425b-b61f-ad2b80183749"]
632025-05-22T21:28:24.980ZINFOcrucible-agent (dropshot): request completed latency_us = 24808 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:42072 req_id = 27469765-4a28-4aa3-a2c8-983a4da1bf96 response_code = 200 uri = /crucible/0/regions/a8ffce74-77d3-49af-ba5c-e99858a2dfe3/snapshots/da1a9ac6-2a61-425b-b61f-ad2b80183749/run