Max level shown:
1 [ Mar 4 03:17:42 Enabled. ]
2 [ Mar 4 03:17:42 Rereading configuration. ]
3 [ Mar 4 03:17:49 Rereading configuration. ]
4 [ Mar 4 03:17:57 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_5fc2c97d-a3e5-4273-98a3-7dcd202c0a40/crucible -l [fd00:1122:3344:101::12]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62025-03-04T03:17:57.596ZINFOcrucible-agent: dataset: "oxp_5fc2c97d-a3e5-4273-98a3-7dcd202c0a40/crucible"
72025-03-04T03:17:57.598ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::12]:32345
82025-03-04T03:17:57.598ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92025-03-04T03:17:57.615ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102025-03-04T03:17:57.758ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::12]:32345
112025-03-04T03:18:47.132ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::b]:60151
122025-03-04T03:18:47.179ZINFOcrucible-agent (datafile): region 47a8f417-ae55-43f9-a510-9b8a9f0dd722 state: Requested
132025-03-04T03:18:47.182ZINFOcrucible-agent (dropshot): request completed latency_us = 21721 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:60151 req_id = af701db1-c72d-4771-a84d-7faca0d164e8 response_code = 200 uri = /crucible/0/regions
142025-03-04T03:18:47.182ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
152025-03-04T03:18:47.191ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_5fc2c97d-a3e5-4273-98a3-7dcd202c0a40/crucible/regions/47a8f417-ae55-43f9-a510-9b8a9f0dd722
162025-03-04T03:18:47.191ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_5fc2c97d-a3e5-4273-98a3-7dcd202c0a40/crucible/regions/47a8f417-ae55-43f9-a510-9b8a9f0dd722
172025-03-04T03:18:47.268ZINFOcrucible-agent (worker): creating region Region { id: RegionId("47a8f417-ae55-43f9-a510-9b8a9f0dd722"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/47a8f417-ae55-43f9-a510-9b8a9f0dd722" region = 47a8f417-ae55-43f9-a510-9b8a9f0dd722
182025-03-04T03:18:47.271ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "47a8f417-ae55-43f9-a510-9b8a9f0dd722" "--data" "/data/regions/47a8f417-ae55-43f9-a510-9b8a9f0dd722" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 47a8f417-ae55-43f9-a510-9b8a9f0dd722
192025-03-04T03:18:47.433ZINFOcrucible-agent (worker): region files created ok region = 47a8f417-ae55-43f9-a510-9b8a9f0dd722
202025-03-04T03:18:47.433ZINFOcrucible-agent (datafile): region 47a8f417-ae55-43f9-a510-9b8a9f0dd722 state: Requested -> Created
212025-03-04T03:18:47.433ZINFOcrucible-agent (worker): applying SMF actions post create...
222025-03-04T03:18:47.434ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-47a8f417-ae55-43f9-a510-9b8a9f0dd722
232025-03-04T03:18:47.440ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-47a8f417-ae55-43f9-a510-9b8a9f0dd722
242025-03-04T03:18:47.454ZINFOcrucible-agent (worker): creating config property group
252025-03-04T03:18:47.461ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-47a8f417-ae55-43f9-a510-9b8a9f0dd722
262025-03-04T03:18:47.462ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/47a8f417-ae55-43f9-a510-9b8a9f0dd722
272025-03-04T03:18:47.462ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
282025-03-04T03:18:47.462ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
292025-03-04T03:18:47.462ZINFOcrucible-agent (worker): commit
302025-03-04T03:18:47.467ZINFOcrucible-agent (worker): ok!
312025-03-04T03:18:47.481ZINFOcrucible-agent (worker): SMF ok!
322025-03-04T03:18:47.504ZINFOcrucible-agent (dropshot): request completed latency_us = 55 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:60151 req_id = db1ec83c-9ea3-479e-b659-323b6bada1db response_code = 200 uri = /crucible/0/regions
332025-03-04T03:29:03.988ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::b]:62420
342025-03-04T03:29:03.991ZINFOcrucible-agent (dropshot): request completed latency_us = 2822 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:62420 req_id = b0b5acc4-8893-4165-a8f2-2aa91ec769ec response_code = 200 uri = /crucible/0/regions/47a8f417-ae55-43f9-a510-9b8a9f0dd722
352025-03-04T03:29:03.998ZINFOcrucible-agent (datafile): path is "/data/regions/47a8f417-ae55-43f9-a510-9b8a9f0dd722"
362025-03-04T03:29:04.008ZINFOcrucible-agent (datafile): dataset is oxp_5fc2c97d-a3e5-4273-98a3-7dcd202c0a40/crucible/regions/47a8f417-ae55-43f9-a510-9b8a9f0dd722
372025-03-04T03:29:04.018ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_5fc2c97d-a3e5-4273-98a3-7dcd202c0a40/crucible/regions/47a8f417-ae55-43f9-a510-9b8a9f0dd722@55fb3511-44f6-4fe1-ac53-f4bc63e8d6b2
382025-03-04T03:29:04.018ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_5fc2c97d-a3e5-4273-98a3-7dcd202c0a40/crucible/regions/47a8f417-ae55-43f9-a510-9b8a9f0dd722", "55fb3511-44f6-4fe1-ac53-f4bc63e8d6b2"]
392025-03-04T03:29:04.029ZINFOcrucible-agent (dropshot): request completed latency_us = 32962 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:62420 req_id = 455df652-e63d-416d-84d2-4bc5bde1be39 response_code = 200 uri = /crucible/0/regions/47a8f417-ae55-43f9-a510-9b8a9f0dd722/snapshots/55fb3511-44f6-4fe1-ac53-f4bc63e8d6b2
402025-03-04T03:29:04.032ZINFOcrucible-agent (datafile): path is "/data/regions/47a8f417-ae55-43f9-a510-9b8a9f0dd722"
412025-03-04T03:29:04.041ZINFOcrucible-agent (datafile): dataset is oxp_5fc2c97d-a3e5-4273-98a3-7dcd202c0a40/crucible/regions/47a8f417-ae55-43f9-a510-9b8a9f0dd722
422025-03-04T03:29:04.049ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_5fc2c97d-a3e5-4273-98a3-7dcd202c0a40/crucible/regions/47a8f417-ae55-43f9-a510-9b8a9f0dd722@55fb3511-44f6-4fe1-ac53-f4bc63e8d6b2
432025-03-04T03:29:04.049ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_5fc2c97d-a3e5-4273-98a3-7dcd202c0a40/crucible/regions/47a8f417-ae55-43f9-a510-9b8a9f0dd722", "55fb3511-44f6-4fe1-ac53-f4bc63e8d6b2"]
442025-03-04T03:29:04.058ZINFOcrucible-agent (datafile): requesting running snapshot 47a8f417-ae55-43f9-a510-9b8a9f0dd722-55fb3511-44f6-4fe1-ac53-f4bc63e8d6b2 state: Requested
452025-03-04T03:29:04.059ZINFOcrucible-agent (worker): applying SMF actions for region 47a8f417-ae55-43f9-a510-9b8a9f0dd722 running snapshot 55fb3511-44f6-4fe1-ac53-f4bc63e8d6b2 (state Requested)...
462025-03-04T03:29:04.059ZINFOcrucible-agent (dropshot): request completed latency_us = 27242 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:62420 req_id = 2c09124f-8660-43ce-a517-a73ade7e3bde response_code = 200 uri = /crucible/0/regions/47a8f417-ae55-43f9-a510-9b8a9f0dd722/snapshots/55fb3511-44f6-4fe1-ac53-f4bc63e8d6b2/run
472025-03-04T03:29:04.064ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-47a8f417-ae55-43f9-a510-9b8a9f0dd722-55fb3511-44f6-4fe1-ac53-f4bc63e8d6b2
482025-03-04T03:29:04.068ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-47a8f417-ae55-43f9-a510-9b8a9f0dd722-55fb3511-44f6-4fe1-ac53-f4bc63e8d6b2
492025-03-04T03:29:04.078ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-47a8f417-ae55-43f9-a510-9b8a9f0dd722-55fb3511-44f6-4fe1-ac53-f4bc63e8d6b2
502025-03-04T03:29:04.078ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/47a8f417-ae55-43f9-a510-9b8a9f0dd722/.zfs/snapshot/55fb3511-44f6-4fe1-ac53-f4bc63e8d6b2
512025-03-04T03:29:04.078ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
522025-03-04T03:29:04.079ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
532025-03-04T03:29:04.079ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
542025-03-04T03:29:04.079ZINFOcrucible-agent (worker): commit
552025-03-04T03:29:04.083ZINFOcrucible-agent (worker): ok!
562025-03-04T03:29:04.087ZINFOcrucible-agent (worker): SMF ok!
572025-03-04T03:29:04.087ZINFOcrucible-agent (datafile): region 47a8f417-ae55-43f9-a510-9b8a9f0dd722 running snapshot 55fb3511-44f6-4fe1-ac53-f4bc63e8d6b2 state: Requested -> Created
582025-03-04T03:29:04.413ZINFOcrucible-agent (datafile): path is "/data/regions/47a8f417-ae55-43f9-a510-9b8a9f0dd722"
592025-03-04T03:29:04.422ZINFOcrucible-agent (datafile): dataset is oxp_5fc2c97d-a3e5-4273-98a3-7dcd202c0a40/crucible/regions/47a8f417-ae55-43f9-a510-9b8a9f0dd722
602025-03-04T03:29:04.430ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_5fc2c97d-a3e5-4273-98a3-7dcd202c0a40/crucible/regions/47a8f417-ae55-43f9-a510-9b8a9f0dd722@55fb3511-44f6-4fe1-ac53-f4bc63e8d6b2
612025-03-04T03:29:04.430ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_5fc2c97d-a3e5-4273-98a3-7dcd202c0a40/crucible/regions/47a8f417-ae55-43f9-a510-9b8a9f0dd722", "55fb3511-44f6-4fe1-ac53-f4bc63e8d6b2"]
622025-03-04T03:29:04.439ZINFOcrucible-agent (dropshot): request completed latency_us = 25437 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:62420 req_id = 2832ea87-d824-462e-9b8e-203270319ba8 response_code = 200 uri = /crucible/0/regions/47a8f417-ae55-43f9-a510-9b8a9f0dd722/snapshots/55fb3511-44f6-4fe1-ac53-f4bc63e8d6b2/run
632025-03-04T03:29:08.611ZINFOcrucible-agent (datafile): region 6ef94d21-55ab-42c4-ab1c-0d240bf901cd state: Requested
642025-03-04T03:29:08.611ZINFOcrucible-agent (dropshot): request completed latency_us = 281 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:62420 req_id = 478dfcbc-58b4-4f95-b34c-d138d918df77 response_code = 200 uri = /crucible/0/regions
652025-03-04T03:29:08.611ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
662025-03-04T03:29:08.621ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_5fc2c97d-a3e5-4273-98a3-7dcd202c0a40/crucible/regions/6ef94d21-55ab-42c4-ab1c-0d240bf901cd
672025-03-04T03:29:08.621ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_5fc2c97d-a3e5-4273-98a3-7dcd202c0a40/crucible/regions/6ef94d21-55ab-42c4-ab1c-0d240bf901cd
682025-03-04T03:29:08.695ZINFOcrucible-agent (worker): creating region Region { id: RegionId("6ef94d21-55ab-42c4-ab1c-0d240bf901cd"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19002, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/6ef94d21-55ab-42c4-ab1c-0d240bf901cd" region = 6ef94d21-55ab-42c4-ab1c-0d240bf901cd
692025-03-04T03:29:08.698ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "6ef94d21-55ab-42c4-ab1c-0d240bf901cd" "--data" "/data/regions/6ef94d21-55ab-42c4-ab1c-0d240bf901cd" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 6ef94d21-55ab-42c4-ab1c-0d240bf901cd
702025-03-04T03:29:08.767ZINFOcrucible-agent (dropshot): request completed latency_us = 70 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:62420 req_id = 677da241-61a3-4555-823e-159489df104b response_code = 200 uri = /crucible/0/regions
712025-03-04T03:29:08.786ZINFOcrucible-agent (worker): region files created ok region = 6ef94d21-55ab-42c4-ab1c-0d240bf901cd
722025-03-04T03:29:08.786ZINFOcrucible-agent (datafile): region 6ef94d21-55ab-42c4-ab1c-0d240bf901cd state: Requested -> Created
732025-03-04T03:29:08.786ZINFOcrucible-agent (worker): applying SMF actions post create...
742025-03-04T03:29:08.790ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-6ef94d21-55ab-42c4-ab1c-0d240bf901cd
752025-03-04T03:29:08.802ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-6ef94d21-55ab-42c4-ab1c-0d240bf901cd
762025-03-04T03:29:08.810ZINFOcrucible-agent (worker): creating config property group
772025-03-04T03:29:08.814ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-6ef94d21-55ab-42c4-ab1c-0d240bf901cd
782025-03-04T03:29:08.814ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/6ef94d21-55ab-42c4-ab1c-0d240bf901cd
792025-03-04T03:29:08.814ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
802025-03-04T03:29:08.814ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
812025-03-04T03:29:08.814ZINFOcrucible-agent (worker): commit
822025-03-04T03:29:08.818ZINFOcrucible-agent (worker): ok!
832025-03-04T03:29:08.824ZINFOcrucible-agent (worker): SMF ok!
842025-03-04T03:29:09.288ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::b]:63231
852025-03-04T03:29:09.289ZINFOcrucible-agent (dropshot): request completed latency_us = 46 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:63231 req_id = bfa779ea-fd93-4d1f-93b1-43ba4fc42730 response_code = 200 uri = /crucible/0/regions/6ef94d21-55ab-42c4-ab1c-0d240bf901cd
862025-03-04T03:29:09.429ZINFOcrucible-agent (dropshot): request completed latency_us = 66 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:62420 req_id = 1decae66-7129-44ff-8b1a-e1453c40aa22 response_code = 200 uri = /crucible/0/regions
872025-03-04T03:29:35.024ZINFOcrucible-agent (dropshot): request completed latency_us = 52 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:62420 req_id = 121946b0-f4cc-4dff-b345-2cbd7eb84698 response_code = 200 uri = /crucible/0/regions/6ef94d21-55ab-42c4-ab1c-0d240bf901cd
882025-03-04T03:29:35.028ZINFOcrucible-agent (datafile): path is "/data/regions/6ef94d21-55ab-42c4-ab1c-0d240bf901cd"
892025-03-04T03:29:35.040ZINFOcrucible-agent (datafile): dataset is oxp_5fc2c97d-a3e5-4273-98a3-7dcd202c0a40/crucible/regions/6ef94d21-55ab-42c4-ab1c-0d240bf901cd
902025-03-04T03:29:35.050ZINFOcrucible-agent (datafile): region 6ef94d21-55ab-42c4-ab1c-0d240bf901cd state: Created -> Tombstoned
912025-03-04T03:29:35.050ZINFOcrucible-agent (worker): applying SMF actions before removal...
922025-03-04T03:29:35.050ZINFOcrucible-agent (dropshot): request completed latency_us = 22072 local_addr = [fd00:1122:3344:101::12]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::b]:62420 req_id = 0c249b95-a57f-49f4-b02b-1ae490f8e52f response_code = 204 uri = /crucible/0/regions/6ef94d21-55ab-42c4-ab1c-0d240bf901cd
932025-03-04T03:29:35.051ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-6ef94d21-55ab-42c4-ab1c-0d240bf901cd (instance states: (Some(Online), None))
942025-03-04T03:29:35.054ZINFOcrucible-agent (dropshot): request completed latency_us = 20 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:62420 req_id = 899468b5-91ec-43cc-93f6-ee342c5572f6 response_code = 200 uri = /crucible/0/regions/6ef94d21-55ab-42c4-ab1c-0d240bf901cd
952025-03-04T03:29:35.057ZINFOcrucible-agent (worker): SMF ok!
962025-03-04T03:29:35.068ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_5fc2c97d-a3e5-4273-98a3-7dcd202c0a40/crucible/regions/6ef94d21-55ab-42c4-ab1c-0d240bf901cd" region = 6ef94d21-55ab-42c4-ab1c-0d240bf901cd
972025-03-04T03:29:35.110ZINFOcrucible-agent (dropshot): request completed latency_us = 81 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:62420 req_id = 88a5f6f9-a4bb-4c97-8cf1-445ae81dfe22 response_code = 200 uri = /crucible/0/regions/6ef94d21-55ab-42c4-ab1c-0d240bf901cd
982025-03-04T03:29:35.130ZINFOcrucible-agent (datafile): region 6ef94d21-55ab-42c4-ab1c-0d240bf901cd state: Tombstoned -> Destroyed
992025-03-04T03:29:35.295ZINFOcrucible-agent (dropshot): request completed latency_us = 29 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:62420 req_id = ff390035-bf9d-49aa-bda5-151ea7c5d9ad response_code = 200 uri = /crucible/0/regions/6ef94d21-55ab-42c4-ab1c-0d240bf901cd