Max level shown:
1 [ Mar 4 03:17:41 Enabled. ]
2 [ Mar 4 03:17:41 Rereading configuration. ]
3 [ Mar 4 03:17:43 Rereading configuration. ]
4 [ Mar 4 03:17:56 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_8bf53b00-dedd-4516-be68-25e52e3985f5/crucible -l [fd00:1122:3344:101::1a]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62025-03-04T03:17:56.353ZINFOcrucible-agent: dataset: "oxp_8bf53b00-dedd-4516-be68-25e52e3985f5/crucible"
72025-03-04T03:17:56.356ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::1a]:32345
82025-03-04T03:17:56.356ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92025-03-04T03:17:56.372ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102025-03-04T03:17:56.485ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::1a]:32345
112025-03-04T03:18:43.213ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::1a]:32345 remote_addr = [fd00:1122:3344:101::b]:51011
122025-03-04T03:18:43.258ZINFOcrucible-agent (datafile): region 8522f663-bfd2-44db-b998-3fd36bc04757 state: Requested
132025-03-04T03:18:43.261ZINFOcrucible-agent (dropshot): request completed latency_us = 20074 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:51011 req_id = eecdf989-af52-46f6-850f-1e50baad7078 response_code = 200 uri = /crucible/0/regions
142025-03-04T03:18:43.261ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
152025-03-04T03:18:43.271ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_8bf53b00-dedd-4516-be68-25e52e3985f5/crucible/regions/8522f663-bfd2-44db-b998-3fd36bc04757
162025-03-04T03:18:43.271ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_8bf53b00-dedd-4516-be68-25e52e3985f5/crucible/regions/8522f663-bfd2-44db-b998-3fd36bc04757
172025-03-04T03:18:43.338ZINFOcrucible-agent (worker): creating region Region { id: RegionId("8522f663-bfd2-44db-b998-3fd36bc04757"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/8522f663-bfd2-44db-b998-3fd36bc04757" region = 8522f663-bfd2-44db-b998-3fd36bc04757
182025-03-04T03:18:43.338ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "8522f663-bfd2-44db-b998-3fd36bc04757" "--data" "/data/regions/8522f663-bfd2-44db-b998-3fd36bc04757" "--block-size" "512" "--extent-size" "131072" "--extent-count" "16" "--encrypted" region = 8522f663-bfd2-44db-b998-3fd36bc04757
192025-03-04T03:18:43.575ZINFOcrucible-agent (dropshot): request completed latency_us = 78 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:51011 req_id = 2ed79d48-20ed-4d86-80b3-63f9abc4fcf4 response_code = 200 uri = /crucible/0/regions
202025-03-04T03:18:43.633ZINFOcrucible-agent (worker): region files created ok region = 8522f663-bfd2-44db-b998-3fd36bc04757
212025-03-04T03:18:43.633ZINFOcrucible-agent (datafile): region 8522f663-bfd2-44db-b998-3fd36bc04757 state: Requested -> Created
222025-03-04T03:18:43.633ZINFOcrucible-agent (worker): applying SMF actions post create...
232025-03-04T03:18:43.633ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-8522f663-bfd2-44db-b998-3fd36bc04757
242025-03-04T03:18:43.639ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-8522f663-bfd2-44db-b998-3fd36bc04757
252025-03-04T03:18:43.656ZINFOcrucible-agent (worker): creating config property group
262025-03-04T03:18:43.659ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-8522f663-bfd2-44db-b998-3fd36bc04757
272025-03-04T03:18:43.660ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/8522f663-bfd2-44db-b998-3fd36bc04757
282025-03-04T03:18:43.660ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
292025-03-04T03:18:43.660ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
302025-03-04T03:18:43.660ZINFOcrucible-agent (worker): commit
312025-03-04T03:18:43.665ZINFOcrucible-agent (worker): ok!
322025-03-04T03:18:43.680ZINFOcrucible-agent (worker): SMF ok!
332025-03-04T03:18:43.886ZINFOcrucible-agent (dropshot): request completed latency_us = 52 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:51011 req_id = 1a2cbc1c-9ce6-423d-8edc-95b0b7224006 response_code = 200 uri = /crucible/0/regions
342025-03-04T03:18:44.634ZINFOcrucible-agent (dropshot): request completed latency_us = 109 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:51011 req_id = d1ce4691-affd-41c1-b4c5-41a9d3a49956 response_code = 200 uri = /crucible/0/regions/8522f663-bfd2-44db-b998-3fd36bc04757
352025-03-04T03:18:44.637ZINFOcrucible-agent (datafile): path is "/data/regions/8522f663-bfd2-44db-b998-3fd36bc04757"
362025-03-04T03:18:44.648ZINFOcrucible-agent (datafile): dataset is oxp_8bf53b00-dedd-4516-be68-25e52e3985f5/crucible/regions/8522f663-bfd2-44db-b998-3fd36bc04757
372025-03-04T03:18:44.658ZINFOcrucible-agent (datafile): region 8522f663-bfd2-44db-b998-3fd36bc04757 state: Created -> Tombstoned
382025-03-04T03:18:44.658ZINFOcrucible-agent (worker): applying SMF actions before removal...
392025-03-04T03:18:44.658ZINFOcrucible-agent (dropshot): request completed latency_us = 21511 local_addr = [fd00:1122:3344:101::1a]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::b]:51011 req_id = 1ac0c7d3-4ae4-47ec-a22f-9e2b0dd47428 response_code = 204 uri = /crucible/0/regions/8522f663-bfd2-44db-b998-3fd36bc04757
402025-03-04T03:18:44.660ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-8522f663-bfd2-44db-b998-3fd36bc04757 (instance states: (Some(Online), None))
412025-03-04T03:18:44.661ZINFOcrucible-agent (dropshot): request completed latency_us = 30 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:51011 req_id = 1829b7e1-9055-4de4-ab8f-3499ba3400a4 response_code = 200 uri = /crucible/0/regions/8522f663-bfd2-44db-b998-3fd36bc04757
422025-03-04T03:18:44.670ZINFOcrucible-agent (worker): SMF ok!
432025-03-04T03:18:44.682ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_8bf53b00-dedd-4516-be68-25e52e3985f5/crucible/regions/8522f663-bfd2-44db-b998-3fd36bc04757" region = 8522f663-bfd2-44db-b998-3fd36bc04757
442025-03-04T03:18:44.749ZINFOcrucible-agent (datafile): region 8522f663-bfd2-44db-b998-3fd36bc04757 state: Tombstoned -> Destroyed
452025-03-04T03:18:44.803ZINFOcrucible-agent (dropshot): request completed latency_us = 65 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:51011 req_id = a86c964d-0eaa-4e4c-b4f9-0f11cea73650 response_code = 200 uri = /crucible/0/regions/8522f663-bfd2-44db-b998-3fd36bc04757
462025-03-04T03:29:02.638ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::1a]:32345 remote_addr = [fd00:1122:3344:101::b]:40323
472025-03-04T03:29:02.638ZINFOcrucible-agent (datafile): region 0a9fc6e0-c356-4a38-ab20-966e3292009a state: Requested
482025-03-04T03:29:02.638ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
492025-03-04T03:29:02.638ZINFOcrucible-agent (dropshot): request completed latency_us = 673 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:40323 req_id = d07b5305-da79-4b47-be36-32bce5f8c534 response_code = 200 uri = /crucible/0/regions
502025-03-04T03:29:02.648ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_8bf53b00-dedd-4516-be68-25e52e3985f5/crucible/regions/0a9fc6e0-c356-4a38-ab20-966e3292009a
512025-03-04T03:29:02.648ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_8bf53b00-dedd-4516-be68-25e52e3985f5/crucible/regions/0a9fc6e0-c356-4a38-ab20-966e3292009a
522025-03-04T03:29:02.703ZINFOcrucible-agent (worker): creating region Region { id: RegionId("0a9fc6e0-c356-4a38-ab20-966e3292009a"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/0a9fc6e0-c356-4a38-ab20-966e3292009a" region = 0a9fc6e0-c356-4a38-ab20-966e3292009a
532025-03-04T03:29:02.705ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "0a9fc6e0-c356-4a38-ab20-966e3292009a" "--data" "/data/regions/0a9fc6e0-c356-4a38-ab20-966e3292009a" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 0a9fc6e0-c356-4a38-ab20-966e3292009a
542025-03-04T03:29:02.796ZINFOcrucible-agent (worker): region files created ok region = 0a9fc6e0-c356-4a38-ab20-966e3292009a
552025-03-04T03:29:02.796ZINFOcrucible-agent (datafile): region 0a9fc6e0-c356-4a38-ab20-966e3292009a state: Requested -> Created
562025-03-04T03:29:02.796ZINFOcrucible-agent (worker): applying SMF actions post create...
572025-03-04T03:29:02.798ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-0a9fc6e0-c356-4a38-ab20-966e3292009a
582025-03-04T03:29:02.805ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-0a9fc6e0-c356-4a38-ab20-966e3292009a
592025-03-04T03:29:02.821ZINFOcrucible-agent (worker): creating config property group
602025-03-04T03:29:02.824ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-0a9fc6e0-c356-4a38-ab20-966e3292009a
612025-03-04T03:29:02.824ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/0a9fc6e0-c356-4a38-ab20-966e3292009a
622025-03-04T03:29:02.824ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
632025-03-04T03:29:02.824ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
642025-03-04T03:29:02.825ZINFOcrucible-agent (worker): commit
652025-03-04T03:29:02.828ZINFOcrucible-agent (worker): ok!
662025-03-04T03:29:02.834ZINFOcrucible-agent (worker): SMF ok!
672025-03-04T03:29:02.974ZINFOcrucible-agent (dropshot): request completed latency_us = 72 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:40323 req_id = 99e70387-39c0-44dc-8812-1ae4b885602f response_code = 200 uri = /crucible/0/regions
682025-03-04T03:29:08.611ZINFOcrucible-agent (datafile): region 1af7cf48-ae5e-4bc9-84fd-edd57f0f89ad state: Requested
692025-03-04T03:29:08.611ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
702025-03-04T03:29:08.612ZINFOcrucible-agent (dropshot): request completed latency_us = 307 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:40323 req_id = ee778235-9853-4882-9a53-c123da0a420d response_code = 200 uri = /crucible/0/regions
712025-03-04T03:29:08.622ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_8bf53b00-dedd-4516-be68-25e52e3985f5/crucible/regions/1af7cf48-ae5e-4bc9-84fd-edd57f0f89ad
722025-03-04T03:29:08.622ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_8bf53b00-dedd-4516-be68-25e52e3985f5/crucible/regions/1af7cf48-ae5e-4bc9-84fd-edd57f0f89ad
732025-03-04T03:29:08.679ZINFOcrucible-agent (worker): creating region Region { id: RegionId("1af7cf48-ae5e-4bc9-84fd-edd57f0f89ad"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/1af7cf48-ae5e-4bc9-84fd-edd57f0f89ad" region = 1af7cf48-ae5e-4bc9-84fd-edd57f0f89ad
742025-03-04T03:29:08.679ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "1af7cf48-ae5e-4bc9-84fd-edd57f0f89ad" "--data" "/data/regions/1af7cf48-ae5e-4bc9-84fd-edd57f0f89ad" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 1af7cf48-ae5e-4bc9-84fd-edd57f0f89ad
752025-03-04T03:29:08.754ZINFOcrucible-agent (dropshot): request completed latency_us = 92 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:40323 req_id = a48f126d-a20e-40ca-9e2b-d3b411f8651d response_code = 200 uri = /crucible/0/regions
762025-03-04T03:29:08.768ZINFOcrucible-agent (worker): region files created ok region = 1af7cf48-ae5e-4bc9-84fd-edd57f0f89ad
772025-03-04T03:29:08.768ZINFOcrucible-agent (datafile): region 1af7cf48-ae5e-4bc9-84fd-edd57f0f89ad state: Requested -> Created
782025-03-04T03:29:08.769ZINFOcrucible-agent (worker): applying SMF actions post create...
792025-03-04T03:29:08.771ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-1af7cf48-ae5e-4bc9-84fd-edd57f0f89ad
802025-03-04T03:29:08.783ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-1af7cf48-ae5e-4bc9-84fd-edd57f0f89ad
812025-03-04T03:29:08.793ZINFOcrucible-agent (worker): creating config property group
822025-03-04T03:29:08.796ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-1af7cf48-ae5e-4bc9-84fd-edd57f0f89ad
832025-03-04T03:29:08.796ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/1af7cf48-ae5e-4bc9-84fd-edd57f0f89ad
842025-03-04T03:29:08.796ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
852025-03-04T03:29:08.796ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
862025-03-04T03:29:08.796ZINFOcrucible-agent (worker): commit
872025-03-04T03:29:08.801ZINFOcrucible-agent (worker): ok!
882025-03-04T03:29:08.807ZINFOcrucible-agent (worker): SMF ok!
892025-03-04T03:29:09.068ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::1a]:32345 remote_addr = [fd00:1122:3344:101::b]:47981
902025-03-04T03:29:09.069ZINFOcrucible-agent (dropshot): request completed latency_us = 770 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:47981 req_id = 7f5ad412-f2fa-4578-b9dd-e1f25f7bd1ad response_code = 200 uri = /crucible/0/regions/1af7cf48-ae5e-4bc9-84fd-edd57f0f89ad
912025-03-04T03:29:09.222ZINFOcrucible-agent (dropshot): request completed latency_us = 54 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::b]:40323 req_id = 340e528d-fd19-40d9-99c0-d521d57df6c4 response_code = 200 uri = /crucible/0/regions
922025-03-04T03:29:35.025ZINFOcrucible-agent (dropshot): request completed latency_us = 41 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:40323 req_id = c5a3d68f-894a-4a0b-a81a-d370bd6acf9d response_code = 200 uri = /crucible/0/regions/1af7cf48-ae5e-4bc9-84fd-edd57f0f89ad
932025-03-04T03:29:35.028ZINFOcrucible-agent (datafile): path is "/data/regions/1af7cf48-ae5e-4bc9-84fd-edd57f0f89ad"
942025-03-04T03:29:35.041ZINFOcrucible-agent (datafile): dataset is oxp_8bf53b00-dedd-4516-be68-25e52e3985f5/crucible/regions/1af7cf48-ae5e-4bc9-84fd-edd57f0f89ad
952025-03-04T03:29:35.051ZINFOcrucible-agent (datafile): region 1af7cf48-ae5e-4bc9-84fd-edd57f0f89ad state: Created -> Tombstoned
962025-03-04T03:29:35.052ZINFOcrucible-agent (worker): applying SMF actions before removal...
972025-03-04T03:29:35.052ZINFOcrucible-agent (dropshot): request completed latency_us = 23751 local_addr = [fd00:1122:3344:101::1a]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::b]:40323 req_id = 280d06bb-e676-4cb6-9ab8-ddde4274852a response_code = 204 uri = /crucible/0/regions/1af7cf48-ae5e-4bc9-84fd-edd57f0f89ad
982025-03-04T03:29:35.053ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-1af7cf48-ae5e-4bc9-84fd-edd57f0f89ad (instance states: (Some(Online), None))
992025-03-04T03:29:35.055ZINFOcrucible-agent (dropshot): request completed latency_us = 24 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:40323 req_id = dea80bed-c861-49d5-a972-c1c6e447f7e8 response_code = 200 uri = /crucible/0/regions/1af7cf48-ae5e-4bc9-84fd-edd57f0f89ad
1002025-03-04T03:29:35.059ZINFOcrucible-agent (worker): SMF ok!
1012025-03-04T03:29:35.070ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_8bf53b00-dedd-4516-be68-25e52e3985f5/crucible/regions/1af7cf48-ae5e-4bc9-84fd-edd57f0f89ad" region = 1af7cf48-ae5e-4bc9-84fd-edd57f0f89ad
1022025-03-04T03:29:35.111ZINFOcrucible-agent (dropshot): request completed latency_us = 685 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:40323 req_id = 9dcbb5e7-41ec-4c58-b0e5-752855bfe5a1 response_code = 200 uri = /crucible/0/regions/1af7cf48-ae5e-4bc9-84fd-edd57f0f89ad
1032025-03-04T03:29:35.129ZINFOcrucible-agent (datafile): region 1af7cf48-ae5e-4bc9-84fd-edd57f0f89ad state: Tombstoned -> Destroyed
1042025-03-04T03:29:35.209ZINFOcrucible-agent (dropshot): request completed latency_us = 27 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::b]:40323 req_id = 6951f3bc-2759-49d0-b8f4-47b052910191 response_code = 200 uri = /crucible/0/regions/1af7cf48-ae5e-4bc9-84fd-edd57f0f89ad