Max level shown:
1 [ Jul 9 03:52:38 Enabled. ]
2 [ Jul 9 03:52:38 Rereading configuration. ]
3 [ Jul 9 03:52:40 Rereading configuration. ]
4 [ Jul 9 03:52:42 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_c5feda06-626f-4148-afb8-488b1ffe21e6/crucible -l [fd00:1122:3344:101::12]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62024-07-09T03:52:42.738ZINFOcrucible-agent: dataset: "oxp_c5feda06-626f-4148-afb8-488b1ffe21e6/crucible"
72024-07-09T03:52:42.744ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::12]:32345
82024-07-09T03:52:42.744ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92024-07-09T03:52:42.773ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102024-07-09T03:52:42.888ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::12]:32345
112024-07-09T03:54:01.689ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:41682
122024-07-09T03:54:01.708ZINFOcrucible-agent (datafile): region e8e2de12-00ba-4f94-8e2f-04cc22541644 state: Requested
132024-07-09T03:54:01.714ZINFOcrucible-agent (dropshot): request completed latency_us = 14037 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:41682 req_id = 77e83fc6-c4a9-4c2d-8507-81482ea8ca62 response_code = 200 uri = /crucible/0/regions
142024-07-09T03:54:01.715ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
152024-07-09T03:54:01.727ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_c5feda06-626f-4148-afb8-488b1ffe21e6/crucible/regions/e8e2de12-00ba-4f94-8e2f-04cc22541644
162024-07-09T03:54:01.727ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_c5feda06-626f-4148-afb8-488b1ffe21e6/crucible/regions/e8e2de12-00ba-4f94-8e2f-04cc22541644
172024-07-09T03:54:01.790ZINFOcrucible-agent (worker): creating region Region { id: RegionId("e8e2de12-00ba-4f94-8e2f-04cc22541644"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/e8e2de12-00ba-4f94-8e2f-04cc22541644" region = e8e2de12-00ba-4f94-8e2f-04cc22541644
182024-07-09T03:54:01.792ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "e8e2de12-00ba-4f94-8e2f-04cc22541644" "--data" "/data/regions/e8e2de12-00ba-4f94-8e2f-04cc22541644" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = e8e2de12-00ba-4f94-8e2f-04cc22541644
192024-07-09T03:54:01.949ZINFOcrucible-agent (worker): region files created ok region = e8e2de12-00ba-4f94-8e2f-04cc22541644
202024-07-09T03:54:01.949ZINFOcrucible-agent (datafile): region e8e2de12-00ba-4f94-8e2f-04cc22541644 state: Requested -> Created
212024-07-09T03:54:01.949ZINFOcrucible-agent (worker): applying SMF actions post create...
222024-07-09T03:54:01.950ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-e8e2de12-00ba-4f94-8e2f-04cc22541644
232024-07-09T03:54:01.956ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-e8e2de12-00ba-4f94-8e2f-04cc22541644
242024-07-09T03:54:01.972ZINFOcrucible-agent (worker): creating config property group
252024-07-09T03:54:01.975ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-e8e2de12-00ba-4f94-8e2f-04cc22541644
262024-07-09T03:54:01.976ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/e8e2de12-00ba-4f94-8e2f-04cc22541644
272024-07-09T03:54:01.980ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
282024-07-09T03:54:01.980ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
292024-07-09T03:54:01.980ZINFOcrucible-agent (worker): commit
302024-07-09T03:54:01.988ZINFOcrucible-agent (worker): ok!
312024-07-09T03:54:01.995ZINFOcrucible-agent (worker): SMF ok!
322024-07-09T03:54:02.095ZINFOcrucible-agent (dropshot): request completed latency_us = 74 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:41682 req_id = bd36d1cc-0b7c-412d-94fb-0130178b742e response_code = 200 uri = /crucible/0/regions
332024-07-09T03:59:13.605ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:63397
342024-07-09T03:59:13.608ZINFOcrucible-agent (dropshot): request completed latency_us = 2792 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:63397 req_id = b73d83fd-9754-459a-b68f-1f425116598e response_code = 200 uri = /crucible/0/regions/e8e2de12-00ba-4f94-8e2f-04cc22541644
352024-07-09T03:59:13.611ZINFOcrucible-agent (datafile): path is "/data/regions/e8e2de12-00ba-4f94-8e2f-04cc22541644"
362024-07-09T03:59:13.620ZINFOcrucible-agent (datafile): dataset is oxp_c5feda06-626f-4148-afb8-488b1ffe21e6/crucible/regions/e8e2de12-00ba-4f94-8e2f-04cc22541644
372024-07-09T03:59:13.628ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_c5feda06-626f-4148-afb8-488b1ffe21e6/crucible/regions/e8e2de12-00ba-4f94-8e2f-04cc22541644@55ed7137-8ad7-4401-a037-50f3d8186e42
382024-07-09T03:59:13.628ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_c5feda06-626f-4148-afb8-488b1ffe21e6/crucible/regions/e8e2de12-00ba-4f94-8e2f-04cc22541644", "55ed7137-8ad7-4401-a037-50f3d8186e42"]
392024-07-09T03:59:13.639ZINFOcrucible-agent (dropshot): request completed latency_us = 30774 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:63397 req_id = b5595fef-556b-43e3-9213-1eb11ebe5270 response_code = 200 uri = /crucible/0/regions/e8e2de12-00ba-4f94-8e2f-04cc22541644/snapshots/55ed7137-8ad7-4401-a037-50f3d8186e42
402024-07-09T03:59:13.639ZINFOcrucible-agent (datafile): path is "/data/regions/e8e2de12-00ba-4f94-8e2f-04cc22541644"
412024-07-09T03:59:13.648ZINFOcrucible-agent (datafile): dataset is oxp_c5feda06-626f-4148-afb8-488b1ffe21e6/crucible/regions/e8e2de12-00ba-4f94-8e2f-04cc22541644
422024-07-09T03:59:13.655ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_c5feda06-626f-4148-afb8-488b1ffe21e6/crucible/regions/e8e2de12-00ba-4f94-8e2f-04cc22541644@55ed7137-8ad7-4401-a037-50f3d8186e42
432024-07-09T03:59:13.655ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_c5feda06-626f-4148-afb8-488b1ffe21e6/crucible/regions/e8e2de12-00ba-4f94-8e2f-04cc22541644", "55ed7137-8ad7-4401-a037-50f3d8186e42"]
442024-07-09T03:59:13.665ZINFOcrucible-agent (datafile): requesting running snapshot e8e2de12-00ba-4f94-8e2f-04cc22541644-55ed7137-8ad7-4401-a037-50f3d8186e42 state: Requested
452024-07-09T03:59:13.666ZINFOcrucible-agent (worker): applying SMF actions for region e8e2de12-00ba-4f94-8e2f-04cc22541644 running snapshot 55ed7137-8ad7-4401-a037-50f3d8186e42 (state Requested)...
462024-07-09T03:59:13.666ZINFOcrucible-agent (dropshot): request completed latency_us = 26595 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:63397 req_id = 33bd2705-01dd-45fd-9817-f7262da04a9c response_code = 200 uri = /crucible/0/regions/e8e2de12-00ba-4f94-8e2f-04cc22541644/snapshots/55ed7137-8ad7-4401-a037-50f3d8186e42/run
472024-07-09T03:59:13.672ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/e8e2de12-00ba-4f94-8e2f-04cc22541644
482024-07-09T03:59:13.673ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
492024-07-09T03:59:13.673ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::12
502024-07-09T03:59:13.673ZINFOcrucible-agent (worker): reconfig required
512024-07-09T03:59:13.673ZINFOcrucible-agent (worker): using existing config property group
522024-07-09T03:59:13.673ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-e8e2de12-00ba-4f94-8e2f-04cc22541644
532024-07-09T03:59:13.673ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/e8e2de12-00ba-4f94-8e2f-04cc22541644
542024-07-09T03:59:13.673ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
552024-07-09T03:59:13.673ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
562024-07-09T03:59:13.673ZINFOcrucible-agent (worker): commit
572024-07-09T03:59:13.682ZINFOcrucible-agent (worker): ok!
582024-07-09T03:59:13.683ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-e8e2de12-00ba-4f94-8e2f-04cc22541644-55ed7137-8ad7-4401-a037-50f3d8186e42
592024-07-09T03:59:13.687ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-e8e2de12-00ba-4f94-8e2f-04cc22541644-55ed7137-8ad7-4401-a037-50f3d8186e42
602024-07-09T03:59:13.699ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-e8e2de12-00ba-4f94-8e2f-04cc22541644-55ed7137-8ad7-4401-a037-50f3d8186e42
612024-07-09T03:59:13.699ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/e8e2de12-00ba-4f94-8e2f-04cc22541644/.zfs/snapshot/55ed7137-8ad7-4401-a037-50f3d8186e42
622024-07-09T03:59:13.699ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
632024-07-09T03:59:13.699ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
642024-07-09T03:59:13.699ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
652024-07-09T03:59:13.699ZINFOcrucible-agent (worker): commit
662024-07-09T03:59:13.703ZINFOcrucible-agent (worker): ok!
672024-07-09T03:59:13.707ZINFOcrucible-agent (worker): SMF ok!
682024-07-09T03:59:13.707ZINFOcrucible-agent (datafile): region e8e2de12-00ba-4f94-8e2f-04cc22541644 running snapshot 55ed7137-8ad7-4401-a037-50f3d8186e42 state: Requested -> Created
692024-07-09T03:59:16.225ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:64324
702024-07-09T03:59:16.225ZINFOcrucible-agent (datafile): region fb128b23-d70c-4126-bd6f-24222b7669eb state: Requested
712024-07-09T03:59:16.227ZINFOcrucible-agent (dropshot): request completed latency_us = 2153 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:64324 req_id = 61442904-c391-46e9-b832-221861b571a2 response_code = 200 uri = /crucible/0/regions
722024-07-09T03:59:16.227ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
732024-07-09T03:59:16.237ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_c5feda06-626f-4148-afb8-488b1ffe21e6/crucible/regions/fb128b23-d70c-4126-bd6f-24222b7669eb
742024-07-09T03:59:16.237ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_c5feda06-626f-4148-afb8-488b1ffe21e6/crucible/regions/fb128b23-d70c-4126-bd6f-24222b7669eb
752024-07-09T03:59:16.304ZINFOcrucible-agent (worker): creating region Region { id: RegionId("fb128b23-d70c-4126-bd6f-24222b7669eb"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19002, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/fb128b23-d70c-4126-bd6f-24222b7669eb" region = fb128b23-d70c-4126-bd6f-24222b7669eb
762024-07-09T03:59:16.305ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "fb128b23-d70c-4126-bd6f-24222b7669eb" "--data" "/data/regions/fb128b23-d70c-4126-bd6f-24222b7669eb" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = fb128b23-d70c-4126-bd6f-24222b7669eb
772024-07-09T03:59:16.374ZINFOcrucible-agent (worker): region files created ok region = fb128b23-d70c-4126-bd6f-24222b7669eb
782024-07-09T03:59:16.374ZINFOcrucible-agent (datafile): region fb128b23-d70c-4126-bd6f-24222b7669eb state: Requested -> Created
792024-07-09T03:59:16.374ZINFOcrucible-agent (worker): applying SMF actions post create...
802024-07-09T03:59:16.375ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/e8e2de12-00ba-4f94-8e2f-04cc22541644
812024-07-09T03:59:16.375ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
822024-07-09T03:59:16.375ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::12
832024-07-09T03:59:16.375ZINFOcrucible-agent (worker): reconfig required
842024-07-09T03:59:16.375ZINFOcrucible-agent (worker): using existing config property group
852024-07-09T03:59:16.375ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-e8e2de12-00ba-4f94-8e2f-04cc22541644
862024-07-09T03:59:16.375ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/e8e2de12-00ba-4f94-8e2f-04cc22541644
872024-07-09T03:59:16.375ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
882024-07-09T03:59:16.375ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
892024-07-09T03:59:16.375ZINFOcrucible-agent (worker): commit
902024-07-09T03:59:16.381ZINFOcrucible-agent (worker): ok!
912024-07-09T03:59:16.381ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-fb128b23-d70c-4126-bd6f-24222b7669eb
922024-07-09T03:59:16.390ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-fb128b23-d70c-4126-bd6f-24222b7669eb
932024-07-09T03:59:16.398ZINFOcrucible-agent (worker): creating config property group
942024-07-09T03:59:16.401ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-fb128b23-d70c-4126-bd6f-24222b7669eb
952024-07-09T03:59:16.401ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/fb128b23-d70c-4126-bd6f-24222b7669eb
962024-07-09T03:59:16.401ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
972024-07-09T03:59:16.401ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
982024-07-09T03:59:16.401ZINFOcrucible-agent (worker): commit
992024-07-09T03:59:16.405ZINFOcrucible-agent (worker): ok!
1002024-07-09T03:59:16.410ZINFOcrucible-agent (worker): SMF ok!
1012024-07-09T03:59:16.587ZINFOcrucible-agent (dropshot): request completed latency_us = 70 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:64324 req_id = 307dab64-422e-4b8d-a651-2987311d24a8 response_code = 200 uri = /crucible/0/regions
1022024-07-09T03:59:46.574ZINFOcrucible-agent (dropshot): request completed latency_us = 70 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:64324 req_id = c9a5f845-a4fe-4bea-8d96-467699ba6aca response_code = 200 uri = /crucible/0/regions/fb128b23-d70c-4126-bd6f-24222b7669eb
1032024-07-09T03:59:46.577ZINFOcrucible-agent (datafile): path is "/data/regions/fb128b23-d70c-4126-bd6f-24222b7669eb"
1042024-07-09T03:59:46.587ZINFOcrucible-agent (datafile): dataset is oxp_c5feda06-626f-4148-afb8-488b1ffe21e6/crucible/regions/fb128b23-d70c-4126-bd6f-24222b7669eb
1052024-07-09T03:59:46.595ZINFOcrucible-agent (datafile): region fb128b23-d70c-4126-bd6f-24222b7669eb state: Created -> Tombstoned
1062024-07-09T03:59:46.595ZINFOcrucible-agent (worker): applying SMF actions before removal...
1072024-07-09T03:59:46.596ZINFOcrucible-agent (dropshot): request completed latency_us = 18844 local_addr = [fd00:1122:3344:101::12]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:64324 req_id = 7d1d0a13-5515-46ff-8010-3c295178bf97 response_code = 204 uri = /crucible/0/regions/fb128b23-d70c-4126-bd6f-24222b7669eb
1082024-07-09T03:59:46.597ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-fb128b23-d70c-4126-bd6f-24222b7669eb (instance states: (Some(Online), None))
1092024-07-09T03:59:46.598ZINFOcrucible-agent (dropshot): request completed latency_us = 25 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:64324 req_id = b72928d8-75ed-4726-8e71-cd337ca5f8ad response_code = 200 uri = /crucible/0/regions/fb128b23-d70c-4126-bd6f-24222b7669eb
1102024-07-09T03:59:46.605ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/e8e2de12-00ba-4f94-8e2f-04cc22541644
1112024-07-09T03:59:46.605ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
1122024-07-09T03:59:46.605ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::12
1132024-07-09T03:59:46.605ZINFOcrucible-agent (worker): reconfig required
1142024-07-09T03:59:46.605ZINFOcrucible-agent (worker): using existing config property group
1152024-07-09T03:59:46.605ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-e8e2de12-00ba-4f94-8e2f-04cc22541644
1162024-07-09T03:59:46.605ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/e8e2de12-00ba-4f94-8e2f-04cc22541644
1172024-07-09T03:59:46.605ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
1182024-07-09T03:59:46.606ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
1192024-07-09T03:59:46.606ZINFOcrucible-agent (worker): commit
1202024-07-09T03:59:46.611ZINFOcrucible-agent (worker): ok!
1212024-07-09T03:59:46.613ZINFOcrucible-agent (worker): SMF ok!
1222024-07-09T03:59:46.622ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_c5feda06-626f-4148-afb8-488b1ffe21e6/crucible/regions/fb128b23-d70c-4126-bd6f-24222b7669eb" region = fb128b23-d70c-4126-bd6f-24222b7669eb
1232024-07-09T03:59:46.682ZINFOcrucible-agent (dropshot): request completed latency_us = 91 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:64324 req_id = 3eee7ca1-679d-4501-b409-0680e5c72eb2 response_code = 200 uri = /crucible/0/regions/fb128b23-d70c-4126-bd6f-24222b7669eb
1242024-07-09T03:59:46.705ZINFOcrucible-agent (datafile): region fb128b23-d70c-4126-bd6f-24222b7669eb state: Tombstoned -> Destroyed
1252024-07-09T03:59:46.761ZINFOcrucible-agent (dropshot): request completed latency_us = 23 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:64324 req_id = c3392279-1752-480b-8420-cc2cab1be0b9 response_code = 200 uri = /crucible/0/regions/fb128b23-d70c-4126-bd6f-24222b7669eb