Max level shown:
1 [ Jun 17 21:19:48 Enabled. ]
2 [ Jun 17 21:19:48 Rereading configuration. ]
3 [ Jun 17 21:19:53 Rereading configuration. ]
4 [ Jun 17 21:19:58 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_e4f51e96-8b35-4542-822a-22c6778e005f/crucible -l [fd00:1122:3344:101::14]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62024-06-17T21:19:58.735ZINFOcrucible-agent: dataset: "oxp_e4f51e96-8b35-4542-822a-22c6778e005f/crucible"
72024-06-17T21:19:58.738ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::14]:32345
82024-06-17T21:19:58.738ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92024-06-17T21:19:58.754ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102024-06-17T21:19:58.977ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::14]:32345
112024-06-17T21:21:09.080ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::14]:32345 remote_addr = [fd00:1122:3344:101::a]:58779
122024-06-17T21:21:09.087ZINFOcrucible-agent (datafile): region d258df11-d3fd-428e-972b-91b3498b4d4b state: Requested
132024-06-17T21:21:09.087ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
142024-06-17T21:21:09.091ZINFOcrucible-agent (dropshot): request completed latency_us = 7354 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:58779 req_id = a1821142-0a43-4a38-844d-d0d9e95b483c response_code = 200 uri = /crucible/0/regions
152024-06-17T21:21:09.098ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_e4f51e96-8b35-4542-822a-22c6778e005f/crucible/regions/d258df11-d3fd-428e-972b-91b3498b4d4b
162024-06-17T21:21:09.098ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_e4f51e96-8b35-4542-822a-22c6778e005f/crucible/regions/d258df11-d3fd-428e-972b-91b3498b4d4b
172024-06-17T21:21:09.151ZINFOcrucible-agent (worker): creating region Region { id: RegionId("d258df11-d3fd-428e-972b-91b3498b4d4b"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/d258df11-d3fd-428e-972b-91b3498b4d4b" region = d258df11-d3fd-428e-972b-91b3498b4d4b
182024-06-17T21:21:09.151ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "d258df11-d3fd-428e-972b-91b3498b4d4b" "--data" "/data/regions/d258df11-d3fd-428e-972b-91b3498b4d4b" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = d258df11-d3fd-428e-972b-91b3498b4d4b
192024-06-17T21:21:09.279ZINFOcrucible-agent (dropshot): request completed latency_us = 80 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:58779 req_id = 4c7fd9a8-e1bc-4e80-8072-847f9f007324 response_code = 200 uri = /crucible/0/regions
202024-06-17T21:21:09.289ZINFOcrucible-agent (worker): region files created ok region = d258df11-d3fd-428e-972b-91b3498b4d4b
212024-06-17T21:21:09.289ZINFOcrucible-agent (datafile): region d258df11-d3fd-428e-972b-91b3498b4d4b state: Requested -> Created
222024-06-17T21:21:09.289ZINFOcrucible-agent (worker): applying SMF actions post create...
232024-06-17T21:21:09.289ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-d258df11-d3fd-428e-972b-91b3498b4d4b
242024-06-17T21:21:09.295ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-d258df11-d3fd-428e-972b-91b3498b4d4b
252024-06-17T21:21:09.302ZINFOcrucible-agent (worker): creating config property group
262024-06-17T21:21:09.307ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-d258df11-d3fd-428e-972b-91b3498b4d4b
272024-06-17T21:21:09.316ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/d258df11-d3fd-428e-972b-91b3498b4d4b
282024-06-17T21:21:09.316ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
292024-06-17T21:21:09.316ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
302024-06-17T21:21:09.316ZINFOcrucible-agent (worker): commit
312024-06-17T21:21:09.320ZINFOcrucible-agent (worker): ok!
322024-06-17T21:21:09.324ZINFOcrucible-agent (worker): SMF ok!
332024-06-17T21:21:09.995ZINFOcrucible-agent (dropshot): request completed latency_us = 78 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:58779 req_id = e4e061ab-48f0-42ea-8a99-06ec6e48e330 response_code = 200 uri = /crucible/0/regions
342024-06-17T21:26:22.645ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::14]:32345 remote_addr = [fd00:1122:3344:101::a]:40045
352024-06-17T21:26:22.648ZINFOcrucible-agent (dropshot): request completed latency_us = 3096 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:40045 req_id = b931e364-c93c-44af-bdc1-ca9dafc4f19c response_code = 200 uri = /crucible/0/regions/d258df11-d3fd-428e-972b-91b3498b4d4b
362024-06-17T21:26:22.649ZINFOcrucible-agent (datafile): path is "/data/regions/d258df11-d3fd-428e-972b-91b3498b4d4b"
372024-06-17T21:26:22.658ZINFOcrucible-agent (datafile): dataset is oxp_e4f51e96-8b35-4542-822a-22c6778e005f/crucible/regions/d258df11-d3fd-428e-972b-91b3498b4d4b
382024-06-17T21:26:22.667ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_e4f51e96-8b35-4542-822a-22c6778e005f/crucible/regions/d258df11-d3fd-428e-972b-91b3498b4d4b@9f154061-576e-45bf-baea-08cf51715545
392024-06-17T21:26:22.667ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_e4f51e96-8b35-4542-822a-22c6778e005f/crucible/regions/d258df11-d3fd-428e-972b-91b3498b4d4b", "9f154061-576e-45bf-baea-08cf51715545"]
402024-06-17T21:26:22.681ZINFOcrucible-agent (dropshot): request completed latency_us = 32536 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:40045 req_id = 111393f9-7ee4-40d0-b70b-d3225b6d81d9 response_code = 200 uri = /crucible/0/regions/d258df11-d3fd-428e-972b-91b3498b4d4b/snapshots/9f154061-576e-45bf-baea-08cf51715545
412024-06-17T21:26:22.682ZINFOcrucible-agent (datafile): path is "/data/regions/d258df11-d3fd-428e-972b-91b3498b4d4b"
422024-06-17T21:26:22.691ZINFOcrucible-agent (datafile): dataset is oxp_e4f51e96-8b35-4542-822a-22c6778e005f/crucible/regions/d258df11-d3fd-428e-972b-91b3498b4d4b
432024-06-17T21:26:22.699ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_e4f51e96-8b35-4542-822a-22c6778e005f/crucible/regions/d258df11-d3fd-428e-972b-91b3498b4d4b@9f154061-576e-45bf-baea-08cf51715545
442024-06-17T21:26:22.699ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_e4f51e96-8b35-4542-822a-22c6778e005f/crucible/regions/d258df11-d3fd-428e-972b-91b3498b4d4b", "9f154061-576e-45bf-baea-08cf51715545"]
452024-06-17T21:26:22.708ZINFOcrucible-agent (datafile): requesting running snapshot d258df11-d3fd-428e-972b-91b3498b4d4b-9f154061-576e-45bf-baea-08cf51715545 state: Requested
462024-06-17T21:26:22.709ZINFOcrucible-agent (worker): applying SMF actions for region d258df11-d3fd-428e-972b-91b3498b4d4b running snapshot 9f154061-576e-45bf-baea-08cf51715545 (state Requested)...
472024-06-17T21:26:22.709ZINFOcrucible-agent (dropshot): request completed latency_us = 27608 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:40045 req_id = 59c4ed93-6f10-4f01-b8d9-9d9b69ac4fc3 response_code = 200 uri = /crucible/0/regions/d258df11-d3fd-428e-972b-91b3498b4d4b/snapshots/9f154061-576e-45bf-baea-08cf51715545/run
482024-06-17T21:26:22.714ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/d258df11-d3fd-428e-972b-91b3498b4d4b
492024-06-17T21:26:22.714ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
502024-06-17T21:26:22.714ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::14
512024-06-17T21:26:22.714ZINFOcrucible-agent (worker): reconfig required
522024-06-17T21:26:22.714ZINFOcrucible-agent (worker): using existing config property group
532024-06-17T21:26:22.714ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-d258df11-d3fd-428e-972b-91b3498b4d4b
542024-06-17T21:26:22.715ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/d258df11-d3fd-428e-972b-91b3498b4d4b
552024-06-17T21:26:22.715ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
562024-06-17T21:26:22.715ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
572024-06-17T21:26:22.715ZINFOcrucible-agent (worker): commit
582024-06-17T21:26:22.720ZINFOcrucible-agent (worker): ok!
592024-06-17T21:26:22.720ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-d258df11-d3fd-428e-972b-91b3498b4d4b-9f154061-576e-45bf-baea-08cf51715545
602024-06-17T21:26:22.723ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-d258df11-d3fd-428e-972b-91b3498b4d4b-9f154061-576e-45bf-baea-08cf51715545
612024-06-17T21:26:22.737ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-d258df11-d3fd-428e-972b-91b3498b4d4b-9f154061-576e-45bf-baea-08cf51715545
622024-06-17T21:26:22.737ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/d258df11-d3fd-428e-972b-91b3498b4d4b/.zfs/snapshot/9f154061-576e-45bf-baea-08cf51715545
632024-06-17T21:26:22.737ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
642024-06-17T21:26:22.737ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
652024-06-17T21:26:22.737ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
662024-06-17T21:26:22.737ZINFOcrucible-agent (worker): commit
672024-06-17T21:26:22.740ZINFOcrucible-agent (worker): ok!
682024-06-17T21:26:22.745ZINFOcrucible-agent (worker): SMF ok!
692024-06-17T21:26:22.745ZINFOcrucible-agent (datafile): region d258df11-d3fd-428e-972b-91b3498b4d4b running snapshot 9f154061-576e-45bf-baea-08cf51715545 state: Requested -> Created
702024-06-17T21:26:25.485ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::14]:32345 remote_addr = [fd00:1122:3344:101::a]:51436
712024-06-17T21:26:25.485ZINFOcrucible-agent (datafile): region c0a396ab-aeb3-4990-9c4e-25af9a375b69 state: Requested
722024-06-17T21:26:25.485ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
732024-06-17T21:26:25.485ZINFOcrucible-agent (dropshot): request completed latency_us = 741 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:51436 req_id = 8de9a1e1-3261-477c-9951-d40167416e3f response_code = 200 uri = /crucible/0/regions
742024-06-17T21:26:25.495ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_e4f51e96-8b35-4542-822a-22c6778e005f/crucible/regions/c0a396ab-aeb3-4990-9c4e-25af9a375b69
752024-06-17T21:26:25.495ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_e4f51e96-8b35-4542-822a-22c6778e005f/crucible/regions/c0a396ab-aeb3-4990-9c4e-25af9a375b69
762024-06-17T21:26:25.566ZINFOcrucible-agent (worker): creating region Region { id: RegionId("c0a396ab-aeb3-4990-9c4e-25af9a375b69"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19002, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/c0a396ab-aeb3-4990-9c4e-25af9a375b69" region = c0a396ab-aeb3-4990-9c4e-25af9a375b69
772024-06-17T21:26:25.567ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "c0a396ab-aeb3-4990-9c4e-25af9a375b69" "--data" "/data/regions/c0a396ab-aeb3-4990-9c4e-25af9a375b69" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = c0a396ab-aeb3-4990-9c4e-25af9a375b69
782024-06-17T21:26:25.634ZINFOcrucible-agent (worker): region files created ok region = c0a396ab-aeb3-4990-9c4e-25af9a375b69
792024-06-17T21:26:25.634ZINFOcrucible-agent (datafile): region c0a396ab-aeb3-4990-9c4e-25af9a375b69 state: Requested -> Created
802024-06-17T21:26:25.634ZINFOcrucible-agent (worker): applying SMF actions post create...
812024-06-17T21:26:25.635ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-c0a396ab-aeb3-4990-9c4e-25af9a375b69
822024-06-17T21:26:25.641ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-c0a396ab-aeb3-4990-9c4e-25af9a375b69
832024-06-17T21:26:25.648ZINFOcrucible-agent (worker): creating config property group
842024-06-17T21:26:25.650ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-c0a396ab-aeb3-4990-9c4e-25af9a375b69
852024-06-17T21:26:25.651ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/c0a396ab-aeb3-4990-9c4e-25af9a375b69
862024-06-17T21:26:25.651ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
872024-06-17T21:26:25.651ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
882024-06-17T21:26:25.651ZINFOcrucible-agent (worker): commit
892024-06-17T21:26:25.654ZINFOcrucible-agent (worker): ok!
902024-06-17T21:26:25.659ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/d258df11-d3fd-428e-972b-91b3498b4d4b
912024-06-17T21:26:25.659ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
922024-06-17T21:26:25.659ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::14
932024-06-17T21:26:25.659ZINFOcrucible-agent (worker): reconfig required
942024-06-17T21:26:25.659ZINFOcrucible-agent (worker): using existing config property group
952024-06-17T21:26:25.659ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-d258df11-d3fd-428e-972b-91b3498b4d4b
962024-06-17T21:26:25.663ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/d258df11-d3fd-428e-972b-91b3498b4d4b
972024-06-17T21:26:25.663ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
982024-06-17T21:26:25.663ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
992024-06-17T21:26:25.663ZINFOcrucible-agent (worker): commit
1002024-06-17T21:26:25.667ZINFOcrucible-agent (worker): ok!
1012024-06-17T21:26:25.669ZINFOcrucible-agent (worker): SMF ok!
1022024-06-17T21:26:25.805ZINFOcrucible-agent (dropshot): request completed latency_us = 65 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:51436 req_id = bcfc7e77-a0f8-4cf0-bdbd-b7063e49d64e response_code = 200 uri = /crucible/0/regions
1032024-06-17T21:26:49.666ZINFOcrucible-agent (dropshot): request completed latency_us = 639 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:51436 req_id = 62cb7792-9c28-4e7b-a170-ce9a1c24a2f2 response_code = 200 uri = /crucible/0/regions/c0a396ab-aeb3-4990-9c4e-25af9a375b69
1042024-06-17T21:26:49.669ZINFOcrucible-agent (datafile): path is "/data/regions/c0a396ab-aeb3-4990-9c4e-25af9a375b69"
1052024-06-17T21:26:49.680ZINFOcrucible-agent (datafile): dataset is oxp_e4f51e96-8b35-4542-822a-22c6778e005f/crucible/regions/c0a396ab-aeb3-4990-9c4e-25af9a375b69
1062024-06-17T21:26:49.689ZINFOcrucible-agent (datafile): region c0a396ab-aeb3-4990-9c4e-25af9a375b69 state: Created -> Tombstoned
1072024-06-17T21:26:49.689ZINFOcrucible-agent (worker): applying SMF actions before removal...
1082024-06-17T21:26:49.692ZINFOcrucible-agent (dropshot): request completed latency_us = 23364 local_addr = [fd00:1122:3344:101::14]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:51436 req_id = b09e810b-bb99-4c33-8674-0b8e66acff49 response_code = 204 uri = /crucible/0/regions/c0a396ab-aeb3-4990-9c4e-25af9a375b69
1092024-06-17T21:26:49.693ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-c0a396ab-aeb3-4990-9c4e-25af9a375b69 (instance states: (Some(Online), None))
1102024-06-17T21:26:49.695ZINFOcrucible-agent (dropshot): request completed latency_us = 34 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:51436 req_id = a3b1621b-df8c-434e-b7c2-e8e6043da841 response_code = 200 uri = /crucible/0/regions/c0a396ab-aeb3-4990-9c4e-25af9a375b69
1112024-06-17T21:26:49.697ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/d258df11-d3fd-428e-972b-91b3498b4d4b
1122024-06-17T21:26:49.697ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
1132024-06-17T21:26:49.697ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::14
1142024-06-17T21:26:49.697ZINFOcrucible-agent (worker): reconfig required
1152024-06-17T21:26:49.697ZINFOcrucible-agent (worker): using existing config property group
1162024-06-17T21:26:49.697ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-d258df11-d3fd-428e-972b-91b3498b4d4b
1172024-06-17T21:26:49.697ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/d258df11-d3fd-428e-972b-91b3498b4d4b
1182024-06-17T21:26:49.698ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
1192024-06-17T21:26:49.698ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
1202024-06-17T21:26:49.698ZINFOcrucible-agent (worker): commit
1212024-06-17T21:26:49.701ZINFOcrucible-agent (worker): ok!
1222024-06-17T21:26:49.703ZINFOcrucible-agent (worker): SMF ok!
1232024-06-17T21:26:49.712ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_e4f51e96-8b35-4542-822a-22c6778e005f/crucible/regions/c0a396ab-aeb3-4990-9c4e-25af9a375b69" region = c0a396ab-aeb3-4990-9c4e-25af9a375b69
1242024-06-17T21:26:49.767ZINFOcrucible-agent (datafile): region c0a396ab-aeb3-4990-9c4e-25af9a375b69 state: Tombstoned -> Destroyed
1252024-06-17T21:26:49.776ZINFOcrucible-agent (dropshot): request completed latency_us = 62 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:51436 req_id = 8be72605-94ca-4ebe-b91b-570175e7b652 response_code = 200 uri = /crucible/0/regions/c0a396ab-aeb3-4990-9c4e-25af9a375b69