Max level shown:
1 [ Oct 9 17:19:23 Enabled. ]
2 [ Oct 9 17:19:23 Rereading configuration. ]
3 [ Oct 9 17:19:25 Rereading configuration. ]
4 [ Oct 9 17:19:27 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_b629ce16-33f2-4e5d-8506-138eb852a76c/crucible -l [fd00:1122:3344:101::1a]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62024-10-09T17:19:27.922ZINFOcrucible-agent: dataset: "oxp_b629ce16-33f2-4e5d-8506-138eb852a76c/crucible"
72024-10-09T17:19:27.923ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::1a]:32345
82024-10-09T17:19:27.923ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92024-10-09T17:19:27.944ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102024-10-09T17:19:28.053ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::1a]:32345
112024-10-09T17:20:41.610ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::1a]:32345 remote_addr = [fd00:1122:3344:101::a]:57598
122024-10-09T17:20:41.641ZINFOcrucible-agent (datafile): region 4d897563-5295-4f57-8013-4f9d906ae518 state: Requested
132024-10-09T17:20:41.645ZINFOcrucible-agent (dropshot): request completed latency_us = 12043 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:57598 req_id = 395c078d-2c4c-462d-8f17-89eca0105d61 response_code = 200 uri = /crucible/0/regions
142024-10-09T17:20:41.646ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
152024-10-09T17:20:41.655ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_b629ce16-33f2-4e5d-8506-138eb852a76c/crucible/regions/4d897563-5295-4f57-8013-4f9d906ae518
162024-10-09T17:20:41.655ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_b629ce16-33f2-4e5d-8506-138eb852a76c/crucible/regions/4d897563-5295-4f57-8013-4f9d906ae518
172024-10-09T17:20:41.704ZINFOcrucible-agent (worker): creating region Region { id: RegionId("4d897563-5295-4f57-8013-4f9d906ae518"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/4d897563-5295-4f57-8013-4f9d906ae518" region = 4d897563-5295-4f57-8013-4f9d906ae518
182024-10-09T17:20:41.708ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "4d897563-5295-4f57-8013-4f9d906ae518" "--data" "/data/regions/4d897563-5295-4f57-8013-4f9d906ae518" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 4d897563-5295-4f57-8013-4f9d906ae518
192024-10-09T17:20:41.845ZINFOcrucible-agent (worker): region files created ok region = 4d897563-5295-4f57-8013-4f9d906ae518
202024-10-09T17:20:41.845ZINFOcrucible-agent (datafile): region 4d897563-5295-4f57-8013-4f9d906ae518 state: Requested -> Created
212024-10-09T17:20:41.845ZINFOcrucible-agent (worker): applying SMF actions post create...
222024-10-09T17:20:41.847ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-4d897563-5295-4f57-8013-4f9d906ae518
232024-10-09T17:20:41.852ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-4d897563-5295-4f57-8013-4f9d906ae518
242024-10-09T17:20:41.866ZINFOcrucible-agent (worker): creating config property group
252024-10-09T17:20:41.869ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-4d897563-5295-4f57-8013-4f9d906ae518
262024-10-09T17:20:41.874ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/4d897563-5295-4f57-8013-4f9d906ae518
272024-10-09T17:20:41.874ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
282024-10-09T17:20:41.874ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
292024-10-09T17:20:41.874ZINFOcrucible-agent (worker): commit
302024-10-09T17:20:41.877ZINFOcrucible-agent (worker): ok!
312024-10-09T17:20:41.881ZINFOcrucible-agent (worker): SMF ok!
322024-10-09T17:20:42.014ZINFOcrucible-agent (dropshot): request completed latency_us = 66 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:57598 req_id = 77a98002-c22f-4095-966f-a4abefb5e2af response_code = 200 uri = /crucible/0/regions
332024-10-09T17:24:00.680ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::1a]:32345 remote_addr = [fd00:1122:3344:101::a]:50326
342024-10-09T17:24:00.680ZINFOcrucible-agent (datafile): region 23712bc5-1a34-4e7c-ad75-599a371afd53 state: Requested
352024-10-09T17:24:00.680ZINFOcrucible-agent (dropshot): request completed latency_us = 596 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:50326 req_id = 37f3eed5-874c-4f38-86aa-8abb3ae054dd response_code = 200 uri = /crucible/0/regions
362024-10-09T17:24:00.680ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
372024-10-09T17:24:00.691ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_b629ce16-33f2-4e5d-8506-138eb852a76c/crucible/regions/23712bc5-1a34-4e7c-ad75-599a371afd53
382024-10-09T17:24:00.691ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_b629ce16-33f2-4e5d-8506-138eb852a76c/crucible/regions/23712bc5-1a34-4e7c-ad75-599a371afd53
392024-10-09T17:24:00.755ZINFOcrucible-agent (worker): creating region Region { id: RegionId("23712bc5-1a34-4e7c-ad75-599a371afd53"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/23712bc5-1a34-4e7c-ad75-599a371afd53" region = 23712bc5-1a34-4e7c-ad75-599a371afd53
402024-10-09T17:24:00.757ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "23712bc5-1a34-4e7c-ad75-599a371afd53" "--data" "/data/regions/23712bc5-1a34-4e7c-ad75-599a371afd53" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 23712bc5-1a34-4e7c-ad75-599a371afd53
412024-10-09T17:24:00.830ZINFOcrucible-agent (worker): region files created ok region = 23712bc5-1a34-4e7c-ad75-599a371afd53
422024-10-09T17:24:00.830ZINFOcrucible-agent (datafile): region 23712bc5-1a34-4e7c-ad75-599a371afd53 state: Requested -> Created
432024-10-09T17:24:00.830ZINFOcrucible-agent (worker): applying SMF actions post create...
442024-10-09T17:24:00.830ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-23712bc5-1a34-4e7c-ad75-599a371afd53
452024-10-09T17:24:00.835ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-23712bc5-1a34-4e7c-ad75-599a371afd53
462024-10-09T17:24:00.849ZINFOcrucible-agent (worker): creating config property group
472024-10-09T17:24:00.853ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-23712bc5-1a34-4e7c-ad75-599a371afd53
482024-10-09T17:24:00.854ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/23712bc5-1a34-4e7c-ad75-599a371afd53
492024-10-09T17:24:00.854ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
502024-10-09T17:24:00.854ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
512024-10-09T17:24:00.854ZINFOcrucible-agent (worker): commit
522024-10-09T17:24:00.858ZINFOcrucible-agent (worker): ok!
532024-10-09T17:24:00.867ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/4d897563-5295-4f57-8013-4f9d906ae518
542024-10-09T17:24:00.867ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
552024-10-09T17:24:00.867ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::1a
562024-10-09T17:24:00.867ZINFOcrucible-agent (worker): reconfig required
572024-10-09T17:24:00.867ZINFOcrucible-agent (worker): using existing config property group
582024-10-09T17:24:00.867ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-4d897563-5295-4f57-8013-4f9d906ae518
592024-10-09T17:24:00.867ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/4d897563-5295-4f57-8013-4f9d906ae518
602024-10-09T17:24:00.868ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
612024-10-09T17:24:00.868ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
622024-10-09T17:24:00.868ZINFOcrucible-agent (worker): commit
632024-10-09T17:24:00.879ZINFOcrucible-agent (worker): ok!
642024-10-09T17:24:00.879ZINFOcrucible-agent (worker): SMF ok!
652024-10-09T17:24:01.033ZINFOcrucible-agent (dropshot): request completed latency_us = 68 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:50326 req_id = b5be7a8f-2300-40c8-ad13-1c6467340a6e response_code = 200 uri = /crucible/0/regions
662024-10-09T17:24:01.479ZINFOcrucible-agent (dropshot): request completed latency_us = 2677 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:50326 req_id = 7908e364-35e3-4f8d-b0db-8361ffe706b0 response_code = 200 uri = /crucible/0/regions/4d897563-5295-4f57-8013-4f9d906ae518
672024-10-09T17:24:01.484ZINFOcrucible-agent (datafile): path is "/data/regions/4d897563-5295-4f57-8013-4f9d906ae518"
682024-10-09T17:24:01.494ZINFOcrucible-agent (datafile): dataset is oxp_b629ce16-33f2-4e5d-8506-138eb852a76c/crucible/regions/4d897563-5295-4f57-8013-4f9d906ae518
692024-10-09T17:24:01.504ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_b629ce16-33f2-4e5d-8506-138eb852a76c/crucible/regions/4d897563-5295-4f57-8013-4f9d906ae518@5a769270-2edb-4f6e-8f69-40585f1d8665
702024-10-09T17:24:01.504ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_b629ce16-33f2-4e5d-8506-138eb852a76c/crucible/regions/4d897563-5295-4f57-8013-4f9d906ae518", "5a769270-2edb-4f6e-8f69-40585f1d8665"]
712024-10-09T17:24:01.517ZINFOcrucible-agent (dropshot): request completed latency_us = 33824 local_addr = [fd00:1122:3344:101::1a]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:50326 req_id = 3f3c4365-31f1-43a9-b1bf-613fd3f97f1a response_code = 200 uri = /crucible/0/regions/4d897563-5295-4f57-8013-4f9d906ae518/snapshots/5a769270-2edb-4f6e-8f69-40585f1d8665
722024-10-09T17:24:01.524ZINFOcrucible-agent (datafile): path is "/data/regions/4d897563-5295-4f57-8013-4f9d906ae518"
732024-10-09T17:24:01.532ZINFOcrucible-agent (datafile): dataset is oxp_b629ce16-33f2-4e5d-8506-138eb852a76c/crucible/regions/4d897563-5295-4f57-8013-4f9d906ae518
742024-10-09T17:24:01.539ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_b629ce16-33f2-4e5d-8506-138eb852a76c/crucible/regions/4d897563-5295-4f57-8013-4f9d906ae518@5a769270-2edb-4f6e-8f69-40585f1d8665
752024-10-09T17:24:01.539ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_b629ce16-33f2-4e5d-8506-138eb852a76c/crucible/regions/4d897563-5295-4f57-8013-4f9d906ae518", "5a769270-2edb-4f6e-8f69-40585f1d8665"]
762024-10-09T17:24:01.549ZINFOcrucible-agent (datafile): requesting running snapshot 4d897563-5295-4f57-8013-4f9d906ae518-5a769270-2edb-4f6e-8f69-40585f1d8665 state: Requested
772024-10-09T17:24:01.549ZINFOcrucible-agent (worker): applying SMF actions for region 4d897563-5295-4f57-8013-4f9d906ae518 running snapshot 5a769270-2edb-4f6e-8f69-40585f1d8665 (state Requested)...
782024-10-09T17:24:01.549ZINFOcrucible-agent (dropshot): request completed latency_us = 25245 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:50326 req_id = 3f2f927b-2fcf-46de-83b2-625567fb75e8 response_code = 200 uri = /crucible/0/regions/4d897563-5295-4f57-8013-4f9d906ae518/snapshots/5a769270-2edb-4f6e-8f69-40585f1d8665/run
792024-10-09T17:24:01.551ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/4d897563-5295-4f57-8013-4f9d906ae518
802024-10-09T17:24:01.551ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
812024-10-09T17:24:01.551ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::1a
822024-10-09T17:24:01.551ZINFOcrucible-agent (worker): reconfig required
832024-10-09T17:24:01.551ZINFOcrucible-agent (worker): using existing config property group
842024-10-09T17:24:01.551ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-4d897563-5295-4f57-8013-4f9d906ae518
852024-10-09T17:24:01.552ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/4d897563-5295-4f57-8013-4f9d906ae518
862024-10-09T17:24:01.552ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
872024-10-09T17:24:01.552ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
882024-10-09T17:24:01.552ZINFOcrucible-agent (worker): commit
892024-10-09T17:24:01.555ZINFOcrucible-agent (worker): ok!
902024-10-09T17:24:01.556ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-4d897563-5295-4f57-8013-4f9d906ae518-5a769270-2edb-4f6e-8f69-40585f1d8665
912024-10-09T17:24:01.560ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-4d897563-5295-4f57-8013-4f9d906ae518-5a769270-2edb-4f6e-8f69-40585f1d8665
922024-10-09T17:24:01.568ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-4d897563-5295-4f57-8013-4f9d906ae518-5a769270-2edb-4f6e-8f69-40585f1d8665
932024-10-09T17:24:01.569ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/4d897563-5295-4f57-8013-4f9d906ae518/.zfs/snapshot/5a769270-2edb-4f6e-8f69-40585f1d8665
942024-10-09T17:24:01.569ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
952024-10-09T17:24:01.569ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
962024-10-09T17:24:01.569ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::1a
972024-10-09T17:24:01.569ZINFOcrucible-agent (worker): commit
982024-10-09T17:24:01.573ZINFOcrucible-agent (worker): ok!
992024-10-09T17:24:01.577ZINFOcrucible-agent (worker): SMF ok!
1002024-10-09T17:24:01.577ZINFOcrucible-agent (datafile): region 4d897563-5295-4f57-8013-4f9d906ae518 running snapshot 5a769270-2edb-4f6e-8f69-40585f1d8665 state: Requested -> Created
1012024-10-09T17:24:01.742ZINFOcrucible-agent (datafile): path is "/data/regions/4d897563-5295-4f57-8013-4f9d906ae518"
1022024-10-09T17:24:01.751ZINFOcrucible-agent (datafile): dataset is oxp_b629ce16-33f2-4e5d-8506-138eb852a76c/crucible/regions/4d897563-5295-4f57-8013-4f9d906ae518
1032024-10-09T17:24:01.759ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_b629ce16-33f2-4e5d-8506-138eb852a76c/crucible/regions/4d897563-5295-4f57-8013-4f9d906ae518@5a769270-2edb-4f6e-8f69-40585f1d8665
1042024-10-09T17:24:01.759ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_b629ce16-33f2-4e5d-8506-138eb852a76c/crucible/regions/4d897563-5295-4f57-8013-4f9d906ae518", "5a769270-2edb-4f6e-8f69-40585f1d8665"]
1052024-10-09T17:24:01.768ZINFOcrucible-agent (dropshot): request completed latency_us = 25641 local_addr = [fd00:1122:3344:101::1a]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:50326 req_id = 7d82f691-1d02-44a6-8b3d-ec144ad896a6 response_code = 200 uri = /crucible/0/regions/4d897563-5295-4f57-8013-4f9d906ae518/snapshots/5a769270-2edb-4f6e-8f69-40585f1d8665/run