Max level shown:
1 [ Sep 4 16:52:06 Enabled. ]
2 [ Sep 4 16:52:06 Rereading configuration. ]
3 [ Sep 4 16:52:08 Rereading configuration. ]
4 [ Sep 4 16:52:12 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_41da29c9-1a43-49cc-b472-5dde9cb41a62/crucible -l [fd00:1122:3344:101::17]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62024-09-04T16:52:12.897ZINFOcrucible-agent: dataset: "oxp_41da29c9-1a43-49cc-b472-5dde9cb41a62/crucible"
72024-09-04T16:52:12.913ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::17]:32345
82024-09-04T16:52:12.913ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92024-09-04T16:52:12.919ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102024-09-04T16:52:13.141ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::17]:32345
112024-09-04T16:52:53.235ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::17]:32345 remote_addr = [fd00:1122:3344:101::a]:40055
122024-09-04T16:52:53.242ZINFOcrucible-agent (datafile): region 81cd44e9-eb62-4d08-8270-c69875b06ec9 state: Requested
132024-09-04T16:52:53.245ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
142024-09-04T16:52:53.245ZINFOcrucible-agent (dropshot): request completed latency_us = 3041 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:40055 req_id = c3da2d9d-f89a-49e3-b391-ace2b138ee77 response_code = 200 uri = /crucible/0/regions
152024-09-04T16:52:53.255ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_41da29c9-1a43-49cc-b472-5dde9cb41a62/crucible/regions/81cd44e9-eb62-4d08-8270-c69875b06ec9
162024-09-04T16:52:53.255ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_41da29c9-1a43-49cc-b472-5dde9cb41a62/crucible/regions/81cd44e9-eb62-4d08-8270-c69875b06ec9
172024-09-04T16:52:53.303ZINFOcrucible-agent (worker): creating region Region { id: RegionId("81cd44e9-eb62-4d08-8270-c69875b06ec9"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/81cd44e9-eb62-4d08-8270-c69875b06ec9" region = 81cd44e9-eb62-4d08-8270-c69875b06ec9
182024-09-04T16:52:53.303ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "81cd44e9-eb62-4d08-8270-c69875b06ec9" "--data" "/data/regions/81cd44e9-eb62-4d08-8270-c69875b06ec9" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 81cd44e9-eb62-4d08-8270-c69875b06ec9
192024-09-04T16:52:53.521ZINFOcrucible-agent (dropshot): request completed latency_us = 90 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:40055 req_id = d65828a9-c05d-4117-85f3-3fe86654410d response_code = 200 uri = /crucible/0/regions
202024-09-04T16:52:53.634ZINFOcrucible-agent (worker): region files created ok region = 81cd44e9-eb62-4d08-8270-c69875b06ec9
212024-09-04T16:52:53.634ZINFOcrucible-agent (datafile): region 81cd44e9-eb62-4d08-8270-c69875b06ec9 state: Requested -> Created
222024-09-04T16:52:53.635ZINFOcrucible-agent (worker): applying SMF actions post create...
232024-09-04T16:52:53.635ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-81cd44e9-eb62-4d08-8270-c69875b06ec9
242024-09-04T16:52:53.640ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-81cd44e9-eb62-4d08-8270-c69875b06ec9
252024-09-04T16:52:53.653ZINFOcrucible-agent (worker): creating config property group
262024-09-04T16:52:53.656ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-81cd44e9-eb62-4d08-8270-c69875b06ec9
272024-09-04T16:52:53.656ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/81cd44e9-eb62-4d08-8270-c69875b06ec9
282024-09-04T16:52:53.657ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
292024-09-04T16:52:53.657ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
302024-09-04T16:52:53.657ZINFOcrucible-agent (worker): commit
312024-09-04T16:52:53.661ZINFOcrucible-agent (worker): ok!
322024-09-04T16:52:53.665ZINFOcrucible-agent (worker): SMF ok!
332024-09-04T16:52:53.839ZINFOcrucible-agent (dropshot): request completed latency_us = 69 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:40055 req_id = c3714136-75d3-4c51-84fb-965ef4575faf response_code = 200 uri = /crucible/0/regions
342024-09-04T16:58:12.398ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::17]:32345 remote_addr = [fd00:1122:3344:101::a]:33223
352024-09-04T16:58:12.398ZINFOcrucible-agent (dropshot): request completed latency_us = 576 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:33223 req_id = d90dd42c-1c74-4e35-8385-e2e2b85ea78e response_code = 200 uri = /crucible/0/regions/81cd44e9-eb62-4d08-8270-c69875b06ec9
362024-09-04T16:58:12.404ZINFOcrucible-agent (datafile): path is "/data/regions/81cd44e9-eb62-4d08-8270-c69875b06ec9"
372024-09-04T16:58:12.414ZINFOcrucible-agent (datafile): dataset is oxp_41da29c9-1a43-49cc-b472-5dde9cb41a62/crucible/regions/81cd44e9-eb62-4d08-8270-c69875b06ec9
382024-09-04T16:58:12.423ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_41da29c9-1a43-49cc-b472-5dde9cb41a62/crucible/regions/81cd44e9-eb62-4d08-8270-c69875b06ec9@f2109d8e-886b-4781-9ff1-7afa29fccf52
392024-09-04T16:58:12.423ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_41da29c9-1a43-49cc-b472-5dde9cb41a62/crucible/regions/81cd44e9-eb62-4d08-8270-c69875b06ec9", "f2109d8e-886b-4781-9ff1-7afa29fccf52"]
402024-09-04T16:58:12.435ZINFOcrucible-agent (dropshot): request completed latency_us = 33267 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:33223 req_id = f47254a1-b094-4a6f-b1a4-ae7a068018f7 response_code = 200 uri = /crucible/0/regions/81cd44e9-eb62-4d08-8270-c69875b06ec9/snapshots/f2109d8e-886b-4781-9ff1-7afa29fccf52
412024-09-04T16:58:12.438ZINFOcrucible-agent (datafile): path is "/data/regions/81cd44e9-eb62-4d08-8270-c69875b06ec9"
422024-09-04T16:58:12.447ZINFOcrucible-agent (datafile): dataset is oxp_41da29c9-1a43-49cc-b472-5dde9cb41a62/crucible/regions/81cd44e9-eb62-4d08-8270-c69875b06ec9
432024-09-04T16:58:12.455ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_41da29c9-1a43-49cc-b472-5dde9cb41a62/crucible/regions/81cd44e9-eb62-4d08-8270-c69875b06ec9@f2109d8e-886b-4781-9ff1-7afa29fccf52
442024-09-04T16:58:12.455ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_41da29c9-1a43-49cc-b472-5dde9cb41a62/crucible/regions/81cd44e9-eb62-4d08-8270-c69875b06ec9", "f2109d8e-886b-4781-9ff1-7afa29fccf52"]
452024-09-04T16:58:12.463ZINFOcrucible-agent (datafile): requesting running snapshot 81cd44e9-eb62-4d08-8270-c69875b06ec9-f2109d8e-886b-4781-9ff1-7afa29fccf52 state: Requested
462024-09-04T16:58:12.464ZINFOcrucible-agent (worker): applying SMF actions for region 81cd44e9-eb62-4d08-8270-c69875b06ec9 running snapshot f2109d8e-886b-4781-9ff1-7afa29fccf52 (state Requested)...
472024-09-04T16:58:12.464ZINFOcrucible-agent (dropshot): request completed latency_us = 25201 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:33223 req_id = f4cf71aa-302c-4e01-9635-bd0c591667b8 response_code = 200 uri = /crucible/0/regions/81cd44e9-eb62-4d08-8270-c69875b06ec9/snapshots/f2109d8e-886b-4781-9ff1-7afa29fccf52/run
482024-09-04T16:58:12.471ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-81cd44e9-eb62-4d08-8270-c69875b06ec9-f2109d8e-886b-4781-9ff1-7afa29fccf52
492024-09-04T16:58:12.474ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-81cd44e9-eb62-4d08-8270-c69875b06ec9-f2109d8e-886b-4781-9ff1-7afa29fccf52
502024-09-04T16:58:12.485ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-81cd44e9-eb62-4d08-8270-c69875b06ec9-f2109d8e-886b-4781-9ff1-7afa29fccf52
512024-09-04T16:58:12.486ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/81cd44e9-eb62-4d08-8270-c69875b06ec9/.zfs/snapshot/f2109d8e-886b-4781-9ff1-7afa29fccf52
522024-09-04T16:58:12.486ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
532024-09-04T16:58:12.486ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
542024-09-04T16:58:12.486ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
552024-09-04T16:58:12.486ZINFOcrucible-agent (worker): commit
562024-09-04T16:58:12.489ZINFOcrucible-agent (worker): ok!
572024-09-04T16:58:12.493ZINFOcrucible-agent (worker): SMF ok!
582024-09-04T16:58:12.493ZINFOcrucible-agent (datafile): region 81cd44e9-eb62-4d08-8270-c69875b06ec9 running snapshot f2109d8e-886b-4781-9ff1-7afa29fccf52 state: Requested -> Created
592024-09-04T16:58:12.835ZINFOcrucible-agent (datafile): path is "/data/regions/81cd44e9-eb62-4d08-8270-c69875b06ec9"
602024-09-04T16:58:12.844ZINFOcrucible-agent (datafile): dataset is oxp_41da29c9-1a43-49cc-b472-5dde9cb41a62/crucible/regions/81cd44e9-eb62-4d08-8270-c69875b06ec9
612024-09-04T16:58:12.853ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_41da29c9-1a43-49cc-b472-5dde9cb41a62/crucible/regions/81cd44e9-eb62-4d08-8270-c69875b06ec9@f2109d8e-886b-4781-9ff1-7afa29fccf52
622024-09-04T16:58:12.853ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_41da29c9-1a43-49cc-b472-5dde9cb41a62/crucible/regions/81cd44e9-eb62-4d08-8270-c69875b06ec9", "f2109d8e-886b-4781-9ff1-7afa29fccf52"]
632024-09-04T16:58:12.860ZINFOcrucible-agent (dropshot): request completed latency_us = 24882 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:33223 req_id = 0f144c61-1a9b-4aac-8911-5e9ded3d3901 response_code = 200 uri = /crucible/0/regions/81cd44e9-eb62-4d08-8270-c69875b06ec9/snapshots/f2109d8e-886b-4781-9ff1-7afa29fccf52/run