Max level shown:
1 [ Sep 3 23:51:12 Enabled. ]
2 [ Sep 3 23:51:12 Rereading configuration. ]
3 [ Sep 3 23:51:14 Rereading configuration. ]
4 [ Sep 3 23:51:15 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_004daad2-caa8-4a89-a0fc-7c1f8227b87e/crucible -l [fd00:1122:3344:101::15]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62024-09-03T23:51:15.783ZINFOcrucible-agent: dataset: "oxp_004daad2-caa8-4a89-a0fc-7c1f8227b87e/crucible"
72024-09-03T23:51:15.783ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::15]:32345
82024-09-03T23:51:15.783ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92024-09-03T23:51:15.877ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102024-09-03T23:51:16.064ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::15]:32345
112024-09-03T23:51:50.460ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::15]:32345 remote_addr = [fd00:1122:3344:101::a]:63693
122024-09-03T23:51:50.489ZINFOcrucible-agent (datafile): region 4520b9e3-cf6a-4de8-b41d-43360d867ca7 state: Requested
132024-09-03T23:51:50.495ZINFOcrucible-agent (dropshot): request completed latency_us = 21343 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:63693 req_id = 4b7ca72d-b969-4ff6-89b1-b9a9bac73210 response_code = 200 uri = /crucible/0/regions
142024-09-03T23:51:50.496ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
152024-09-03T23:51:50.510ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_004daad2-caa8-4a89-a0fc-7c1f8227b87e/crucible/regions/4520b9e3-cf6a-4de8-b41d-43360d867ca7
162024-09-03T23:51:50.510ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_004daad2-caa8-4a89-a0fc-7c1f8227b87e/crucible/regions/4520b9e3-cf6a-4de8-b41d-43360d867ca7
172024-09-03T23:51:50.601ZINFOcrucible-agent (worker): creating region Region { id: RegionId("4520b9e3-cf6a-4de8-b41d-43360d867ca7"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/4520b9e3-cf6a-4de8-b41d-43360d867ca7" region = 4520b9e3-cf6a-4de8-b41d-43360d867ca7
182024-09-03T23:51:50.603ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "4520b9e3-cf6a-4de8-b41d-43360d867ca7" "--data" "/data/regions/4520b9e3-cf6a-4de8-b41d-43360d867ca7" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 4520b9e3-cf6a-4de8-b41d-43360d867ca7
192024-09-03T23:51:50.869ZINFOcrucible-agent (dropshot): request completed latency_us = 69 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:63693 req_id = 5abc4df1-de14-426e-a59f-e01eb629d27a response_code = 200 uri = /crucible/0/regions
202024-09-03T23:51:50.872ZINFOcrucible-agent (worker): region files created ok region = 4520b9e3-cf6a-4de8-b41d-43360d867ca7
212024-09-03T23:51:50.872ZINFOcrucible-agent (datafile): region 4520b9e3-cf6a-4de8-b41d-43360d867ca7 state: Requested -> Created
222024-09-03T23:51:50.872ZINFOcrucible-agent (worker): applying SMF actions post create...
232024-09-03T23:51:50.872ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-4520b9e3-cf6a-4de8-b41d-43360d867ca7
242024-09-03T23:51:50.878ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-4520b9e3-cf6a-4de8-b41d-43360d867ca7
252024-09-03T23:51:50.891ZINFOcrucible-agent (worker): creating config property group
262024-09-03T23:51:50.894ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-4520b9e3-cf6a-4de8-b41d-43360d867ca7
272024-09-03T23:51:50.902ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/4520b9e3-cf6a-4de8-b41d-43360d867ca7
282024-09-03T23:51:50.904ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
292024-09-03T23:51:50.904ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
302024-09-03T23:51:50.904ZINFOcrucible-agent (worker): commit
312024-09-03T23:51:50.908ZINFOcrucible-agent (worker): ok!
322024-09-03T23:51:50.914ZINFOcrucible-agent (worker): SMF ok!
332024-09-03T23:51:51.322ZINFOcrucible-agent (dropshot): request completed latency_us = 59 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:63693 req_id = 864fcf35-ff95-40ae-b251-3a57f75408d6 response_code = 200 uri = /crucible/0/regions
342024-09-03T23:56:55.921ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::15]:32345 remote_addr = [fd00:1122:3344:101::a]:64994
352024-09-03T23:56:55.921ZINFOcrucible-agent (datafile): region bf71b1db-e4aa-4f00-b327-7c0f51cd253e state: Requested
362024-09-03T23:56:55.922ZINFOcrucible-agent (dropshot): request completed latency_us = 872 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:64994 req_id = 8098b1de-1002-4013-aef1-e48b98f39934 response_code = 200 uri = /crucible/0/regions
372024-09-03T23:56:55.922ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
382024-09-03T23:56:55.932ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_004daad2-caa8-4a89-a0fc-7c1f8227b87e/crucible/regions/bf71b1db-e4aa-4f00-b327-7c0f51cd253e
392024-09-03T23:56:55.932ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_004daad2-caa8-4a89-a0fc-7c1f8227b87e/crucible/regions/bf71b1db-e4aa-4f00-b327-7c0f51cd253e
402024-09-03T23:56:55.983ZINFOcrucible-agent (worker): creating region Region { id: RegionId("bf71b1db-e4aa-4f00-b327-7c0f51cd253e"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/bf71b1db-e4aa-4f00-b327-7c0f51cd253e" region = bf71b1db-e4aa-4f00-b327-7c0f51cd253e
412024-09-03T23:56:55.985ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "bf71b1db-e4aa-4f00-b327-7c0f51cd253e" "--data" "/data/regions/bf71b1db-e4aa-4f00-b327-7c0f51cd253e" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = bf71b1db-e4aa-4f00-b327-7c0f51cd253e
422024-09-03T23:56:56.101ZINFOcrucible-agent (worker): region files created ok region = bf71b1db-e4aa-4f00-b327-7c0f51cd253e
432024-09-03T23:56:56.101ZINFOcrucible-agent (datafile): region bf71b1db-e4aa-4f00-b327-7c0f51cd253e state: Requested -> Created
442024-09-03T23:56:56.102ZINFOcrucible-agent (worker): applying SMF actions post create...
452024-09-03T23:56:56.110ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/4520b9e3-cf6a-4de8-b41d-43360d867ca7
462024-09-03T23:56:56.110ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
472024-09-03T23:56:56.110ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::15
482024-09-03T23:56:56.110ZINFOcrucible-agent (worker): reconfig required
492024-09-03T23:56:56.110ZINFOcrucible-agent (worker): using existing config property group
502024-09-03T23:56:56.110ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-4520b9e3-cf6a-4de8-b41d-43360d867ca7
512024-09-03T23:56:56.110ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/4520b9e3-cf6a-4de8-b41d-43360d867ca7
522024-09-03T23:56:56.113ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
532024-09-03T23:56:56.113ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
542024-09-03T23:56:56.113ZINFOcrucible-agent (worker): commit
552024-09-03T23:56:56.121ZINFOcrucible-agent (worker): ok!
562024-09-03T23:56:56.121ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-bf71b1db-e4aa-4f00-b327-7c0f51cd253e
572024-09-03T23:56:56.124ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-bf71b1db-e4aa-4f00-b327-7c0f51cd253e
582024-09-03T23:56:56.130ZINFOcrucible-agent (worker): creating config property group
592024-09-03T23:56:56.134ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-bf71b1db-e4aa-4f00-b327-7c0f51cd253e
602024-09-03T23:56:56.134ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/bf71b1db-e4aa-4f00-b327-7c0f51cd253e
612024-09-03T23:56:56.134ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
622024-09-03T23:56:56.134ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
632024-09-03T23:56:56.134ZINFOcrucible-agent (worker): commit
642024-09-03T23:56:56.138ZINFOcrucible-agent (worker): ok!
652024-09-03T23:56:56.143ZINFOcrucible-agent (worker): SMF ok!
662024-09-03T23:56:56.208ZINFOcrucible-agent (dropshot): request completed latency_us = 65 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:64994 req_id = 452c3c6b-808a-4ee2-938e-6a8033211d0a response_code = 200 uri = /crucible/0/regions
672024-09-03T23:56:57.259ZINFOcrucible-agent (dropshot): request completed latency_us = 3441 local_addr = [fd00:1122:3344:101::15]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:64994 req_id = 38ed9475-1179-4e4d-941f-67b18045886d response_code = 200 uri = /crucible/0/regions/4520b9e3-cf6a-4de8-b41d-43360d867ca7
682024-09-03T23:56:57.262ZINFOcrucible-agent (datafile): path is "/data/regions/4520b9e3-cf6a-4de8-b41d-43360d867ca7"
692024-09-03T23:56:57.271ZINFOcrucible-agent (datafile): dataset is oxp_004daad2-caa8-4a89-a0fc-7c1f8227b87e/crucible/regions/4520b9e3-cf6a-4de8-b41d-43360d867ca7
702024-09-03T23:56:57.279ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_004daad2-caa8-4a89-a0fc-7c1f8227b87e/crucible/regions/4520b9e3-cf6a-4de8-b41d-43360d867ca7@6cb8a04e-0263-457c-acc7-970602f6584b
712024-09-03T23:56:57.279ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_004daad2-caa8-4a89-a0fc-7c1f8227b87e/crucible/regions/4520b9e3-cf6a-4de8-b41d-43360d867ca7", "6cb8a04e-0263-457c-acc7-970602f6584b"]
722024-09-03T23:56:57.290ZINFOcrucible-agent (dropshot): request completed latency_us = 28239 local_addr = [fd00:1122:3344:101::15]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:64994 req_id = d2546f43-2f26-4569-8a2f-f66ee98de3b6 response_code = 200 uri = /crucible/0/regions/4520b9e3-cf6a-4de8-b41d-43360d867ca7/snapshots/6cb8a04e-0263-457c-acc7-970602f6584b
732024-09-03T23:56:57.294ZINFOcrucible-agent (datafile): path is "/data/regions/4520b9e3-cf6a-4de8-b41d-43360d867ca7"
742024-09-03T23:56:57.301ZINFOcrucible-agent (datafile): dataset is oxp_004daad2-caa8-4a89-a0fc-7c1f8227b87e/crucible/regions/4520b9e3-cf6a-4de8-b41d-43360d867ca7
752024-09-03T23:56:57.309ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_004daad2-caa8-4a89-a0fc-7c1f8227b87e/crucible/regions/4520b9e3-cf6a-4de8-b41d-43360d867ca7@6cb8a04e-0263-457c-acc7-970602f6584b
762024-09-03T23:56:57.309ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_004daad2-caa8-4a89-a0fc-7c1f8227b87e/crucible/regions/4520b9e3-cf6a-4de8-b41d-43360d867ca7", "6cb8a04e-0263-457c-acc7-970602f6584b"]
772024-09-03T23:56:57.318ZINFOcrucible-agent (datafile): requesting running snapshot 4520b9e3-cf6a-4de8-b41d-43360d867ca7-6cb8a04e-0263-457c-acc7-970602f6584b state: Requested
782024-09-03T23:56:57.318ZINFOcrucible-agent (worker): applying SMF actions for region 4520b9e3-cf6a-4de8-b41d-43360d867ca7 running snapshot 6cb8a04e-0263-457c-acc7-970602f6584b (state Requested)...
792024-09-03T23:56:57.318ZINFOcrucible-agent (dropshot): request completed latency_us = 24965 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:64994 req_id = f7b49675-e224-440f-8ffc-40d509155957 response_code = 200 uri = /crucible/0/regions/4520b9e3-cf6a-4de8-b41d-43360d867ca7/snapshots/6cb8a04e-0263-457c-acc7-970602f6584b/run
802024-09-03T23:56:57.319ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/4520b9e3-cf6a-4de8-b41d-43360d867ca7
812024-09-03T23:56:57.319ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
822024-09-03T23:56:57.320ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::15
832024-09-03T23:56:57.320ZINFOcrucible-agent (worker): reconfig required
842024-09-03T23:56:57.320ZINFOcrucible-agent (worker): using existing config property group
852024-09-03T23:56:57.320ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-4520b9e3-cf6a-4de8-b41d-43360d867ca7
862024-09-03T23:56:57.320ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/4520b9e3-cf6a-4de8-b41d-43360d867ca7
872024-09-03T23:56:57.320ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
882024-09-03T23:56:57.320ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
892024-09-03T23:56:57.320ZINFOcrucible-agent (worker): commit
902024-09-03T23:56:57.324ZINFOcrucible-agent (worker): ok!
912024-09-03T23:56:57.326ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-4520b9e3-cf6a-4de8-b41d-43360d867ca7-6cb8a04e-0263-457c-acc7-970602f6584b
922024-09-03T23:56:57.331ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-4520b9e3-cf6a-4de8-b41d-43360d867ca7-6cb8a04e-0263-457c-acc7-970602f6584b
932024-09-03T23:56:57.341ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-4520b9e3-cf6a-4de8-b41d-43360d867ca7-6cb8a04e-0263-457c-acc7-970602f6584b
942024-09-03T23:56:57.341ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/4520b9e3-cf6a-4de8-b41d-43360d867ca7/.zfs/snapshot/6cb8a04e-0263-457c-acc7-970602f6584b
952024-09-03T23:56:57.341ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
962024-09-03T23:56:57.341ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
972024-09-03T23:56:57.341ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
982024-09-03T23:56:57.341ZINFOcrucible-agent (worker): commit
992024-09-03T23:56:57.344ZINFOcrucible-agent (worker): ok!
1002024-09-03T23:56:57.348ZINFOcrucible-agent (worker): SMF ok!
1012024-09-03T23:56:57.348ZINFOcrucible-agent (datafile): region 4520b9e3-cf6a-4de8-b41d-43360d867ca7 running snapshot 6cb8a04e-0263-457c-acc7-970602f6584b state: Requested -> Created
1022024-09-03T23:56:57.647ZINFOcrucible-agent (datafile): path is "/data/regions/4520b9e3-cf6a-4de8-b41d-43360d867ca7"
1032024-09-03T23:56:57.656ZINFOcrucible-agent (datafile): dataset is oxp_004daad2-caa8-4a89-a0fc-7c1f8227b87e/crucible/regions/4520b9e3-cf6a-4de8-b41d-43360d867ca7
1042024-09-03T23:56:57.664ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_004daad2-caa8-4a89-a0fc-7c1f8227b87e/crucible/regions/4520b9e3-cf6a-4de8-b41d-43360d867ca7@6cb8a04e-0263-457c-acc7-970602f6584b
1052024-09-03T23:56:57.664ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_004daad2-caa8-4a89-a0fc-7c1f8227b87e/crucible/regions/4520b9e3-cf6a-4de8-b41d-43360d867ca7", "6cb8a04e-0263-457c-acc7-970602f6584b"]
1062024-09-03T23:56:57.672ZINFOcrucible-agent (dropshot): request completed latency_us = 24681 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:64994 req_id = 28c7614c-f0fd-4589-9992-a0290b9ea2d9 response_code = 200 uri = /crucible/0/regions/4520b9e3-cf6a-4de8-b41d-43360d867ca7/snapshots/6cb8a04e-0263-457c-acc7-970602f6584b/run