Max level shown:
1 [ Sep 9 17:51:55 Enabled. ]
2 [ Sep 9 17:51:55 Rereading configuration. ]
3 [ Sep 9 17:51:56 Rereading configuration. ]
4 [ Sep 9 17:51:57 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_89ffd24b-1dd8-4f52-85d7-6f3c66fce582/crucible -l [fd00:1122:3344:101::15]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62024-09-09T17:51:57.449ZINFOcrucible-agent: dataset: "oxp_89ffd24b-1dd8-4f52-85d7-6f3c66fce582/crucible"
72024-09-09T17:51:57.450ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::15]:32345
82024-09-09T17:51:57.450ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92024-09-09T17:51:57.467ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102024-09-09T17:51:57.590ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::15]:32345
112024-09-09T17:53:25.570ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::15]:32345 remote_addr = [fd00:1122:3344:101::a]:61308
122024-09-09T17:53:25.591ZINFOcrucible-agent (datafile): region 78394767-b1be-480c-95e6-a7808d755dd4 state: Requested
132024-09-09T17:53:25.596ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
142024-09-09T17:53:25.600ZINFOcrucible-agent (dropshot): request completed latency_us = 15941 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:61308 req_id = cd3ba618-4219-40b6-94c9-462a9ba4f8e8 response_code = 200 uri = /crucible/0/regions
152024-09-09T17:53:25.608ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_89ffd24b-1dd8-4f52-85d7-6f3c66fce582/crucible/regions/78394767-b1be-480c-95e6-a7808d755dd4
162024-09-09T17:53:25.608ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_89ffd24b-1dd8-4f52-85d7-6f3c66fce582/crucible/regions/78394767-b1be-480c-95e6-a7808d755dd4
172024-09-09T17:53:25.667ZINFOcrucible-agent (worker): creating region Region { id: RegionId("78394767-b1be-480c-95e6-a7808d755dd4"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/78394767-b1be-480c-95e6-a7808d755dd4" region = 78394767-b1be-480c-95e6-a7808d755dd4
182024-09-09T17:53:25.669ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "78394767-b1be-480c-95e6-a7808d755dd4" "--data" "/data/regions/78394767-b1be-480c-95e6-a7808d755dd4" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 78394767-b1be-480c-95e6-a7808d755dd4
192024-09-09T17:53:25.795ZINFOcrucible-agent (worker): region files created ok region = 78394767-b1be-480c-95e6-a7808d755dd4
202024-09-09T17:53:25.795ZINFOcrucible-agent (datafile): region 78394767-b1be-480c-95e6-a7808d755dd4 state: Requested -> Created
212024-09-09T17:53:25.795ZINFOcrucible-agent (worker): applying SMF actions post create...
222024-09-09T17:53:25.795ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-78394767-b1be-480c-95e6-a7808d755dd4
232024-09-09T17:53:25.799ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-78394767-b1be-480c-95e6-a7808d755dd4
242024-09-09T17:53:25.817ZINFOcrucible-agent (worker): creating config property group
252024-09-09T17:53:25.820ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-78394767-b1be-480c-95e6-a7808d755dd4
262024-09-09T17:53:25.828ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/78394767-b1be-480c-95e6-a7808d755dd4
272024-09-09T17:53:25.830ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
282024-09-09T17:53:25.830ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
292024-09-09T17:53:25.830ZINFOcrucible-agent (worker): commit
302024-09-09T17:53:25.834ZINFOcrucible-agent (worker): ok!
312024-09-09T17:53:25.838ZINFOcrucible-agent (worker): SMF ok!
322024-09-09T17:53:25.858ZINFOcrucible-agent (dropshot): request completed latency_us = 66 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:61308 req_id = 9f7bf872-8986-4c4a-bdf9-5ae172e54408 response_code = 200 uri = /crucible/0/regions
332024-09-09T17:58:49.486ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::15]:32345 remote_addr = [fd00:1122:3344:101::a]:65238
342024-09-09T17:58:49.486ZINFOcrucible-agent (datafile): region 8ff8670c-4423-4617-a7cb-31393494a6c1 state: Requested
352024-09-09T17:58:49.487ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
362024-09-09T17:58:49.487ZINFOcrucible-agent (dropshot): request completed latency_us = 974 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:65238 req_id = cdd56012-0bc2-4b27-b6b4-0146a1c55334 response_code = 200 uri = /crucible/0/regions
372024-09-09T17:58:49.498ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_89ffd24b-1dd8-4f52-85d7-6f3c66fce582/crucible/regions/8ff8670c-4423-4617-a7cb-31393494a6c1
382024-09-09T17:58:49.498ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_89ffd24b-1dd8-4f52-85d7-6f3c66fce582/crucible/regions/8ff8670c-4423-4617-a7cb-31393494a6c1
392024-09-09T17:58:49.557ZINFOcrucible-agent (worker): creating region Region { id: RegionId("8ff8670c-4423-4617-a7cb-31393494a6c1"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/8ff8670c-4423-4617-a7cb-31393494a6c1" region = 8ff8670c-4423-4617-a7cb-31393494a6c1
402024-09-09T17:58:49.557ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "8ff8670c-4423-4617-a7cb-31393494a6c1" "--data" "/data/regions/8ff8670c-4423-4617-a7cb-31393494a6c1" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 8ff8670c-4423-4617-a7cb-31393494a6c1
412024-09-09T17:58:49.676ZINFOcrucible-agent (worker): region files created ok region = 8ff8670c-4423-4617-a7cb-31393494a6c1
422024-09-09T17:58:49.676ZINFOcrucible-agent (datafile): region 8ff8670c-4423-4617-a7cb-31393494a6c1 state: Requested -> Created
432024-09-09T17:58:49.677ZINFOcrucible-agent (worker): applying SMF actions post create...
442024-09-09T17:58:49.684ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/78394767-b1be-480c-95e6-a7808d755dd4
452024-09-09T17:58:49.684ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
462024-09-09T17:58:49.684ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::15
472024-09-09T17:58:49.684ZINFOcrucible-agent (worker): reconfig required
482024-09-09T17:58:49.684ZINFOcrucible-agent (worker): using existing config property group
492024-09-09T17:58:49.684ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-78394767-b1be-480c-95e6-a7808d755dd4
502024-09-09T17:58:49.684ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/78394767-b1be-480c-95e6-a7808d755dd4
512024-09-09T17:58:49.684ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
522024-09-09T17:58:49.684ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
532024-09-09T17:58:49.684ZINFOcrucible-agent (worker): commit
542024-09-09T17:58:49.692ZINFOcrucible-agent (worker): ok!
552024-09-09T17:58:49.692ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-8ff8670c-4423-4617-a7cb-31393494a6c1
562024-09-09T17:58:49.695ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-8ff8670c-4423-4617-a7cb-31393494a6c1
572024-09-09T17:58:49.701ZINFOcrucible-agent (worker): creating config property group
582024-09-09T17:58:49.708ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-8ff8670c-4423-4617-a7cb-31393494a6c1
592024-09-09T17:58:49.708ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/8ff8670c-4423-4617-a7cb-31393494a6c1
602024-09-09T17:58:49.708ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
612024-09-09T17:58:49.708ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
622024-09-09T17:58:49.708ZINFOcrucible-agent (worker): commit
632024-09-09T17:58:49.713ZINFOcrucible-agent (worker): ok!
642024-09-09T17:58:49.717ZINFOcrucible-agent (worker): SMF ok!
652024-09-09T17:58:49.732ZINFOcrucible-agent (dropshot): request completed latency_us = 67 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:65238 req_id = ba28db46-ab11-4bc9-a83f-213fc3e9b972 response_code = 200 uri = /crucible/0/regions
662024-09-09T17:58:51.354ZINFOcrucible-agent (dropshot): request completed latency_us = 58 local_addr = [fd00:1122:3344:101::15]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:65238 req_id = e082e6cd-47ec-4e0e-9f34-4b96b7aa0af6 response_code = 200 uri = /crucible/0/regions/78394767-b1be-480c-95e6-a7808d755dd4
672024-09-09T17:58:51.359ZINFOcrucible-agent (datafile): path is "/data/regions/78394767-b1be-480c-95e6-a7808d755dd4"
682024-09-09T17:58:51.369ZINFOcrucible-agent (datafile): dataset is oxp_89ffd24b-1dd8-4f52-85d7-6f3c66fce582/crucible/regions/78394767-b1be-480c-95e6-a7808d755dd4
692024-09-09T17:58:51.377ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_89ffd24b-1dd8-4f52-85d7-6f3c66fce582/crucible/regions/78394767-b1be-480c-95e6-a7808d755dd4@ebdc67ef-6bb2-44c8-b1d0-da862a2ac38a
702024-09-09T17:58:51.377ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_89ffd24b-1dd8-4f52-85d7-6f3c66fce582/crucible/regions/78394767-b1be-480c-95e6-a7808d755dd4", "ebdc67ef-6bb2-44c8-b1d0-da862a2ac38a"]
712024-09-09T17:58:51.388ZINFOcrucible-agent (dropshot): request completed latency_us = 31667 local_addr = [fd00:1122:3344:101::15]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:65238 req_id = 96aaa381-c55a-475d-ba03-6e9fe67343eb response_code = 200 uri = /crucible/0/regions/78394767-b1be-480c-95e6-a7808d755dd4/snapshots/ebdc67ef-6bb2-44c8-b1d0-da862a2ac38a
722024-09-09T17:58:51.392ZINFOcrucible-agent (datafile): path is "/data/regions/78394767-b1be-480c-95e6-a7808d755dd4"
732024-09-09T17:58:51.401ZINFOcrucible-agent (datafile): dataset is oxp_89ffd24b-1dd8-4f52-85d7-6f3c66fce582/crucible/regions/78394767-b1be-480c-95e6-a7808d755dd4
742024-09-09T17:58:51.409ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_89ffd24b-1dd8-4f52-85d7-6f3c66fce582/crucible/regions/78394767-b1be-480c-95e6-a7808d755dd4@ebdc67ef-6bb2-44c8-b1d0-da862a2ac38a
752024-09-09T17:58:51.409ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_89ffd24b-1dd8-4f52-85d7-6f3c66fce582/crucible/regions/78394767-b1be-480c-95e6-a7808d755dd4", "ebdc67ef-6bb2-44c8-b1d0-da862a2ac38a"]
762024-09-09T17:58:51.419ZINFOcrucible-agent (datafile): requesting running snapshot 78394767-b1be-480c-95e6-a7808d755dd4-ebdc67ef-6bb2-44c8-b1d0-da862a2ac38a state: Requested
772024-09-09T17:58:51.419ZINFOcrucible-agent (worker): applying SMF actions for region 78394767-b1be-480c-95e6-a7808d755dd4 running snapshot ebdc67ef-6bb2-44c8-b1d0-da862a2ac38a (state Requested)...
782024-09-09T17:58:51.419ZINFOcrucible-agent (dropshot): request completed latency_us = 26761 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:65238 req_id = e24160e2-0fe8-4111-9cee-fd8c0818a8fa response_code = 200 uri = /crucible/0/regions/78394767-b1be-480c-95e6-a7808d755dd4/snapshots/ebdc67ef-6bb2-44c8-b1d0-da862a2ac38a/run
792024-09-09T17:58:51.420ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/78394767-b1be-480c-95e6-a7808d755dd4
802024-09-09T17:58:51.420ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
812024-09-09T17:58:51.420ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::15
822024-09-09T17:58:51.420ZINFOcrucible-agent (worker): reconfig required
832024-09-09T17:58:51.420ZINFOcrucible-agent (worker): using existing config property group
842024-09-09T17:58:51.420ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-78394767-b1be-480c-95e6-a7808d755dd4
852024-09-09T17:58:51.420ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/78394767-b1be-480c-95e6-a7808d755dd4
862024-09-09T17:58:51.420ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
872024-09-09T17:58:51.420ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
882024-09-09T17:58:51.420ZINFOcrucible-agent (worker): commit
892024-09-09T17:58:51.424ZINFOcrucible-agent (worker): ok!
902024-09-09T17:58:51.425ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-78394767-b1be-480c-95e6-a7808d755dd4-ebdc67ef-6bb2-44c8-b1d0-da862a2ac38a
912024-09-09T17:58:51.429ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-78394767-b1be-480c-95e6-a7808d755dd4-ebdc67ef-6bb2-44c8-b1d0-da862a2ac38a
922024-09-09T17:58:51.438ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-78394767-b1be-480c-95e6-a7808d755dd4-ebdc67ef-6bb2-44c8-b1d0-da862a2ac38a
932024-09-09T17:58:51.439ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/78394767-b1be-480c-95e6-a7808d755dd4/.zfs/snapshot/ebdc67ef-6bb2-44c8-b1d0-da862a2ac38a
942024-09-09T17:58:51.439ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
952024-09-09T17:58:51.439ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
962024-09-09T17:58:51.439ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::15
972024-09-09T17:58:51.439ZINFOcrucible-agent (worker): commit
982024-09-09T17:58:51.443ZINFOcrucible-agent (worker): ok!
992024-09-09T17:58:51.448ZINFOcrucible-agent (worker): SMF ok!
1002024-09-09T17:58:51.448ZINFOcrucible-agent (datafile): region 78394767-b1be-480c-95e6-a7808d755dd4 running snapshot ebdc67ef-6bb2-44c8-b1d0-da862a2ac38a state: Requested -> Created
1012024-09-09T17:58:51.786ZINFOcrucible-agent (datafile): path is "/data/regions/78394767-b1be-480c-95e6-a7808d755dd4"
1022024-09-09T17:58:51.796ZINFOcrucible-agent (datafile): dataset is oxp_89ffd24b-1dd8-4f52-85d7-6f3c66fce582/crucible/regions/78394767-b1be-480c-95e6-a7808d755dd4
1032024-09-09T17:58:51.805ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_89ffd24b-1dd8-4f52-85d7-6f3c66fce582/crucible/regions/78394767-b1be-480c-95e6-a7808d755dd4@ebdc67ef-6bb2-44c8-b1d0-da862a2ac38a
1042024-09-09T17:58:51.805ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_89ffd24b-1dd8-4f52-85d7-6f3c66fce582/crucible/regions/78394767-b1be-480c-95e6-a7808d755dd4", "ebdc67ef-6bb2-44c8-b1d0-da862a2ac38a"]
1052024-09-09T17:58:51.813ZINFOcrucible-agent (dropshot): request completed latency_us = 27416 local_addr = [fd00:1122:3344:101::15]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:65238 req_id = 47042f48-7cc1-47a2-ac0c-b1680a1ff88b response_code = 200 uri = /crucible/0/regions/78394767-b1be-480c-95e6-a7808d755dd4/snapshots/ebdc67ef-6bb2-44c8-b1d0-da862a2ac38a/run