Max level shown:
12024-12-13T22:30:11.012ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::13]:32345 remote_addr = [fd00:1122:3344:101::c]:40745
22024-12-13T22:30:11.043ZINFOcrucible-agent (datafile): region 3f384ef5-b126-40f1-a399-de6e63551da9 state: Requested
32024-12-13T22:30:11.049ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
42024-12-13T22:30:11.053ZINFOcrucible-agent (dropshot): request completed latency_us = 16295 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:40745 req_id = 5cf84b7e-9a29-4240-b7c2-47a276c93956 response_code = 200 uri = /crucible/0/regions
52024-12-13T22:30:11.061ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_0d689a16-0a28-4b27-af5a-3b4bf99f365f/crucible/regions/3f384ef5-b126-40f1-a399-de6e63551da9
62024-12-13T22:30:11.061ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_0d689a16-0a28-4b27-af5a-3b4bf99f365f/crucible/regions/3f384ef5-b126-40f1-a399-de6e63551da9
72024-12-13T22:30:11.124ZINFOcrucible-agent (worker): creating region Region { id: RegionId("3f384ef5-b126-40f1-a399-de6e63551da9"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/3f384ef5-b126-40f1-a399-de6e63551da9" region = 3f384ef5-b126-40f1-a399-de6e63551da9
82024-12-13T22:30:11.126ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "3f384ef5-b126-40f1-a399-de6e63551da9" "--data" "/data/regions/3f384ef5-b126-40f1-a399-de6e63551da9" "--block-size" "512" "--extent-size" "131072" "--extent-count" "16" "--encrypted" region = 3f384ef5-b126-40f1-a399-de6e63551da9
92024-12-13T22:30:11.203ZINFOcrucible-agent (dropshot): request completed latency_us = 78 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:40745 req_id = 118fea0e-18fa-4707-95cd-8cb56656cbf4 response_code = 200 uri = /crucible/0/regions
102024-12-13T22:30:11.240ZINFOcrucible-agent (worker): region files created ok region = 3f384ef5-b126-40f1-a399-de6e63551da9
112024-12-13T22:30:11.240ZINFOcrucible-agent (datafile): region 3f384ef5-b126-40f1-a399-de6e63551da9 state: Requested -> Created
122024-12-13T22:30:11.240ZINFOcrucible-agent (worker): applying SMF actions post create...
132024-12-13T22:30:11.240ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-3f384ef5-b126-40f1-a399-de6e63551da9
142024-12-13T22:30:11.246ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-3f384ef5-b126-40f1-a399-de6e63551da9
152024-12-13T22:30:11.257ZINFOcrucible-agent (worker): creating config property group
162024-12-13T22:30:11.262ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-3f384ef5-b126-40f1-a399-de6e63551da9
172024-12-13T22:30:11.263ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/3f384ef5-b126-40f1-a399-de6e63551da9
182024-12-13T22:30:11.263ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
192024-12-13T22:30:11.263ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::13
202024-12-13T22:30:11.263ZINFOcrucible-agent (worker): commit
212024-12-13T22:30:11.267ZINFOcrucible-agent (worker): ok!
222024-12-13T22:30:11.273ZINFOcrucible-agent (worker): SMF ok!
232024-12-13T22:30:11.863ZINFOcrucible-agent (dropshot): request completed latency_us = 59 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:40745 req_id = 1ae2df90-a834-4bc2-a65e-f107d4245b46 response_code = 200 uri = /crucible/0/regions
242024-12-13T22:30:12.210ZINFOcrucible-agent (dropshot): request completed latency_us = 77 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:40745 req_id = a150008a-c92f-4c4b-b7da-735a514bf7b7 response_code = 200 uri = /crucible/0/regions/3f384ef5-b126-40f1-a399-de6e63551da9
252024-12-13T22:30:12.215ZINFOcrucible-agent (datafile): path is "/data/regions/3f384ef5-b126-40f1-a399-de6e63551da9"
262024-12-13T22:30:12.226ZINFOcrucible-agent (datafile): dataset is oxp_0d689a16-0a28-4b27-af5a-3b4bf99f365f/crucible/regions/3f384ef5-b126-40f1-a399-de6e63551da9
272024-12-13T22:30:12.236ZINFOcrucible-agent (datafile): region 3f384ef5-b126-40f1-a399-de6e63551da9 state: Created -> Tombstoned
282024-12-13T22:30:12.236ZINFOcrucible-agent (worker): applying SMF actions before removal...
292024-12-13T22:30:12.236ZINFOcrucible-agent (dropshot): request completed latency_us = 21003 local_addr = [fd00:1122:3344:101::13]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::c]:40745 req_id = 761dacae-251c-451e-9f54-5e59fc0e9590 response_code = 204 uri = /crucible/0/regions/3f384ef5-b126-40f1-a399-de6e63551da9
302024-12-13T22:30:12.239ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-3f384ef5-b126-40f1-a399-de6e63551da9 (instance states: (Some(Online), None))
312024-12-13T22:30:12.239ZINFOcrucible-agent (dropshot): request completed latency_us = 47 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:40745 req_id = e699d4ee-ec93-4873-85e4-f19debfe04bf response_code = 200 uri = /crucible/0/regions/3f384ef5-b126-40f1-a399-de6e63551da9
322024-12-13T22:30:12.246ZINFOcrucible-agent (worker): SMF ok!
332024-12-13T22:30:12.257ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_0d689a16-0a28-4b27-af5a-3b4bf99f365f/crucible/regions/3f384ef5-b126-40f1-a399-de6e63551da9" region = 3f384ef5-b126-40f1-a399-de6e63551da9
342024-12-13T22:30:12.322ZINFOcrucible-agent (datafile): region 3f384ef5-b126-40f1-a399-de6e63551da9 state: Tombstoned -> Destroyed
352024-12-13T22:30:12.337ZINFOcrucible-agent (dropshot): request completed latency_us = 81 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:40745 req_id = bb85862b-1436-4e73-8fab-a104512d8943 response_code = 200 uri = /crucible/0/regions/3f384ef5-b126-40f1-a399-de6e63551da9
362024-12-13T22:30:47.427ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::13]:32345 remote_addr = [fd00:1122:3344:101::a]:56570
372024-12-13T22:30:47.427ZINFOcrucible-agent (datafile): region 105f76bc-b792-46fa-a474-35e976fb8bb4 state: Requested
382024-12-13T22:30:47.427ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
392024-12-13T22:30:47.427ZINFOcrucible-agent (dropshot): request completed latency_us = 211 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:56570 req_id = 36bc92d8-efd5-43e0-9834-1fbdb92fff0d response_code = 200 uri = /crucible/0/regions
402024-12-13T22:30:47.437ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_0d689a16-0a28-4b27-af5a-3b4bf99f365f/crucible/regions/105f76bc-b792-46fa-a474-35e976fb8bb4
412024-12-13T22:30:47.437ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_0d689a16-0a28-4b27-af5a-3b4bf99f365f/crucible/regions/105f76bc-b792-46fa-a474-35e976fb8bb4
422024-12-13T22:30:47.496ZINFOcrucible-agent (worker): creating region Region { id: RegionId("105f76bc-b792-46fa-a474-35e976fb8bb4"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/105f76bc-b792-46fa-a474-35e976fb8bb4" region = 105f76bc-b792-46fa-a474-35e976fb8bb4
432024-12-13T22:30:47.496ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "105f76bc-b792-46fa-a474-35e976fb8bb4" "--data" "/data/regions/105f76bc-b792-46fa-a474-35e976fb8bb4" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 105f76bc-b792-46fa-a474-35e976fb8bb4
442024-12-13T22:30:47.568ZINFOcrucible-agent (worker): region files created ok region = 105f76bc-b792-46fa-a474-35e976fb8bb4
452024-12-13T22:30:47.568ZINFOcrucible-agent (datafile): region 105f76bc-b792-46fa-a474-35e976fb8bb4 state: Requested -> Created
462024-12-13T22:30:47.568ZINFOcrucible-agent (worker): applying SMF actions post create...
472024-12-13T22:30:47.569ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-105f76bc-b792-46fa-a474-35e976fb8bb4
482024-12-13T22:30:47.573ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-105f76bc-b792-46fa-a474-35e976fb8bb4
492024-12-13T22:30:47.581ZINFOcrucible-agent (worker): creating config property group
502024-12-13T22:30:47.583ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-105f76bc-b792-46fa-a474-35e976fb8bb4
512024-12-13T22:30:47.584ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/105f76bc-b792-46fa-a474-35e976fb8bb4
522024-12-13T22:30:47.584ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
532024-12-13T22:30:47.584ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::13
542024-12-13T22:30:47.584ZINFOcrucible-agent (worker): commit
552024-12-13T22:30:47.587ZINFOcrucible-agent (worker): ok!
562024-12-13T22:30:47.591ZINFOcrucible-agent (worker): SMF ok!
572024-12-13T22:30:47.594ZINFOcrucible-agent (dropshot): request completed latency_us = 53 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:56570 req_id = 6aaa0b1e-2371-4c3c-aa48-6c5574cf9061 response_code = 200 uri = /crucible/0/regions
582024-12-13T22:41:00.269ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::13]:32345 remote_addr = [fd00:1122:3344:101::a]:40620
592024-12-13T22:41:00.269ZINFOcrucible-agent (datafile): region fa98f23d-b392-4351-bde7-11f3a573f656 state: Requested
602024-12-13T22:41:00.270ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
612024-12-13T22:41:00.270ZINFOcrucible-agent (dropshot): request completed latency_us = 964 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:40620 req_id = 3f604f02-225f-49b9-bba2-8dd8f2d1f4bc response_code = 200 uri = /crucible/0/regions
622024-12-13T22:41:00.280ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_0d689a16-0a28-4b27-af5a-3b4bf99f365f/crucible/regions/fa98f23d-b392-4351-bde7-11f3a573f656
632024-12-13T22:41:00.280ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_0d689a16-0a28-4b27-af5a-3b4bf99f365f/crucible/regions/fa98f23d-b392-4351-bde7-11f3a573f656
642024-12-13T22:41:00.339ZINFOcrucible-agent (worker): creating region Region { id: RegionId("fa98f23d-b392-4351-bde7-11f3a573f656"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/fa98f23d-b392-4351-bde7-11f3a573f656" region = fa98f23d-b392-4351-bde7-11f3a573f656
652024-12-13T22:41:00.339ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "fa98f23d-b392-4351-bde7-11f3a573f656" "--data" "/data/regions/fa98f23d-b392-4351-bde7-11f3a573f656" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = fa98f23d-b392-4351-bde7-11f3a573f656
662024-12-13T22:41:00.419ZINFOcrucible-agent (worker): region files created ok region = fa98f23d-b392-4351-bde7-11f3a573f656
672024-12-13T22:41:00.419ZINFOcrucible-agent (datafile): region fa98f23d-b392-4351-bde7-11f3a573f656 state: Requested -> Created
682024-12-13T22:41:00.420ZINFOcrucible-agent (worker): applying SMF actions post create...
692024-12-13T22:41:00.427ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-fa98f23d-b392-4351-bde7-11f3a573f656
702024-12-13T22:41:00.435ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-fa98f23d-b392-4351-bde7-11f3a573f656
712024-12-13T22:41:00.444ZINFOcrucible-agent (worker): creating config property group
722024-12-13T22:41:00.447ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-fa98f23d-b392-4351-bde7-11f3a573f656
732024-12-13T22:41:00.447ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/fa98f23d-b392-4351-bde7-11f3a573f656
742024-12-13T22:41:00.447ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
752024-12-13T22:41:00.447ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::13
762024-12-13T22:41:00.447ZINFOcrucible-agent (worker): commit
772024-12-13T22:41:00.451ZINFOcrucible-agent (worker): ok!
782024-12-13T22:41:00.455ZINFOcrucible-agent (worker): SMF ok!
792024-12-13T22:41:00.468ZINFOcrucible-agent (dropshot): request completed latency_us = 940 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:40620 req_id = 4aa21f7a-0495-4105-9385-cab187a38591 response_code = 200 uri = /crucible/0/regions
802024-12-13T22:41:01.897ZINFOcrucible-agent (dropshot): request completed latency_us = 71 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:40620 req_id = 952c1947-1f88-4275-b6f9-5323a876c1c0 response_code = 200 uri = /crucible/0/regions/105f76bc-b792-46fa-a474-35e976fb8bb4
812024-12-13T22:41:01.903ZINFOcrucible-agent (datafile): path is "/data/regions/105f76bc-b792-46fa-a474-35e976fb8bb4"
822024-12-13T22:41:01.916ZINFOcrucible-agent (datafile): dataset is oxp_0d689a16-0a28-4b27-af5a-3b4bf99f365f/crucible/regions/105f76bc-b792-46fa-a474-35e976fb8bb4
832024-12-13T22:41:01.928ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_0d689a16-0a28-4b27-af5a-3b4bf99f365f/crucible/regions/105f76bc-b792-46fa-a474-35e976fb8bb4@7c997cb0-2332-44d9-b550-9dd58574f965
842024-12-13T22:41:01.928ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_0d689a16-0a28-4b27-af5a-3b4bf99f365f/crucible/regions/105f76bc-b792-46fa-a474-35e976fb8bb4", "7c997cb0-2332-44d9-b550-9dd58574f965"]
852024-12-13T22:41:01.945ZINFOcrucible-agent (dropshot): request completed latency_us = 42082 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:40620 req_id = 5b412eaf-1cd1-407f-bc7c-2866cf2c8cc4 response_code = 200 uri = /crucible/0/regions/105f76bc-b792-46fa-a474-35e976fb8bb4/snapshots/7c997cb0-2332-44d9-b550-9dd58574f965
862024-12-13T22:41:01.950ZINFOcrucible-agent (datafile): path is "/data/regions/105f76bc-b792-46fa-a474-35e976fb8bb4"
872024-12-13T22:41:01.962ZINFOcrucible-agent (datafile): dataset is oxp_0d689a16-0a28-4b27-af5a-3b4bf99f365f/crucible/regions/105f76bc-b792-46fa-a474-35e976fb8bb4
882024-12-13T22:41:01.974ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_0d689a16-0a28-4b27-af5a-3b4bf99f365f/crucible/regions/105f76bc-b792-46fa-a474-35e976fb8bb4@7c997cb0-2332-44d9-b550-9dd58574f965
892024-12-13T22:41:01.974ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_0d689a16-0a28-4b27-af5a-3b4bf99f365f/crucible/regions/105f76bc-b792-46fa-a474-35e976fb8bb4", "7c997cb0-2332-44d9-b550-9dd58574f965"]
902024-12-13T22:41:01.988ZINFOcrucible-agent (datafile): requesting running snapshot 105f76bc-b792-46fa-a474-35e976fb8bb4-7c997cb0-2332-44d9-b550-9dd58574f965 state: Requested
912024-12-13T22:41:01.989ZINFOcrucible-agent (worker): applying SMF actions for region 105f76bc-b792-46fa-a474-35e976fb8bb4 running snapshot 7c997cb0-2332-44d9-b550-9dd58574f965 (state Requested)...
922024-12-13T22:41:01.989ZINFOcrucible-agent (dropshot): request completed latency_us = 38430 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:40620 req_id = fadeb9fe-efe6-4f60-bb93-95a9c24cde38 response_code = 200 uri = /crucible/0/regions/105f76bc-b792-46fa-a474-35e976fb8bb4/snapshots/7c997cb0-2332-44d9-b550-9dd58574f965/run
932024-12-13T22:41:01.993ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-105f76bc-b792-46fa-a474-35e976fb8bb4-7c997cb0-2332-44d9-b550-9dd58574f965
942024-12-13T22:41:01.996ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-105f76bc-b792-46fa-a474-35e976fb8bb4-7c997cb0-2332-44d9-b550-9dd58574f965
952024-12-13T22:41:02.008ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-105f76bc-b792-46fa-a474-35e976fb8bb4-7c997cb0-2332-44d9-b550-9dd58574f965
962024-12-13T22:41:02.009ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/105f76bc-b792-46fa-a474-35e976fb8bb4/.zfs/snapshot/7c997cb0-2332-44d9-b550-9dd58574f965
972024-12-13T22:41:02.009ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
982024-12-13T22:41:02.009ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
992024-12-13T22:41:02.009ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::13
1002024-12-13T22:41:02.009ZINFOcrucible-agent (worker): commit
1012024-12-13T22:41:02.014ZINFOcrucible-agent (worker): ok!
1022024-12-13T22:41:02.019ZINFOcrucible-agent (worker): SMF ok!
1032024-12-13T22:41:02.019ZINFOcrucible-agent (datafile): region 105f76bc-b792-46fa-a474-35e976fb8bb4 running snapshot 7c997cb0-2332-44d9-b550-9dd58574f965 state: Requested -> Created
1042024-12-13T22:41:02.338ZINFOcrucible-agent (datafile): path is "/data/regions/105f76bc-b792-46fa-a474-35e976fb8bb4"
1052024-12-13T22:41:02.347ZINFOcrucible-agent (datafile): dataset is oxp_0d689a16-0a28-4b27-af5a-3b4bf99f365f/crucible/regions/105f76bc-b792-46fa-a474-35e976fb8bb4
1062024-12-13T22:41:02.355ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_0d689a16-0a28-4b27-af5a-3b4bf99f365f/crucible/regions/105f76bc-b792-46fa-a474-35e976fb8bb4@7c997cb0-2332-44d9-b550-9dd58574f965
1072024-12-13T22:41:02.355ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_0d689a16-0a28-4b27-af5a-3b4bf99f365f/crucible/regions/105f76bc-b792-46fa-a474-35e976fb8bb4", "7c997cb0-2332-44d9-b550-9dd58574f965"]
1082024-12-13T22:41:02.364ZINFOcrucible-agent (dropshot): request completed latency_us = 26493 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:40620 req_id = 749fc36e-3b8e-4af5-ae0d-999f95d6a919 response_code = 200 uri = /crucible/0/regions/105f76bc-b792-46fa-a474-35e976fb8bb4/snapshots/7c997cb0-2332-44d9-b550-9dd58574f965/run