Max level shown:
12024-12-21T01:49:45.843ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::17]:32345 remote_addr = [fd00:1122:3344:101::a]:63595
22024-12-21T01:49:45.846ZINFOcrucible-agent (dropshot): request completed latency_us = 3122 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:63595 req_id = c89e8a38-7733-4a72-bd0b-9d18fc48aa1f response_code = 200 uri = /crucible/0/regions/45c98cb1-61b7-4e45-8efc-68848d490822
32024-12-21T01:49:45.850ZINFOcrucible-agent (datafile): path is "/data/regions/45c98cb1-61b7-4e45-8efc-68848d490822"
42024-12-21T01:49:45.859ZINFOcrucible-agent (datafile): dataset is oxp_335ed515-415f-4302-9f2a-54aed0987aee/crucible/regions/45c98cb1-61b7-4e45-8efc-68848d490822
52024-12-21T01:49:45.869ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_335ed515-415f-4302-9f2a-54aed0987aee/crucible/regions/45c98cb1-61b7-4e45-8efc-68848d490822@58ce16cb-c870-4038-938f-1f2801c98a8c
62024-12-21T01:49:45.869ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_335ed515-415f-4302-9f2a-54aed0987aee/crucible/regions/45c98cb1-61b7-4e45-8efc-68848d490822", "58ce16cb-c870-4038-938f-1f2801c98a8c"]
72024-12-21T01:49:45.880ZINFOcrucible-agent (dropshot): request completed latency_us = 30290 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:63595 req_id = 5399c493-0883-4ab0-8c96-98de959ca681 response_code = 200 uri = /crucible/0/regions/45c98cb1-61b7-4e45-8efc-68848d490822/snapshots/58ce16cb-c870-4038-938f-1f2801c98a8c
82024-12-21T01:49:45.884ZINFOcrucible-agent (datafile): path is "/data/regions/45c98cb1-61b7-4e45-8efc-68848d490822"
92024-12-21T01:49:45.892ZINFOcrucible-agent (datafile): dataset is oxp_335ed515-415f-4302-9f2a-54aed0987aee/crucible/regions/45c98cb1-61b7-4e45-8efc-68848d490822
102024-12-21T01:49:45.901ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_335ed515-415f-4302-9f2a-54aed0987aee/crucible/regions/45c98cb1-61b7-4e45-8efc-68848d490822@58ce16cb-c870-4038-938f-1f2801c98a8c
112024-12-21T01:49:45.901ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_335ed515-415f-4302-9f2a-54aed0987aee/crucible/regions/45c98cb1-61b7-4e45-8efc-68848d490822", "58ce16cb-c870-4038-938f-1f2801c98a8c"]
122024-12-21T01:49:45.911ZINFOcrucible-agent (datafile): requesting running snapshot 45c98cb1-61b7-4e45-8efc-68848d490822-58ce16cb-c870-4038-938f-1f2801c98a8c state: Requested
132024-12-21T01:49:45.911ZINFOcrucible-agent (worker): applying SMF actions for region 45c98cb1-61b7-4e45-8efc-68848d490822 running snapshot 58ce16cb-c870-4038-938f-1f2801c98a8c (state Requested)...
142024-12-21T01:49:45.911ZINFOcrucible-agent (dropshot): request completed latency_us = 27802 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:63595 req_id = c307cc11-580c-4363-98ec-5c50fd8b841c response_code = 200 uri = /crucible/0/regions/45c98cb1-61b7-4e45-8efc-68848d490822/snapshots/58ce16cb-c870-4038-938f-1f2801c98a8c/run
152024-12-21T01:49:45.921ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-45c98cb1-61b7-4e45-8efc-68848d490822-58ce16cb-c870-4038-938f-1f2801c98a8c
162024-12-21T01:49:45.925ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-45c98cb1-61b7-4e45-8efc-68848d490822-58ce16cb-c870-4038-938f-1f2801c98a8c
172024-12-21T01:49:45.936ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-45c98cb1-61b7-4e45-8efc-68848d490822-58ce16cb-c870-4038-938f-1f2801c98a8c
182024-12-21T01:49:45.937ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/45c98cb1-61b7-4e45-8efc-68848d490822/.zfs/snapshot/58ce16cb-c870-4038-938f-1f2801c98a8c
192024-12-21T01:49:45.937ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
202024-12-21T01:49:45.937ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
212024-12-21T01:49:45.937ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
222024-12-21T01:49:45.937ZINFOcrucible-agent (worker): commit
232024-12-21T01:49:45.940ZINFOcrucible-agent (worker): ok!
242024-12-21T01:49:45.945ZINFOcrucible-agent (worker): SMF ok!
252024-12-21T01:49:45.945ZINFOcrucible-agent (datafile): region 45c98cb1-61b7-4e45-8efc-68848d490822 running snapshot 58ce16cb-c870-4038-938f-1f2801c98a8c state: Requested -> Created
262024-12-21T01:49:46.074ZINFOcrucible-agent (datafile): path is "/data/regions/45c98cb1-61b7-4e45-8efc-68848d490822"
272024-12-21T01:49:46.083ZINFOcrucible-agent (datafile): dataset is oxp_335ed515-415f-4302-9f2a-54aed0987aee/crucible/regions/45c98cb1-61b7-4e45-8efc-68848d490822
282024-12-21T01:49:46.092ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_335ed515-415f-4302-9f2a-54aed0987aee/crucible/regions/45c98cb1-61b7-4e45-8efc-68848d490822@58ce16cb-c870-4038-938f-1f2801c98a8c
292024-12-21T01:49:46.092ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_335ed515-415f-4302-9f2a-54aed0987aee/crucible/regions/45c98cb1-61b7-4e45-8efc-68848d490822", "58ce16cb-c870-4038-938f-1f2801c98a8c"]
302024-12-21T01:49:46.101ZINFOcrucible-agent (dropshot): request completed latency_us = 27089 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:63595 req_id = 343cc16d-9695-4ebf-9eab-20f22e4b7218 response_code = 200 uri = /crucible/0/regions/45c98cb1-61b7-4e45-8efc-68848d490822/snapshots/58ce16cb-c870-4038-938f-1f2801c98a8c/run
312024-12-21T01:49:49.162ZINFOcrucible-agent (datafile): region 1dcd91ac-bac5-44dd-8e9e-95b0c59c7c5e state: Requested
322024-12-21T01:49:49.162ZINFOcrucible-agent (dropshot): request completed latency_us = 248 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:63595 req_id = 39ad4ca0-96fb-45ba-a2a6-1ae05e592011 response_code = 200 uri = /crucible/0/regions
332024-12-21T01:49:49.162ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
342024-12-21T01:49:49.171ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_335ed515-415f-4302-9f2a-54aed0987aee/crucible/regions/1dcd91ac-bac5-44dd-8e9e-95b0c59c7c5e
352024-12-21T01:49:49.171ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_335ed515-415f-4302-9f2a-54aed0987aee/crucible/regions/1dcd91ac-bac5-44dd-8e9e-95b0c59c7c5e
362024-12-21T01:49:49.225ZINFOcrucible-agent (worker): creating region Region { id: RegionId("1dcd91ac-bac5-44dd-8e9e-95b0c59c7c5e"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19002, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/1dcd91ac-bac5-44dd-8e9e-95b0c59c7c5e" region = 1dcd91ac-bac5-44dd-8e9e-95b0c59c7c5e
372024-12-21T01:49:49.225ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "1dcd91ac-bac5-44dd-8e9e-95b0c59c7c5e" "--data" "/data/regions/1dcd91ac-bac5-44dd-8e9e-95b0c59c7c5e" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 1dcd91ac-bac5-44dd-8e9e-95b0c59c7c5e
382024-12-21T01:49:49.295ZINFOcrucible-agent (worker): region files created ok region = 1dcd91ac-bac5-44dd-8e9e-95b0c59c7c5e
392024-12-21T01:49:49.295ZINFOcrucible-agent (datafile): region 1dcd91ac-bac5-44dd-8e9e-95b0c59c7c5e state: Requested -> Created
402024-12-21T01:49:49.295ZINFOcrucible-agent (worker): applying SMF actions post create...
412024-12-21T01:49:49.295ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-1dcd91ac-bac5-44dd-8e9e-95b0c59c7c5e
422024-12-21T01:49:49.302ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-1dcd91ac-bac5-44dd-8e9e-95b0c59c7c5e
432024-12-21T01:49:49.310ZINFOcrucible-agent (worker): creating config property group
442024-12-21T01:49:49.314ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-1dcd91ac-bac5-44dd-8e9e-95b0c59c7c5e
452024-12-21T01:49:49.314ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/1dcd91ac-bac5-44dd-8e9e-95b0c59c7c5e
462024-12-21T01:49:49.314ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
472024-12-21T01:49:49.314ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
482024-12-21T01:49:49.314ZINFOcrucible-agent (worker): commit
492024-12-21T01:49:49.318ZINFOcrucible-agent (worker): ok!
502024-12-21T01:49:49.329ZINFOcrucible-agent (worker): SMF ok!
512024-12-21T01:49:49.388ZINFOcrucible-agent (dropshot): request completed latency_us = 67 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:63595 req_id = d649b524-3e80-4c91-9d10-569458b0fa0c response_code = 200 uri = /crucible/0/regions
522024-12-21T01:50:14.410ZINFOcrucible-agent (dropshot): request completed latency_us = 54 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:63595 req_id = 947f6fc6-9ccb-421f-a725-76e587948dd9 response_code = 200 uri = /crucible/0/regions/1dcd91ac-bac5-44dd-8e9e-95b0c59c7c5e
532024-12-21T01:50:14.412ZINFOcrucible-agent (datafile): path is "/data/regions/1dcd91ac-bac5-44dd-8e9e-95b0c59c7c5e"
542024-12-21T01:50:14.425ZINFOcrucible-agent (datafile): dataset is oxp_335ed515-415f-4302-9f2a-54aed0987aee/crucible/regions/1dcd91ac-bac5-44dd-8e9e-95b0c59c7c5e
552024-12-21T01:50:14.436ZINFOcrucible-agent (datafile): region 1dcd91ac-bac5-44dd-8e9e-95b0c59c7c5e state: Created -> Tombstoned
562024-12-21T01:50:14.436ZINFOcrucible-agent (worker): applying SMF actions before removal...
572024-12-21T01:50:14.436ZINFOcrucible-agent (dropshot): request completed latency_us = 23738 local_addr = [fd00:1122:3344:101::17]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:63595 req_id = 9b61fa87-243a-443b-996e-59f3149f9db9 response_code = 204 uri = /crucible/0/regions/1dcd91ac-bac5-44dd-8e9e-95b0c59c7c5e
582024-12-21T01:50:14.439ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-1dcd91ac-bac5-44dd-8e9e-95b0c59c7c5e (instance states: (Some(Online), None))
592024-12-21T01:50:14.440ZINFOcrucible-agent (dropshot): request completed latency_us = 381 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:63595 req_id = fa10a06b-ab81-414c-b6f6-cd8588170fc2 response_code = 200 uri = /crucible/0/regions/1dcd91ac-bac5-44dd-8e9e-95b0c59c7c5e
602024-12-21T01:50:14.445ZINFOcrucible-agent (worker): SMF ok!
612024-12-21T01:50:14.456ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_335ed515-415f-4302-9f2a-54aed0987aee/crucible/regions/1dcd91ac-bac5-44dd-8e9e-95b0c59c7c5e" region = 1dcd91ac-bac5-44dd-8e9e-95b0c59c7c5e
622024-12-21T01:50:14.501ZINFOcrucible-agent (datafile): region 1dcd91ac-bac5-44dd-8e9e-95b0c59c7c5e state: Tombstoned -> Destroyed
632024-12-21T01:50:14.574ZINFOcrucible-agent (dropshot): request completed latency_us = 64 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:63595 req_id = a2e71483-60f0-40f1-bce2-c8252cda851b response_code = 200 uri = /crucible/0/regions/1dcd91ac-bac5-44dd-8e9e-95b0c59c7c5e