Max level shown:
12024-08-28T23:34:20.233ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::14]:32345 remote_addr = [fd00:1122:3344:101::a]:54630
22024-08-28T23:34:20.233ZINFOcrucible-agent (dropshot): request completed latency_us = 59 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:54630 req_id = 3fd3e2fe-8b1d-4823-8094-a29e7cdad413 response_code = 200 uri = /crucible/0/regions/1decdc63-67fc-43bb-acac-78de4439314f
32024-08-28T23:34:20.239ZINFOcrucible-agent (datafile): path is "/data/regions/1decdc63-67fc-43bb-acac-78de4439314f"
42024-08-28T23:34:20.249ZINFOcrucible-agent (datafile): dataset is oxp_4eb71b25-4675-4051-abf2-3274c1ba98b3/crucible/regions/1decdc63-67fc-43bb-acac-78de4439314f
52024-08-28T23:34:20.257ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_4eb71b25-4675-4051-abf2-3274c1ba98b3/crucible/regions/1decdc63-67fc-43bb-acac-78de4439314f@6eed8bd4-27c2-47c7-bd5e-414d008b3c4a
62024-08-28T23:34:20.257ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_4eb71b25-4675-4051-abf2-3274c1ba98b3/crucible/regions/1decdc63-67fc-43bb-acac-78de4439314f", "6eed8bd4-27c2-47c7-bd5e-414d008b3c4a"]
72024-08-28T23:34:20.267ZINFOcrucible-agent (dropshot): request completed latency_us = 30558 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:54630 req_id = 8b662177-e9bc-4aa4-86eb-62d7159a10f5 response_code = 200 uri = /crucible/0/regions/1decdc63-67fc-43bb-acac-78de4439314f/snapshots/6eed8bd4-27c2-47c7-bd5e-414d008b3c4a
82024-08-28T23:34:20.272ZINFOcrucible-agent (datafile): path is "/data/regions/1decdc63-67fc-43bb-acac-78de4439314f"
92024-08-28T23:34:20.280ZINFOcrucible-agent (datafile): dataset is oxp_4eb71b25-4675-4051-abf2-3274c1ba98b3/crucible/regions/1decdc63-67fc-43bb-acac-78de4439314f
102024-08-28T23:34:20.288ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_4eb71b25-4675-4051-abf2-3274c1ba98b3/crucible/regions/1decdc63-67fc-43bb-acac-78de4439314f@6eed8bd4-27c2-47c7-bd5e-414d008b3c4a
112024-08-28T23:34:20.288ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_4eb71b25-4675-4051-abf2-3274c1ba98b3/crucible/regions/1decdc63-67fc-43bb-acac-78de4439314f", "6eed8bd4-27c2-47c7-bd5e-414d008b3c4a"]
122024-08-28T23:34:20.298ZINFOcrucible-agent (datafile): requesting running snapshot 1decdc63-67fc-43bb-acac-78de4439314f-6eed8bd4-27c2-47c7-bd5e-414d008b3c4a state: Requested
132024-08-28T23:34:20.299ZINFOcrucible-agent (worker): applying SMF actions for region 1decdc63-67fc-43bb-acac-78de4439314f running snapshot 6eed8bd4-27c2-47c7-bd5e-414d008b3c4a (state Requested)...
142024-08-28T23:34:20.299ZINFOcrucible-agent (dropshot): request completed latency_us = 27795 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:54630 req_id = f2d4a696-eb7a-44f3-b415-a2a968afe37d response_code = 200 uri = /crucible/0/regions/1decdc63-67fc-43bb-acac-78de4439314f/snapshots/6eed8bd4-27c2-47c7-bd5e-414d008b3c4a/run
152024-08-28T23:34:20.307ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/1decdc63-67fc-43bb-acac-78de4439314f
162024-08-28T23:34:20.307ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
172024-08-28T23:34:20.307ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::14
182024-08-28T23:34:20.307ZINFOcrucible-agent (worker): reconfig required
192024-08-28T23:34:20.307ZINFOcrucible-agent (worker): using existing config property group
202024-08-28T23:34:20.307ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-1decdc63-67fc-43bb-acac-78de4439314f
212024-08-28T23:34:20.307ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/1decdc63-67fc-43bb-acac-78de4439314f
222024-08-28T23:34:20.307ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
232024-08-28T23:34:20.307ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
242024-08-28T23:34:20.307ZINFOcrucible-agent (worker): commit
252024-08-28T23:34:20.316ZINFOcrucible-agent (worker): ok!
262024-08-28T23:34:20.316ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-1decdc63-67fc-43bb-acac-78de4439314f-6eed8bd4-27c2-47c7-bd5e-414d008b3c4a
272024-08-28T23:34:20.320ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-1decdc63-67fc-43bb-acac-78de4439314f-6eed8bd4-27c2-47c7-bd5e-414d008b3c4a
282024-08-28T23:34:20.337ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-1decdc63-67fc-43bb-acac-78de4439314f-6eed8bd4-27c2-47c7-bd5e-414d008b3c4a
292024-08-28T23:34:20.337ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/1decdc63-67fc-43bb-acac-78de4439314f/.zfs/snapshot/6eed8bd4-27c2-47c7-bd5e-414d008b3c4a
302024-08-28T23:34:20.337ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
312024-08-28T23:34:20.337ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
322024-08-28T23:34:20.337ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
332024-08-28T23:34:20.338ZINFOcrucible-agent (worker): commit
342024-08-28T23:34:20.344ZINFOcrucible-agent (worker): ok!
352024-08-28T23:34:20.352ZINFOcrucible-agent (worker): SMF ok!
362024-08-28T23:34:20.352ZINFOcrucible-agent (datafile): region 1decdc63-67fc-43bb-acac-78de4439314f running snapshot 6eed8bd4-27c2-47c7-bd5e-414d008b3c4a state: Requested -> Created
372024-08-28T23:34:20.517ZINFOcrucible-agent (datafile): path is "/data/regions/1decdc63-67fc-43bb-acac-78de4439314f"
382024-08-28T23:34:20.526ZINFOcrucible-agent (datafile): dataset is oxp_4eb71b25-4675-4051-abf2-3274c1ba98b3/crucible/regions/1decdc63-67fc-43bb-acac-78de4439314f
392024-08-28T23:34:20.534ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_4eb71b25-4675-4051-abf2-3274c1ba98b3/crucible/regions/1decdc63-67fc-43bb-acac-78de4439314f@6eed8bd4-27c2-47c7-bd5e-414d008b3c4a
402024-08-28T23:34:20.534ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_4eb71b25-4675-4051-abf2-3274c1ba98b3/crucible/regions/1decdc63-67fc-43bb-acac-78de4439314f", "6eed8bd4-27c2-47c7-bd5e-414d008b3c4a"]
412024-08-28T23:34:20.543ZINFOcrucible-agent (dropshot): request completed latency_us = 25515 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:54630 req_id = c7ad2ff6-2b88-4dcf-997f-82a7592f0cbb response_code = 200 uri = /crucible/0/regions/1decdc63-67fc-43bb-acac-78de4439314f/snapshots/6eed8bd4-27c2-47c7-bd5e-414d008b3c4a/run
422024-08-28T23:34:24.633ZINFOcrucible-agent (datafile): region 9bb70d45-ae73-4e9e-bd1c-a466592bcf4a state: Requested
432024-08-28T23:34:24.633ZINFOcrucible-agent (dropshot): request completed latency_us = 280 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:54630 req_id = aa3a7e70-ede5-48a6-b4c4-45b95c34186d response_code = 200 uri = /crucible/0/regions
442024-08-28T23:34:24.633ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
452024-08-28T23:34:24.643ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_4eb71b25-4675-4051-abf2-3274c1ba98b3/crucible/regions/9bb70d45-ae73-4e9e-bd1c-a466592bcf4a
462024-08-28T23:34:24.643ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_4eb71b25-4675-4051-abf2-3274c1ba98b3/crucible/regions/9bb70d45-ae73-4e9e-bd1c-a466592bcf4a
472024-08-28T23:34:24.714ZINFOcrucible-agent (worker): creating region Region { id: RegionId("9bb70d45-ae73-4e9e-bd1c-a466592bcf4a"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19002, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/9bb70d45-ae73-4e9e-bd1c-a466592bcf4a" region = 9bb70d45-ae73-4e9e-bd1c-a466592bcf4a
482024-08-28T23:34:24.714ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "9bb70d45-ae73-4e9e-bd1c-a466592bcf4a" "--data" "/data/regions/9bb70d45-ae73-4e9e-bd1c-a466592bcf4a" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 9bb70d45-ae73-4e9e-bd1c-a466592bcf4a
492024-08-28T23:34:24.869ZINFOcrucible-agent (worker): region files created ok region = 9bb70d45-ae73-4e9e-bd1c-a466592bcf4a
502024-08-28T23:34:24.869ZINFOcrucible-agent (datafile): region 9bb70d45-ae73-4e9e-bd1c-a466592bcf4a state: Requested -> Created
512024-08-28T23:34:24.869ZINFOcrucible-agent (worker): applying SMF actions post create...
522024-08-28T23:34:24.870ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/1decdc63-67fc-43bb-acac-78de4439314f
532024-08-28T23:34:24.870ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
542024-08-28T23:34:24.870ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::14
552024-08-28T23:34:24.870ZINFOcrucible-agent (worker): reconfig required
562024-08-28T23:34:24.870ZINFOcrucible-agent (worker): using existing config property group
572024-08-28T23:34:24.870ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-1decdc63-67fc-43bb-acac-78de4439314f
582024-08-28T23:34:24.870ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/1decdc63-67fc-43bb-acac-78de4439314f
592024-08-28T23:34:24.871ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
602024-08-28T23:34:24.871ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
612024-08-28T23:34:24.871ZINFOcrucible-agent (worker): commit
622024-08-28T23:34:24.880ZINFOcrucible-agent (worker): ok!
632024-08-28T23:34:24.880ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-9bb70d45-ae73-4e9e-bd1c-a466592bcf4a
642024-08-28T23:34:24.887ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-9bb70d45-ae73-4e9e-bd1c-a466592bcf4a
652024-08-28T23:34:24.903ZINFOcrucible-agent (worker): creating config property group
662024-08-28T23:34:24.909ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-9bb70d45-ae73-4e9e-bd1c-a466592bcf4a
672024-08-28T23:34:24.910ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/9bb70d45-ae73-4e9e-bd1c-a466592bcf4a
682024-08-28T23:34:24.910ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
692024-08-28T23:34:24.910ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
702024-08-28T23:34:24.910ZINFOcrucible-agent (worker): commit
712024-08-28T23:34:24.933ZINFOcrucible-agent (worker): ok!
722024-08-28T23:34:24.954ZINFOcrucible-agent (worker): SMF ok!
732024-08-28T23:34:25.005ZINFOcrucible-agent (dropshot): request completed latency_us = 69 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:54630 req_id = 0af91f53-a0bf-420f-82a5-0f36fd68756d response_code = 200 uri = /crucible/0/regions
742024-08-28T23:34:57.676ZINFOcrucible-agent (dropshot): request completed latency_us = 54 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:54630 req_id = c4a85736-72e7-4a1b-abbb-29827cf7f72d response_code = 200 uri = /crucible/0/regions/9bb70d45-ae73-4e9e-bd1c-a466592bcf4a
752024-08-28T23:34:57.680ZINFOcrucible-agent (datafile): path is "/data/regions/9bb70d45-ae73-4e9e-bd1c-a466592bcf4a"
762024-08-28T23:34:57.690ZINFOcrucible-agent (datafile): dataset is oxp_4eb71b25-4675-4051-abf2-3274c1ba98b3/crucible/regions/9bb70d45-ae73-4e9e-bd1c-a466592bcf4a
772024-08-28T23:34:57.699ZINFOcrucible-agent (datafile): region 9bb70d45-ae73-4e9e-bd1c-a466592bcf4a state: Created -> Tombstoned
782024-08-28T23:34:57.699ZINFOcrucible-agent (worker): applying SMF actions before removal...
792024-08-28T23:34:57.699ZINFOcrucible-agent (dropshot): request completed latency_us = 19525 local_addr = [fd00:1122:3344:101::14]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:54630 req_id = 90c49907-954e-4871-b6a2-fda64f2e1e34 response_code = 204 uri = /crucible/0/regions/9bb70d45-ae73-4e9e-bd1c-a466592bcf4a
802024-08-28T23:34:57.700ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-9bb70d45-ae73-4e9e-bd1c-a466592bcf4a (instance states: (Some(Online), None))
812024-08-28T23:34:57.702ZINFOcrucible-agent (dropshot): request completed latency_us = 17 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:54630 req_id = 85f41889-dc6f-4779-b8be-0bc00ccd045c response_code = 200 uri = /crucible/0/regions/9bb70d45-ae73-4e9e-bd1c-a466592bcf4a
822024-08-28T23:34:57.705ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/1decdc63-67fc-43bb-acac-78de4439314f
832024-08-28T23:34:57.706ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
842024-08-28T23:34:57.706ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::14
852024-08-28T23:34:57.706ZINFOcrucible-agent (worker): reconfig required
862024-08-28T23:34:57.706ZINFOcrucible-agent (worker): using existing config property group
872024-08-28T23:34:57.706ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-1decdc63-67fc-43bb-acac-78de4439314f
882024-08-28T23:34:57.706ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/1decdc63-67fc-43bb-acac-78de4439314f
892024-08-28T23:34:57.706ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
902024-08-28T23:34:57.706ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
912024-08-28T23:34:57.706ZINFOcrucible-agent (worker): commit
922024-08-28T23:34:57.710ZINFOcrucible-agent (worker): ok!
932024-08-28T23:34:57.711ZINFOcrucible-agent (worker): SMF ok!
942024-08-28T23:34:57.722ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_4eb71b25-4675-4051-abf2-3274c1ba98b3/crucible/regions/9bb70d45-ae73-4e9e-bd1c-a466592bcf4a" region = 9bb70d45-ae73-4e9e-bd1c-a466592bcf4a
952024-08-28T23:34:57.776ZINFOcrucible-agent (datafile): region 9bb70d45-ae73-4e9e-bd1c-a466592bcf4a state: Tombstoned -> Destroyed
962024-08-28T23:34:57.815ZINFOcrucible-agent (dropshot): request completed latency_us = 75 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:54630 req_id = e47c6705-9bfd-44fb-8948-0facc800f3a2 response_code = 200 uri = /crucible/0/regions/9bb70d45-ae73-4e9e-bd1c-a466592bcf4a