Max level shown:
12024-02-24T05:03:54.853ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::17]:32345 remote_addr = [fd00:1122:3344:101::a]:49145
22024-02-24T05:03:54.912ZINFOcrucible-agent (datafile): region ce972d76-4fe7-4d4b-b46e-7928b1bb870a state: Requested
32024-02-24T05:03:54.913ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
42024-02-24T05:03:54.913ZINFOcrucible-agent (dropshot): request completed latency_us = 743 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:49145 req_id = d799fe29-b2d3-4989-b238-bcd72c934b04 response_code = 200 uri = /crucible/0/regions
52024-02-24T05:03:54.922ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60/crucible/regions/ce972d76-4fe7-4d4b-b46e-7928b1bb870a
62024-02-24T05:03:54.922ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60/crucible/regions/ce972d76-4fe7-4d4b-b46e-7928b1bb870a
72024-02-24T05:03:54.975ZINFOcrucible-agent (worker): creating region Region { id: RegionId("ce972d76-4fe7-4d4b-b46e-7928b1bb870a"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None } at "/data/regions/ce972d76-4fe7-4d4b-b46e-7928b1bb870a" region = ce972d76-4fe7-4d4b-b46e-7928b1bb870a
82024-02-24T05:03:55.040ZINFOcrucible-agent (worker): region files created ok region = ce972d76-4fe7-4d4b-b46e-7928b1bb870a
92024-02-24T05:03:55.040ZINFOcrucible-agent (datafile): region ce972d76-4fe7-4d4b-b46e-7928b1bb870a state: Requested -> Created
102024-02-24T05:03:55.040ZINFOcrucible-agent (worker): applying SMF actions post create...
112024-02-24T05:03:55.044ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/3afaa2d5-82c5-498f-9558-c2092959d7d0
122024-02-24T05:03:55.044ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
132024-02-24T05:03:55.044ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::17
142024-02-24T05:03:55.044ZINFOcrucible-agent (worker): reconfig required
152024-02-24T05:03:55.044ZINFOcrucible-agent (worker): using existing config property group
162024-02-24T05:03:55.044ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-3afaa2d5-82c5-498f-9558-c2092959d7d0
172024-02-24T05:03:55.044ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/3afaa2d5-82c5-498f-9558-c2092959d7d0
182024-02-24T05:03:55.044ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
192024-02-24T05:03:55.044ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
202024-02-24T05:03:55.044ZINFOcrucible-agent (worker): commit
212024-02-24T05:03:55.052ZINFOcrucible-agent (worker): ok!
222024-02-24T05:03:55.052ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-ce972d76-4fe7-4d4b-b46e-7928b1bb870a
232024-02-24T05:03:55.056ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-ce972d76-4fe7-4d4b-b46e-7928b1bb870a
242024-02-24T05:03:55.062ZINFOcrucible-agent (worker): creating config property group
252024-02-24T05:03:55.065ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-ce972d76-4fe7-4d4b-b46e-7928b1bb870a
262024-02-24T05:03:55.065ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/ce972d76-4fe7-4d4b-b46e-7928b1bb870a
272024-02-24T05:03:55.065ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
282024-02-24T05:03:55.065ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
292024-02-24T05:03:55.065ZINFOcrucible-agent (worker): commit
302024-02-24T05:03:55.069ZINFOcrucible-agent (worker): ok!
312024-02-24T05:03:55.073ZINFOcrucible-agent (worker): SMF ok!
322024-02-24T05:03:55.191ZINFOcrucible-agent (dropshot): request completed latency_us = 61 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:49145 req_id = 51f7fc67-70f5-446f-9be2-1c1bbc7ad801 response_code = 200 uri = /crucible/0/regions
332024-02-24T05:03:55.726ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::17]:32345 remote_addr = [fd00:1122:3344:101::a]:56171
342024-02-24T05:03:55.729ZINFOcrucible-agent (dropshot): request completed latency_us = 2693 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:56171 req_id = 1bb314ea-c24f-4da3-a4e7-86f942f26021 response_code = 200 uri = /crucible/0/regions/3afaa2d5-82c5-498f-9558-c2092959d7d0
352024-02-24T05:03:55.729ZINFOcrucible-agent (datafile): path is "/data/regions/3afaa2d5-82c5-498f-9558-c2092959d7d0"
362024-02-24T05:03:55.737ZINFOcrucible-agent (datafile): dataset is oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60/crucible/regions/3afaa2d5-82c5-498f-9558-c2092959d7d0
372024-02-24T05:03:55.745ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60/crucible/regions/3afaa2d5-82c5-498f-9558-c2092959d7d0@767f3fa6-de57-4fd6-ad9a-678ff91528ca
382024-02-24T05:03:55.745ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60/crucible/regions/3afaa2d5-82c5-498f-9558-c2092959d7d0", "767f3fa6-de57-4fd6-ad9a-678ff91528ca"]
392024-02-24T05:03:55.755ZINFOcrucible-agent (dropshot): request completed latency_us = 26542 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:56171 req_id = ce0a451d-5e2b-4e4f-b3d4-f1a21c1fdc71 response_code = 200 uri = /crucible/0/regions/3afaa2d5-82c5-498f-9558-c2092959d7d0/snapshots/767f3fa6-de57-4fd6-ad9a-678ff91528ca
402024-02-24T05:03:55.755ZINFOcrucible-agent (datafile): path is "/data/regions/3afaa2d5-82c5-498f-9558-c2092959d7d0"
412024-02-24T05:03:55.763ZINFOcrucible-agent (datafile): dataset is oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60/crucible/regions/3afaa2d5-82c5-498f-9558-c2092959d7d0
422024-02-24T05:03:55.771ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60/crucible/regions/3afaa2d5-82c5-498f-9558-c2092959d7d0@767f3fa6-de57-4fd6-ad9a-678ff91528ca
432024-02-24T05:03:55.771ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60/crucible/regions/3afaa2d5-82c5-498f-9558-c2092959d7d0", "767f3fa6-de57-4fd6-ad9a-678ff91528ca"]
442024-02-24T05:03:55.780ZINFOcrucible-agent (datafile): requesting running snapshot 3afaa2d5-82c5-498f-9558-c2092959d7d0-767f3fa6-de57-4fd6-ad9a-678ff91528ca state: Requested
452024-02-24T05:03:55.780ZINFOcrucible-agent (worker): applying SMF actions for region 3afaa2d5-82c5-498f-9558-c2092959d7d0 running snapshot 767f3fa6-de57-4fd6-ad9a-678ff91528ca (state Requested)...
462024-02-24T05:03:55.780ZINFOcrucible-agent (dropshot): request completed latency_us = 25022 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:56171 req_id = 7040c321-6711-48fc-a8db-2d97945f52e6 response_code = 200 uri = /crucible/0/regions/3afaa2d5-82c5-498f-9558-c2092959d7d0/snapshots/767f3fa6-de57-4fd6-ad9a-678ff91528ca/run
472024-02-24T05:03:55.781ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/3afaa2d5-82c5-498f-9558-c2092959d7d0
482024-02-24T05:03:55.781ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
492024-02-24T05:03:55.781ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::17
502024-02-24T05:03:55.781ZINFOcrucible-agent (worker): reconfig required
512024-02-24T05:03:55.781ZINFOcrucible-agent (worker): using existing config property group
522024-02-24T05:03:55.781ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-3afaa2d5-82c5-498f-9558-c2092959d7d0
532024-02-24T05:03:55.782ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/3afaa2d5-82c5-498f-9558-c2092959d7d0
542024-02-24T05:03:55.782ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
552024-02-24T05:03:55.782ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
562024-02-24T05:03:55.782ZINFOcrucible-agent (worker): commit
572024-02-24T05:03:55.785ZINFOcrucible-agent (worker): ok!
582024-02-24T05:03:55.787ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-3afaa2d5-82c5-498f-9558-c2092959d7d0-767f3fa6-de57-4fd6-ad9a-678ff91528ca
592024-02-24T05:03:55.795ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-3afaa2d5-82c5-498f-9558-c2092959d7d0-767f3fa6-de57-4fd6-ad9a-678ff91528ca
602024-02-24T05:03:55.804ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-3afaa2d5-82c5-498f-9558-c2092959d7d0-767f3fa6-de57-4fd6-ad9a-678ff91528ca
612024-02-24T05:03:55.804ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/3afaa2d5-82c5-498f-9558-c2092959d7d0/.zfs/snapshot/767f3fa6-de57-4fd6-ad9a-678ff91528ca
622024-02-24T05:03:55.804ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
632024-02-24T05:03:55.804ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
642024-02-24T05:03:55.804ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
652024-02-24T05:03:55.804ZINFOcrucible-agent (worker): commit
662024-02-24T05:03:55.808ZINFOcrucible-agent (worker): ok!
672024-02-24T05:03:55.812ZINFOcrucible-agent (worker): SMF ok!
682024-02-24T05:03:55.812ZINFOcrucible-agent (datafile): region 3afaa2d5-82c5-498f-9558-c2092959d7d0 running snapshot 767f3fa6-de57-4fd6-ad9a-678ff91528ca state: Requested -> Created
692024-02-24T05:03:59.858ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::17]:32345 remote_addr = [fd00:1122:3344:101::a]:33669
702024-02-24T05:03:59.917ZINFOcrucible-agent (datafile): region 0e740589-d6ba-4995-b400-87869f9a9b82 state: Requested
712024-02-24T05:03:59.917ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
722024-02-24T05:03:59.917ZINFOcrucible-agent (dropshot): request completed latency_us = 278 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:33669 req_id = ee439e39-0586-43ae-8b80-986cd7ab246e response_code = 200 uri = /crucible/0/regions
732024-02-24T05:03:59.927ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60/crucible/regions/0e740589-d6ba-4995-b400-87869f9a9b82
742024-02-24T05:03:59.927ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60/crucible/regions/0e740589-d6ba-4995-b400-87869f9a9b82
752024-02-24T05:04:00.119ZINFOcrucible-agent (worker): creating region Region { id: RegionId("0e740589-d6ba-4995-b400-87869f9a9b82"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19003, cert_pem: None, key_pem: None, root_pem: None } at "/data/regions/0e740589-d6ba-4995-b400-87869f9a9b82" region = 0e740589-d6ba-4995-b400-87869f9a9b82
762024-02-24T05:04:00.123ZINFOcrucible-agent (dropshot): request completed latency_us = 71 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:33669 req_id = 26fd22b2-3be1-483e-a77e-b6e2c26800dd response_code = 200 uri = /crucible/0/regions
772024-02-24T05:04:00.526ZINFOcrucible-agent (worker): region files created ok region = 0e740589-d6ba-4995-b400-87869f9a9b82
782024-02-24T05:04:00.526ZINFOcrucible-agent (datafile): region 0e740589-d6ba-4995-b400-87869f9a9b82 state: Requested -> Created
792024-02-24T05:04:00.527ZINFOcrucible-agent (worker): applying SMF actions post create...
802024-02-24T05:04:00.527ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-0e740589-d6ba-4995-b400-87869f9a9b82
812024-02-24T05:04:00.570ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-0e740589-d6ba-4995-b400-87869f9a9b82
822024-02-24T05:04:00.607ZINFOcrucible-agent (dropshot): request completed latency_us = 54 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:33669 req_id = 22b440c7-cdcf-459d-b2c6-9eb2127664ee response_code = 200 uri = /crucible/0/regions
832024-02-24T05:04:00.651ZINFOcrucible-agent (worker): creating config property group
842024-02-24T05:04:00.665ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-0e740589-d6ba-4995-b400-87869f9a9b82
852024-02-24T05:04:00.666ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/0e740589-d6ba-4995-b400-87869f9a9b82
862024-02-24T05:04:00.666ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19003
872024-02-24T05:04:00.666ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
882024-02-24T05:04:00.666ZINFOcrucible-agent (worker): commit
892024-02-24T05:04:00.690ZINFOcrucible-agent (worker): ok!
902024-02-24T05:04:00.702ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/3afaa2d5-82c5-498f-9558-c2092959d7d0
912024-02-24T05:04:00.702ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
922024-02-24T05:04:00.702ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::17
932024-02-24T05:04:00.702ZINFOcrucible-agent (worker): reconfig required
942024-02-24T05:04:00.702ZINFOcrucible-agent (worker): using existing config property group
952024-02-24T05:04:00.702ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-3afaa2d5-82c5-498f-9558-c2092959d7d0
962024-02-24T05:04:00.752ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/3afaa2d5-82c5-498f-9558-c2092959d7d0
972024-02-24T05:04:00.752ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
982024-02-24T05:04:00.752ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
992024-02-24T05:04:00.752ZINFOcrucible-agent (worker): commit
1002024-02-24T05:04:00.793ZINFOcrucible-agent (worker): ok!
1012024-02-24T05:04:00.796ZINFOcrucible-agent (worker): SMF ok!
1022024-02-24T05:04:41.721ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::17]:32345 remote_addr = [fd00:1122:3344:101::a]:63817
1032024-02-24T05:04:41.722ZINFOcrucible-agent (dropshot): request completed latency_us = 711 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:63817 req_id = 8b3e5448-3870-4cb4-87a6-d1ddcc9733a1 response_code = 200 uri = /crucible/0/regions/0e740589-d6ba-4995-b400-87869f9a9b82
1042024-02-24T05:04:41.722ZINFOcrucible-agent (datafile): path is "/data/regions/0e740589-d6ba-4995-b400-87869f9a9b82"
1052024-02-24T05:04:41.732ZINFOcrucible-agent (datafile): dataset is oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60/crucible/regions/0e740589-d6ba-4995-b400-87869f9a9b82
1062024-02-24T05:04:41.743ZINFOcrucible-agent (datafile): region 0e740589-d6ba-4995-b400-87869f9a9b82 state: Created -> Tombstoned
1072024-02-24T05:04:41.743ZINFOcrucible-agent (worker): applying SMF actions before removal...
1082024-02-24T05:04:41.743ZINFOcrucible-agent (dropshot): request completed latency_us = 20902 local_addr = [fd00:1122:3344:101::17]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:63817 req_id = d25d4f1e-415d-4182-8dc2-25b67a1a79ba response_code = 204 uri = /crucible/0/regions/0e740589-d6ba-4995-b400-87869f9a9b82
1092024-02-24T05:04:41.743ZINFOcrucible-agent (dropshot): request completed latency_us = 20 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:63817 req_id = 240d6bb6-f27a-4afe-9bd0-2300e9c78885 response_code = 200 uri = /crucible/0/regions/0e740589-d6ba-4995-b400-87869f9a9b82
1102024-02-24T05:04:41.744ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-0e740589-d6ba-4995-b400-87869f9a9b82 (instance states: (Some(Online), None))
1112024-02-24T05:04:41.755ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/3afaa2d5-82c5-498f-9558-c2092959d7d0
1122024-02-24T05:04:41.755ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
1132024-02-24T05:04:41.755ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::17
1142024-02-24T05:04:41.755ZINFOcrucible-agent (worker): reconfig required
1152024-02-24T05:04:41.755ZINFOcrucible-agent (worker): using existing config property group
1162024-02-24T05:04:41.755ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-3afaa2d5-82c5-498f-9558-c2092959d7d0
1172024-02-24T05:04:41.755ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/3afaa2d5-82c5-498f-9558-c2092959d7d0
1182024-02-24T05:04:41.755ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
1192024-02-24T05:04:41.755ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
1202024-02-24T05:04:41.756ZINFOcrucible-agent (worker): commit
1212024-02-24T05:04:41.763ZINFOcrucible-agent (worker): ok!
1222024-02-24T05:04:41.766ZINFOcrucible-agent (worker): SMF ok!
1232024-02-24T05:04:41.774ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60/crucible/regions/0e740589-d6ba-4995-b400-87869f9a9b82" region = 0e740589-d6ba-4995-b400-87869f9a9b82
1242024-02-24T05:04:41.797ZINFOcrucible-agent (dropshot): request completed latency_us = 74 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:63817 req_id = 730d7f16-426b-48bb-b119-c157b6f11cc5 response_code = 200 uri = /crucible/0/regions/0e740589-d6ba-4995-b400-87869f9a9b82
1252024-02-24T05:04:41.833ZINFOcrucible-agent (datafile): region 0e740589-d6ba-4995-b400-87869f9a9b82 state: Tombstoned -> Destroyed
1262024-02-24T05:04:41.941ZINFOcrucible-agent (dropshot): request completed latency_us = 39 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:63817 req_id = ed71664a-23c4-47e6-be46-b56d21328fc3 response_code = 200 uri = /crucible/0/regions/0e740589-d6ba-4995-b400-87869f9a9b82