Max level shown:
12024-03-11T14:45:44.129ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::19]:32345 remote_addr = [fd00:1122:3344:101::a]:37287
22024-03-11T14:45:44.189ZINFOcrucible-agent (datafile): region ceb33e57-9744-47c0-bab4-aaf68a2701a4 state: Requested
32024-03-11T14:45:44.190ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
42024-03-11T14:45:44.190ZINFOcrucible-agent (dropshot): request completed latency_us = 1336 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:37287 req_id = 3f0070e7-9ab3-4328-8bbf-4943b6504c33 response_code = 200 uri = /crucible/0/regions
52024-03-11T14:45:44.199ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/ceb33e57-9744-47c0-bab4-aaf68a2701a4
62024-03-11T14:45:44.199ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/ceb33e57-9744-47c0-bab4-aaf68a2701a4
72024-03-11T14:45:44.259ZINFOcrucible-agent (worker): creating region Region { id: RegionId("ceb33e57-9744-47c0-bab4-aaf68a2701a4"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None } at "/data/regions/ceb33e57-9744-47c0-bab4-aaf68a2701a4" region = ceb33e57-9744-47c0-bab4-aaf68a2701a4
82024-03-11T14:45:44.352ZINFOcrucible-agent (worker): region files created ok region = ceb33e57-9744-47c0-bab4-aaf68a2701a4
92024-03-11T14:45:44.352ZINFOcrucible-agent (datafile): region ceb33e57-9744-47c0-bab4-aaf68a2701a4 state: Requested -> Created
102024-03-11T14:45:44.352ZINFOcrucible-agent (worker): applying SMF actions post create...
112024-03-11T14:45:44.361ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/79bda774-159f-4095-a620-1c1da81a9191
122024-03-11T14:45:44.361ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
132024-03-11T14:45:44.361ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::19
142024-03-11T14:45:44.361ZINFOcrucible-agent (worker): reconfig required
152024-03-11T14:45:44.361ZINFOcrucible-agent (worker): using existing config property group
162024-03-11T14:45:44.361ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-79bda774-159f-4095-a620-1c1da81a9191
172024-03-11T14:45:44.362ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/79bda774-159f-4095-a620-1c1da81a9191
182024-03-11T14:45:44.362ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
192024-03-11T14:45:44.362ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
202024-03-11T14:45:44.362ZINFOcrucible-agent (worker): commit
212024-03-11T14:45:44.367ZINFOcrucible-agent (worker): ok!
222024-03-11T14:45:44.368ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-ceb33e57-9744-47c0-bab4-aaf68a2701a4
232024-03-11T14:45:44.371ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-ceb33e57-9744-47c0-bab4-aaf68a2701a4
242024-03-11T14:45:44.381ZINFOcrucible-agent (worker): creating config property group
252024-03-11T14:45:44.384ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-ceb33e57-9744-47c0-bab4-aaf68a2701a4
262024-03-11T14:45:44.384ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/ceb33e57-9744-47c0-bab4-aaf68a2701a4
272024-03-11T14:45:44.384ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
282024-03-11T14:45:44.384ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
292024-03-11T14:45:44.384ZINFOcrucible-agent (worker): commit
302024-03-11T14:45:44.389ZINFOcrucible-agent (worker): ok!
312024-03-11T14:45:44.393ZINFOcrucible-agent (worker): SMF ok!
322024-03-11T14:45:44.533ZINFOcrucible-agent (dropshot): request completed latency_us = 74 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:37287 req_id = 431acb66-e0f2-4bdb-89bb-b0eeafc115d5 response_code = 200 uri = /crucible/0/regions
332024-03-11T14:45:45.766ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::19]:32345 remote_addr = [fd00:1122:3344:101::a]:39585
342024-03-11T14:45:45.766ZINFOcrucible-agent (dropshot): request completed latency_us = 83 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:39585 req_id = 418a52f5-c353-4026-851d-49289b05d832 response_code = 200 uri = /crucible/0/regions/79bda774-159f-4095-a620-1c1da81a9191
352024-03-11T14:45:45.766ZINFOcrucible-agent (datafile): path is "/data/regions/79bda774-159f-4095-a620-1c1da81a9191"
362024-03-11T14:45:45.775ZINFOcrucible-agent (datafile): dataset is oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/79bda774-159f-4095-a620-1c1da81a9191
372024-03-11T14:45:45.784ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/79bda774-159f-4095-a620-1c1da81a9191@a54a324d-800f-4645-83ae-56c8f6240caa
382024-03-11T14:45:45.784ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/79bda774-159f-4095-a620-1c1da81a9191", "a54a324d-800f-4645-83ae-56c8f6240caa"]
392024-03-11T14:45:45.796ZINFOcrucible-agent (dropshot): request completed latency_us = 30054 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:39585 req_id = 4f75f5af-d827-440a-8805-38d388428a07 response_code = 200 uri = /crucible/0/regions/79bda774-159f-4095-a620-1c1da81a9191/snapshots/a54a324d-800f-4645-83ae-56c8f6240caa
402024-03-11T14:45:45.797ZINFOcrucible-agent (datafile): path is "/data/regions/79bda774-159f-4095-a620-1c1da81a9191"
412024-03-11T14:45:45.805ZINFOcrucible-agent (datafile): dataset is oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/79bda774-159f-4095-a620-1c1da81a9191
422024-03-11T14:45:45.814ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/79bda774-159f-4095-a620-1c1da81a9191@a54a324d-800f-4645-83ae-56c8f6240caa
432024-03-11T14:45:45.814ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/79bda774-159f-4095-a620-1c1da81a9191", "a54a324d-800f-4645-83ae-56c8f6240caa"]
442024-03-11T14:45:45.826ZINFOcrucible-agent (datafile): requesting running snapshot 79bda774-159f-4095-a620-1c1da81a9191-a54a324d-800f-4645-83ae-56c8f6240caa state: Requested
452024-03-11T14:45:45.827ZINFOcrucible-agent (worker): applying SMF actions for region 79bda774-159f-4095-a620-1c1da81a9191 running snapshot a54a324d-800f-4645-83ae-56c8f6240caa (state Requested)...
462024-03-11T14:45:45.827ZINFOcrucible-agent (dropshot): request completed latency_us = 30369 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:39585 req_id = 67d585d5-7d53-4545-8a76-5e9dc4955d42 response_code = 200 uri = /crucible/0/regions/79bda774-159f-4095-a620-1c1da81a9191/snapshots/a54a324d-800f-4645-83ae-56c8f6240caa/run
472024-03-11T14:45:45.827ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/79bda774-159f-4095-a620-1c1da81a9191
482024-03-11T14:45:45.827ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
492024-03-11T14:45:45.827ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::19
502024-03-11T14:45:45.827ZINFOcrucible-agent (worker): reconfig required
512024-03-11T14:45:45.827ZINFOcrucible-agent (worker): using existing config property group
522024-03-11T14:45:45.828ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-79bda774-159f-4095-a620-1c1da81a9191
532024-03-11T14:45:45.828ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/79bda774-159f-4095-a620-1c1da81a9191
542024-03-11T14:45:45.828ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
552024-03-11T14:45:45.828ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
562024-03-11T14:45:45.828ZINFOcrucible-agent (worker): commit
572024-03-11T14:45:45.832ZINFOcrucible-agent (worker): ok!
582024-03-11T14:45:45.834ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-79bda774-159f-4095-a620-1c1da81a9191-a54a324d-800f-4645-83ae-56c8f6240caa
592024-03-11T14:45:45.839ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-79bda774-159f-4095-a620-1c1da81a9191-a54a324d-800f-4645-83ae-56c8f6240caa
602024-03-11T14:45:45.850ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-79bda774-159f-4095-a620-1c1da81a9191-a54a324d-800f-4645-83ae-56c8f6240caa
612024-03-11T14:45:45.850ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/79bda774-159f-4095-a620-1c1da81a9191/.zfs/snapshot/a54a324d-800f-4645-83ae-56c8f6240caa
622024-03-11T14:45:45.850ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
632024-03-11T14:45:45.850ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
642024-03-11T14:45:45.850ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
652024-03-11T14:45:45.850ZINFOcrucible-agent (worker): commit
662024-03-11T14:45:45.854ZINFOcrucible-agent (worker): ok!
672024-03-11T14:45:45.858ZINFOcrucible-agent (worker): SMF ok!
682024-03-11T14:45:45.858ZINFOcrucible-agent (datafile): region 79bda774-159f-4095-a620-1c1da81a9191 running snapshot a54a324d-800f-4645-83ae-56c8f6240caa state: Requested -> Created
692024-03-11T14:45:48.662ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::19]:32345 remote_addr = [fd00:1122:3344:101::a]:51905
702024-03-11T14:45:48.692ZINFOcrucible-agent (datafile): region 4b37e53e-38fd-4b1b-afd7-2790f2a70b8d state: Requested
712024-03-11T14:45:48.692ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
722024-03-11T14:45:48.692ZINFOcrucible-agent (dropshot): request completed latency_us = 377 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:51905 req_id = b833e323-14b1-44c3-870c-8d9a2adce0f9 response_code = 200 uri = /crucible/0/regions
732024-03-11T14:45:48.702ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/4b37e53e-38fd-4b1b-afd7-2790f2a70b8d
742024-03-11T14:45:48.702ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/4b37e53e-38fd-4b1b-afd7-2790f2a70b8d
752024-03-11T14:45:48.774ZINFOcrucible-agent (worker): creating region Region { id: RegionId("4b37e53e-38fd-4b1b-afd7-2790f2a70b8d"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19003, cert_pem: None, key_pem: None, root_pem: None } at "/data/regions/4b37e53e-38fd-4b1b-afd7-2790f2a70b8d" region = 4b37e53e-38fd-4b1b-afd7-2790f2a70b8d
762024-03-11T14:45:48.845ZINFOcrucible-agent (worker): region files created ok region = 4b37e53e-38fd-4b1b-afd7-2790f2a70b8d
772024-03-11T14:45:48.845ZINFOcrucible-agent (datafile): region 4b37e53e-38fd-4b1b-afd7-2790f2a70b8d state: Requested -> Created
782024-03-11T14:45:48.845ZINFOcrucible-agent (worker): applying SMF actions post create...
792024-03-11T14:45:48.845ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-4b37e53e-38fd-4b1b-afd7-2790f2a70b8d
802024-03-11T14:45:48.848ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-4b37e53e-38fd-4b1b-afd7-2790f2a70b8d
812024-03-11T14:45:48.856ZINFOcrucible-agent (worker): creating config property group
822024-03-11T14:45:48.860ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-4b37e53e-38fd-4b1b-afd7-2790f2a70b8d
832024-03-11T14:45:48.860ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/4b37e53e-38fd-4b1b-afd7-2790f2a70b8d
842024-03-11T14:45:48.860ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19003
852024-03-11T14:45:48.860ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
862024-03-11T14:45:48.860ZINFOcrucible-agent (worker): commit
872024-03-11T14:45:48.864ZINFOcrucible-agent (worker): ok!
882024-03-11T14:45:48.870ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/79bda774-159f-4095-a620-1c1da81a9191
892024-03-11T14:45:48.870ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
902024-03-11T14:45:48.870ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::19
912024-03-11T14:45:48.870ZINFOcrucible-agent (worker): reconfig required
922024-03-11T14:45:48.870ZINFOcrucible-agent (worker): using existing config property group
932024-03-11T14:45:48.870ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-79bda774-159f-4095-a620-1c1da81a9191
942024-03-11T14:45:48.874ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/79bda774-159f-4095-a620-1c1da81a9191
952024-03-11T14:45:48.875ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
962024-03-11T14:45:48.875ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
972024-03-11T14:45:48.875ZINFOcrucible-agent (worker): commit
982024-03-11T14:45:48.878ZINFOcrucible-agent (worker): ok!
992024-03-11T14:45:48.881ZINFOcrucible-agent (worker): SMF ok!
1002024-03-11T14:45:48.933ZINFOcrucible-agent (dropshot): request completed latency_us = 86 local_addr = [fd00:1122:3344:101::19]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:51905 req_id = 011376a4-1b6c-4273-b431-746d46a0e7e5 response_code = 200 uri = /crucible/0/regions
1012024-03-11T14:46:20.895ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::19]:32345 remote_addr = [fd00:1122:3344:101::a]:46375
1022024-03-11T14:46:20.895ZINFOcrucible-agent (dropshot): request completed latency_us = 84 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:46375 req_id = c217c05b-d529-4227-940e-26230acf9040 response_code = 200 uri = /crucible/0/regions/4b37e53e-38fd-4b1b-afd7-2790f2a70b8d
1032024-03-11T14:46:20.895ZINFOcrucible-agent (datafile): path is "/data/regions/4b37e53e-38fd-4b1b-afd7-2790f2a70b8d"
1042024-03-11T14:46:20.908ZINFOcrucible-agent (datafile): dataset is oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/4b37e53e-38fd-4b1b-afd7-2790f2a70b8d
1052024-03-11T14:46:20.916ZINFOcrucible-agent (datafile): region 4b37e53e-38fd-4b1b-afd7-2790f2a70b8d state: Created -> Tombstoned
1062024-03-11T14:46:20.916ZINFOcrucible-agent (worker): applying SMF actions before removal...
1072024-03-11T14:46:20.916ZINFOcrucible-agent (dropshot): request completed latency_us = 21130 local_addr = [fd00:1122:3344:101::19]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:46375 req_id = 357832b8-5fe2-4caa-96b8-e9c1d502a7b9 response_code = 204 uri = /crucible/0/regions/4b37e53e-38fd-4b1b-afd7-2790f2a70b8d
1082024-03-11T14:46:20.917ZINFOcrucible-agent (dropshot): request completed latency_us = 21 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:46375 req_id = cd4f9789-8d69-4a7d-8fb5-62639c885f9f response_code = 200 uri = /crucible/0/regions/4b37e53e-38fd-4b1b-afd7-2790f2a70b8d
1092024-03-11T14:46:20.918ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-4b37e53e-38fd-4b1b-afd7-2790f2a70b8d (instance states: (Some(Online), None))
1102024-03-11T14:46:20.922ZINFOcrucible-agent (worker): existing directory value /dev/null does not match /data/regions/79bda774-159f-4095-a620-1c1da81a9191
1112024-03-11T14:46:20.922ZINFOcrucible-agent (worker): existing port value 0 does not match 19000
1122024-03-11T14:46:20.922ZINFOcrucible-agent (worker): existing address value 0.0.0.0 does not match fd00:1122:3344:101::19
1132024-03-11T14:46:20.922ZINFOcrucible-agent (worker): reconfig required
1142024-03-11T14:46:20.922ZINFOcrucible-agent (worker): using existing config property group
1152024-03-11T14:46:20.922ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-79bda774-159f-4095-a620-1c1da81a9191
1162024-03-11T14:46:20.923ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/79bda774-159f-4095-a620-1c1da81a9191
1172024-03-11T14:46:20.923ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
1182024-03-11T14:46:20.923ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::19
1192024-03-11T14:46:20.923ZINFOcrucible-agent (worker): commit
1202024-03-11T14:46:20.926ZINFOcrucible-agent (worker): ok!
1212024-03-11T14:46:20.929ZINFOcrucible-agent (worker): SMF ok!
1222024-03-11T14:46:20.939ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/4b37e53e-38fd-4b1b-afd7-2790f2a70b8d" region = 4b37e53e-38fd-4b1b-afd7-2790f2a70b8d
1232024-03-11T14:46:20.999ZINFOcrucible-agent (datafile): region 4b37e53e-38fd-4b1b-afd7-2790f2a70b8d state: Tombstoned -> Destroyed
1242024-03-11T14:46:21.056ZINFOcrucible-agent (dropshot): request completed latency_us = 549 local_addr = [fd00:1122:3344:101::19]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:46375 req_id = c0db05a6-8325-49fa-951d-614ef57d6d8e response_code = 200 uri = /crucible/0/regions/4b37e53e-38fd-4b1b-afd7-2790f2a70b8d