Max level shown:
12024-12-20T23:05:23.015ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::14]:32345 remote_addr = [fd00:1122:3344:101::a]:42409
22024-12-20T23:05:23.015ZINFOcrucible-agent (datafile): region 1448a217-9d00-4a33-8d31-c447039073aa state: Requested
32024-12-20T23:05:23.015ZINFOcrucible-agent (dropshot): request completed latency_us = 863 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:42409 req_id = d853f0d5-51f3-4639-a179-cc9877ee0a52 response_code = 200 uri = /crucible/0/regions
42024-12-20T23:05:23.015ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
52024-12-20T23:05:23.027ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_d31b2b69-efd5-461d-912c-5052088aed91/crucible/regions/1448a217-9d00-4a33-8d31-c447039073aa
62024-12-20T23:05:23.027ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_d31b2b69-efd5-461d-912c-5052088aed91/crucible/regions/1448a217-9d00-4a33-8d31-c447039073aa
72024-12-20T23:05:23.089ZINFOcrucible-agent (worker): creating region Region { id: RegionId("1448a217-9d00-4a33-8d31-c447039073aa"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/1448a217-9d00-4a33-8d31-c447039073aa" region = 1448a217-9d00-4a33-8d31-c447039073aa
82024-12-20T23:05:23.089ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "1448a217-9d00-4a33-8d31-c447039073aa" "--data" "/data/regions/1448a217-9d00-4a33-8d31-c447039073aa" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 1448a217-9d00-4a33-8d31-c447039073aa
92024-12-20T23:05:23.166ZINFOcrucible-agent (worker): region files created ok region = 1448a217-9d00-4a33-8d31-c447039073aa
102024-12-20T23:05:23.166ZINFOcrucible-agent (datafile): region 1448a217-9d00-4a33-8d31-c447039073aa state: Requested -> Created
112024-12-20T23:05:23.166ZINFOcrucible-agent (worker): applying SMF actions post create...
122024-12-20T23:05:23.167ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-1448a217-9d00-4a33-8d31-c447039073aa
132024-12-20T23:05:23.174ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-1448a217-9d00-4a33-8d31-c447039073aa
142024-12-20T23:05:23.189ZINFOcrucible-agent (worker): creating config property group
152024-12-20T23:05:23.192ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-1448a217-9d00-4a33-8d31-c447039073aa
162024-12-20T23:05:23.192ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/1448a217-9d00-4a33-8d31-c447039073aa
172024-12-20T23:05:23.192ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
182024-12-20T23:05:23.192ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
192024-12-20T23:05:23.192ZINFOcrucible-agent (worker): commit
202024-12-20T23:05:23.196ZINFOcrucible-agent (worker): ok!
212024-12-20T23:05:23.204ZINFOcrucible-agent (worker): SMF ok!
222024-12-20T23:05:23.315ZINFOcrucible-agent (dropshot): request completed latency_us = 77 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:42409 req_id = 9cb2df9b-6171-41a4-b621-f4b115f25f75 response_code = 200 uri = /crucible/0/regions
232024-12-20T23:05:24.848ZINFOcrucible-agent (dropshot): request completed latency_us = 2527 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:42409 req_id = b3d8010b-ce97-47a7-8a91-ec79ee24f8b1 response_code = 200 uri = /crucible/0/regions/942e408b-abed-4367-8507-3066484bde2f
242024-12-20T23:05:24.852ZINFOcrucible-agent (datafile): path is "/data/regions/942e408b-abed-4367-8507-3066484bde2f"
252024-12-20T23:05:24.861ZINFOcrucible-agent (datafile): dataset is oxp_d31b2b69-efd5-461d-912c-5052088aed91/crucible/regions/942e408b-abed-4367-8507-3066484bde2f
262024-12-20T23:05:24.871ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_d31b2b69-efd5-461d-912c-5052088aed91/crucible/regions/942e408b-abed-4367-8507-3066484bde2f@9c5cc29d-1ebc-433e-8a96-f229cda2e28c
272024-12-20T23:05:24.871ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_d31b2b69-efd5-461d-912c-5052088aed91/crucible/regions/942e408b-abed-4367-8507-3066484bde2f", "9c5cc29d-1ebc-433e-8a96-f229cda2e28c"]
282024-12-20T23:05:24.885ZINFOcrucible-agent (dropshot): request completed latency_us = 33024 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:42409 req_id = f99917f7-574c-4e32-8331-6b941e3fdf9f response_code = 200 uri = /crucible/0/regions/942e408b-abed-4367-8507-3066484bde2f/snapshots/9c5cc29d-1ebc-433e-8a96-f229cda2e28c
292024-12-20T23:05:24.888ZINFOcrucible-agent (datafile): path is "/data/regions/942e408b-abed-4367-8507-3066484bde2f"
302024-12-20T23:05:24.899ZINFOcrucible-agent (datafile): dataset is oxp_d31b2b69-efd5-461d-912c-5052088aed91/crucible/regions/942e408b-abed-4367-8507-3066484bde2f
312024-12-20T23:05:24.908ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_d31b2b69-efd5-461d-912c-5052088aed91/crucible/regions/942e408b-abed-4367-8507-3066484bde2f@9c5cc29d-1ebc-433e-8a96-f229cda2e28c
322024-12-20T23:05:24.908ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_d31b2b69-efd5-461d-912c-5052088aed91/crucible/regions/942e408b-abed-4367-8507-3066484bde2f", "9c5cc29d-1ebc-433e-8a96-f229cda2e28c"]
332024-12-20T23:05:24.918ZINFOcrucible-agent (datafile): requesting running snapshot 942e408b-abed-4367-8507-3066484bde2f-9c5cc29d-1ebc-433e-8a96-f229cda2e28c state: Requested
342024-12-20T23:05:24.918ZINFOcrucible-agent (worker): applying SMF actions for region 942e408b-abed-4367-8507-3066484bde2f running snapshot 9c5cc29d-1ebc-433e-8a96-f229cda2e28c (state Requested)...
352024-12-20T23:05:24.918ZINFOcrucible-agent (dropshot): request completed latency_us = 29891 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:42409 req_id = 421c4ce1-1c08-4b8d-9e83-802078ddc9b4 response_code = 200 uri = /crucible/0/regions/942e408b-abed-4367-8507-3066484bde2f/snapshots/9c5cc29d-1ebc-433e-8a96-f229cda2e28c/run
362024-12-20T23:05:24.921ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-942e408b-abed-4367-8507-3066484bde2f-9c5cc29d-1ebc-433e-8a96-f229cda2e28c
372024-12-20T23:05:24.930ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-942e408b-abed-4367-8507-3066484bde2f-9c5cc29d-1ebc-433e-8a96-f229cda2e28c
382024-12-20T23:05:24.940ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-942e408b-abed-4367-8507-3066484bde2f-9c5cc29d-1ebc-433e-8a96-f229cda2e28c
392024-12-20T23:05:24.941ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/942e408b-abed-4367-8507-3066484bde2f/.zfs/snapshot/9c5cc29d-1ebc-433e-8a96-f229cda2e28c
402024-12-20T23:05:24.941ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
412024-12-20T23:05:24.941ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
422024-12-20T23:05:24.941ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
432024-12-20T23:05:24.941ZINFOcrucible-agent (worker): commit
442024-12-20T23:05:24.944ZINFOcrucible-agent (worker): ok!
452024-12-20T23:05:24.949ZINFOcrucible-agent (worker): SMF ok!
462024-12-20T23:05:24.949ZINFOcrucible-agent (datafile): region 942e408b-abed-4367-8507-3066484bde2f running snapshot 9c5cc29d-1ebc-433e-8a96-f229cda2e28c state: Requested -> Created
472024-12-20T23:05:25.181ZINFOcrucible-agent (datafile): path is "/data/regions/942e408b-abed-4367-8507-3066484bde2f"
482024-12-20T23:05:25.190ZINFOcrucible-agent (datafile): dataset is oxp_d31b2b69-efd5-461d-912c-5052088aed91/crucible/regions/942e408b-abed-4367-8507-3066484bde2f
492024-12-20T23:05:25.199ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_d31b2b69-efd5-461d-912c-5052088aed91/crucible/regions/942e408b-abed-4367-8507-3066484bde2f@9c5cc29d-1ebc-433e-8a96-f229cda2e28c
502024-12-20T23:05:25.199ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_d31b2b69-efd5-461d-912c-5052088aed91/crucible/regions/942e408b-abed-4367-8507-3066484bde2f", "9c5cc29d-1ebc-433e-8a96-f229cda2e28c"]
512024-12-20T23:05:25.208ZINFOcrucible-agent (dropshot): request completed latency_us = 27317 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:42409 req_id = 9fed71c8-a2eb-4677-b28d-a069ff955dc9 response_code = 200 uri = /crucible/0/regions/942e408b-abed-4367-8507-3066484bde2f/snapshots/9c5cc29d-1ebc-433e-8a96-f229cda2e28c/run
522024-12-20T23:05:28.634ZINFOcrucible-agent (datafile): region ae6a20a5-4672-41f9-8d90-df8482d29402 state: Requested
532024-12-20T23:05:28.634ZINFOcrucible-agent (dropshot): request completed latency_us = 233 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:42409 req_id = 0079f69e-07b1-4afb-a8dd-e87a9f159502 response_code = 200 uri = /crucible/0/regions
542024-12-20T23:05:28.634ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
552024-12-20T23:05:28.643ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_d31b2b69-efd5-461d-912c-5052088aed91/crucible/regions/ae6a20a5-4672-41f9-8d90-df8482d29402
562024-12-20T23:05:28.643ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_d31b2b69-efd5-461d-912c-5052088aed91/crucible/regions/ae6a20a5-4672-41f9-8d90-df8482d29402
572024-12-20T23:05:28.705ZINFOcrucible-agent (worker): creating region Region { id: RegionId("ae6a20a5-4672-41f9-8d90-df8482d29402"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19003, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/ae6a20a5-4672-41f9-8d90-df8482d29402" region = ae6a20a5-4672-41f9-8d90-df8482d29402
582024-12-20T23:05:28.707ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "ae6a20a5-4672-41f9-8d90-df8482d29402" "--data" "/data/regions/ae6a20a5-4672-41f9-8d90-df8482d29402" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = ae6a20a5-4672-41f9-8d90-df8482d29402
592024-12-20T23:05:28.787ZINFOcrucible-agent (worker): region files created ok region = ae6a20a5-4672-41f9-8d90-df8482d29402
602024-12-20T23:05:28.787ZINFOcrucible-agent (datafile): region ae6a20a5-4672-41f9-8d90-df8482d29402 state: Requested -> Created
612024-12-20T23:05:28.787ZINFOcrucible-agent (worker): applying SMF actions post create...
622024-12-20T23:05:28.789ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-ae6a20a5-4672-41f9-8d90-df8482d29402
632024-12-20T23:05:28.792ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-ae6a20a5-4672-41f9-8d90-df8482d29402
642024-12-20T23:05:28.799ZINFOcrucible-agent (worker): creating config property group
652024-12-20T23:05:28.802ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-ae6a20a5-4672-41f9-8d90-df8482d29402
662024-12-20T23:05:28.802ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/ae6a20a5-4672-41f9-8d90-df8482d29402
672024-12-20T23:05:28.802ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19003
682024-12-20T23:05:28.802ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::14
692024-12-20T23:05:28.802ZINFOcrucible-agent (worker): commit
702024-12-20T23:05:28.805ZINFOcrucible-agent (worker): ok!
712024-12-20T23:05:28.811ZINFOcrucible-agent (worker): SMF ok!
722024-12-20T23:05:29.009ZINFOcrucible-agent (dropshot): request completed latency_us = 72 local_addr = [fd00:1122:3344:101::14]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:42409 req_id = 2fb70022-f1d8-4bff-85bc-0392ed40afc0 response_code = 200 uri = /crucible/0/regions
732024-12-20T23:05:54.225ZINFOcrucible-agent (dropshot): request completed latency_us = 114 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:42409 req_id = f0c4e9e1-1be6-4a37-9e2e-e0cd42c10e8c response_code = 200 uri = /crucible/0/regions/ae6a20a5-4672-41f9-8d90-df8482d29402
742024-12-20T23:05:54.228ZINFOcrucible-agent (datafile): path is "/data/regions/ae6a20a5-4672-41f9-8d90-df8482d29402"
752024-12-20T23:05:54.242ZINFOcrucible-agent (datafile): dataset is oxp_d31b2b69-efd5-461d-912c-5052088aed91/crucible/regions/ae6a20a5-4672-41f9-8d90-df8482d29402
762024-12-20T23:05:54.252ZINFOcrucible-agent (datafile): region ae6a20a5-4672-41f9-8d90-df8482d29402 state: Created -> Tombstoned
772024-12-20T23:05:54.253ZINFOcrucible-agent (worker): applying SMF actions before removal...
782024-12-20T23:05:54.253ZINFOcrucible-agent (dropshot): request completed latency_us = 24405 local_addr = [fd00:1122:3344:101::14]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:42409 req_id = 6611b912-0c3e-4741-b71a-8e5a602ea3bd response_code = 204 uri = /crucible/0/regions/ae6a20a5-4672-41f9-8d90-df8482d29402
792024-12-20T23:05:54.256ZINFOcrucible-agent (dropshot): request completed latency_us = 36 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:42409 req_id = 9419082d-6d3e-4d82-85a2-f2908d972356 response_code = 200 uri = /crucible/0/regions/ae6a20a5-4672-41f9-8d90-df8482d29402
802024-12-20T23:05:54.256ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-ae6a20a5-4672-41f9-8d90-df8482d29402 (instance states: (Some(Online), None))
812024-12-20T23:05:54.264ZINFOcrucible-agent (worker): SMF ok!
822024-12-20T23:05:54.275ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_d31b2b69-efd5-461d-912c-5052088aed91/crucible/regions/ae6a20a5-4672-41f9-8d90-df8482d29402" region = ae6a20a5-4672-41f9-8d90-df8482d29402
832024-12-20T23:05:54.328ZINFOcrucible-agent (datafile): region ae6a20a5-4672-41f9-8d90-df8482d29402 state: Tombstoned -> Destroyed
842024-12-20T23:05:54.353ZINFOcrucible-agent (dropshot): request completed latency_us = 84 local_addr = [fd00:1122:3344:101::14]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:42409 req_id = af45b162-4023-4350-aec8-a52a46eaa5ef response_code = 200 uri = /crucible/0/regions/ae6a20a5-4672-41f9-8d90-df8482d29402