Max level shown:
1 [ Jul 19 18:45:09 Enabled. ]
2 [ Jul 19 18:45:09 Rereading configuration. ]
3 [ Jul 19 18:45:11 Rereading configuration. ]
4 [ Jul 19 18:45:13 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_ceb75e0c-fc60-474c-85f7-e6b93e4d0425/crucible -l [fd00:1122:3344:101::17]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62024-07-19T18:45:13.787ZINFOcrucible-agent: dataset: "oxp_ceb75e0c-fc60-474c-85f7-e6b93e4d0425/crucible"
72024-07-19T18:45:13.791ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::17]:32345
82024-07-19T18:45:13.791ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92024-07-19T18:45:13.805ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102024-07-19T18:45:13.940ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::17]:32345
112024-07-19T18:45:33.258ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::17]:32345 remote_addr = [fd00:1122:3344:101::a]:61900
122024-07-19T18:45:33.268ZINFOcrucible-agent (datafile): region d5d8e980-8daf-48a5-982c-43a33cda9032 state: Requested
132024-07-19T18:45:33.271ZINFOcrucible-agent (dropshot): request completed latency_us = 4962 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:61900 req_id = 9dacea65-79d2-4640-857a-2b5600ba9277 response_code = 200 uri = /crucible/0/regions
142024-07-19T18:45:33.273ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
152024-07-19T18:45:33.283ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_ceb75e0c-fc60-474c-85f7-e6b93e4d0425/crucible/regions/d5d8e980-8daf-48a5-982c-43a33cda9032
162024-07-19T18:45:33.283ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_ceb75e0c-fc60-474c-85f7-e6b93e4d0425/crucible/regions/d5d8e980-8daf-48a5-982c-43a33cda9032
172024-07-19T18:45:33.352ZINFOcrucible-agent (worker): creating region Region { id: RegionId("d5d8e980-8daf-48a5-982c-43a33cda9032"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/d5d8e980-8daf-48a5-982c-43a33cda9032" region = d5d8e980-8daf-48a5-982c-43a33cda9032
182024-07-19T18:45:33.354ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "d5d8e980-8daf-48a5-982c-43a33cda9032" "--data" "/data/regions/d5d8e980-8daf-48a5-982c-43a33cda9032" "--block-size" "512" "--extent-size" "131072" "--extent-count" "16" "--encrypted" region = d5d8e980-8daf-48a5-982c-43a33cda9032
192024-07-19T18:45:33.477ZINFOcrucible-agent (worker): region files created ok region = d5d8e980-8daf-48a5-982c-43a33cda9032
202024-07-19T18:45:33.477ZINFOcrucible-agent (datafile): region d5d8e980-8daf-48a5-982c-43a33cda9032 state: Requested -> Created
212024-07-19T18:45:33.477ZINFOcrucible-agent (worker): applying SMF actions post create...
222024-07-19T18:45:33.477ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-d5d8e980-8daf-48a5-982c-43a33cda9032
232024-07-19T18:45:33.483ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-d5d8e980-8daf-48a5-982c-43a33cda9032
242024-07-19T18:45:33.497ZINFOcrucible-agent (worker): creating config property group
252024-07-19T18:45:33.501ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-d5d8e980-8daf-48a5-982c-43a33cda9032
262024-07-19T18:45:33.501ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/d5d8e980-8daf-48a5-982c-43a33cda9032
272024-07-19T18:45:33.501ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
282024-07-19T18:45:33.501ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
292024-07-19T18:45:33.502ZINFOcrucible-agent (worker): commit
302024-07-19T18:45:33.507ZINFOcrucible-agent (worker): ok!
312024-07-19T18:45:33.519ZINFOcrucible-agent (worker): SMF ok!
322024-07-19T18:45:33.555ZINFOcrucible-agent (dropshot): request completed latency_us = 81 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:61900 req_id = a70eeb95-9bc6-4033-ab43-3571ad0085d2 response_code = 200 uri = /crucible/0/regions
332024-07-19T18:45:34.086ZINFOcrucible-agent (dropshot): request completed latency_us = 86 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:61900 req_id = acc3dc39-7718-490c-9050-7e777fb9e601 response_code = 200 uri = /crucible/0/regions/d5d8e980-8daf-48a5-982c-43a33cda9032
342024-07-19T18:45:34.088ZINFOcrucible-agent (datafile): path is "/data/regions/d5d8e980-8daf-48a5-982c-43a33cda9032"
352024-07-19T18:45:34.098ZINFOcrucible-agent (datafile): dataset is oxp_ceb75e0c-fc60-474c-85f7-e6b93e4d0425/crucible/regions/d5d8e980-8daf-48a5-982c-43a33cda9032
362024-07-19T18:45:34.106ZINFOcrucible-agent (datafile): region d5d8e980-8daf-48a5-982c-43a33cda9032 state: Created -> Tombstoned
372024-07-19T18:45:34.106ZINFOcrucible-agent (worker): applying SMF actions before removal...
382024-07-19T18:45:34.106ZINFOcrucible-agent (dropshot): request completed latency_us = 18240 local_addr = [fd00:1122:3344:101::17]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:61900 req_id = f6921592-b878-4838-98a8-c54d304516a1 response_code = 204 uri = /crucible/0/regions/d5d8e980-8daf-48a5-982c-43a33cda9032
392024-07-19T18:45:34.107ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-d5d8e980-8daf-48a5-982c-43a33cda9032 (instance states: (Some(Online), None))
402024-07-19T18:45:34.109ZINFOcrucible-agent (dropshot): request completed latency_us = 585 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:61900 req_id = d5bf72ed-85d9-4ada-b43e-93166b791727 response_code = 200 uri = /crucible/0/regions/d5d8e980-8daf-48a5-982c-43a33cda9032
412024-07-19T18:45:34.112ZINFOcrucible-agent (worker): SMF ok!
422024-07-19T18:45:34.121ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_ceb75e0c-fc60-474c-85f7-e6b93e4d0425/crucible/regions/d5d8e980-8daf-48a5-982c-43a33cda9032" region = d5d8e980-8daf-48a5-982c-43a33cda9032
432024-07-19T18:45:34.169ZINFOcrucible-agent (datafile): region d5d8e980-8daf-48a5-982c-43a33cda9032 state: Tombstoned -> Destroyed
442024-07-19T18:45:34.192ZINFOcrucible-agent (dropshot): request completed latency_us = 62 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:61900 req_id = bbb16968-1211-4960-b89a-8320e5939d38 response_code = 200 uri = /crucible/0/regions/d5d8e980-8daf-48a5-982c-43a33cda9032
452024-07-19T18:45:35.822ZINFOcrucible-agent (datafile): region cf24c0a1-091b-41da-952f-aaddb5237975 state: Requested
462024-07-19T18:45:35.824ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
472024-07-19T18:45:35.824ZINFOcrucible-agent (dropshot): request completed latency_us = 2097 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:61900 req_id = b799e712-aaf9-42c1-abc6-76087c59bbe0 response_code = 200 uri = /crucible/0/regions
482024-07-19T18:45:35.832ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_ceb75e0c-fc60-474c-85f7-e6b93e4d0425/crucible/regions/cf24c0a1-091b-41da-952f-aaddb5237975
492024-07-19T18:45:35.832ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_ceb75e0c-fc60-474c-85f7-e6b93e4d0425/crucible/regions/cf24c0a1-091b-41da-952f-aaddb5237975
502024-07-19T18:45:35.888ZINFOcrucible-agent (worker): creating region Region { id: RegionId("cf24c0a1-091b-41da-952f-aaddb5237975"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/cf24c0a1-091b-41da-952f-aaddb5237975" region = cf24c0a1-091b-41da-952f-aaddb5237975
512024-07-19T18:45:35.888ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "cf24c0a1-091b-41da-952f-aaddb5237975" "--data" "/data/regions/cf24c0a1-091b-41da-952f-aaddb5237975" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = cf24c0a1-091b-41da-952f-aaddb5237975
522024-07-19T18:45:35.974ZINFOcrucible-agent (worker): region files created ok region = cf24c0a1-091b-41da-952f-aaddb5237975
532024-07-19T18:45:35.975ZINFOcrucible-agent (datafile): region cf24c0a1-091b-41da-952f-aaddb5237975 state: Requested -> Created
542024-07-19T18:45:35.975ZINFOcrucible-agent (worker): applying SMF actions post create...
552024-07-19T18:45:35.976ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-cf24c0a1-091b-41da-952f-aaddb5237975
562024-07-19T18:45:35.980ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-cf24c0a1-091b-41da-952f-aaddb5237975
572024-07-19T18:45:35.989ZINFOcrucible-agent (worker): creating config property group
582024-07-19T18:45:35.996ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-cf24c0a1-091b-41da-952f-aaddb5237975
592024-07-19T18:45:35.996ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/cf24c0a1-091b-41da-952f-aaddb5237975
602024-07-19T18:45:35.996ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
612024-07-19T18:45:35.996ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
622024-07-19T18:45:35.996ZINFOcrucible-agent (worker): commit
632024-07-19T18:45:36.000ZINFOcrucible-agent (worker): ok!
642024-07-19T18:45:36.005ZINFOcrucible-agent (worker): SMF ok!
652024-07-19T18:45:36.068ZINFOcrucible-agent (dropshot): request completed latency_us = 74 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:61900 req_id = 627c4399-e454-44b8-9896-a95efd0f8630 response_code = 200 uri = /crucible/0/regions
662024-07-19T18:50:51.195ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::17]:32345 remote_addr = [fd00:1122:3344:101::a]:45288
672024-07-19T18:50:51.195ZINFOcrucible-agent (datafile): region caf83295-72e6-4960-a6c0-9794bdbd24ca state: Requested
682024-07-19T18:50:51.196ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
692024-07-19T18:50:51.196ZINFOcrucible-agent (dropshot): request completed latency_us = 957 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:45288 req_id = b82aa496-9271-4120-a293-f612a8ccb22f response_code = 200 uri = /crucible/0/regions
702024-07-19T18:50:51.206ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_ceb75e0c-fc60-474c-85f7-e6b93e4d0425/crucible/regions/caf83295-72e6-4960-a6c0-9794bdbd24ca
712024-07-19T18:50:51.206ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_ceb75e0c-fc60-474c-85f7-e6b93e4d0425/crucible/regions/caf83295-72e6-4960-a6c0-9794bdbd24ca
722024-07-19T18:50:51.261ZINFOcrucible-agent (worker): creating region Region { id: RegionId("caf83295-72e6-4960-a6c0-9794bdbd24ca"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/caf83295-72e6-4960-a6c0-9794bdbd24ca" region = caf83295-72e6-4960-a6c0-9794bdbd24ca
732024-07-19T18:50:51.261ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "caf83295-72e6-4960-a6c0-9794bdbd24ca" "--data" "/data/regions/caf83295-72e6-4960-a6c0-9794bdbd24ca" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = caf83295-72e6-4960-a6c0-9794bdbd24ca
742024-07-19T18:50:51.339ZINFOcrucible-agent (worker): region files created ok region = caf83295-72e6-4960-a6c0-9794bdbd24ca
752024-07-19T18:50:51.339ZINFOcrucible-agent (datafile): region caf83295-72e6-4960-a6c0-9794bdbd24ca state: Requested -> Created
762024-07-19T18:50:51.339ZINFOcrucible-agent (worker): applying SMF actions post create...
772024-07-19T18:50:51.340ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-caf83295-72e6-4960-a6c0-9794bdbd24ca
782024-07-19T18:50:51.349ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-caf83295-72e6-4960-a6c0-9794bdbd24ca
792024-07-19T18:50:51.363ZINFOcrucible-agent (worker): creating config property group
802024-07-19T18:50:51.366ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-caf83295-72e6-4960-a6c0-9794bdbd24ca
812024-07-19T18:50:51.366ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/caf83295-72e6-4960-a6c0-9794bdbd24ca
822024-07-19T18:50:51.366ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
832024-07-19T18:50:51.366ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
842024-07-19T18:50:51.366ZINFOcrucible-agent (worker): commit
852024-07-19T18:50:51.370ZINFOcrucible-agent (worker): ok!
862024-07-19T18:50:51.376ZINFOcrucible-agent (worker): SMF ok!
872024-07-19T18:50:51.379ZINFOcrucible-agent (dropshot): request completed latency_us = 76 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:45288 req_id = 3dc8bd93-08a5-4987-85c8-d33fd3f20388 response_code = 200 uri = /crucible/0/regions
882024-07-19T18:50:52.667ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::17]:32345 remote_addr = [fd00:1122:3344:101::a]:63244
892024-07-19T18:50:52.667ZINFOcrucible-agent (dropshot): request completed latency_us = 48 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:63244 req_id = 39e743df-febc-45c1-af01-3eb0162671dd response_code = 200 uri = /crucible/0/regions/cf24c0a1-091b-41da-952f-aaddb5237975
902024-07-19T18:50:52.668ZINFOcrucible-agent (datafile): path is "/data/regions/cf24c0a1-091b-41da-952f-aaddb5237975"
912024-07-19T18:50:52.677ZINFOcrucible-agent (datafile): dataset is oxp_ceb75e0c-fc60-474c-85f7-e6b93e4d0425/crucible/regions/cf24c0a1-091b-41da-952f-aaddb5237975
922024-07-19T18:50:52.686ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_ceb75e0c-fc60-474c-85f7-e6b93e4d0425/crucible/regions/cf24c0a1-091b-41da-952f-aaddb5237975@a465ea46-8db8-45a4-b078-b65a8c9cfbf3
932024-07-19T18:50:52.686ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_ceb75e0c-fc60-474c-85f7-e6b93e4d0425/crucible/regions/cf24c0a1-091b-41da-952f-aaddb5237975", "a465ea46-8db8-45a4-b078-b65a8c9cfbf3"]
942024-07-19T18:50:52.696ZINFOcrucible-agent (dropshot): request completed latency_us = 27961 local_addr = [fd00:1122:3344:101::17]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:63244 req_id = c1425553-420e-4126-8e8c-1c64864c252b response_code = 200 uri = /crucible/0/regions/cf24c0a1-091b-41da-952f-aaddb5237975/snapshots/a465ea46-8db8-45a4-b078-b65a8c9cfbf3
952024-07-19T18:50:52.696ZINFOcrucible-agent (datafile): path is "/data/regions/cf24c0a1-091b-41da-952f-aaddb5237975"
962024-07-19T18:50:52.705ZINFOcrucible-agent (datafile): dataset is oxp_ceb75e0c-fc60-474c-85f7-e6b93e4d0425/crucible/regions/cf24c0a1-091b-41da-952f-aaddb5237975
972024-07-19T18:50:52.713ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_ceb75e0c-fc60-474c-85f7-e6b93e4d0425/crucible/regions/cf24c0a1-091b-41da-952f-aaddb5237975@a465ea46-8db8-45a4-b078-b65a8c9cfbf3
982024-07-19T18:50:52.713ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_ceb75e0c-fc60-474c-85f7-e6b93e4d0425/crucible/regions/cf24c0a1-091b-41da-952f-aaddb5237975", "a465ea46-8db8-45a4-b078-b65a8c9cfbf3"]
992024-07-19T18:50:52.726ZINFOcrucible-agent (datafile): requesting running snapshot cf24c0a1-091b-41da-952f-aaddb5237975-a465ea46-8db8-45a4-b078-b65a8c9cfbf3 state: Requested
1002024-07-19T18:50:52.726ZINFOcrucible-agent (worker): applying SMF actions for region cf24c0a1-091b-41da-952f-aaddb5237975 running snapshot a465ea46-8db8-45a4-b078-b65a8c9cfbf3 (state Requested)...
1012024-07-19T18:50:52.726ZINFOcrucible-agent (dropshot): request completed latency_us = 30088 local_addr = [fd00:1122:3344:101::17]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:63244 req_id = 462aed1c-c1e6-434b-9c49-79457ec2d3d8 response_code = 200 uri = /crucible/0/regions/cf24c0a1-091b-41da-952f-aaddb5237975/snapshots/a465ea46-8db8-45a4-b078-b65a8c9cfbf3/run
1022024-07-19T18:50:52.729ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-cf24c0a1-091b-41da-952f-aaddb5237975-a465ea46-8db8-45a4-b078-b65a8c9cfbf3
1032024-07-19T18:50:52.734ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-cf24c0a1-091b-41da-952f-aaddb5237975-a465ea46-8db8-45a4-b078-b65a8c9cfbf3
1042024-07-19T18:50:52.745ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-cf24c0a1-091b-41da-952f-aaddb5237975-a465ea46-8db8-45a4-b078-b65a8c9cfbf3
1052024-07-19T18:50:52.746ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/cf24c0a1-091b-41da-952f-aaddb5237975/.zfs/snapshot/a465ea46-8db8-45a4-b078-b65a8c9cfbf3
1062024-07-19T18:50:52.746ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
1072024-07-19T18:50:52.746ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
1082024-07-19T18:50:52.746ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::17
1092024-07-19T18:50:52.746ZINFOcrucible-agent (worker): commit
1102024-07-19T18:50:52.750ZINFOcrucible-agent (worker): ok!
1112024-07-19T18:50:52.753ZINFOcrucible-agent (worker): SMF ok!
1122024-07-19T18:50:52.753ZINFOcrucible-agent (datafile): region cf24c0a1-091b-41da-952f-aaddb5237975 running snapshot a465ea46-8db8-45a4-b078-b65a8c9cfbf3 state: Requested -> Created