Max level shown:
1 [ Jun 20 21:03:50 Enabled. ]
2 [ Jun 20 21:03:50 Rereading configuration. ]
3 [ Jun 20 21:03:51 Rereading configuration. ]
4 [ Jun 20 21:03:54 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_cc5f6a04-2111-4941-923c-8867d737d5fc/crucible -l [fd00:1122:3344:101::12]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62024-06-20T21:03:54.182ZINFOcrucible-agent: dataset: "oxp_cc5f6a04-2111-4941-923c-8867d737d5fc/crucible"
72024-06-20T21:03:54.183ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::12]:32345
82024-06-20T21:03:54.183ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92024-06-20T21:03:54.203ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102024-06-20T21:03:54.294ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::12]:32345
112024-06-20T21:04:16.578ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::c]:36685
122024-06-20T21:04:16.594ZINFOcrucible-agent (datafile): region 57ade24d-dc07-464b-bfd2-408fbf16bd5f state: Requested
132024-06-20T21:04:16.597ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
142024-06-20T21:04:16.601ZINFOcrucible-agent (dropshot): request completed latency_us = 13836 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:36685 req_id = c5e59be2-c7fc-4e7f-8a2f-20347748a57e response_code = 200 uri = /crucible/0/regions
152024-06-20T21:04:16.608ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_cc5f6a04-2111-4941-923c-8867d737d5fc/crucible/regions/57ade24d-dc07-464b-bfd2-408fbf16bd5f
162024-06-20T21:04:16.608ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_cc5f6a04-2111-4941-923c-8867d737d5fc/crucible/regions/57ade24d-dc07-464b-bfd2-408fbf16bd5f
172024-06-20T21:04:16.691ZINFOcrucible-agent (worker): creating region Region { id: RegionId("57ade24d-dc07-464b-bfd2-408fbf16bd5f"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/57ade24d-dc07-464b-bfd2-408fbf16bd5f" region = 57ade24d-dc07-464b-bfd2-408fbf16bd5f
182024-06-20T21:04:16.693ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "57ade24d-dc07-464b-bfd2-408fbf16bd5f" "--data" "/data/regions/57ade24d-dc07-464b-bfd2-408fbf16bd5f" "--block-size" "512" "--extent-size" "131072" "--extent-count" "16" "--encrypted" region = 57ade24d-dc07-464b-bfd2-408fbf16bd5f
192024-06-20T21:04:16.767ZINFOcrucible-agent (dropshot): request completed latency_us = 84 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:36685 req_id = d791f2f4-4fbc-4a07-9e76-c1a53d09b1ed response_code = 200 uri = /crucible/0/regions
202024-06-20T21:04:16.822ZINFOcrucible-agent (worker): region files created ok region = 57ade24d-dc07-464b-bfd2-408fbf16bd5f
212024-06-20T21:04:16.822ZINFOcrucible-agent (datafile): region 57ade24d-dc07-464b-bfd2-408fbf16bd5f state: Requested -> Created
222024-06-20T21:04:16.822ZINFOcrucible-agent (worker): applying SMF actions post create...
232024-06-20T21:04:16.822ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-57ade24d-dc07-464b-bfd2-408fbf16bd5f
242024-06-20T21:04:16.827ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-57ade24d-dc07-464b-bfd2-408fbf16bd5f
252024-06-20T21:04:16.841ZINFOcrucible-agent (worker): creating config property group
262024-06-20T21:04:16.854ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-57ade24d-dc07-464b-bfd2-408fbf16bd5f
272024-06-20T21:04:16.855ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/57ade24d-dc07-464b-bfd2-408fbf16bd5f
282024-06-20T21:04:16.855ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
292024-06-20T21:04:16.855ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
302024-06-20T21:04:16.855ZINFOcrucible-agent (worker): commit
312024-06-20T21:04:16.859ZINFOcrucible-agent (worker): ok!
322024-06-20T21:04:16.865ZINFOcrucible-agent (worker): SMF ok!
332024-06-20T21:04:17.313ZINFOcrucible-agent (dropshot): request completed latency_us = 79 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::c]:36685 req_id = fca0b0ed-9b30-42cb-9ee2-8c437b8435e8 response_code = 200 uri = /crucible/0/regions
342024-06-20T21:04:17.733ZINFOcrucible-agent (dropshot): request completed latency_us = 517 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:36685 req_id = d2c4125b-f987-4762-a3ed-4c38e9dd1f86 response_code = 200 uri = /crucible/0/regions/57ade24d-dc07-464b-bfd2-408fbf16bd5f
352024-06-20T21:04:17.735ZINFOcrucible-agent (datafile): path is "/data/regions/57ade24d-dc07-464b-bfd2-408fbf16bd5f"
362024-06-20T21:04:17.744ZINFOcrucible-agent (datafile): dataset is oxp_cc5f6a04-2111-4941-923c-8867d737d5fc/crucible/regions/57ade24d-dc07-464b-bfd2-408fbf16bd5f
372024-06-20T21:04:17.753ZINFOcrucible-agent (datafile): region 57ade24d-dc07-464b-bfd2-408fbf16bd5f state: Created -> Tombstoned
382024-06-20T21:04:17.753ZINFOcrucible-agent (worker): applying SMF actions before removal...
392024-06-20T21:04:17.754ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-57ade24d-dc07-464b-bfd2-408fbf16bd5f (instance states: (Some(Online), None))
402024-06-20T21:04:17.755ZINFOcrucible-agent (dropshot): request completed latency_us = 20057 local_addr = [fd00:1122:3344:101::12]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::c]:36685 req_id = e95313ef-3525-4c3c-936e-eb3c678ea98e response_code = 204 uri = /crucible/0/regions/57ade24d-dc07-464b-bfd2-408fbf16bd5f
412024-06-20T21:04:17.758ZINFOcrucible-agent (dropshot): request completed latency_us = 21 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:36685 req_id = 08a3cdaa-19b5-42ed-bb22-8fbe122616d5 response_code = 200 uri = /crucible/0/regions/57ade24d-dc07-464b-bfd2-408fbf16bd5f
422024-06-20T21:04:17.759ZINFOcrucible-agent (worker): SMF ok!
432024-06-20T21:04:17.770ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_cc5f6a04-2111-4941-923c-8867d737d5fc/crucible/regions/57ade24d-dc07-464b-bfd2-408fbf16bd5f" region = 57ade24d-dc07-464b-bfd2-408fbf16bd5f
442024-06-20T21:04:17.837ZINFOcrucible-agent (datafile): region 57ade24d-dc07-464b-bfd2-408fbf16bd5f state: Tombstoned -> Destroyed
452024-06-20T21:04:17.883ZINFOcrucible-agent (dropshot): request completed latency_us = 51 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::c]:36685 req_id = 85cc8182-d089-41db-af1d-c04d1edfba05 response_code = 200 uri = /crucible/0/regions/57ade24d-dc07-464b-bfd2-408fbf16bd5f
462024-06-20T21:05:06.162ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:52941
472024-06-20T21:05:06.162ZINFOcrucible-agent (datafile): region 5736af07-0dde-43d6-90dd-091400281fa4 state: Requested
482024-06-20T21:05:06.162ZINFOcrucible-agent (dropshot): request completed latency_us = 201 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:52941 req_id = 9091836a-98b1-414e-94b5-f173ca4d0840 response_code = 200 uri = /crucible/0/regions
492024-06-20T21:05:06.162ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
502024-06-20T21:05:06.171ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_cc5f6a04-2111-4941-923c-8867d737d5fc/crucible/regions/5736af07-0dde-43d6-90dd-091400281fa4
512024-06-20T21:05:06.172ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_cc5f6a04-2111-4941-923c-8867d737d5fc/crucible/regions/5736af07-0dde-43d6-90dd-091400281fa4
522024-06-20T21:05:06.228ZINFOcrucible-agent (worker): creating region Region { id: RegionId("5736af07-0dde-43d6-90dd-091400281fa4"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/5736af07-0dde-43d6-90dd-091400281fa4" region = 5736af07-0dde-43d6-90dd-091400281fa4
532024-06-20T21:05:06.230ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "5736af07-0dde-43d6-90dd-091400281fa4" "--data" "/data/regions/5736af07-0dde-43d6-90dd-091400281fa4" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 5736af07-0dde-43d6-90dd-091400281fa4
542024-06-20T21:05:06.295ZINFOcrucible-agent (worker): region files created ok region = 5736af07-0dde-43d6-90dd-091400281fa4
552024-06-20T21:05:06.295ZINFOcrucible-agent (datafile): region 5736af07-0dde-43d6-90dd-091400281fa4 state: Requested -> Created
562024-06-20T21:05:06.295ZINFOcrucible-agent (worker): applying SMF actions post create...
572024-06-20T21:05:06.297ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-5736af07-0dde-43d6-90dd-091400281fa4
582024-06-20T21:05:06.299ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-5736af07-0dde-43d6-90dd-091400281fa4
592024-06-20T21:05:06.306ZINFOcrucible-agent (worker): creating config property group
602024-06-20T21:05:06.308ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-5736af07-0dde-43d6-90dd-091400281fa4
612024-06-20T21:05:06.309ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/5736af07-0dde-43d6-90dd-091400281fa4
622024-06-20T21:05:06.309ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
632024-06-20T21:05:06.309ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
642024-06-20T21:05:06.309ZINFOcrucible-agent (worker): commit
652024-06-20T21:05:06.312ZINFOcrucible-agent (worker): ok!
662024-06-20T21:05:06.316ZINFOcrucible-agent (worker): SMF ok!
672024-06-20T21:05:06.340ZINFOcrucible-agent (dropshot): request completed latency_us = 58 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:52941 req_id = aa7c6f4b-dce7-4f03-82a9-b3a463748929 response_code = 200 uri = /crucible/0/regions
682024-06-20T21:10:17.841ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:55858
692024-06-20T21:10:17.842ZINFOcrucible-agent (datafile): region ac21495c-209d-44c2-acaa-73a3f1e91b93 state: Requested
702024-06-20T21:10:17.842ZINFOcrucible-agent (dropshot): request completed latency_us = 1168 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:55858 req_id = 5a74f2fd-c3e5-4710-b9e4-6208df21a623 response_code = 200 uri = /crucible/0/regions
712024-06-20T21:10:17.842ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
722024-06-20T21:10:17.854ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_cc5f6a04-2111-4941-923c-8867d737d5fc/crucible/regions/ac21495c-209d-44c2-acaa-73a3f1e91b93
732024-06-20T21:10:17.854ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_cc5f6a04-2111-4941-923c-8867d737d5fc/crucible/regions/ac21495c-209d-44c2-acaa-73a3f1e91b93
742024-06-20T21:10:17.925ZINFOcrucible-agent (worker): creating region Region { id: RegionId("ac21495c-209d-44c2-acaa-73a3f1e91b93"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19001, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/ac21495c-209d-44c2-acaa-73a3f1e91b93" region = ac21495c-209d-44c2-acaa-73a3f1e91b93
752024-06-20T21:10:17.927ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "ac21495c-209d-44c2-acaa-73a3f1e91b93" "--data" "/data/regions/ac21495c-209d-44c2-acaa-73a3f1e91b93" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = ac21495c-209d-44c2-acaa-73a3f1e91b93
762024-06-20T21:10:18.024ZINFOcrucible-agent (worker): region files created ok region = ac21495c-209d-44c2-acaa-73a3f1e91b93
772024-06-20T21:10:18.024ZINFOcrucible-agent (datafile): region ac21495c-209d-44c2-acaa-73a3f1e91b93 state: Requested -> Created
782024-06-20T21:10:18.024ZINFOcrucible-agent (worker): applying SMF actions post create...
792024-06-20T21:10:18.030ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-ac21495c-209d-44c2-acaa-73a3f1e91b93
802024-06-20T21:10:18.049ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-ac21495c-209d-44c2-acaa-73a3f1e91b93
812024-06-20T21:10:18.065ZINFOcrucible-agent (worker): creating config property group
822024-06-20T21:10:18.070ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-ac21495c-209d-44c2-acaa-73a3f1e91b93
832024-06-20T21:10:18.070ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/ac21495c-209d-44c2-acaa-73a3f1e91b93
842024-06-20T21:10:18.070ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
852024-06-20T21:10:18.070ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
862024-06-20T21:10:18.070ZINFOcrucible-agent (worker): commit
872024-06-20T21:10:18.076ZINFOcrucible-agent (worker): ok!
882024-06-20T21:10:18.086ZINFOcrucible-agent (worker): SMF ok!
892024-06-20T21:10:18.095ZINFOcrucible-agent (dropshot): request completed latency_us = 77 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:55858 req_id = b594d67a-1d33-4f72-bfe9-e62e474ccf94 response_code = 200 uri = /crucible/0/regions
902024-06-20T21:10:19.146ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:32952
912024-06-20T21:10:19.146ZINFOcrucible-agent (dropshot): request completed latency_us = 62 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:32952 req_id = 93e77f5e-9331-4fe4-90a5-09cc5e0020b2 response_code = 200 uri = /crucible/0/regions/5736af07-0dde-43d6-90dd-091400281fa4
922024-06-20T21:10:19.149ZINFOcrucible-agent (datafile): path is "/data/regions/5736af07-0dde-43d6-90dd-091400281fa4"
932024-06-20T21:10:19.157ZINFOcrucible-agent (datafile): dataset is oxp_cc5f6a04-2111-4941-923c-8867d737d5fc/crucible/regions/5736af07-0dde-43d6-90dd-091400281fa4
942024-06-20T21:10:19.166ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_cc5f6a04-2111-4941-923c-8867d737d5fc/crucible/regions/5736af07-0dde-43d6-90dd-091400281fa4@1754a985-200c-4441-a679-cc9cc7979745
952024-06-20T21:10:19.166ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_cc5f6a04-2111-4941-923c-8867d737d5fc/crucible/regions/5736af07-0dde-43d6-90dd-091400281fa4", "1754a985-200c-4441-a679-cc9cc7979745"]
962024-06-20T21:10:19.180ZINFOcrucible-agent (dropshot): request completed latency_us = 33655 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:32952 req_id = 8db09d4a-1e2b-4291-8ba3-a92c45400ced response_code = 200 uri = /crucible/0/regions/5736af07-0dde-43d6-90dd-091400281fa4/snapshots/1754a985-200c-4441-a679-cc9cc7979745
972024-06-20T21:10:19.180ZINFOcrucible-agent (datafile): path is "/data/regions/5736af07-0dde-43d6-90dd-091400281fa4"
982024-06-20T21:10:19.189ZINFOcrucible-agent (datafile): dataset is oxp_cc5f6a04-2111-4941-923c-8867d737d5fc/crucible/regions/5736af07-0dde-43d6-90dd-091400281fa4
992024-06-20T21:10:19.198ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_cc5f6a04-2111-4941-923c-8867d737d5fc/crucible/regions/5736af07-0dde-43d6-90dd-091400281fa4@1754a985-200c-4441-a679-cc9cc7979745
1002024-06-20T21:10:19.198ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_cc5f6a04-2111-4941-923c-8867d737d5fc/crucible/regions/5736af07-0dde-43d6-90dd-091400281fa4", "1754a985-200c-4441-a679-cc9cc7979745"]
1012024-06-20T21:10:19.207ZINFOcrucible-agent (datafile): requesting running snapshot 5736af07-0dde-43d6-90dd-091400281fa4-1754a985-200c-4441-a679-cc9cc7979745 state: Requested
1022024-06-20T21:10:19.208ZINFOcrucible-agent (worker): applying SMF actions for region 5736af07-0dde-43d6-90dd-091400281fa4 running snapshot 1754a985-200c-4441-a679-cc9cc7979745 (state Requested)...
1032024-06-20T21:10:19.208ZINFOcrucible-agent (dropshot): request completed latency_us = 27871 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:32952 req_id = 34fb3299-42ff-44d8-80ec-7acc6bbc37db response_code = 200 uri = /crucible/0/regions/5736af07-0dde-43d6-90dd-091400281fa4/snapshots/1754a985-200c-4441-a679-cc9cc7979745/run
1042024-06-20T21:10:19.211ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-5736af07-0dde-43d6-90dd-091400281fa4-1754a985-200c-4441-a679-cc9cc7979745
1052024-06-20T21:10:19.217ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-5736af07-0dde-43d6-90dd-091400281fa4-1754a985-200c-4441-a679-cc9cc7979745
1062024-06-20T21:10:19.237ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-5736af07-0dde-43d6-90dd-091400281fa4-1754a985-200c-4441-a679-cc9cc7979745
1072024-06-20T21:10:19.237ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/5736af07-0dde-43d6-90dd-091400281fa4/.zfs/snapshot/1754a985-200c-4441-a679-cc9cc7979745
1082024-06-20T21:10:19.238ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
1092024-06-20T21:10:19.238ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
1102024-06-20T21:10:19.238ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
1112024-06-20T21:10:19.238ZINFOcrucible-agent (worker): commit
1122024-06-20T21:10:19.242ZINFOcrucible-agent (worker): ok!
1132024-06-20T21:10:19.249ZINFOcrucible-agent (worker): SMF ok!
1142024-06-20T21:10:19.249ZINFOcrucible-agent (datafile): region 5736af07-0dde-43d6-90dd-091400281fa4 running snapshot 1754a985-200c-4441-a679-cc9cc7979745 state: Requested -> Created