Max level shown:
1 [ Sep 3 23:51:13 Enabled. ]
2 [ Sep 3 23:51:13 Rereading configuration. ]
3 [ Sep 3 23:51:14 Rereading configuration. ]
4 [ Sep 3 23:51:16 Executing start method ("/opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_2cbe1d4d-d15f-4df5-bb04-76918350f53e/crucible -l [fd00:1122:3344:101::13]:32345 -P 19000 -p downstairs -s snapshot"). ]
5 note: configured to log to "/dev/stdout"
62024-09-03T23:51:16.287ZINFOcrucible-agent: dataset: "oxp_2cbe1d4d-d15f-4df5-bb04-76918350f53e/crucible"
72024-09-03T23:51:16.291ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::13]:32345
82024-09-03T23:51:16.291ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
92024-09-03T23:51:16.332ZINFOcrucible-agent (datafile): Using conf_path:"/data/crucible.json"
102024-09-03T23:51:16.445ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::13]:32345
112024-09-03T23:51:47.124ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::13]:32345 remote_addr = [fd00:1122:3344:101::a]:50489
122024-09-03T23:51:47.158ZINFOcrucible-agent (datafile): region 3ae70797-13ff-41da-8f10-f1bdf5ea2a2a state: Requested
132024-09-03T23:51:47.163ZINFOcrucible-agent (dropshot): request completed latency_us = 20721 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:50489 req_id = d3b8755e-cdec-4c80-87c3-68502d3ce1ab response_code = 200 uri = /crucible/0/regions
142024-09-03T23:51:47.168ZINFOcrucible-agent (worker): Region size:1073741824 reservation:1342177280 quota:3221225472
152024-09-03T23:51:47.177ZINFOcrucible-agent (worker): zfs set reservation of 1342177280 for oxp_2cbe1d4d-d15f-4df5-bb04-76918350f53e/crucible/regions/3ae70797-13ff-41da-8f10-f1bdf5ea2a2a
162024-09-03T23:51:47.177ZINFOcrucible-agent (worker): zfs set quota of 3221225472 for oxp_2cbe1d4d-d15f-4df5-bb04-76918350f53e/crucible/regions/3ae70797-13ff-41da-8f10-f1bdf5ea2a2a
172024-09-03T23:51:47.255ZINFOcrucible-agent (worker): creating region Region { id: RegionId("3ae70797-13ff-41da-8f10-f1bdf5ea2a2a"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 16, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/3ae70797-13ff-41da-8f10-f1bdf5ea2a2a" region = 3ae70797-13ff-41da-8f10-f1bdf5ea2a2a
182024-09-03T23:51:47.257ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "3ae70797-13ff-41da-8f10-f1bdf5ea2a2a" "--data" "/data/regions/3ae70797-13ff-41da-8f10-f1bdf5ea2a2a" "--block-size" "512" "--extent-size" "131072" "--extent-count" "16" "--encrypted" region = 3ae70797-13ff-41da-8f10-f1bdf5ea2a2a
192024-09-03T23:51:47.458ZINFOcrucible-agent (dropshot): request completed latency_us = 769 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:50489 req_id = 29c899dd-32d0-4fd7-a84a-db0d1321615d response_code = 200 uri = /crucible/0/regions
202024-09-03T23:51:47.526ZINFOcrucible-agent (worker): region files created ok region = 3ae70797-13ff-41da-8f10-f1bdf5ea2a2a
212024-09-03T23:51:47.526ZINFOcrucible-agent (datafile): region 3ae70797-13ff-41da-8f10-f1bdf5ea2a2a state: Requested -> Created
222024-09-03T23:51:47.526ZINFOcrucible-agent (worker): applying SMF actions post create...
232024-09-03T23:51:47.527ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-3ae70797-13ff-41da-8f10-f1bdf5ea2a2a
242024-09-03T23:51:47.530ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-3ae70797-13ff-41da-8f10-f1bdf5ea2a2a
252024-09-03T23:51:47.543ZINFOcrucible-agent (worker): creating config property group
262024-09-03T23:51:47.548ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-3ae70797-13ff-41da-8f10-f1bdf5ea2a2a
272024-09-03T23:51:47.557ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/3ae70797-13ff-41da-8f10-f1bdf5ea2a2a
282024-09-03T23:51:47.560ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
292024-09-03T23:51:47.560ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::13
302024-09-03T23:51:47.560ZINFOcrucible-agent (worker): commit
312024-09-03T23:51:47.567ZINFOcrucible-agent (worker): ok!
322024-09-03T23:51:47.573ZINFOcrucible-agent (worker): SMF ok!
332024-09-03T23:51:47.998ZINFOcrucible-agent (dropshot): request completed latency_us = 71 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:50489 req_id = e7e9f1d4-e155-4afa-a960-dcecadaf078f response_code = 200 uri = /crucible/0/regions
342024-09-03T23:51:48.457ZINFOcrucible-agent (dropshot): request completed latency_us = 75 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:50489 req_id = abd55783-42fa-468e-b361-7f7e6fcd8627 response_code = 200 uri = /crucible/0/regions/3ae70797-13ff-41da-8f10-f1bdf5ea2a2a
352024-09-03T23:51:48.460ZINFOcrucible-agent (datafile): path is "/data/regions/3ae70797-13ff-41da-8f10-f1bdf5ea2a2a"
362024-09-03T23:51:48.470ZINFOcrucible-agent (datafile): dataset is oxp_2cbe1d4d-d15f-4df5-bb04-76918350f53e/crucible/regions/3ae70797-13ff-41da-8f10-f1bdf5ea2a2a
372024-09-03T23:51:48.479ZINFOcrucible-agent (datafile): region 3ae70797-13ff-41da-8f10-f1bdf5ea2a2a state: Created -> Tombstoned
382024-09-03T23:51:48.480ZINFOcrucible-agent (worker): applying SMF actions before removal...
392024-09-03T23:51:48.480ZINFOcrucible-agent (dropshot): request completed latency_us = 19403 local_addr = [fd00:1122:3344:101::13]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:50489 req_id = 1a2cdcea-845e-4d75-bac6-0df5010bdc6d response_code = 204 uri = /crucible/0/regions/3ae70797-13ff-41da-8f10-f1bdf5ea2a2a
402024-09-03T23:51:48.481ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-3ae70797-13ff-41da-8f10-f1bdf5ea2a2a (instance states: (Some(Online), None))
412024-09-03T23:51:48.483ZINFOcrucible-agent (dropshot): request completed latency_us = 28 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:50489 req_id = ca05eb6e-bd1a-4625-a218-5bcf9d08b816 response_code = 200 uri = /crucible/0/regions/3ae70797-13ff-41da-8f10-f1bdf5ea2a2a
422024-09-03T23:51:48.487ZINFOcrucible-agent (worker): SMF ok!
432024-09-03T23:51:48.497ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_2cbe1d4d-d15f-4df5-bb04-76918350f53e/crucible/regions/3ae70797-13ff-41da-8f10-f1bdf5ea2a2a" region = 3ae70797-13ff-41da-8f10-f1bdf5ea2a2a
442024-09-03T23:51:48.542ZINFOcrucible-agent (dropshot): request completed latency_us = 669 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:50489 req_id = bf67ab42-4e6c-46c9-a4ce-ea1df341f027 response_code = 200 uri = /crucible/0/regions/3ae70797-13ff-41da-8f10-f1bdf5ea2a2a
452024-09-03T23:51:48.564ZINFOcrucible-agent (datafile): region 3ae70797-13ff-41da-8f10-f1bdf5ea2a2a state: Tombstoned -> Destroyed
462024-09-03T23:51:48.607ZINFOcrucible-agent (dropshot): request completed latency_us = 26 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:50489 req_id = 6f653707-f083-4070-92f2-4d651636ded8 response_code = 200 uri = /crucible/0/regions/3ae70797-13ff-41da-8f10-f1bdf5ea2a2a
472024-09-03T23:51:50.458ZINFOcrucible-agent (datafile): region 53e4d5d6-055b-4b3a-9232-c0c69948368c state: Requested
482024-09-03T23:51:50.458ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
492024-09-03T23:51:50.458ZINFOcrucible-agent (dropshot): request completed latency_us = 1004 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:50489 req_id = d8a6ecf2-6229-47d3-b216-66157ebed41a response_code = 200 uri = /crucible/0/regions
502024-09-03T23:51:50.467ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_2cbe1d4d-d15f-4df5-bb04-76918350f53e/crucible/regions/53e4d5d6-055b-4b3a-9232-c0c69948368c
512024-09-03T23:51:50.467ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_2cbe1d4d-d15f-4df5-bb04-76918350f53e/crucible/regions/53e4d5d6-055b-4b3a-9232-c0c69948368c
522024-09-03T23:51:50.562ZINFOcrucible-agent (worker): creating region Region { id: RegionId("53e4d5d6-055b-4b3a-9232-c0c69948368c"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None, source: None, read_only: false } at "/data/regions/53e4d5d6-055b-4b3a-9232-c0c69948368c" region = 53e4d5d6-055b-4b3a-9232-c0c69948368c
532024-09-03T23:51:50.565ZINFOcrucible-agent (worker): downstairs create with: env -i "/opt/oxide/crucible/bin/crucible-downstairs" "create" "--uuid" "53e4d5d6-055b-4b3a-9232-c0c69948368c" "--data" "/data/regions/53e4d5d6-055b-4b3a-9232-c0c69948368c" "--block-size" "512" "--extent-size" "131072" "--extent-count" "32" "--encrypted" region = 53e4d5d6-055b-4b3a-9232-c0c69948368c
542024-09-03T23:51:50.639ZINFOcrucible-agent (worker): region files created ok region = 53e4d5d6-055b-4b3a-9232-c0c69948368c
552024-09-03T23:51:50.639ZINFOcrucible-agent (datafile): region 53e4d5d6-055b-4b3a-9232-c0c69948368c state: Requested -> Created
562024-09-03T23:51:50.639ZINFOcrucible-agent (worker): applying SMF actions post create...
572024-09-03T23:51:50.640ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-53e4d5d6-055b-4b3a-9232-c0c69948368c
582024-09-03T23:51:50.643ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-53e4d5d6-055b-4b3a-9232-c0c69948368c
592024-09-03T23:51:50.651ZINFOcrucible-agent (worker): creating config property group
602024-09-03T23:51:50.654ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-53e4d5d6-055b-4b3a-9232-c0c69948368c
612024-09-03T23:51:50.655ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/53e4d5d6-055b-4b3a-9232-c0c69948368c
622024-09-03T23:51:50.655ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
632024-09-03T23:51:50.655ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::13
642024-09-03T23:51:50.655ZINFOcrucible-agent (worker): commit
652024-09-03T23:51:50.659ZINFOcrucible-agent (worker): ok!
662024-09-03T23:51:50.664ZINFOcrucible-agent (worker): SMF ok!
672024-09-03T23:51:50.823ZINFOcrucible-agent (dropshot): request completed latency_us = 68 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:50489 req_id = 02e0bf6d-6a84-4787-925e-3afb26d82fd7 response_code = 200 uri = /crucible/0/regions
682024-09-03T23:56:58.044ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::13]:32345 remote_addr = [fd00:1122:3344:101::a]:52782
692024-09-03T23:56:58.045ZINFOcrucible-agent (dropshot): request completed latency_us = 588 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:52782 req_id = f13ea91b-237f-43fb-848a-7b87f999f480 response_code = 200 uri = /crucible/0/regions/53e4d5d6-055b-4b3a-9232-c0c69948368c
702024-09-03T23:56:58.051ZINFOcrucible-agent (datafile): path is "/data/regions/53e4d5d6-055b-4b3a-9232-c0c69948368c"
712024-09-03T23:56:58.061ZINFOcrucible-agent (datafile): dataset is oxp_2cbe1d4d-d15f-4df5-bb04-76918350f53e/crucible/regions/53e4d5d6-055b-4b3a-9232-c0c69948368c
722024-09-03T23:56:58.069ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_2cbe1d4d-d15f-4df5-bb04-76918350f53e/crucible/regions/53e4d5d6-055b-4b3a-9232-c0c69948368c@6cb8a04e-0263-457c-acc7-970602f6584b
732024-09-03T23:56:58.069ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_2cbe1d4d-d15f-4df5-bb04-76918350f53e/crucible/regions/53e4d5d6-055b-4b3a-9232-c0c69948368c", "6cb8a04e-0263-457c-acc7-970602f6584b"]
742024-09-03T23:56:58.080ZINFOcrucible-agent (dropshot): request completed latency_us = 32248 local_addr = [fd00:1122:3344:101::13]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:52782 req_id = 039a5a81-0ad2-4a3c-a02f-9c06ed6246b2 response_code = 200 uri = /crucible/0/regions/53e4d5d6-055b-4b3a-9232-c0c69948368c/snapshots/6cb8a04e-0263-457c-acc7-970602f6584b
752024-09-03T23:56:58.084ZINFOcrucible-agent (datafile): path is "/data/regions/53e4d5d6-055b-4b3a-9232-c0c69948368c"
762024-09-03T23:56:58.092ZINFOcrucible-agent (datafile): dataset is oxp_2cbe1d4d-d15f-4df5-bb04-76918350f53e/crucible/regions/53e4d5d6-055b-4b3a-9232-c0c69948368c
772024-09-03T23:56:58.100ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_2cbe1d4d-d15f-4df5-bb04-76918350f53e/crucible/regions/53e4d5d6-055b-4b3a-9232-c0c69948368c@6cb8a04e-0263-457c-acc7-970602f6584b
782024-09-03T23:56:58.100ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_2cbe1d4d-d15f-4df5-bb04-76918350f53e/crucible/regions/53e4d5d6-055b-4b3a-9232-c0c69948368c", "6cb8a04e-0263-457c-acc7-970602f6584b"]
792024-09-03T23:56:58.109ZINFOcrucible-agent (datafile): requesting running snapshot 53e4d5d6-055b-4b3a-9232-c0c69948368c-6cb8a04e-0263-457c-acc7-970602f6584b state: Requested
802024-09-03T23:56:58.109ZINFOcrucible-agent (worker): applying SMF actions for region 53e4d5d6-055b-4b3a-9232-c0c69948368c running snapshot 6cb8a04e-0263-457c-acc7-970602f6584b (state Requested)...
812024-09-03T23:56:58.109ZINFOcrucible-agent (dropshot): request completed latency_us = 25599 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:52782 req_id = 2d004664-bbee-4dbd-a9b9-c1b5e16a202b response_code = 200 uri = /crucible/0/regions/53e4d5d6-055b-4b3a-9232-c0c69948368c/snapshots/6cb8a04e-0263-457c-acc7-970602f6584b/run
822024-09-03T23:56:58.118ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-53e4d5d6-055b-4b3a-9232-c0c69948368c-6cb8a04e-0263-457c-acc7-970602f6584b
832024-09-03T23:56:58.124ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-53e4d5d6-055b-4b3a-9232-c0c69948368c-6cb8a04e-0263-457c-acc7-970602f6584b
842024-09-03T23:56:58.136ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-53e4d5d6-055b-4b3a-9232-c0c69948368c-6cb8a04e-0263-457c-acc7-970602f6584b
852024-09-03T23:56:58.136ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/53e4d5d6-055b-4b3a-9232-c0c69948368c/.zfs/snapshot/6cb8a04e-0263-457c-acc7-970602f6584b
862024-09-03T23:56:58.136ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
872024-09-03T23:56:58.136ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
882024-09-03T23:56:58.136ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::13
892024-09-03T23:56:58.137ZINFOcrucible-agent (worker): commit
902024-09-03T23:56:58.140ZINFOcrucible-agent (worker): ok!
912024-09-03T23:56:58.145ZINFOcrucible-agent (worker): SMF ok!
922024-09-03T23:56:58.145ZINFOcrucible-agent (datafile): region 53e4d5d6-055b-4b3a-9232-c0c69948368c running snapshot 6cb8a04e-0263-457c-acc7-970602f6584b state: Requested -> Created
932024-09-03T23:56:58.397ZINFOcrucible-agent (datafile): path is "/data/regions/53e4d5d6-055b-4b3a-9232-c0c69948368c"
942024-09-03T23:56:58.405ZINFOcrucible-agent (datafile): dataset is oxp_2cbe1d4d-d15f-4df5-bb04-76918350f53e/crucible/regions/53e4d5d6-055b-4b3a-9232-c0c69948368c
952024-09-03T23:56:58.414ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_2cbe1d4d-d15f-4df5-bb04-76918350f53e/crucible/regions/53e4d5d6-055b-4b3a-9232-c0c69948368c@6cb8a04e-0263-457c-acc7-970602f6584b
962024-09-03T23:56:58.414ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_2cbe1d4d-d15f-4df5-bb04-76918350f53e/crucible/regions/53e4d5d6-055b-4b3a-9232-c0c69948368c", "6cb8a04e-0263-457c-acc7-970602f6584b"]
972024-09-03T23:56:58.422ZINFOcrucible-agent (dropshot): request completed latency_us = 25434 local_addr = [fd00:1122:3344:101::13]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:52782 req_id = 6bbbbfe4-3be9-4d81-9c6c-c256d40563a6 response_code = 200 uri = /crucible/0/regions/53e4d5d6-055b-4b3a-9232-c0c69948368c/snapshots/6cb8a04e-0263-457c-acc7-970602f6584b/run