Max level shown:
1 [ Feb 2 02:33:15 Enabled. ]
2 [ Feb 2 02:33:15 Rereading configuration. ]
3 [ Feb 2 02:33:18 Rereading configuration. ]
4 [ Feb 2 02:33:23 Executing start method ("/opt/oxide/lib/svc/manifest/crucible/agent.sh"). ]
5 + . /lib/svc/share/smf_include.sh
6 ++ SMF_EXIT_OK=0
7 ++ SMF_EXIT_NODAEMON=94
8 ++ SMF_EXIT_ERR_FATAL=95
9 ++ SMF_EXIT_ERR_CONFIG=96
10 ++ SMF_EXIT_MON_DEGRADE=97
11 ++ SMF_EXIT_MON_OFFLINE=98
12 ++ SMF_EXIT_ERR_NOSMF=99
13 ++ SMF_EXIT_ERR_PERM=100
14 ++ svcprop -c -p config/datalink svc:/oxide/crucible/agent:default
15 + DATALINK=oxControlService15
16 ++ svcprop -c -p config/gateway svc:/oxide/crucible/agent:default
17 + GATEWAY=fd00:1122:3344:101::1
18 ++ svcprop -c -p config/dataset svc:/oxide/crucible/agent:default
19 + DATASET=oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crucible
20 ++ svcprop -c -p config/listen_addr svc:/oxide/crucible/agent:default
21 + LISTEN_ADDR=fd00:1122:3344:101::16
22 ++ svcprop -c -p config/listen_port svc:/oxide/crucible/agent:default
23 + LISTEN_PORT=32345
24 ++ svcprop -c -p config/portbase svc:/oxide/crucible/agent:default
25 + PORTBASE=19000
26 ++ svcprop -c -p config/downstairs_prefix svc:/oxide/crucible/agent:default
27 + DOWNSTAIRS_PREFIX=downstairs
28 ++ svcprop -c -p config/snapshot_prefix svc:/oxide/crucible/agent:default
29 + SNAPSHOT_PREFIX=snapshot
30 + [[ oxControlService15 == unknown ]]
31 + [[ fd00:1122:3344:101::1 == unknown ]]
32 + ipadm delete-if oxControlService15
33 ipadm: Could not delete oxControlService15: Interface does not exist
34 + true
35 + ipadm create-if -t oxControlService15
36 + ipadm set-ifprop -t -p mtu=9000 -m ipv4 oxControlService15
37 + ipadm set-ifprop -t -p mtu=9000 -m ipv6 oxControlService15
38 + ipadm show-addr oxControlService15/ll
39 ipadm: Address object not found
40 + ipadm create-addr -t -T addrconf oxControlService15/ll
41 + ipadm show-addr oxControlService15/omicron6
42 ipadm: Address object not found
43 + ipadm create-addr -t -T static -a fd00:1122:3344:101::16 oxControlService15/omicron6
44 + route get -inet6 default -inet6 fd00:1122:3344:101::1
45 default: not in table
46 + route add -inet6 default -inet6 fd00:1122:3344:101::1
47 add net default: gateway fd00:1122:3344:101::1
48 + args=('-D' '/opt/oxide/crucible/bin/crucible-downstairs' '--dataset' "$DATASET" '-l' "[$LISTEN_ADDR]:$LISTEN_PORT" '-P' "$PORTBASE" '-p' "$DOWNSTAIRS_PREFIX" '-s' "$SNAPSHOT_PREFIX")
49 + exec /opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crucible -l '[fd00:1122:3344:101::16]:32345' -P 19000 -p downstairs -s snapshot
50 note: configured to log to "/dev/stdout"
512024-02-02T02:33:24.354ZINFOcrucible-agent: dataset: "oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crucible"
522024-02-02T02:33:24.380ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::16]:32345
532024-02-02T02:33:24.380ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
542024-02-02T02:33:24.747ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::16]:32345
552024-02-02T02:34:27.726ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::16]:32345 remote_addr = [fd00:1122:3344:101::a]:58886
562024-02-02T02:34:27.790ZINFOcrucible-agent (datafile): region e4cfe51f-c926-4d1a-a47a-20a931173f1a state: Requested
572024-02-02T02:34:27.790ZINFOcrucible-agent (dropshot): request completed latency_us = 1266 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:58886 req_id = 052c50af-5a0e-47d7-898c-15666581ab6a response_code = 200 uri = /crucible/0/regions
582024-02-02T02:34:27.794ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
592024-02-02T02:34:27.804ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crucible/regions/e4cfe51f-c926-4d1a-a47a-20a931173f1a
602024-02-02T02:34:27.804ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crucible/regions/e4cfe51f-c926-4d1a-a47a-20a931173f1a
612024-02-02T02:34:27.864ZINFOcrucible-agent (worker): creating region Region { id: RegionId("e4cfe51f-c926-4d1a-a47a-20a931173f1a"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None } at "/data/regions/e4cfe51f-c926-4d1a-a47a-20a931173f1a" region = e4cfe51f-c926-4d1a-a47a-20a931173f1a
622024-02-02T02:34:27.981ZINFOcrucible-agent (dropshot): request completed latency_us = 73 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:58886 req_id = 30663b01-2452-4138-ad90-74f74de69513 response_code = 200 uri = /crucible/0/regions
632024-02-02T02:34:28.208ZINFOcrucible-agent (worker): region files created ok region = e4cfe51f-c926-4d1a-a47a-20a931173f1a
642024-02-02T02:34:28.208ZINFOcrucible-agent (datafile): region e4cfe51f-c926-4d1a-a47a-20a931173f1a state: Requested -> Created
652024-02-02T02:34:28.208ZINFOcrucible-agent (worker): applying SMF actions post create...
662024-02-02T02:34:28.208ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-e4cfe51f-c926-4d1a-a47a-20a931173f1a
672024-02-02T02:34:28.216ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-e4cfe51f-c926-4d1a-a47a-20a931173f1a
682024-02-02T02:34:28.229ZINFOcrucible-agent (worker): creating config property group
692024-02-02T02:34:28.234ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-e4cfe51f-c926-4d1a-a47a-20a931173f1a
702024-02-02T02:34:28.235ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/e4cfe51f-c926-4d1a-a47a-20a931173f1a
712024-02-02T02:34:28.235ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
722024-02-02T02:34:28.235ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::16
732024-02-02T02:34:28.235ZINFOcrucible-agent (worker): commit
742024-02-02T02:34:28.242ZINFOcrucible-agent (worker): ok!
752024-02-02T02:34:28.247ZINFOcrucible-agent (worker): SMF ok!
762024-02-02T02:34:28.349ZINFOcrucible-agent (dropshot): request completed latency_us = 62 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:58886 req_id = c8915b91-4906-49e9-b1f6-68009fdc8833 response_code = 200 uri = /crucible/0/regions
772024-02-02T02:40:06.272ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::16]:32345 remote_addr = [fd00:1122:3344:101::a]:45371
782024-02-02T02:40:06.272ZINFOcrucible-agent (dropshot): request completed latency_us = 75 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:45371 req_id = f1d0bbb9-d51b-4b79-9516-a9ed2a1404a7 response_code = 200 uri = /crucible/0/regions/e4cfe51f-c926-4d1a-a47a-20a931173f1a
792024-02-02T02:40:06.280ZINFOcrucible-agent (datafile): path is "/data/regions/e4cfe51f-c926-4d1a-a47a-20a931173f1a"
802024-02-02T02:40:06.289ZINFOcrucible-agent (datafile): dataset is oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crucible/regions/e4cfe51f-c926-4d1a-a47a-20a931173f1a
812024-02-02T02:40:06.298ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crucible/regions/e4cfe51f-c926-4d1a-a47a-20a931173f1a@043d67e6-7244-453f-b2d4-b5c90f72c8d3
822024-02-02T02:40:06.299ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crucible/regions/e4cfe51f-c926-4d1a-a47a-20a931173f1a", "043d67e6-7244-453f-b2d4-b5c90f72c8d3"]
832024-02-02T02:40:06.308ZINFOcrucible-agent (dropshot): request completed latency_us = 35499 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:45371 req_id = ec0e248b-16b9-4fc6-9335-dd6190fbcd10 response_code = 200 uri = /crucible/0/regions/e4cfe51f-c926-4d1a-a47a-20a931173f1a/snapshots/043d67e6-7244-453f-b2d4-b5c90f72c8d3
842024-02-02T02:40:06.308ZINFOcrucible-agent (datafile): path is "/data/regions/e4cfe51f-c926-4d1a-a47a-20a931173f1a"
852024-02-02T02:40:06.317ZINFOcrucible-agent (datafile): dataset is oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crucible/regions/e4cfe51f-c926-4d1a-a47a-20a931173f1a
862024-02-02T02:40:06.326ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crucible/regions/e4cfe51f-c926-4d1a-a47a-20a931173f1a@043d67e6-7244-453f-b2d4-b5c90f72c8d3
872024-02-02T02:40:06.326ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crucible/regions/e4cfe51f-c926-4d1a-a47a-20a931173f1a", "043d67e6-7244-453f-b2d4-b5c90f72c8d3"]
882024-02-02T02:40:06.336ZINFOcrucible-agent (datafile): requesting running snapshot e4cfe51f-c926-4d1a-a47a-20a931173f1a-043d67e6-7244-453f-b2d4-b5c90f72c8d3 state: Requested
892024-02-02T02:40:06.337ZINFOcrucible-agent (worker): applying SMF actions for region e4cfe51f-c926-4d1a-a47a-20a931173f1a running snapshot 043d67e6-7244-453f-b2d4-b5c90f72c8d3 (state Requested)...
902024-02-02T02:40:06.337ZINFOcrucible-agent (dropshot): request completed latency_us = 29037 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:45371 req_id = e3eeec10-ed96-46ca-86b8-bcb0be8a1bbd response_code = 200 uri = /crucible/0/regions/e4cfe51f-c926-4d1a-a47a-20a931173f1a/snapshots/043d67e6-7244-453f-b2d4-b5c90f72c8d3/run
912024-02-02T02:40:06.343ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-e4cfe51f-c926-4d1a-a47a-20a931173f1a-043d67e6-7244-453f-b2d4-b5c90f72c8d3
922024-02-02T02:40:06.349ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-e4cfe51f-c926-4d1a-a47a-20a931173f1a-043d67e6-7244-453f-b2d4-b5c90f72c8d3
932024-02-02T02:40:06.368ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-e4cfe51f-c926-4d1a-a47a-20a931173f1a-043d67e6-7244-453f-b2d4-b5c90f72c8d3
942024-02-02T02:40:06.368ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/e4cfe51f-c926-4d1a-a47a-20a931173f1a/.zfs/snapshot/043d67e6-7244-453f-b2d4-b5c90f72c8d3
952024-02-02T02:40:06.368ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
962024-02-02T02:40:06.368ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
972024-02-02T02:40:06.368ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::16
982024-02-02T02:40:06.368ZINFOcrucible-agent (worker): commit
992024-02-02T02:40:06.375ZINFOcrucible-agent (worker): ok!
1002024-02-02T02:40:06.382ZINFOcrucible-agent (worker): SMF ok!
1012024-02-02T02:40:06.382ZINFOcrucible-agent (datafile): region e4cfe51f-c926-4d1a-a47a-20a931173f1a running snapshot 043d67e6-7244-453f-b2d4-b5c90f72c8d3 state: Requested -> Created
1022024-02-02T02:40:08.895ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::16]:32345 remote_addr = [fd00:1122:3344:101::a]:43854
1032024-02-02T02:40:08.956ZINFOcrucible-agent (datafile): region cc6a4f86-e5d2-445a-8886-0d0d7d6b65ea state: Requested
1042024-02-02T02:40:08.957ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
1052024-02-02T02:40:08.957ZINFOcrucible-agent (dropshot): request completed latency_us = 227 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:43854 req_id = a0c9c3c3-ec1b-4126-b8fb-6afef5c3e7ae response_code = 200 uri = /crucible/0/regions
1062024-02-02T02:40:08.966ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crucible/regions/cc6a4f86-e5d2-445a-8886-0d0d7d6b65ea
1072024-02-02T02:40:08.966ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crucible/regions/cc6a4f86-e5d2-445a-8886-0d0d7d6b65ea
1082024-02-02T02:40:09.045ZINFOcrucible-agent (worker): creating region Region { id: RegionId("cc6a4f86-e5d2-445a-8886-0d0d7d6b65ea"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19002, cert_pem: None, key_pem: None, root_pem: None } at "/data/regions/cc6a4f86-e5d2-445a-8886-0d0d7d6b65ea" region = cc6a4f86-e5d2-445a-8886-0d0d7d6b65ea
1092024-02-02T02:40:09.143ZINFOcrucible-agent (worker): region files created ok region = cc6a4f86-e5d2-445a-8886-0d0d7d6b65ea
1102024-02-02T02:40:09.143ZINFOcrucible-agent (datafile): region cc6a4f86-e5d2-445a-8886-0d0d7d6b65ea state: Requested -> Created
1112024-02-02T02:40:09.143ZINFOcrucible-agent (worker): applying SMF actions post create...
1122024-02-02T02:40:09.143ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-cc6a4f86-e5d2-445a-8886-0d0d7d6b65ea
1132024-02-02T02:40:09.150ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-cc6a4f86-e5d2-445a-8886-0d0d7d6b65ea
1142024-02-02T02:40:09.162ZINFOcrucible-agent (worker): creating config property group
1152024-02-02T02:40:09.164ZINFOcrucible-agent (dropshot): request completed latency_us = 65 local_addr = [fd00:1122:3344:101::16]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:43854 req_id = af9872c2-4faf-46d4-bd6d-a438c8ee6895 response_code = 200 uri = /crucible/0/regions
1162024-02-02T02:40:09.167ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-cc6a4f86-e5d2-445a-8886-0d0d7d6b65ea
1172024-02-02T02:40:09.167ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/cc6a4f86-e5d2-445a-8886-0d0d7d6b65ea
1182024-02-02T02:40:09.167ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
1192024-02-02T02:40:09.167ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::16
1202024-02-02T02:40:09.167ZINFOcrucible-agent (worker): commit
1212024-02-02T02:40:09.174ZINFOcrucible-agent (worker): ok!
1222024-02-02T02:40:09.190ZINFOcrucible-agent (worker): SMF ok!
1232024-02-02T02:40:40.231ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::16]:32345 remote_addr = [fd00:1122:3344:101::a]:54379
1242024-02-02T02:40:40.231ZINFOcrucible-agent (dropshot): request completed latency_us = 157 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:54379 req_id = cb813a6f-c262-44a5-bb1f-ab9ce7c799ba response_code = 200 uri = /crucible/0/regions/cc6a4f86-e5d2-445a-8886-0d0d7d6b65ea
1252024-02-02T02:40:40.231ZINFOcrucible-agent (datafile): path is "/data/regions/cc6a4f86-e5d2-445a-8886-0d0d7d6b65ea"
1262024-02-02T02:40:40.244ZINFOcrucible-agent (datafile): dataset is oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crucible/regions/cc6a4f86-e5d2-445a-8886-0d0d7d6b65ea
1272024-02-02T02:40:40.253ZINFOcrucible-agent (datafile): region cc6a4f86-e5d2-445a-8886-0d0d7d6b65ea state: Created -> Tombstoned
1282024-02-02T02:40:40.253ZINFOcrucible-agent (worker): applying SMF actions before removal...
1292024-02-02T02:40:40.253ZINFOcrucible-agent (dropshot): request completed latency_us = 21888 local_addr = [fd00:1122:3344:101::16]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:54379 req_id = 0065db88-eff2-444d-9115-6721175a16e2 response_code = 204 uri = /crucible/0/regions/cc6a4f86-e5d2-445a-8886-0d0d7d6b65ea
1302024-02-02T02:40:40.253ZINFOcrucible-agent (dropshot): request completed latency_us = 30 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:54379 req_id = 4d9a0d92-5c79-4bb2-82a4-b74ed99f0d02 response_code = 200 uri = /crucible/0/regions/cc6a4f86-e5d2-445a-8886-0d0d7d6b65ea
1312024-02-02T02:40:40.254ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-cc6a4f86-e5d2-445a-8886-0d0d7d6b65ea (instance states: (Some(Online), None))
1322024-02-02T02:40:40.261ZINFOcrucible-agent (worker): SMF ok!
1332024-02-02T02:40:40.272ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crucible/regions/cc6a4f86-e5d2-445a-8886-0d0d7d6b65ea" region = cc6a4f86-e5d2-445a-8886-0d0d7d6b65ea
1342024-02-02T02:40:40.319ZINFOcrucible-agent (datafile): region cc6a4f86-e5d2-445a-8886-0d0d7d6b65ea state: Tombstoned -> Destroyed
1352024-02-02T02:40:40.324ZINFOcrucible-agent (dropshot): request completed latency_us = 86 local_addr = [fd00:1122:3344:101::16]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:54379 req_id = 8e7bc3f1-eb33-40f4-a301-325c340ade57 response_code = 200 uri = /crucible/0/regions/cc6a4f86-e5d2-445a-8886-0d0d7d6b65ea