Max level shown:
1 [ Dec 19 23:01:57 Enabled. ]
2 [ Dec 19 23:01:57 Rereading configuration. ]
3 [ Dec 19 23:02:00 Rereading configuration. ]
4 [ Dec 19 23:02:06 Executing start method ("/opt/oxide/lib/svc/manifest/crucible/agent.sh"). ]
5 + . /lib/svc/share/smf_include.sh
6 ++ SMF_EXIT_OK=0
7 ++ SMF_EXIT_NODAEMON=94
8 ++ SMF_EXIT_ERR_FATAL=95
9 ++ SMF_EXIT_ERR_CONFIG=96
10 ++ SMF_EXIT_MON_DEGRADE=97
11 ++ SMF_EXIT_MON_OFFLINE=98
12 ++ SMF_EXIT_ERR_NOSMF=99
13 ++ SMF_EXIT_ERR_PERM=100
14 ++ svcprop -c -p config/datalink svc:/oxide/crucible/agent:default
15 + DATALINK=oxControlService14
16 ++ svcprop -c -p config/gateway svc:/oxide/crucible/agent:default
17 + GATEWAY=fd00:1122:3344:101::1
18 ++ svcprop -c -p config/dataset svc:/oxide/crucible/agent:default
19 + DATASET=oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible
20 ++ svcprop -c -p config/listen_addr svc:/oxide/crucible/agent:default
21 + LISTEN_ADDR=fd00:1122:3344:101::12
22 ++ svcprop -c -p config/listen_port svc:/oxide/crucible/agent:default
23 + LISTEN_PORT=32345
24 ++ svcprop -c -p config/portbase svc:/oxide/crucible/agent:default
25 + PORTBASE=19000
26 ++ svcprop -c -p config/downstairs_prefix svc:/oxide/crucible/agent:default
27 + DOWNSTAIRS_PREFIX=downstairs
28 ++ svcprop -c -p config/snapshot_prefix svc:/oxide/crucible/agent:default
29 + SNAPSHOT_PREFIX=snapshot
30 + [[ oxControlService14 == unknown ]]
31 + [[ fd00:1122:3344:101::1 == unknown ]]
32 + ipadm delete-if oxControlService14
33 ipadm: Could not delete oxControlService14: Interface does not exist
34 + true
35 + ipadm create-if -t oxControlService14
36 + ipadm set-ifprop -t -p mtu=9000 -m ipv4 oxControlService14
37 + ipadm set-ifprop -t -p mtu=9000 -m ipv6 oxControlService14
38 + ipadm show-addr oxControlService14/ll
39 ipadm: Address object not found
40 + ipadm create-addr -t -T addrconf oxControlService14/ll
41 + ipadm show-addr oxControlService14/omicron6
42 ipadm: Address object not found
43 + ipadm create-addr -t -T static -a fd00:1122:3344:101::12 oxControlService14/omicron6
44 + route get -inet6 default -inet6 fd00:1122:3344:101::1
45 default: not in table
46 + route add -inet6 default -inet6 fd00:1122:3344:101::1
47 add net default: gateway fd00:1122:3344:101::1
48 + args=('-D' '/opt/oxide/crucible/bin/crucible-downstairs' '--dataset' "$DATASET" '-l' "[$LISTEN_ADDR]:$LISTEN_PORT" '-P' "$PORTBASE" '-p' "$DOWNSTAIRS_PREFIX" '-s' "$SNAPSHOT_PREFIX")
49 + exec /opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible -l '[fd00:1122:3344:101::12]:32345' -P 19000 -p downstairs -s snapshot
50 note: configured to log to "/dev/stdout"
512023-12-19T23:02:07.281ZINFOcrucible-agent: dataset: "oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible"
522023-12-19T23:02:07.283ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::12]:32345
532023-12-19T23:02:07.283ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
542023-12-19T23:02:07.612ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::12]:32345
552023-12-19T23:03:48.917ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:64784
562023-12-19T23:03:48.952ZINFOcrucible-agent (datafile): region 6d063c65-4c9c-46ab-bb56-111af27dd7b1 state: Requested
572023-12-19T23:03:48.955ZINFOcrucible-agent (dropshot): request completed latency_us = 17623 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:64784 req_id = d697540c-1d72-4c9a-bf23-32ea24cd5054 response_code = 200 uri = /crucible/0/regions
582023-12-19T23:03:48.955ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
592023-12-19T23:03:48.963ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/6d063c65-4c9c-46ab-bb56-111af27dd7b1
602023-12-19T23:03:48.963ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/6d063c65-4c9c-46ab-bb56-111af27dd7b1
612023-12-19T23:03:49.008ZINFOcrucible-agent (worker): creating region Region { id: RegionId("6d063c65-4c9c-46ab-bb56-111af27dd7b1"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None } at "/data/regions/6d063c65-4c9c-46ab-bb56-111af27dd7b1" region = 6d063c65-4c9c-46ab-bb56-111af27dd7b1
622023-12-19T23:03:49.095ZINFOcrucible-agent (dropshot): request completed latency_us = 91 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:64784 req_id = 8b67c365-35a0-4320-9208-6f30095eff6c response_code = 200 uri = /crucible/0/regions
632023-12-19T23:03:49.315ZINFOcrucible-agent (worker): region files created ok region = 6d063c65-4c9c-46ab-bb56-111af27dd7b1
642023-12-19T23:03:49.315ZINFOcrucible-agent (datafile): region 6d063c65-4c9c-46ab-bb56-111af27dd7b1 state: Requested -> Created
652023-12-19T23:03:49.315ZINFOcrucible-agent (worker): applying SMF actions post create...
662023-12-19T23:03:49.318ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-6d063c65-4c9c-46ab-bb56-111af27dd7b1
672023-12-19T23:03:49.323ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-6d063c65-4c9c-46ab-bb56-111af27dd7b1
682023-12-19T23:03:49.342ZINFOcrucible-agent (worker): creating config property group
692023-12-19T23:03:49.345ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-6d063c65-4c9c-46ab-bb56-111af27dd7b1
702023-12-19T23:03:49.345ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/6d063c65-4c9c-46ab-bb56-111af27dd7b1
712023-12-19T23:03:49.345ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
722023-12-19T23:03:49.345ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
732023-12-19T23:03:49.345ZINFOcrucible-agent (worker): commit
742023-12-19T23:03:49.350ZINFOcrucible-agent (worker): ok!
752023-12-19T23:03:49.359ZINFOcrucible-agent (worker): SMF ok!
762023-12-19T23:03:49.696ZINFOcrucible-agent (dropshot): request completed latency_us = 101 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:64784 req_id = 8aa3a32e-ea69-48fb-af97-3c36d6b72a96 response_code = 200 uri = /crucible/0/regions
772023-12-19T23:09:16.107ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:39070
782023-12-19T23:09:16.110ZINFOcrucible-agent (dropshot): request completed latency_us = 2655 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:39070 req_id = 78ced2b8-c90c-4dc0-9d20-6a0446decc43 response_code = 200 uri = /crucible/0/regions/6d063c65-4c9c-46ab-bb56-111af27dd7b1
792023-12-19T23:09:16.110ZINFOcrucible-agent (datafile): path is "/data/regions/6d063c65-4c9c-46ab-bb56-111af27dd7b1"
802023-12-19T23:09:16.119ZINFOcrucible-agent (datafile): dataset is oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/6d063c65-4c9c-46ab-bb56-111af27dd7b1
812023-12-19T23:09:16.129ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/6d063c65-4c9c-46ab-bb56-111af27dd7b1@f52bac84-3c6e-4589-a377-3af5b7e2cbc8
822023-12-19T23:09:16.129ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/6d063c65-4c9c-46ab-bb56-111af27dd7b1", "f52bac84-3c6e-4589-a377-3af5b7e2cbc8"]
832023-12-19T23:09:16.143ZINFOcrucible-agent (dropshot): request completed latency_us = 33343 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:39070 req_id = f93fe5d3-d6c2-4477-846f-ce62ae861c1a response_code = 200 uri = /crucible/0/regions/6d063c65-4c9c-46ab-bb56-111af27dd7b1/snapshots/f52bac84-3c6e-4589-a377-3af5b7e2cbc8
842023-12-19T23:09:16.144ZINFOcrucible-agent (datafile): path is "/data/regions/6d063c65-4c9c-46ab-bb56-111af27dd7b1"
852023-12-19T23:09:16.153ZINFOcrucible-agent (datafile): dataset is oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/6d063c65-4c9c-46ab-bb56-111af27dd7b1
862023-12-19T23:09:16.162ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/6d063c65-4c9c-46ab-bb56-111af27dd7b1@f52bac84-3c6e-4589-a377-3af5b7e2cbc8
872023-12-19T23:09:16.162ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/6d063c65-4c9c-46ab-bb56-111af27dd7b1", "f52bac84-3c6e-4589-a377-3af5b7e2cbc8"]
882023-12-19T23:09:16.172ZINFOcrucible-agent (datafile): requesting running snapshot 6d063c65-4c9c-46ab-bb56-111af27dd7b1-f52bac84-3c6e-4589-a377-3af5b7e2cbc8 state: Requested
892023-12-19T23:09:16.172ZINFOcrucible-agent (worker): applying SMF actions for region 6d063c65-4c9c-46ab-bb56-111af27dd7b1 running snapshot f52bac84-3c6e-4589-a377-3af5b7e2cbc8 (state Requested)...
902023-12-19T23:09:16.172ZINFOcrucible-agent (dropshot): request completed latency_us = 28665 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:39070 req_id = b479a049-7da5-4a37-ae35-938f222e8364 response_code = 200 uri = /crucible/0/regions/6d063c65-4c9c-46ab-bb56-111af27dd7b1/snapshots/f52bac84-3c6e-4589-a377-3af5b7e2cbc8/run
912023-12-19T23:09:16.181ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-6d063c65-4c9c-46ab-bb56-111af27dd7b1-f52bac84-3c6e-4589-a377-3af5b7e2cbc8
922023-12-19T23:09:16.185ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-6d063c65-4c9c-46ab-bb56-111af27dd7b1-f52bac84-3c6e-4589-a377-3af5b7e2cbc8
932023-12-19T23:09:16.200ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-6d063c65-4c9c-46ab-bb56-111af27dd7b1-f52bac84-3c6e-4589-a377-3af5b7e2cbc8
942023-12-19T23:09:16.200ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/6d063c65-4c9c-46ab-bb56-111af27dd7b1/.zfs/snapshot/f52bac84-3c6e-4589-a377-3af5b7e2cbc8
952023-12-19T23:09:16.201ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
962023-12-19T23:09:16.201ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
972023-12-19T23:09:16.201ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
982023-12-19T23:09:16.201ZINFOcrucible-agent (worker): commit
992023-12-19T23:09:16.204ZINFOcrucible-agent (worker): ok!
1002023-12-19T23:09:16.209ZINFOcrucible-agent (worker): SMF ok!
1012023-12-19T23:09:16.209ZINFOcrucible-agent (datafile): region 6d063c65-4c9c-46ab-bb56-111af27dd7b1 running snapshot f52bac84-3c6e-4589-a377-3af5b7e2cbc8 state: Requested -> Created
1022023-12-19T23:09:18.773ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:51898
1032023-12-19T23:09:18.834ZINFOcrucible-agent (datafile): region 854ce8d1-3d8d-42fb-b37f-50ffe42a9688 state: Requested
1042023-12-19T23:09:18.837ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
1052023-12-19T23:09:18.837ZINFOcrucible-agent (dropshot): request completed latency_us = 3425 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:51898 req_id = 4fbe4d03-6f0e-4f7c-9512-d61eb679bd55 response_code = 200 uri = /crucible/0/regions
1062023-12-19T23:09:18.846ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/854ce8d1-3d8d-42fb-b37f-50ffe42a9688
1072023-12-19T23:09:18.846ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/854ce8d1-3d8d-42fb-b37f-50ffe42a9688
1082023-12-19T23:09:18.904ZINFOcrucible-agent (worker): creating region Region { id: RegionId("854ce8d1-3d8d-42fb-b37f-50ffe42a9688"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19002, cert_pem: None, key_pem: None, root_pem: None } at "/data/regions/854ce8d1-3d8d-42fb-b37f-50ffe42a9688" region = 854ce8d1-3d8d-42fb-b37f-50ffe42a9688
1092023-12-19T23:09:19.005ZINFOcrucible-agent (worker): region files created ok region = 854ce8d1-3d8d-42fb-b37f-50ffe42a9688
1102023-12-19T23:09:19.005ZINFOcrucible-agent (datafile): region 854ce8d1-3d8d-42fb-b37f-50ffe42a9688 state: Requested -> Created
1112023-12-19T23:09:19.005ZINFOcrucible-agent (worker): applying SMF actions post create...
1122023-12-19T23:09:19.007ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-854ce8d1-3d8d-42fb-b37f-50ffe42a9688
1132023-12-19T23:09:19.009ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-854ce8d1-3d8d-42fb-b37f-50ffe42a9688
1142023-12-19T23:09:19.016ZINFOcrucible-agent (worker): creating config property group
1152023-12-19T23:09:19.019ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-854ce8d1-3d8d-42fb-b37f-50ffe42a9688
1162023-12-19T23:09:19.019ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/854ce8d1-3d8d-42fb-b37f-50ffe42a9688
1172023-12-19T23:09:19.019ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
1182023-12-19T23:09:19.019ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
1192023-12-19T23:09:19.019ZINFOcrucible-agent (worker): commit
1202023-12-19T23:09:19.023ZINFOcrucible-agent (worker): ok!
1212023-12-19T23:09:19.029ZINFOcrucible-agent (worker): SMF ok!
1222023-12-19T23:09:19.046ZINFOcrucible-agent (dropshot): request completed latency_us = 79 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:51898 req_id = 119b8155-97cc-4f5e-a6f3-6b0ede36c555 response_code = 200 uri = /crucible/0/regions
1232023-12-19T23:09:50.312ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:34341
1242023-12-19T23:09:50.312ZINFOcrucible-agent (dropshot): request completed latency_us = 54 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:34341 req_id = d7ed1279-f5a2-4d5a-ace6-47d20fb0b01e response_code = 200 uri = /crucible/0/regions/854ce8d1-3d8d-42fb-b37f-50ffe42a9688
1252023-12-19T23:09:50.312ZINFOcrucible-agent (datafile): path is "/data/regions/854ce8d1-3d8d-42fb-b37f-50ffe42a9688"
1262023-12-19T23:09:50.324ZINFOcrucible-agent (datafile): dataset is oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/854ce8d1-3d8d-42fb-b37f-50ffe42a9688
1272023-12-19T23:09:50.333ZINFOcrucible-agent (datafile): region 854ce8d1-3d8d-42fb-b37f-50ffe42a9688 state: Created -> Tombstoned
1282023-12-19T23:09:50.333ZINFOcrucible-agent (worker): applying SMF actions before removal...
1292023-12-19T23:09:50.335ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-854ce8d1-3d8d-42fb-b37f-50ffe42a9688 (instance states: (Some(Online), None))
1302023-12-19T23:09:50.336ZINFOcrucible-agent (dropshot): request completed latency_us = 24009 local_addr = [fd00:1122:3344:101::12]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:34341 req_id = 42295f80-e813-4336-b44d-983eb1da9daf response_code = 204 uri = /crucible/0/regions/854ce8d1-3d8d-42fb-b37f-50ffe42a9688
1312023-12-19T23:09:50.336ZINFOcrucible-agent (dropshot): request completed latency_us = 39 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:34341 req_id = 66971d2d-d460-4e9e-8535-41975dc8f70e response_code = 200 uri = /crucible/0/regions/854ce8d1-3d8d-42fb-b37f-50ffe42a9688
1322023-12-19T23:09:50.344ZINFOcrucible-agent (worker): SMF ok!
1332023-12-19T23:09:50.354ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/854ce8d1-3d8d-42fb-b37f-50ffe42a9688" region = 854ce8d1-3d8d-42fb-b37f-50ffe42a9688
1342023-12-19T23:09:50.397ZINFOcrucible-agent (datafile): region 854ce8d1-3d8d-42fb-b37f-50ffe42a9688 state: Tombstoned -> Destroyed
1352023-12-19T23:09:50.480ZINFOcrucible-agent (dropshot): request completed latency_us = 58 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:34341 req_id = feb5535c-4233-483e-b5f9-f53ffde77d91 response_code = 200 uri = /crucible/0/regions/854ce8d1-3d8d-42fb-b37f-50ffe42a9688