Max level shown:
1 [ Feb 16 07:03:00 Enabled. ]
2 [ Feb 16 07:03:00 Rereading configuration. ]
3 [ Feb 16 07:03:01 Rereading configuration. ]
4 [ Feb 16 07:03:02 Executing start method ("/opt/oxide/lib/svc/manifest/crucible/agent.sh"). ]
5 + . /lib/svc/share/smf_include.sh
6 ++ SMF_EXIT_OK=0
7 ++ SMF_EXIT_NODAEMON=94
8 ++ SMF_EXIT_ERR_FATAL=95
9 ++ SMF_EXIT_ERR_CONFIG=96
10 ++ SMF_EXIT_MON_DEGRADE=97
11 ++ SMF_EXIT_MON_OFFLINE=98
12 ++ SMF_EXIT_ERR_NOSMF=99
13 ++ SMF_EXIT_ERR_PERM=100
14 ++ svcprop -c -p config/datalink svc:/oxide/crucible/agent:default
15 + DATALINK=oxControlService15
16 ++ svcprop -c -p config/gateway svc:/oxide/crucible/agent:default
17 + GATEWAY=fd00:1122:3344:101::1
18 ++ svcprop -c -p config/dataset svc:/oxide/crucible/agent:default
19 + DATASET=oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible
20 ++ svcprop -c -p config/listen_addr svc:/oxide/crucible/agent:default
21 + LISTEN_ADDR=fd00:1122:3344:101::12
22 ++ svcprop -c -p config/listen_port svc:/oxide/crucible/agent:default
23 + LISTEN_PORT=32345
24 ++ svcprop -c -p config/portbase svc:/oxide/crucible/agent:default
25 + PORTBASE=19000
26 ++ svcprop -c -p config/downstairs_prefix svc:/oxide/crucible/agent:default
27 + DOWNSTAIRS_PREFIX=downstairs
28 ++ svcprop -c -p config/snapshot_prefix svc:/oxide/crucible/agent:default
29 + SNAPSHOT_PREFIX=snapshot
30 + [[ oxControlService15 == unknown ]]
31 + [[ fd00:1122:3344:101::1 == unknown ]]
32 + ipadm delete-if oxControlService15
33 ipadm: Could not delete oxControlService15: Interface does not exist
34 + true
35 + ipadm create-if -t oxControlService15
36 + ipadm set-ifprop -t -p mtu=9000 -m ipv4 oxControlService15
37 + ipadm set-ifprop -t -p mtu=9000 -m ipv6 oxControlService15
38 + ipadm show-addr oxControlService15/ll
39 ipadm: Address object not found
40 + ipadm create-addr -t -T addrconf oxControlService15/ll
41 + ipadm show-addr oxControlService15/omicron6
42 ipadm: Address object not found
43 + ipadm create-addr -t -T static -a fd00:1122:3344:101::12 oxControlService15/omicron6
44 + route get -inet6 default -inet6 fd00:1122:3344:101::1
45 default: not in table
46 + route add -inet6 default -inet6 fd00:1122:3344:101::1
47 add net default: gateway fd00:1122:3344:101::1
48 + args=('-D' '/opt/oxide/crucible/bin/crucible-downstairs' '--dataset' "$DATASET" '-l' "[$LISTEN_ADDR]:$LISTEN_PORT" '-P' "$PORTBASE" '-p' "$DOWNSTAIRS_PREFIX" '-s' "$SNAPSHOT_PREFIX")
49 + exec /opt/oxide/crucible/bin/crucible-agent run -D /opt/oxide/crucible/bin/crucible-downstairs --dataset oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible -l '[fd00:1122:3344:101::12]:32345' -P 19000 -p downstairs -s snapshot
50 note: configured to log to "/dev/stdout"
512024-02-16T07:03:03.186ZINFOcrucible-agent: dataset: "oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible"
522024-02-16T07:03:03.194ZINFOcrucible-agent: listen IP: [fd00:1122:3344:101::12]:32345
532024-02-16T07:03:03.194ZINFOcrucible-agent: SMF instance name downstairs_prefix: "downstairs"
542024-02-16T07:03:03.376ZINFOcrucible-agent (dropshot): listening local_addr = [fd00:1122:3344:101::12]:32345
552024-02-16T07:03:55.254ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:55034
562024-02-16T07:03:55.314ZINFOcrucible-agent (datafile): region c709deda-1414-4f58-a3d6-38ba03679d53 state: Requested
572024-02-16T07:03:55.315ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
582024-02-16T07:03:55.315ZINFOcrucible-agent (dropshot): request completed latency_us = 1888 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:55034 req_id = f6c7c019-ac2f-4654-b72d-4423d5d93017 response_code = 200 uri = /crucible/0/regions
592024-02-16T07:03:55.325ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/c709deda-1414-4f58-a3d6-38ba03679d53
602024-02-16T07:03:55.325ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/c709deda-1414-4f58-a3d6-38ba03679d53
612024-02-16T07:03:55.380ZINFOcrucible-agent (worker): creating region Region { id: RegionId("c709deda-1414-4f58-a3d6-38ba03679d53"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19000, cert_pem: None, key_pem: None, root_pem: None } at "/data/regions/c709deda-1414-4f58-a3d6-38ba03679d53" region = c709deda-1414-4f58-a3d6-38ba03679d53
622024-02-16T07:03:55.526ZINFOcrucible-agent (worker): region files created ok region = c709deda-1414-4f58-a3d6-38ba03679d53
632024-02-16T07:03:55.526ZINFOcrucible-agent (datafile): region c709deda-1414-4f58-a3d6-38ba03679d53 state: Requested -> Created
642024-02-16T07:03:55.526ZINFOcrucible-agent (worker): applying SMF actions post create...
652024-02-16T07:03:55.526ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-c709deda-1414-4f58-a3d6-38ba03679d53
662024-02-16T07:03:55.529ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-c709deda-1414-4f58-a3d6-38ba03679d53
672024-02-16T07:03:55.536ZINFOcrucible-agent (worker): creating config property group
682024-02-16T07:03:55.540ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-c709deda-1414-4f58-a3d6-38ba03679d53
692024-02-16T07:03:55.541ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/c709deda-1414-4f58-a3d6-38ba03679d53
702024-02-16T07:03:55.541ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19000
712024-02-16T07:03:55.541ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
722024-02-16T07:03:55.541ZINFOcrucible-agent (worker): commit
732024-02-16T07:03:55.545ZINFOcrucible-agent (worker): ok!
742024-02-16T07:03:55.549ZINFOcrucible-agent (worker): SMF ok!
752024-02-16T07:03:55.564ZINFOcrucible-agent (dropshot): request completed latency_us = 672 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:55034 req_id = fd9684ba-def4-4594-9047-a161f0ada852 response_code = 200 uri = /crucible/0/regions
762024-02-16T07:09:37.564ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:49788
772024-02-16T07:09:37.565ZINFOcrucible-agent (dropshot): request completed latency_us = 701 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:49788 req_id = d7409ac6-17bc-4482-9d77-7637ad1eeca7 response_code = 200 uri = /crucible/0/regions/c709deda-1414-4f58-a3d6-38ba03679d53
782024-02-16T07:09:37.567ZINFOcrucible-agent (datafile): path is "/data/regions/c709deda-1414-4f58-a3d6-38ba03679d53"
792024-02-16T07:09:37.577ZINFOcrucible-agent (datafile): dataset is oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/c709deda-1414-4f58-a3d6-38ba03679d53
802024-02-16T07:09:37.586ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/c709deda-1414-4f58-a3d6-38ba03679d53@c5ff36fd-fda4-4d5b-8854-5b065543fc40
812024-02-16T07:09:37.586ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/c709deda-1414-4f58-a3d6-38ba03679d53", "c5ff36fd-fda4-4d5b-8854-5b065543fc40"]
822024-02-16T07:09:37.597ZINFOcrucible-agent (dropshot): request completed latency_us = 29781 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:49788 req_id = 39164278-2ade-4ef8-9561-57fd7d3fc997 response_code = 200 uri = /crucible/0/regions/c709deda-1414-4f58-a3d6-38ba03679d53/snapshots/c5ff36fd-fda4-4d5b-8854-5b065543fc40
832024-02-16T07:09:37.597ZINFOcrucible-agent (datafile): path is "/data/regions/c709deda-1414-4f58-a3d6-38ba03679d53"
842024-02-16T07:09:37.606ZINFOcrucible-agent (datafile): dataset is oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/c709deda-1414-4f58-a3d6-38ba03679d53
852024-02-16T07:09:37.614ZINFOcrucible-agent (ZfsSnapshotInterface): snapshot is oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/c709deda-1414-4f58-a3d6-38ba03679d53@c5ff36fd-fda4-4d5b-8854-5b065543fc40
862024-02-16T07:09:37.614ZINFOcrucible-agent (ZfsSnapshotInterface): parts is ["oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/c709deda-1414-4f58-a3d6-38ba03679d53", "c5ff36fd-fda4-4d5b-8854-5b065543fc40"]
872024-02-16T07:09:37.622ZINFOcrucible-agent (datafile): requesting running snapshot c709deda-1414-4f58-a3d6-38ba03679d53-c5ff36fd-fda4-4d5b-8854-5b065543fc40 state: Requested
882024-02-16T07:09:37.623ZINFOcrucible-agent (worker): applying SMF actions for region c709deda-1414-4f58-a3d6-38ba03679d53 running snapshot c5ff36fd-fda4-4d5b-8854-5b065543fc40 (state Requested)...
892024-02-16T07:09:37.623ZINFOcrucible-agent (dropshot): request completed latency_us = 25561 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:49788 req_id = 2a1531b5-fe54-4db9-89ce-22c37533cc54 response_code = 200 uri = /crucible/0/regions/c709deda-1414-4f58-a3d6-38ba03679d53/snapshots/c5ff36fd-fda4-4d5b-8854-5b065543fc40/run
902024-02-16T07:09:37.631ZINFOcrucible-agent (worker): creating missing snapshot instance snapshot-c709deda-1414-4f58-a3d6-38ba03679d53-c5ff36fd-fda4-4d5b-8854-5b065543fc40
912024-02-16T07:09:37.635ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:snapshot-c709deda-1414-4f58-a3d6-38ba03679d53-c5ff36fd-fda4-4d5b-8854-5b065543fc40
922024-02-16T07:09:37.645ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:snapshot-c709deda-1414-4f58-a3d6-38ba03679d53-c5ff36fd-fda4-4d5b-8854-5b065543fc40
932024-02-16T07:09:37.646ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/c709deda-1414-4f58-a3d6-38ba03679d53/.zfs/snapshot/c5ff36fd-fda4-4d5b-8854-5b065543fc40
942024-02-16T07:09:37.646ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19001
952024-02-16T07:09:37.646ZINFOcrucible-agent (worker): ensure mode SCF_TYPE_ASTRING ro
962024-02-16T07:09:37.646ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
972024-02-16T07:09:37.646ZINFOcrucible-agent (worker): commit
982024-02-16T07:09:37.651ZINFOcrucible-agent (worker): ok!
992024-02-16T07:09:37.655ZINFOcrucible-agent (worker): SMF ok!
1002024-02-16T07:09:37.655ZINFOcrucible-agent (datafile): region c709deda-1414-4f58-a3d6-38ba03679d53 running snapshot c5ff36fd-fda4-4d5b-8854-5b065543fc40 state: Requested -> Created
1012024-02-16T07:09:40.405ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:35634
1022024-02-16T07:09:40.464ZINFOcrucible-agent (datafile): region 30ef8df5-4489-421e-9a5e-98893b355ad4 state: Requested
1032024-02-16T07:09:40.464ZINFOcrucible-agent (worker): Region size:2147483648 reservation:2684354560 quota:6442450944
1042024-02-16T07:09:40.464ZINFOcrucible-agent (dropshot): request completed latency_us = 303 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:35634 req_id = 7c2e019d-457a-4832-9393-0484253b90eb response_code = 200 uri = /crucible/0/regions
1052024-02-16T07:09:40.472ZINFOcrucible-agent (worker): zfs set reservation of 2684354560 for oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/30ef8df5-4489-421e-9a5e-98893b355ad4
1062024-02-16T07:09:40.472ZINFOcrucible-agent (worker): zfs set quota of 6442450944 for oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/30ef8df5-4489-421e-9a5e-98893b355ad4
1072024-02-16T07:09:40.546ZINFOcrucible-agent (worker): creating region Region { id: RegionId("30ef8df5-4489-421e-9a5e-98893b355ad4"), state: Requested, block_size: 512, extent_size: 131072, extent_count: 32, encrypted: true, port_number: 19002, cert_pem: None, key_pem: None, root_pem: None } at "/data/regions/30ef8df5-4489-421e-9a5e-98893b355ad4" region = 30ef8df5-4489-421e-9a5e-98893b355ad4
1082024-02-16T07:09:40.639ZINFOcrucible-agent (worker): region files created ok region = 30ef8df5-4489-421e-9a5e-98893b355ad4
1092024-02-16T07:09:40.639ZINFOcrucible-agent (datafile): region 30ef8df5-4489-421e-9a5e-98893b355ad4 state: Requested -> Created
1102024-02-16T07:09:40.639ZINFOcrucible-agent (worker): applying SMF actions post create...
1112024-02-16T07:09:40.639ZINFOcrucible-agent (worker): creating missing downstairs instance downstairs-30ef8df5-4489-421e-9a5e-98893b355ad4
1122024-02-16T07:09:40.646ZINFOcrucible-agent (worker): ok, have svc:/oxide/crucible/downstairs:downstairs-30ef8df5-4489-421e-9a5e-98893b355ad4
1132024-02-16T07:09:40.652ZINFOcrucible-agent (worker): creating config property group
1142024-02-16T07:09:40.655ZINFOcrucible-agent (worker): reconfiguring svc:/oxide/crucible/downstairs:downstairs-30ef8df5-4489-421e-9a5e-98893b355ad4
1152024-02-16T07:09:40.655ZINFOcrucible-agent (worker): ensure directory SCF_TYPE_ASTRING /data/regions/30ef8df5-4489-421e-9a5e-98893b355ad4
1162024-02-16T07:09:40.655ZINFOcrucible-agent (worker): ensure port SCF_TYPE_COUNT 19002
1172024-02-16T07:09:40.655ZINFOcrucible-agent (worker): ensure address SCF_TYPE_ASTRING fd00:1122:3344:101::12
1182024-02-16T07:09:40.655ZINFOcrucible-agent (worker): commit
1192024-02-16T07:09:40.658ZINFOcrucible-agent (worker): ok!
1202024-02-16T07:09:40.668ZINFOcrucible-agent (worker): SMF ok!
1212024-02-16T07:09:40.706ZINFOcrucible-agent (dropshot): request completed latency_us = 720 local_addr = [fd00:1122:3344:101::12]:32345 method = POST remote_addr = [fd00:1122:3344:101::a]:35634 req_id = e6ed5bb3-cb6a-4931-9e65-df759af45c24 response_code = 200 uri = /crucible/0/regions
1222024-02-16T07:10:21.357ZINFOcrucible-agent (dropshot): accepted connection local_addr = [fd00:1122:3344:101::12]:32345 remote_addr = [fd00:1122:3344:101::a]:42967
1232024-02-16T07:10:21.357ZINFOcrucible-agent (dropshot): request completed latency_us = 550 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:42967 req_id = f1830369-c6ec-4639-bb0d-575646ee5d36 response_code = 200 uri = /crucible/0/regions/30ef8df5-4489-421e-9a5e-98893b355ad4
1242024-02-16T07:10:21.357ZINFOcrucible-agent (datafile): path is "/data/regions/30ef8df5-4489-421e-9a5e-98893b355ad4"
1252024-02-16T07:10:21.375ZINFOcrucible-agent (datafile): dataset is oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/30ef8df5-4489-421e-9a5e-98893b355ad4
1262024-02-16T07:10:21.383ZINFOcrucible-agent (datafile): region 30ef8df5-4489-421e-9a5e-98893b355ad4 state: Created -> Tombstoned
1272024-02-16T07:10:21.383ZINFOcrucible-agent (worker): applying SMF actions before removal...
1282024-02-16T07:10:21.383ZINFOcrucible-agent (dropshot): request completed latency_us = 26148 local_addr = [fd00:1122:3344:101::12]:32345 method = DELETE remote_addr = [fd00:1122:3344:101::a]:42967 req_id = 4c640908-ccdd-445a-b29f-2d8ed725b835 response_code = 204 uri = /crucible/0/regions/30ef8df5-4489-421e-9a5e-98893b355ad4
1292024-02-16T07:10:21.384ZINFOcrucible-agent (dropshot): request completed latency_us = 20 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:42967 req_id = 681fe2fb-721b-4611-8954-a8bd4e9e9ea7 response_code = 200 uri = /crucible/0/regions/30ef8df5-4489-421e-9a5e-98893b355ad4
1302024-02-16T07:10:21.391ZINFOcrucible-agent (worker): disabling downstairs instance: downstairs-30ef8df5-4489-421e-9a5e-98893b355ad4 (instance states: (Some(Online), None))
1312024-02-16T07:10:21.409ZINFOcrucible-agent (worker): SMF ok!
1322024-02-16T07:10:21.419ZINFOcrucible-agent (worker): deleting zfs dataset "oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03/crucible/regions/30ef8df5-4489-421e-9a5e-98893b355ad4" region = 30ef8df5-4489-421e-9a5e-98893b355ad4
1332024-02-16T07:10:21.459ZINFOcrucible-agent (dropshot): request completed latency_us = 69 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:42967 req_id = c4a566e6-33fd-4a2e-8f92-3f28f7635684 response_code = 200 uri = /crucible/0/regions/30ef8df5-4489-421e-9a5e-98893b355ad4
1342024-02-16T07:10:21.541ZINFOcrucible-agent (datafile): region 30ef8df5-4489-421e-9a5e-98893b355ad4 state: Tombstoned -> Destroyed
1352024-02-16T07:10:21.597ZINFOcrucible-agent (dropshot): request completed latency_us = 45 local_addr = [fd00:1122:3344:101::12]:32345 method = GET remote_addr = [fd00:1122:3344:101::a]:42967 req_id = ca99b49f-5f7d-46c9-99e0-ad4f0ba1d946 response_code = 200 uri = /crucible/0/regions/30ef8df5-4489-421e-9a5e-98893b355ad4