Commit Graph

776 Commits (3d1653f95084b5326e038aa32224ae128dad5c0f)

Author SHA1 Message Date
jeremy-thuon 4a03d13d08
[cilium] fix rbac and upgrade hubble v0.11.0 (#3) (#9959)
* [cilium] fix rbac and upgrade hubble v0.11.0 (#3)

* [cilium] fix rbac for LB bgp ipam

* [cilium] Upgrade Hubble to v0.11.0 and add mTLS between Hubble UI and Hubble Relay

* fix dns domain hubble for tls

---------

Co-authored-by: Thuon Jeremy <d107869@olinfra1.infra.bdm.outscale.c1.dav.fr>

* Fix blank line

---------

Co-authored-by: Thuon Jeremy <d107869@olinfra1.infra.bdm.outscale.c1.dav.fr>
2023-04-09 22:07:15 -07:00
Anton 1bb4f88af1
cilium: Additional fix the configuration of tls for hubble #9880 (#9932) 2023-03-27 08:48:27 -07:00
xiuguang.huang 4020a93d7e
delete the probe option of cilium_kube_proxy_replacement (#9929) 2023-03-27 08:08:28 -07:00
Toru Komatsu 8afd74ce1f
cilium: Fix the configuration of tls for hubble (#9880)
Signed-off-by: utam0k <k0ma@utam0k.jp>
2023-03-24 01:10:31 -07:00
Jiri Fiala 9b37699d0d
Cilium Operator replicas configuration (#9894)
Signed-off-by: Fiala, JiriX <jirix.fiala@intel.com>
2023-03-22 08:28:38 -07:00
Will Hegedus 97dfdcd8fe
feat: support cilium 1.13.1 (#9914)
Cilium 1.13.1 changed how the cilium-cni binary gets placed in /opt/cni/bin,
so that it takes place in an init container rather than in the main agent.
2023-03-21 12:56:12 -07:00
prashantchitta a9f52060c9
Fix ciliums hubble relay configuration (#9876)
* Fix ciliums hubble relay configuration

* Fixed the tls from code review

* Updated to dna_domain instead of hardcoding
2023-03-21 12:50:12 -07:00
蒋航 83c3ce7f8f
Add Retry for Checking calico exists (#9883)
Signed-off-by: hang.jiang <hang.jiang@daocloud.io>
2023-03-20 21:51:06 -07:00
Kay Yan 309aaee427
fix-cilium-error (#9902) 2023-03-20 02:41:17 -07:00
Samuel Liu df9aba6298 fix typo word 2023-03-14 15:49:22 +01:00
biqiang Wu 2ae3ea9ee3
Modified the default value of cilium IPAM and added the support for related parameters (#9443)
Signed-off-by: dcwbq <biqiang.wu@daocloud.io>
2023-03-13 17:45:10 -07:00
蒋航 99115ad04b
Fix Get current calico version (#9873)
Signed-off-by: hang.jiang <hang.jiang@daocloud.io>
2023-03-10 05:48:40 -08:00
Jack 1ca50f3eea
Update check calico version command (#9861) 2023-03-08 00:31:12 -08:00
Arthur Outhenin-Chalandre 82f68ca395
calico: cilium: use localhost lb by default on kube-proxy replacement (#9718)
This commit removes the variable `use_localhost_as_kubeapi_loadbalancer`
and rather detects that we are in a situation where we can use the
localhost apiserver loadbalancer (meaning that we use the localhost load
balancer and that the same ports are used for both the load balancer and
the kube-apiserver).

This also cleanups the calico code to use `kube_apiserver_global_endpoint`
rather than implementing the same logic all over again.

Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@proton.ch>
2023-03-07 04:28:36 -08:00
DRAGON2002 1d9502e01d
update args (#9856)
Signed-off-by: Anant Vijay <anantvijay3@gmail.com>
2023-03-05 18:38:57 -08:00
Arthur Outhenin-Chalandre 6769bb32b1
Network plugin custom (#9819)
* network_plugin/custom_cni: add CNI to apply provided manifests

Add a new simple custom_cni to install provided Kubernetes manifests.
This could be useful to use manifests directly provided by a CNI when
there are not support by Kubespray (i.e.: helm chart or any other manifests
generation method).

Co-authored-by: James Landrein <james.landrein@proton.ch>
Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@proton.ch>

* network_plugin/custom_cni: add test with cilium

Co-authored-by: James Landrein <james.landrein@proton.ch>
Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@proton.ch>

---------

Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@proton.ch>
Co-authored-by: James Landrein <james.landrein@proton.ch>
2023-03-03 00:23:08 -08:00
JaneLiuL 4aacec4542
fix calico rbac issue (#9806) 2023-02-20 01:43:40 -08:00
Bas 2c93c997cf
pre-commit autocorrected files (#9750) 2023-02-06 01:35:16 -08:00
Chauncey 7730cfd619
fix: add ipamconfigs resource for calico (#9755)
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
2023-02-05 15:50:30 -08:00
James 36c6de9abd
Fix cilium's hubble ui configuration (#9735)
This fixes the CrashLoopBackoff error that appears because envoy
configuration has changed a lot and upstream removed the envoy proxy to
use nginx only instead. Those changes are based on upstream cilium helm.
2023-01-31 00:28:48 -08:00
MatthieuFin 374438a3d6
feat(calico): add possibility to enable calico floatingIPs feature (#9680)
Add a variable `calico_felix_floatingIPs` which permit to enable calico feature `floatingIPs`
(disabled per default).

Signed-off-by: MatthieuFin <matthieu2717@gmail.com>

 #9679
2023-01-18 15:42:34 -08:00
Vitaly Yakovenko d919c58e21
[multus] added support for mixed type of container engine (#9224)
* [multus] added support for mixed type of container engine

* [multus] fixed for using with cluster/upgrade-cluster/scale playbooks
2023-01-16 23:30:33 -08:00
Jochen Friedrich 6f61f3d9cb
Support OVN Interconnect (#9599)
Mostly taken from: https://raw.githubusercontent.com/kubeovn/kube-ovn/master/yamls/ovn-ic.yaml.j2
2023-01-16 00:08:52 -08:00
Kay Yan 843e908fa4
update-calico-VXLAN-docs (#9639) 2023-01-06 00:00:00 -08:00
tu1h a8cef962e2
Add retry to avoid 'unknown' state for calicoctl (#9633)
Signed-off-by: tu1h <lihai.tu@daocloud.io>

Signed-off-by: tu1h <lihai.tu@daocloud.io>
2023-01-05 05:09:58 -08:00
my-git9 85fa6af313
cleanup: replace node-role.kubernetes.io/master (#9627)
Signed-off-by: xin.li <xin.li@daocloud.io>

Signed-off-by: xin.li <xin.li@daocloud.io>
2023-01-01 13:59:32 -08:00
Jochen Friedrich 744c81d451
Remove ovn.kubernetes.io/ovs_dp_type from nodeSelector (#9594)
Remove extra tag requirement preventing openvswitch container to start.
2022-12-29 01:37:29 -08:00
ERIK 406fbdb4e7
Update the tag of the flannel image (#9528)
Signed-off-by: bo.jiang <bo.jiang@daocloud.io>

Signed-off-by: bo.jiang <bo.jiang@daocloud.io>
2022-12-28 00:55:27 -08:00
Jochen Friedrich 136f14dec4
Cluster support for ovn-central. (#9596)
Without minimal cluster configuration, even on a one node control plane,
the health check of the ovn-cental container always fails as it queries the
cluster/status.
2022-12-27 20:51:27 -08:00
Kay Yan 93f71df628
Remove CNI BIN dependency for cilium (#9563)
* remove-cni-denpendeny-for-cilium

* remove-cni-denpendeny-for-cilium
2022-12-27 01:31:28 -08:00
janaurka 491e260d20
Feature/add flannel wireguard encryption backend as option (#9583)
* feat(): Add wireguard backend to flannel cni

As described in the flannel docs:
https://github.com/flannel-io/flannel/blob/master/Documentation/backends.md#wireguard

This does not support optional configuration methods like:
- setting a psk (will be autogenerated by default)
- chang listening ports
- change mode (defaults to 'separate')
- change PersistentKeepaliveInterval (defaults to 0)

* Add supported backends to flannel docs

* Fix markdown in docs
2022-12-18 15:39:43 -08:00
Xieql c4d753c931 Fix annotation typo
Signed-off-by: Xieql <xieqianglong@huawei.com>
2022-12-15 18:40:30 +08:00
Book shu ff331f4eba
support flannel dual stack (#9564) 2022-12-13 20:47:35 -08:00
Samuel Liu f1d0d1a9fe
[kube-ovn]: update version v1.10.7 (#9527)
* [kube-ovn]: update version

* update readme
2022-12-08 15:58:11 -08:00
ERIK 8a654b6955
Add cni bin when installing calico (#9367)
Signed-off-by: bo.jiang <bo.jiang@daocloud.io>

Signed-off-by: bo.jiang <bo.jiang@daocloud.io>
2022-11-08 17:46:13 -08:00
Fred Rolland 58faef6ff6
Flannel: fix init container image arch (#9461)
The install-cni-plugin image was not updated to the corresponding
arch when building the different DS.

Fixes issue #9460

Signed-off-by: Fred Rolland <frolland@nvidia.com>

Signed-off-by: Fred Rolland <frolland@nvidia.com>
2022-11-03 02:41:36 -07:00
charlychiu 6bff338bad
fix: hubble relay tls error (#9457) 2022-11-02 00:14:46 -07:00
William Turner 1f54cef71c
Add variable to set direct routing on flannel VXLAN (#9438) 2022-10-31 13:16:45 -07:00
biqiang Wu 78624c5bcb
When using cilium CNI, install Cilium CLI (#9436)
Signed-off-by: dcwbq <biqiang.wu@daocloud.io>

Signed-off-by: dcwbq <biqiang.wu@daocloud.io>
2022-10-30 17:02:45 -07:00
biqiang Wu c681435432
Add switch cilium_enable_bandwidth_manager (#9441)
Signed-off-by: dcwbq <biqiang.wu@daocloud.io>

Signed-off-by: dcwbq <biqiang.wu@daocloud.io>
2022-10-28 03:08:31 -07:00
Kay Yan 32f3d92d6b
Remove PodSecurityPolicies in Calico (#9395) 2022-10-17 05:51:07 -07:00
Kay Yan 859df84b45
remove-psp-in-flannel (#9365) 2022-10-14 00:16:47 -07:00
Kay Yan 4701abff4c
upgrade-api-version-for-PodDisruptionBudget (#9369) 2022-10-10 17:51:02 -07:00
Rene Luria 3646dc0bd2
fix: remove trailing backslash and yaml indent (#9339)
* fix: remove trailing backslash

* fixed indent in cilium config template
2022-09-27 19:45:35 -07:00
biqiang Wu 31caab5f92
Fix: The Hubble certificate is faulty because the cluster name is hard coded (#9340)
Signed-off-by: dcwbq <biqiang.wu@daocloud.io>

Signed-off-by: dcwbq <biqiang.wu@daocloud.io>
2022-09-27 05:57:52 -07:00
Shelming.Song d62c67a5f5
allow user to set env: FELIX_MTUIFACEPATTERN in calico-node.yml (#9330) 2022-09-26 21:57:45 -07:00
Ho Kim 18efdc2c51
Fix typos in calico (#9327) 2022-09-26 00:11:44 -07:00
Krystian Młynek 8acd33d0df
Calico: add wireguard support for Rocky Linux 9 (#9287) 2022-09-20 00:29:20 -07:00
Necatican Yıldırım 7da3dbcb39
Cilium 1.12 Upgrade (#9225)
* Drop support for Cilium < 1.10

Signed-off-by: necatican <necaticanyildirim@gmail.com>

* Synchronize Cilium templates for 1.11.7

Signed-off-by: necatican <contact@necatican.com>

* Set Cilium v1.12.1 as the default version

Signed-off-by: necatican <contact@necatican.com>

Signed-off-by: necatican <necaticanyildirim@gmail.com>
Signed-off-by: necatican <contact@necatican.com>
2022-09-19 02:14:31 -07:00
Ho Kim 952cad8d63
Remove mutual exclusivity in calico: NAT and router mode (#9255)
* Add optional NAT support in calico router mode

* Add a blank line in front of lists

* Remove mutual exclusivity: NAT and router mode

* Ignore router mode from NAT

* Update calico doc
2022-09-13 00:19:07 -07:00
Ho Kim 09d9bc910e
Fix typos in calico comments (#9254) 2022-09-05 18:46:54 -07:00
lou-lan 133a7a0e1b
Add featureDetectOverride configration of calico (#9249) 2022-09-02 04:58:05 -07:00
蒋航 7ebb8c3f2e
make calico installation more stable (#9227)
Signed-off-by: hang.jiang <hang.jiang@daocloud.io>

Signed-off-by: hang.jiang <hang.jiang@daocloud.io>
2022-08-30 21:13:01 -07:00
tasekida 220f149299
Fix abort because calicoctl.sh is not a full path (#9217) 2022-08-30 08:07:02 -07:00
Florian Ruynat 617b17ad46
Fix kube_ovn_hw_offload value (#9218) 2022-08-30 03:21:01 -07:00
Chad Swenson de762400ad
Fixes for calico_datastore: etcd (#9228)
It seems that PR #8839 broke `calico_datastore: etcd` when it removed ipamconfig support for etcd mode.

This PR fixes some failing tasks when `calico_datastore == etcd`, but it does not restore ipamconfig support for calico in etcd mode. If someone wants to restore ipamconfig support for `calico_datastore: etcd` please submit a follow up PR for that.
2022-08-29 22:41:00 -07:00
Tristan bbd1161147
9035: Make Cilium rolling-restart delay/timeout configurable (#9176)
See #9035
2022-08-22 02:37:44 -07:00
Thearas ea29cd0890
add list nodes rules to cilium-operator clusterrole (#9178) 2022-08-18 01:02:36 -07:00
Samuel Liu b36bb9115a
[calico] calico rr supports multiple groups (#9134)
* update calico rr

* fix bgppeer conf

* fix yamllint

* fix ansible lint

* fix calico deploy

* fix yamllint

* fix some typo
2022-08-18 00:52:37 -07:00
Florian Ruynat 307f598bc8 Move flannel to etcd datastore 2022-08-02 16:55:52 -07:00
Florian Ruynat eb10249a75 Align canal templates with calico official ones (k8s datastore) 2022-08-02 16:55:52 -07:00
Denis Khachyan 8306adb102
update cilium to v1.11.7 (#9119) 2022-07-26 10:33:11 -07:00
忘尘 6525461d97
Add reset tasks specific to calico network_plugin (#9103) 2022-07-19 13:15:27 -07:00
Cyclinder 2e1863af78
feat: change default blockSize for calico (#9055)
Signed-off-by: cyclinder qifeng.guo@daocloud.io
2022-07-19 13:05:27 -07:00
Samuel Liu d821bed2ea
Fix some typo (#9056)
* fix ingress controller task name

* fix calico word

* add check typo
2022-07-11 09:49:48 -07:00
Mohamed Zaian a7ba7cdcd5
[calico] add v3.23.2 and make it default (#9041) 2022-07-08 10:41:48 -07:00
Emin AKTAS 5071529a74
feat: upgrade cilium and add default variables (#9065)
Signed-off-by: eminaktas <eminaktas34@gmail.com>
Signed-off-by: Emin Aktas <emin.aktas@trendyol.com>
2022-07-07 10:35:34 -07:00
Tom Stian Berget e1f06dd406
Add support for the updated (startup|liveness|readiness)Probe.Port numbers in Cilium (#9031) 2022-06-27 11:00:59 -07:00
orange-llajeanne b33896844e
apply calico bgp peer definition task to all nodes, but delegate to (#8974)
first control plane node
2022-06-24 19:42:57 -07:00
Cyclinder c3c9a42502
support multus multi-architecture installation (#9012)
Signed-off-by: cyclinder qifeng.guo@daocloud.io
2022-06-21 10:56:26 -07:00
Alessio Greggi 97b4d79ed5
feat: make kubernetes owner parametrized (#8952)
* feat: make kubernetes owner parametrized

* docs: update hardening guide with configuration for CIS 1.1.19

* fix: set etcd data directory permissions to be compliant to CIS 1.1.12
2022-06-17 01:34:32 -07:00
Calin Cristian Andrei 2de5c4821c [calico] clean up workarounds for older versions 2022-06-15 00:57:20 -07:00
Calin Cristian Andrei 9efe145688 [calico] make 3.23.1 the default and drop 3.20.x and 3.19.x 2022-06-15 00:57:20 -07:00
Viktor Jacynycz 77de7cb785
Expose calico-typha metrics port (#8855) 2022-06-14 07:17:33 -07:00
Ho Kim 889454f2bc
Fix typo in calico check (#8969) 2022-06-13 14:10:12 -07:00
Ilya Margolin cc6cbfbe71
Allow disabling calico CNI logs with calico_cni_log_file_path (#8921)
* Allow disabling calico CNI logs with calico_cni_log_file_path

Calico CNI logs up to 1G if it log a lot with current default settings:
log_file_max_size	100	Max file size in MB log files can reach before they are rotated.
log_file_max_age	30	Max age in days that old log files will be kept on the host before they are removed.
log_file_max_count	10	Max number of rotated log files allowed on the host before they are cleaned up.

See https://projectcalico.docs.tigera.io/reference/cni-plugin/configuration#logging

To save disk space, make the path configurable and allow disabling this log by setting
`calico_cni_log_file_path: false`

* Fix markdown

* Update roles/network_plugin/canal/templates/cni-canal.conflist.j2

Co-authored-by: Kenichi Omichi <ken1ohmichi@gmail.com>

Co-authored-by: Kenichi Omichi <ken1ohmichi@gmail.com>
2022-06-07 09:22:56 -07:00
mahjonp 8030e6f76c
fix 8893#issuecomment-1147154353 (#8933)
Signed-off-by: mahjonp <junpeng.man@gmail.com>
2022-06-06 12:40:21 -07:00
Kenichi Omichi 814760ba25
Use blocks for macvlan tasks for each distribution (#8918)
For the code readability, this adds blocks for each distribution.
2022-06-06 07:50:24 -07:00
vanyasvl a4f752fb02
Add subjectAltName to calico-apiserver certificate (#8907)
* Add AltName to calico-apiserver certificate

* fix support for centos7 openssl
2022-06-06 07:38:23 -07:00
Daniil Muidinov 5c136ae3af
[calico] add 3.22.3 and 3.23.1 (#8897)
* [calico]
* add 3.22.3 and 3.23.1
* set 3.22.3 default
* fix download crd for calico 3.22.3 and upper

* update calico README.md
2022-05-31 13:27:23 -07:00
mahjonp c927da00e0
Support cilium ip-masq-agent configuration (#8893)
* fix deploy Cilium with eBPF-based Masquerading failed

Signed-off-by: mahjonp <junpeng.man@gmail.com>

* forget to add the enable-ip-masq-agent flag

Signed-off-by: mahjonp <junpeng.man@gmail.com>
2022-05-31 09:26:53 -07:00
Kenichi Omichi dc2a18e436
Merge pull request #8815 from simplekube-ro/dont_clobber_calico
[calico] don't clobber calico options set by the user
2022-05-24 10:25:48 -07:00
Ross Kusler 4c97ce747c
Adding support for the kube-router flag --cluster-asn flag (#8837) 2022-05-23 16:39:10 -07:00
Tamas Pasztor 9d3a894991
Possible remove ippools from cni config (#8845)
* Possible remove ippools from cni config

* Typo

* Update roles/network_plugin/calico/templates/cni-calico.conflist.j2

Co-authored-by: Kenichi Omichi <ken1ohmichi@gmail.com>

* Update cni-calico.conflist.j2

Incorrectly deleted calico forwarding content.

* Update roles/network_plugin/calico/templates/cni-calico.conflist.j2

Co-authored-by: Kenichi Omichi <ken1ohmichi@gmail.com>

Co-authored-by: Kenichi Omichi <ken1ohmichi@gmail.com>
2022-05-19 23:45:13 -07:00
Samuel Liu a28b58dbd0
[calico]use ipamconfig instead of calico ipam command (#8839)
* use ipamconfig instead of calico ipam command

* fix ansible lint
2022-05-19 11:13:20 -07:00
Samuel Liu eea7bb7692
only need run this once (#8833)
calicoctl ipam xx
calicoctl apply xx
2022-05-17 09:52:27 -07:00
Calin Cristian Andrei 569a319ff5 [calico] don't clobber user set bgp configuration options that are not managed by kubespray 2022-05-12 15:50:38 +00:00
Calin Cristian Andrei 47812ec002 [calico] don't clobber user set ippool options that are not managed by kubespray 2022-05-12 15:50:05 +00:00
Calin Cristian Andrei c27dee57ea [calico] don't clobber user set felixconfig options that are not managed by kubespray 2022-05-12 15:49:24 +00:00
Cyclinder 3eb0a4071a
set default value of name to "k8s-pod-network" (#8813)
Signed-off-by: cyclinder qifeng.guo@daocloud.io
2022-05-12 08:29:14 -07:00
Samuel Liu f26f544ff6
[kube-ovn]: update kube-ovn version and sync some feature (#8790)
* [kube-ovn]: some feature

kube-ovn vlan mode
ipv6/ipv4 dual stack
...

* remove unused env

* fix readinessprobe
2022-05-11 21:35:15 -07:00
Necatican Yıldırım 13443b05a6
Overhaul Cilium manifests to match the newer versions (#8717)
* [cilium] Separate templates for cilium, cilium-operator, and hubble installations

Signed-off-by: necatican <necaticanyildirim@gmail.com>

* [cilium] Update cilium-operator templates

Signed-off-by: necatican <necaticanyildirim@gmail.com>

* [cilium] Allow using custom args and mounting extra volumes for the Cilium Operator

Signed-off-by: necatican <necaticanyildirim@gmail.com>

* [cilium] Update the cilium configmap to filter out the deprecated variables, and add the new variables

Signed-off-by: necatican <necaticanyildirim@gmail.com>

* [cilium] Add an option to use Wireguard encryption on Cilium 1.10 and up

Signed-off-by: necatican <necaticanyildirim@gmail.com>

* [cilium] Update cilium-agent templates

Signed-off-by: necatican <necaticanyildirim@gmail.com>

* [cilium] Bump Cilium version to 1.11.3

Signed-off-by: necatican <necaticanyildirim@gmail.com>
2022-05-11 06:23:04 -07:00
Samuel Liu 1294fd5730
check calico ipv6 (#8738)
* check calico ipv6

* just check ipip mode for ipv6
2022-04-29 00:35:13 -07:00
zhengtianbao 937e64d296
Update flannel use install-cni-plugin to fit upstream (#8714)
* Update flannel use install-cni-plugin to fit upstream

* Replace flannel cni repo

* Remove download flannel binary
2022-04-18 09:44:41 -07:00
Cristian Calin 45262da726
[calico] call calico checks early on to prevent altering the cluster with bad configuration (#8707) 2022-04-14 01:08:46 -07:00
Samuel Liu 424ef3b3f9
[calico] add calico apiserver (#8690)
* [calico] add calico apiserver

* fix yamllint

* remove addext argument

* Configure API server with the CA bundle

* add check kdd
2022-04-08 00:02:42 -07:00
Cristian Calin dd2d95ecdf
[calico] don't enable ipip encapsulation by default and use vxlan in CI (#8434)
* [calico] make vxlan encapsulation the default

* don't enable ipip encapsulation by default
* set calico_network_backend by default to vxlan
* update sample inventory and documentation

* [CI] pin default calico parameters for upgrade tests to ensure proper upgrade

* [CI] improve netchecker connectivity testing

* [CI] show logs for tests

* [calico] tweak task name

* [CI] Don't run the provisioner from vagrant since we run it in testcases_run.sh

* [CI] move kube-router tests to vagrant to avoid network connectivity issues during netchecker check

* service proxy mode still fails connectivity tests so keeping it manual mode

* [kube-router] account for containerd use-case
2022-03-17 18:05:39 -07:00
Qasim Mehmood 5a486a5cca
Calico: Fix Wireguard support for CentOS Stream 9/RHEL 9 Beta (#8625) 2022-03-17 04:11:20 -07:00
Toni Tauro 5a49ac52f9
feat(calico): add configurable ipam strictaffinity (#8581)
Signed-off-by: Toni Tauro <toni.tauro@adfinis.com>
2022-03-07 22:58:33 -08:00
Tom Janson ddef7e1139
missing "check_mode: no"s for several read-only tasks (#8584)
this is not complete -- there are almost certainly more instances of
this issue
2022-03-02 09:29:14 -08:00