Commit Graph

467 Commits (a0a2f402955ecfc72ecfa8c0b2948d00da464dea)

Author SHA1 Message Date
Kenichi Omichi ed38d8d3a1
Add ingress-nginx check for updating README (#9533)
To detect the version mismatch.
2022-12-01 01:16:27 -08:00
Kenichi Omichi cbf4586c4c
Specify Quick mode for sonobuoy test (#9499)
The certified-conformance mode took 2+ hours and that was too long
by comparing Quick mode which was specified previously.
So this updates the mode to Quick again.
2022-11-16 21:54:39 -08:00
Kenichi Omichi b60f65c1e8
Update sonobuoy version (#9485)
The latest version of sonobuoy is v0.56.11.
This updates the version to the latest.

As the file name, this makes it use certified-conformance mode
clearly for the latest version of sonobuoy.
2022-11-15 00:46:41 -08:00
William Turner eeb376460d
Fix inconsistent handling of admission plugin list (#9407)
* Fix inconsistent handling of admission plugin list

* Adjust hardening doc with the normalized admission plugin list

* Add pre-check for admission plugins format change

* Ignore checking admission plugins value when variable is not defined
2022-10-26 00:28:37 -07:00
Kenichi Omichi e5ec0f18c0
Add packet_ubuntu20-calico-aio-hardening (#9359)
To verify the hardening method works always.
The configuration comes from docs/hardening.md

Fix yaml format of hardening.yml

Add condition to skip 040 test for hardening
2022-10-19 05:35:04 -07:00
Kenichi Omichi 72b45eec2e
Use agnhost instead of busybox for network test (#9390)
busybox container requires a root permission for ping.
For testing hardening method at CI, we need to switch to another image
which doesn't require the root permission for network testing.
On kubernetes/kubernetes repo, we are using agnhost which doesn't
require it. So this makes the test use aghhost image.

In addition, this updates the test manifest to specify securityContext
without any privilege.
2022-10-14 06:10:46 -07:00
Kenichi Omichi 24632ae81b
Add check_typo job (#9361)
To block merging pull requests which contain typo automatically.
2022-10-07 02:21:53 -07:00
Kay Yan 97ca2f3c78
add-timezone-support (#9263) 2022-09-14 21:11:22 -07:00
Kenichi Omichi 5fa885b150
Remove unused cri_dockerd_enabled configuration (#9259)
Since the commit fad296616c cri_dockerd_enabled
has not been used. But the packet_ubuntu22-aio-docker.yml still contains
the configuration and causes confusions.
This removes the configuration for cleanup.
2022-09-08 00:06:05 -07:00
Kay Yan e2f1f8d69d
add-Rocky-9-support (#9212) 2022-09-04 16:54:36 -07:00
Florian Ruynat 1baabb3c05
Fix cloud_init files for different distros (#9232) 2022-08-30 08:03:02 -07:00
ERIK 9ad2d24ad8
Add unsafe_show_logs switch (#9164)
Signed-off-by: bo.jiang <bo.jiang@daocloud.io>

Signed-off-by: bo.jiang <bo.jiang@daocloud.io>
2022-08-16 18:52:48 -07:00
Kenichi Omichi 7e862939db
Add kube-vip check to check_readme_versions.sh (#9155)
To check the kube-vip version between readme.md and the default value
on the role, this updates check_readme_versions.sh
2022-08-06 08:26:20 -07:00
Florian Ruynat 4df6e35270 Move oracle7-canal to centos7-canal 2022-08-02 16:55:52 -07:00
Florian Ruynat 7fa6314791
Add ignore_assert_error to ubuntu20 etcd ha job (#9108) 2022-07-26 10:45:09 -07:00
Florian Ruynat ff5e487e32 Add retries to api servers response 2022-07-21 23:03:38 -07:00
Florian Ruynat 9c51ac5157 Switch fedora36se to 35 and 35docker to 36 2022-07-21 23:03:38 -07:00
Florian Ruynat 07eab539a6 Add Fedora 36 support and CI, remove Fedora 34 (eol) 2022-07-21 23:03:38 -07:00
Cyclinder 2e1863af78
feat: change default blockSize for calico (#9055)
Signed-off-by: cyclinder qifeng.guo@daocloud.io
2022-07-19 13:05:27 -07:00
Cyclinder e6154998fd
fix calico tunl0 routes test (#9061)
Signed-off-by: cyclinder qifeng.guo@daocloud.io
2022-07-06 04:52:49 -07:00
Kenichi Omichi 7414409aa0
Add target components on check_readme_versions.sh (#9045)
This adds target components on check_readme_versions.sh after
merging https://github.com/kubernetes-sigs/kubespray/pull/9044
In addition, this fixes typo on check_readme_versions.sh

This adds `foo_version` variables for some components because
check_readme_versions.sh verifies the corresponding version for
`<component name>_version` from main.yml. This change also makes
consistency in the main.yml. In long-term, we will be able to
remove the existing `foo_image_tag` variables, but that is not now
for backwards compatibility for users.
2022-07-05 08:02:47 -07:00
Kay Yan adfd77f11d
add-test-for-kubeadm-etcd-deployment (#9007) 2022-07-05 07:58:47 -07:00
Kenichi Omichi f3ea8cf45e
Add Rocky Linux 8 support for vagrant (#8905)
To test Kubespray on Rocky Linux 8 with vagrant, this adds it to
the Vagrantfile.
2022-07-05 07:50:47 -07:00
Kenichi Omichi 6cd243f14e
Add component version check for README.md (#9042)
During code-review, reviwers needed to take care of README.md also
should be updated when the pull request updated component versions.
This adds the corresponding check to reduce reviwer's burden.
2022-06-29 00:14:05 -07:00
Kay Yan 4b03f6c20f
add-managed-ntp-support (#9027) 2022-06-28 13:15:34 -07:00
Samuel Liu e8ccbebd6f
add ingress nginx webhook (#9033)
* add ingress nginx webhook

* fix ingress nginx template
2022-06-28 11:55:35 -07:00
Calin Cristian Andrei 56d83c931b [CI] use debian-11 image with more disk space to ensure successful upgrade tests 2022-06-17 08:00:32 -07:00
Calin Cristian Andrei a22ae6143a [CI] ensure upgrade tests cover defaults (containerd currently) 2022-06-17 08:00:32 -07:00
Calin Cristian Andrei 5dc8be9aa2 [CI] kube 1.24 requires at least 1775Mi of memory, might as well leave the default of 2048 2022-06-15 00:57:20 -07:00
Calin Cristian Andrei fad296616c [docker] use cri-dockerd instead of dockershim for any kubernetes version deployed with docker as the container_manager 2022-06-15 00:57:20 -07:00
Kay Yan 85271fc2e5
add-ci-for-ubuntu2204 (#8958) 2022-06-15 00:47:19 -07:00
Kay Yan 4726a110fc
remove-support-for-ansible-2.9-2.10 (#8951) 2022-06-10 03:35:47 -07:00
Kenichi Omichi 024a3ee551
Replace callback_whitelist with callbacks_enabled (#8759)
When running molecule jobs, we saw the folloing warning message:

 [DEPRECATION WARNING]: [defaults]callback_whitelist option, normalizing names
 to new standard, use callbacks_enabled instead. This feature will be removed
 from ansible-core in version 2.15. Deprecation warnings can be disabled by
 setting deprecation_warnings=False in ansible.cfg.

callbacks_enabled has been added since Ansible 2.11 and Kubespray is using
Ansible 2.12 at master branch. So we can use callbacks_enabled safely to
avoid the warning message.
2022-06-09 13:15:45 -07:00
Kenichi Omichi cd7381d8de
Drop Ansible support for v2.9 and v2.10 (#8925)
Ansible v2.9 and v2.10 are EOL as [1].
This drops those version supports by following the upstream Ansible.

This sets use_ssh_args true always because that is required to use
ssh_args on ansible.cfg on Ansible v2.11 or later[2].

ansible_ssh_host is replaced with ansible_host because ansible_ssh_host
has been deprecated already and cenots7 jobs were failed due to the
deprecated ansible_ssh_host.

[1]: https://docs.ansible.com/ansible/devel/reference_appendices/release_and_maintenance.html#ansible-core-changelogs
[2]: https://docs.ansible.com/ansible/latest/collections/ansible/posix/synchronize_module.html#parameter-use_ssh_args
2022-06-09 07:07:42 -07:00
Mohamed Zaian bb530da5c2 [registry] Switch registry to use registry.k8s.io
Please see the conversation here: https://groups.google.com/a/kubernetes.io/g/dev/c/DYZYNQ_A6_c
2022-06-08 14:12:22 +02:00
Kay Yan 3d8f3bc0b7
Fix the invalid kube vip manifest (#8831)
* add Feature synchronized time checking

* fix-invalid-kube-vip-manifest
2022-05-17 23:48:55 -07:00
Cristian Calin 3a89e31dee
[ansible] update ansible and cryptography requirements to work on ubuntu 22.04 (#8826) 2022-05-16 11:14:17 -07:00
Calin Cristian Andrei 9143810a4d [CI] add remove node job 2022-05-04 06:35:51 -07:00
Florian Ruynat 74c7e009b7
Move flannel to kubespray/quay for CI (#8774) 2022-05-04 00:11:30 -07:00
Kenichi Omichi f184725c5f
Use ansible 2.12 for testcases_prepare (#8763)
tests/requirements.txt links to tests/requirements-2.12.txt, so
Kubespray uses ansible 2.12 by default for testing. However we
forgot to update testcases_prepare.sh to use ansible 2.12.
This updates testcases_prepare to use ansible 2.12.
2022-05-02 11:34:31 -07:00
Cristian Calin 835fd86a08
[CI] split molecule testes to run in parallel (#8756)
* add parametrization to molecule_run.sh

* [CI] split molecule tests to allow parallelization of work
2022-04-29 00:09:12 -07:00
Cristian Calin aa12f1c56b
[CI] fix packet_ubuntu20-calico-etcd-kubeadm-upgrade-ha job (#8752) 2022-04-27 12:39:36 -07:00
Cristian Calin 3261d26181
[etcd] ensure etcd is properly upgraded when managed by kubeadm (#8722)
* [etcd] ensure etcd is properly upgraded when managed by kubeadm

* [CI] add periodic job to test upgrade of etcd managed by kubeadm
2022-04-17 10:32:41 -07:00
Samuel Liu 424ef3b3f9
[calico] add calico apiserver (#8690)
* [calico] add calico apiserver

* fix yamllint

* remove addext argument

* Configure API server with the CA bundle

* add check kdd
2022-04-08 00:02:42 -07:00
Florian Ruynat 6af9cae0a5
Add missing 2.10 ansible test (#8665) 2022-03-30 08:12:27 -07:00
Cristian Calin ef29455652
[ansible] make ansible 5.x the new default version (#8660)
* [ansible] make ansible 5.x the new default version and move different versions tested to nightly jobs

* [CI] jobs were missing proper ansible cleanup
2022-03-29 15:36:11 -07:00
Cristian Calin 113de8381c
[ansible] add support for ansible 5 (ansible-core 2.12) (#8512) 2022-03-28 08:49:22 -07:00
Florian Ruynat 97cb64c62d Remove k8s module for ns creation 2022-03-25 16:57:58 -07:00
Cristian Calin dd2d95ecdf
[calico] don't enable ipip encapsulation by default and use vxlan in CI (#8434)
* [calico] make vxlan encapsulation the default

* don't enable ipip encapsulation by default
* set calico_network_backend by default to vxlan
* update sample inventory and documentation

* [CI] pin default calico parameters for upgrade tests to ensure proper upgrade

* [CI] improve netchecker connectivity testing

* [CI] show logs for tests

* [calico] tweak task name

* [CI] Don't run the provisioner from vagrant since we run it in testcases_run.sh

* [CI] move kube-router tests to vagrant to avoid network connectivity issues during netchecker check

* service proxy mode still fails connectivity tests so keeping it manual mode

* [kube-router] account for containerd use-case
2022-03-17 18:05:39 -07:00
Cristian Calin 394857b5ce
[docker] add support for cri-dockerd as a replacement for dockershim (#8623) 2022-03-16 16:28:11 -07:00
Takuya Murakami 39acb2b84d
Update ansible-lint to 5.4.0 (#8607) (#8608)
* Update ansible-lint to 5.4.0 (#8607)

It seems that the Rich version 11.0.0 has a breaking change.
So need to update ansible-lint to 5.3.2 or later.

* Fix for ansible-lint no-changed-when rule (#8607)
2022-03-07 05:35:55 -08:00
Cristian Calin a090038d02
[CI] add ara to collect CI job logs (#8545) 2022-02-23 07:36:19 -08:00
Necatican Yıldırım e9c8913248
Add kubeadm option to etcd_deployment_type to replace the etcd_kubeadm_enabled variable (#8317)
* Add kubeadm option to etcd_deployment_type to replace the etcd_kubeadm_enabled variable

Signed-off-by: necatican <necaticanyildirim@gmail.com>

* Add etcd kubeadm deployment documentation

Signed-off-by: necatican <necaticanyildirim@gmail.com>

* Refactor warning for the deprecated 'etcd_kubeadm_enabled' variable

Signed-off-by: necatican <necaticanyildirim@gmail.com>
2022-02-22 08:53:16 -08:00
Florian Ruynat d580014c66
Fix CI for Fedora (followup) + OpenSUSE Leap (update to 15.3) (#8407)
* Fix fedora jobs - followup

* Update OpenSUSE Leap to 15.3

* Fix cilium version in README + update minor 1.11.1
2022-01-24 23:24:30 -08:00
Cristian Calin ef34f5fe7d
[calico] switch default iptables backend detection to Auto (#8429) 2022-01-23 23:47:57 -08:00
Florian Ruynat d007132655
Fix Fedora CI following ipset version in kube-proxy for k8s 1.23 (#8397) 2022-01-11 05:01:17 -08:00
Cristian Calin 3eab1129b9
CI: Replace CentOS 8 with AlmaLinux 8 before CentOS 8 EOL end of 2021 (#8297) 2022-01-05 02:20:33 -08:00
Florian Ruynat 2f9a8c04dc
Add nginx_image_repo to mirrored image on quay (#8364) 2022-01-03 10:03:00 -08:00
Cristian Calin 4daa824b3c
CI: fix test name debian10-aio was a 2 instance default (#8286)
* CI: fix test name debian10-aio was a 2 instance default

* CI: Fix running ubuntu20-aio-docker

* CI: Fix running ubuntu18-aio-docker
2021-12-13 14:50:25 -08:00
Cristian Calin 682c8a59c2
containerd: change default resolvconf_mode to host_resolvconf (#8247)
* containerd: change default resolvconf_mode to host_resolvconf

* Wait for kube-apiserver to come back after pod refresh

* Handle resolv.conf gracefully

* Retain currently configured DNS entries to ensure we don't break the resolvers

* Suse uses wickedd for network management so no dhcp hooks

* Molecule: increase ansible timeout

* CI: Increase ansible timeout to 120s for Packet jobs
2021-12-09 14:09:06 -08:00
Florian Ruynat e19ce27352
Remove ovn4nfv support (#8265) 2021-12-03 11:56:35 -08:00
Florian Ruynat e82443241b
Move opensuse CI to docker and fix ubuntu16 containerd version for docker (#8257) 2021-12-02 08:01:34 -08:00
Florian Ruynat b38382a68f
Move cri-o default package to 1.22 (#8258) 2021-12-02 06:21:34 -08:00
Florian Ruynat 30ec03259d
Remove fedora33 - eol (#8246) 2021-11-30 15:53:17 -08:00
Cristian Calin ee882fa462
Add capability to use swap, requires Kube 1.22 (#8241)
* Alpha-NodeSwap: allow nodes to use swap

* CI: Add Fedora 35 with experimental swap job
2021-11-30 00:52:56 -08:00
Cristian Calin 2f44b40d68
OEL7: Fix CentOS7 Extras for OEL7 (#8219)
* OEL7: Fix CentOS7 Extras for OEL7

* Molecule: add logs collection for jobs
2021-11-29 13:39:21 -08:00
Florian Ruynat a5f88e14d0
Cleanup tests (#8234)
* Add Fedora 35 image, support and CI

* Cleanup tests and allow_failure for vagrant
2021-11-26 09:00:51 -08:00
Cristian Calin e78bda65fe
Defaults: replace docker with containerd as our default container_manager (#8175)
* Defaults: replace docker with containerd as our default container_manager

* CI: Use docker for download_localhost test

* Defaults: with container_manager=containerd we need etcd_deployment_type=host

* CI: Run weave jobs with docker

* CI: Vagrant don't download_force_cache

* CI: Fix upgrade tests

* should run compatible with old settings, this means docker
* we need to run with a distro that has at least modern containerd,
  this means move from debian9 to debian10 to allow `containerd_version`
  to match between 2.17 and master
2021-11-25 06:54:33 -08:00
Cristian Calin c74e1c9db3
CI: use images from quay.io to prevent being throttled by docker hub (#8209)
* CI: use netchecker images from quay to prevent throttling

* Molecule: use hello-world image from quay.io
2021-11-19 13:23:40 -08:00
Cristian Calin 28b5281c45
Python: bring back python 2.7 support for ansible 2.9 in supported EL distributions (#8192) 2021-11-15 08:06:48 -08:00
EDGsheryl 4d79a55904
Remove extra parameter kube_proxy_remove (#8158)
Signed-off-by: EDGsheryl <edgsheryl@gmail.com>
2021-11-15 00:02:48 -08:00
Cristian Calin 0d0468e127
Exercise multiple ansible versions in CI (#8172)
* Ansible: separate requirements files for supported ansible versions

* Ansible: allow using ansible 2.11

* CI: Exercise Ansible 2.9 and Ansible 2.11 in a basic AIO CI job

* CI: Allow running a reset test outside of idempotency tests and running it in stage1

* CI: move ubuntu18-calico-aio job to stage2 and relay only on ubuntu20 with the variously supported ansible versions for stage1

* CI: add capability to install collections or roles from ansible-galaxy to mitigate missing behavior in older ansible versions
2021-11-10 16:11:50 -08:00
Cristian Calin 039205560a
nodelocaldns: allow a secondary pod for nodelocaldns for local-HA (#8100)
* nodelocaldns: allow a secondary pod for nodelocaldns for local-HA

* CI: add job to test nodelocaldns secondary
2021-11-09 09:57:47 -08:00
Cristian Calin 4a8757161e
Docker: replace the use of containerd_version with docker_containerd_version to avoid causing conflicts when bumping containerd_version (#8130) 2021-11-08 15:56:49 -08:00
Cristian Calin 8d553f7e91
Mitogen: deprecate the use of mitogen and remove coverage from CI (#8147) 2021-11-05 00:57:52 -07:00
Cristian Calin 2ed211ba15
Fix-CI: python was upgraded in CI to 3.10 and pathlib is now included in python base making this dependency break the CI (#8153) 2021-11-03 12:52:32 -07:00
Florian Ruynat 331647f4ab
Remove deprecated Ambassador ingress code (#8086) 2021-10-26 15:19:09 -07:00
Cristian Calin ea8e2fc651
containerd: download containerd from upstream instead of using distro specific packages (#7970)
* Containerd: download containerd from upstream instead of using distro specific packages

split runc download to separate role
make bootstrap-os role deploy container-selinux and seccomp libraries
clean up package manager provided containerd
move variables to docker role that are no longer common with containerd

* Containerd: make molecule testing more relevant

* replace ubuntu18 with ubuntu20
* add centos8 and debian11 to molecule tests
* run kubernetes/preinstall role to ensure relevancy
  of test including dependency packages

* CI: adjust test scenarios for downloaded containerd
2021-10-20 08:47:58 -07:00
Iago Santos 43958614e3
Fix kubespray flatcar ansible_os_family and ansible_distribution (#8029)
Closes https://github.com/kubernetes-sigs/kubespray/issues/8028

Signed-off-by: Iago Santos <iago.santos.pardo@adfinis.com>
2021-10-01 09:11:23 -07:00
Cristian Calin 33146b9481
CI: Add Calico eBPF in HA mode test (#7710)
* Sample-Inventory: add sample for calico_bpf_enabled

* Calico-Doc: note about CONFIG_NET_SCHED for eBPF support

* CI: Add Calico eBPF in HA mode test
2021-09-24 09:57:23 -07:00
Florian Ruynat 30a7dfa4f8
Fix ubuntu16/centos8 CI jobs (#7972) 2021-09-16 23:39:01 -07:00
Cristian Calin d57ddf0be8
Feature DynamicKubeletConfig is deprecated in 1.22 and will not move to GA (#7938)
* Feature DynamicKubeletConfig is deprecated in 1.22 and will not move to GA

* Add check for dynamic_kubelet_configuration with kube >= 1.22
2021-09-07 10:47:16 -07:00
Florian Ruynat c7529270ff
Fix CI script for Terraform >0.15 (#7928) 2021-09-01 04:30:59 -07:00
Florian Ruynat 207d3e7b4e
Add Debian-11 image and CI (#7919) 2021-08-31 14:02:22 -07:00
Florian Ruynat 77a74adedd
Bump centos8 CI job memory to 3go and remove mitogen for fedora CI (#7921) 2021-08-30 08:25:13 -07:00
Cristian Calin 1afdb05ea9
Fedora and RHEL use etc_t and the convention is <type_name>_t (#7891)
* Fedora and RHEL use etc_t and the convention is <type_name>_t

* Docs: specify all values for preinstall_selinux_state

* CI: Add Fedora 34 with SELinux in enforcing mode
2021-08-27 14:20:53 -07:00
jayonlau 6b5da84014
Clean up extra spaces last one (#7904)
Although these errors are not important, they affect the code specification.
2021-08-25 12:06:09 -07:00
Cristian Calin 7516fe142f
Move to Ansible 3.4.0 (#7672)
* Ansible: move to Ansible 3.4.0 which uses ansible-base 2.10.10

* Docs: add a note about ansible upgrade post 2.9.x

* CI: ensure ansible is removed before ansible 3.x is installed to avoid pip failures

* Ansible: use newer ansible-lint

* Fix ansible-lint 5.0.11 found issues

* syntax issues
* risky-file-permissions
* var-naming
* role-name
* molecule tests

* Mitogen: use 0.3.0rc1 which adds support for ansible 2.10+

* Pin ansible-base to 2.10.11 to get package fix on RHEL8
2021-07-12 00:00:47 -07:00
Florian Ruynat d1388d69d0
Fix tests following python change (#7775)
* Fix ansible detection for python3 and ubuntu

* Fix oracle missing centos-extras repo for containerd/docker dependencies
2021-07-08 18:52:53 -07:00
Cristian Calin a2cf6816ce
Calico wireguard (#7638)
* Calico: add Wireguard support

* CI: Add Calico Wireguard scenario
2021-06-25 03:22:45 -07:00
Kenichi Omichi 8d7327c188
Remove old groups from test inventory (#7656)
We have released v2.16 of Kubespray already, so we can remove those
old groups from the test inventory as the TODO says.
2021-06-09 02:45:48 -07:00
Florian Ruynat 54cda80018
Fix debian docker available version (#7668) 2021-06-01 20:58:39 -07:00
Florian Ruynat 7896bc7831
Add Fedora 33 image and CI, remove Fedora 31 (EOL) + update docker packages (#7657)
* Update docker package to 20.10.6

* Add Fedora 33 image and CI, remove Fedora 31 (EOL)
2021-05-28 08:04:25 -07:00
Florian Ruynat fd8ae54fa7 Docker default version is now 20.10 2021-05-27 11:18:24 -07:00
Fatih Sarhan 59fc17f4e3
Override the default value of containerd's root, state, and oom_score (#7622)
* Override the default value of containerd's root, state, and oom_score configurations

* Add tests data for containerd_storage_dir, containerd_state_dir and containerd_oom_score variables
2021-05-19 08:24:53 -07:00
Samuel Liu 96e6a6ac3f
Add krew support (#7464)
* Add krew support

* Add reset for krew

* Update install krew(local)

* ansible lint

* yamllint

* fix krew default vars

* fix kubectl_localhost mode

* replace include

* fix e206
2021-05-03 07:16:03 -07:00
Florian Ruynat d29ea386d6
Fix issue with api token wait check not working (#7566) 2021-04-30 07:47:59 -07:00
Florian Ruynat ffc38a2237
Fix busybox for tests to reduce dockerhub calls (#7571) 2021-04-29 17:39:57 -07:00
Cristian Calin 360aff4a57
Rename ansible groups to use _ instead of - (#7552)
* rename ansible groups to use _ instead of -

k8s-cluster -> k8s_cluster
k8s-node -> k8s_node
calico-rr -> calico_rr
no-floating -> no_floating

Note: kube-node,k8s-cluster groups in upgrade CI
      need clean-up after v2.16 is tagged

* ensure old groups are mapped to the new ones
2021-04-29 05:20:50 -07:00
Florian Ruynat 42382e2cde
Update Terraform/Vagrant + increase tf_ovh retries (#7477) 2021-04-12 09:47:39 -07:00