Commit Graph

214 Commits (2ec1c93897dadf2141c8a40595a6216d68b2516d)

Author SHA1 Message Date
Kenichi Omichi 9c28f61dbd
Enable shellcheck for contrib/ (#9122)
Today we have many contributions to contrib/offline/ and some PRs
contained invalid coding style for those scripts.
This enables shellcheck to make such invalid coding style easily.
2022-07-26 23:32:32 -07:00
Florian Ruynat 9c51ac5157 Switch fedora36se to 35 and 35docker to 36 2022-07-21 23:03:38 -07:00
Florian Ruynat 07eab539a6 Add Fedora 36 support and CI, remove Fedora 34 (eol) 2022-07-21 23:03:38 -07:00
Kay Yan adfd77f11d
add-test-for-kubeadm-etcd-deployment (#9007) 2022-07-05 07:58:47 -07:00
Kenichi Omichi f3ea8cf45e
Add Rocky Linux 8 support for vagrant (#8905)
To test Kubespray on Rocky Linux 8 with vagrant, this adds it to
the Vagrantfile.
2022-07-05 07:50:47 -07:00
Kenichi Omichi 6cd243f14e
Add component version check for README.md (#9042)
During code-review, reviwers needed to take care of README.md also
should be updated when the pull request updated component versions.
This adds the corresponding check to reduce reviwer's burden.
2022-06-29 00:14:05 -07:00
Calin Cristian Andrei 56d83c931b [CI] use debian-11 image with more disk space to ensure successful upgrade tests 2022-06-17 08:00:32 -07:00
Calin Cristian Andrei 589823bdc1 [CI] remove docker stand-alone molecule test 2022-06-15 00:57:20 -07:00
Kay Yan 85271fc2e5
add-ci-for-ubuntu2204 (#8958) 2022-06-15 00:47:19 -07:00
Kay Yan 4726a110fc
remove-support-for-ansible-2.9-2.10 (#8951) 2022-06-10 03:35:47 -07:00
Calin Cristian Andrei 9143810a4d [CI] add remove node job 2022-05-04 06:35:51 -07:00
Cristian Calin 835fd86a08
[CI] split molecule testes to run in parallel (#8756)
* add parametrization to molecule_run.sh

* [CI] split molecule tests to allow parallelization of work
2022-04-29 00:09:12 -07:00
Cristian Calin 6cc5b38a2e
[terraform] use modern day equinix metal provider (#8748)
* [terraform] use modern day equinix metal provider

* [CI] ensure packet job tests metal
2022-04-27 10:34:13 -07:00
Cristian Calin 3261d26181
[etcd] ensure etcd is properly upgraded when managed by kubeadm (#8722)
* [etcd] ensure etcd is properly upgraded when managed by kubeadm

* [CI] add periodic job to test upgrade of etcd managed by kubeadm
2022-04-17 10:32:41 -07:00
Cristian Calin ef29455652
[ansible] make ansible 5.x the new default version (#8660)
* [ansible] make ansible 5.x the new default version and move different versions tested to nightly jobs

* [CI] jobs were missing proper ansible cleanup
2022-03-29 15:36:11 -07:00
Cristian Calin 113de8381c
[ansible] add support for ansible 5 (ansible-core 2.12) (#8512) 2022-03-28 08:49:22 -07:00
Cristian Calin dd2d95ecdf
[calico] don't enable ipip encapsulation by default and use vxlan in CI (#8434)
* [calico] make vxlan encapsulation the default

* don't enable ipip encapsulation by default
* set calico_network_backend by default to vxlan
* update sample inventory and documentation

* [CI] pin default calico parameters for upgrade tests to ensure proper upgrade

* [CI] improve netchecker connectivity testing

* [CI] show logs for tests

* [calico] tweak task name

* [CI] Don't run the provisioner from vagrant since we run it in testcases_run.sh

* [CI] move kube-router tests to vagrant to avoid network connectivity issues during netchecker check

* service proxy mode still fails connectivity tests so keeping it manual mode

* [kube-router] account for containerd use-case
2022-03-17 18:05:39 -07:00
Kenichi Omichi 73c889eb10
Fix failures of ansible-lint (#8401)
This fixes the following types of failures:
- empty-string-compare
- literal-compare
- risky-file-permissions
- risky-shell-pipe
- var-spacing

In addition, this changes .gitlab-ci/lint.yml to block the same issue
by using the same method at Kubespray CI.
2022-01-11 00:45:16 -08:00
Cristian Calin 3eab1129b9
CI: Replace CentOS 8 with AlmaLinux 8 before CentOS 8 EOL end of 2021 (#8297) 2022-01-05 02:20:33 -08:00
Cristian Calin 4daa824b3c
CI: fix test name debian10-aio was a 2 instance default (#8286)
* CI: fix test name debian10-aio was a 2 instance default

* CI: Fix running ubuntu20-aio-docker

* CI: Fix running ubuntu18-aio-docker
2021-12-13 14:50:25 -08:00
Cristian Calin 682c8a59c2
containerd: change default resolvconf_mode to host_resolvconf (#8247)
* containerd: change default resolvconf_mode to host_resolvconf

* Wait for kube-apiserver to come back after pod refresh

* Handle resolv.conf gracefully

* Retain currently configured DNS entries to ensure we don't break the resolvers

* Suse uses wickedd for network management so no dhcp hooks

* Molecule: increase ansible timeout

* CI: Increase ansible timeout to 120s for Packet jobs
2021-12-09 14:09:06 -08:00
Florian Ruynat e19ce27352
Remove ovn4nfv support (#8265) 2021-12-03 11:56:35 -08:00
Cristian Calin a24162f596
CI: upgrade vagrant to 2.2.19 (#8264) 2021-12-02 13:23:44 -08:00
Florian Ruynat e82443241b
Move opensuse CI to docker and fix ubuntu16 containerd version for docker (#8257) 2021-12-02 08:01:34 -08:00
Florian Ruynat b38382a68f
Move cri-o default package to 1.22 (#8258) 2021-12-02 06:21:34 -08:00
Florian Ruynat 30ec03259d
Remove fedora33 - eol (#8246) 2021-11-30 15:53:17 -08:00
Cristian Calin ee882fa462
Add capability to use swap, requires Kube 1.22 (#8241)
* Alpha-NodeSwap: allow nodes to use swap

* CI: Add Fedora 35 with experimental swap job
2021-11-30 00:52:56 -08:00
Cristian Calin 2f44b40d68
OEL7: Fix CentOS7 Extras for OEL7 (#8219)
* OEL7: Fix CentOS7 Extras for OEL7

* Molecule: add logs collection for jobs
2021-11-29 13:39:21 -08:00
Florian Ruynat a5f88e14d0
Cleanup tests (#8234)
* Add Fedora 35 image, support and CI

* Cleanup tests and allow_failure for vagrant
2021-11-26 09:00:51 -08:00
Cristian Calin e78bda65fe
Defaults: replace docker with containerd as our default container_manager (#8175)
* Defaults: replace docker with containerd as our default container_manager

* CI: Use docker for download_localhost test

* Defaults: with container_manager=containerd we need etcd_deployment_type=host

* CI: Run weave jobs with docker

* CI: Vagrant don't download_force_cache

* CI: Fix upgrade tests

* should run compatible with old settings, this means docker
* we need to run with a distro that has at least modern containerd,
  this means move from debian9 to debian10 to allow `containerd_version`
  to match between 2.17 and master
2021-11-25 06:54:33 -08:00
Cristian Calin acd5185ad4
Fix fedora reset (#8205)
* Reset: Fedora uses NetworkManager

* CI: test reset on fedora
2021-11-18 16:46:51 -08:00
Cristian Calin 0d0468e127
Exercise multiple ansible versions in CI (#8172)
* Ansible: separate requirements files for supported ansible versions

* Ansible: allow using ansible 2.11

* CI: Exercise Ansible 2.9 and Ansible 2.11 in a basic AIO CI job

* CI: Allow running a reset test outside of idempotency tests and running it in stage1

* CI: move ubuntu18-calico-aio job to stage2 and relay only on ubuntu20 with the variously supported ansible versions for stage1

* CI: add capability to install collections or roles from ansible-galaxy to mitigate missing behavior in older ansible versions
2021-11-10 16:11:50 -08:00
Cristian Calin 039205560a
nodelocaldns: allow a secondary pod for nodelocaldns for local-HA (#8100)
* nodelocaldns: allow a secondary pod for nodelocaldns for local-HA

* CI: add job to test nodelocaldns secondary
2021-11-09 09:57:47 -08:00
Cristian Calin 8d553f7e91
Mitogen: deprecate the use of mitogen and remove coverage from CI (#8147) 2021-11-05 00:57:52 -07:00
Florian Ruynat 9ffe5940fe
Remove TF 0.14/0.15 support - Add TF 1.x support only (#8062) 2021-10-08 09:01:06 -07:00
Cristian Calin 33146b9481
CI: Add Calico eBPF in HA mode test (#7710)
* Sample-Inventory: add sample for calico_bpf_enabled

* Calico-Doc: note about CONFIG_NET_SCHED for eBPF support

* CI: Add Calico eBPF in HA mode test
2021-09-24 09:57:23 -07:00
Florian Ruynat 6f8b24f367 Allow failure in cert manager job 2021-09-22 09:50:01 -07:00
Florian Ruynat c8f3d88288 Retry vagrant and periodic packet jobs too 2021-09-06 02:58:29 -07:00
Florian Ruynat c7529270ff
Fix CI script for Terraform >0.15 (#7928) 2021-09-01 04:30:59 -07:00
Florian Ruynat 0171c71de0 Update Terraform 0.14 to .11, remove 0.13 jobs and add 0.15 2021-08-31 16:32:59 -07:00
Florian Ruynat 46d0df394f Add one retry to packet_pr jobs 2021-08-31 16:32:59 -07:00
Florian Ruynat 207d3e7b4e
Add Debian-11 image and CI (#7919) 2021-08-31 14:02:22 -07:00
Florian Ruynat 77a74adedd
Bump centos8 CI job memory to 3go and remove mitogen for fedora CI (#7921) 2021-08-30 08:25:13 -07:00
Cristian Calin 1afdb05ea9
Fedora and RHEL use etc_t and the convention is <type_name>_t (#7891)
* Fedora and RHEL use etc_t and the convention is <type_name>_t

* Docs: specify all values for preinstall_selinux_state

* CI: Add Fedora 34 with SELinux in enforcing mode
2021-08-27 14:20:53 -07:00
Maxime Guyot 97d95775a5
Disable OVH CI until voucher situation is cleared up (#7824) 2021-07-26 06:16:33 -07:00
Kenichi Omichi e5ee47408e
Allow failure on tf-elax_ubuntu18-calico (#7814)
tf-elax_ubuntu18-calico is so flake today. The test job is failed
due to SSH connectivity check error after deploying virtual machines
which are used for Kubernetes nodes.
This allows failure on the job to see the test situation without
pull request merger failures.
2021-07-22 07:47:52 -07:00
Cristian Calin 7516fe142f
Move to Ansible 3.4.0 (#7672)
* Ansible: move to Ansible 3.4.0 which uses ansible-base 2.10.10

* Docs: add a note about ansible upgrade post 2.9.x

* CI: ensure ansible is removed before ansible 3.x is installed to avoid pip failures

* Ansible: use newer ansible-lint

* Fix ansible-lint 5.0.11 found issues

* syntax issues
* risky-file-permissions
* var-naming
* role-name
* molecule tests

* Mitogen: use 0.3.0rc1 which adds support for ansible 2.10+

* Pin ansible-base to 2.10.11 to get package fix on RHEL8
2021-07-12 00:00:47 -07:00
Cristian Calin a2cf6816ce
Calico wireguard (#7638)
* Calico: add Wireguard support

* CI: Add Calico Wireguard scenario
2021-06-25 03:22:45 -07:00
Florian Ruynat 7896bc7831
Add Fedora 33 image and CI, remove Fedora 31 (EOL) + update docker packages (#7657)
* Update docker package to 20.10.6

* Add Fedora 33 image and CI, remove Fedora 31 (EOL)
2021-05-28 08:04:25 -07:00
Florian Ruynat 42382e2cde
Update Terraform/Vagrant + increase tf_ovh retries (#7477) 2021-04-12 09:47:39 -07:00
Kenichi Omichi 486b223e01
Replace kube-master with kube_control_plane (#7256)
This replaces kube-master with kube_control_plane because of [1]:

  The Kubernetes project is moving away from wording that is
  considered offensive. A new working group WG Naming was created
  to track this work, and the word "master" was declared as offensive.
  A proposal was formalized for replacing the word "master" with
  "control plane". This means it should be removed from source code,
  documentation, and user-facing configuration from Kubernetes and
  its sub-projects.

NOTE: The reason why this changes it to kube_control_plane not
      kube-control-plane is for valid group names on ansible.

[1]: https://github.com/kubernetes/enhancements/blob/master/keps/sig-cluster-lifecycle/kubeadm/2067-rename-master-label-taint/README.md#motivation
2021-03-23 17:26:05 -07:00
Ewnetu Bayuh Lakew 5c5bf41afe
Terraform support for UpCloud (#7360)
* terraform support for UpCloud

* terraform support for UpCloud

* terraform support for UpCloud

* terraform support for UpCloud

* terraform support for UpCloud

* terraform support for UpCloud

* terraform support for UpCloud

* Updates to README.md and main.tf files

* formatting and updating readme

* added a .terraform_validate CI job

* fixed format issue

* added sample inventory

* added symbolic link to group_vars

* added missing tf variables and minor fixes

* added text formatting

* minor formatting fixes
2021-03-15 01:41:04 -07:00
Florian Ruynat e35becebf8
Move centos7-crio CI job to centos8 (#7327) 2021-03-01 09:57:26 -08:00
Jakub Krzywda 0a0156c946
Vsphere (#7306)
* Add terraform scripts for vSphere

* Fixup: Add terraform scripts for vSphere

* Add inventory generation

* Use machines var to provide IPs

* Add README file

* Add default.tfvars file

* Fix newlines at the end of files

* Remove master.count and worker.count variables

* Fixup cloud-init formatting

* Fixes after initial review

* Add warning about disabled DHCP

* Fixes after second review

* Add sample-inventory
2021-02-26 04:20:15 -08:00
Matt Calvert c2c97c36bc Add in tests for Calico with dual-stack networking 2021-02-05 00:04:52 -08:00
Cristian Klein b77460ec34
contrib/terraform/exoscale: Rework SSH public keys (#7242)
* contrib/terraform/exoscale: Rework SSH public keys

Exoscale has a few limitations with `exoscale_ssh_keypair` resources.
Creating several clusters with these scripts may lead to an error like:

```
Error: API error ParamError 431 (InvalidParameterValueException 4350): The key pair "lj-sc-ssh-key" already has this fingerprint
```

This patch reworks handling of SSH public keys. Specifically, we rely on
the more cloud-agnostic way of configuring SSH public keys via
`cloud-init`.

* contrib/terraform/exoscale: terraform fmt

* contrib/terraform/exoscale: Add terraform validate

* contrib/terraform/exoscale: Inline public SSH keys

The Terraform scripts need to install some SSH key, so that Kubespray
(i.e., the "Ansible part") can take over. Initially, we pointed the
Terraform scripts to `~/.ssh/id_rsa.pub`. This proved to be suboptimal:
Operators sharing responbility for a cluster risk unnecessarily replacing resources.

Therefore, it has been determined that it's best to inline the public
SSH keys. The chosen variable `ssh_public_keys` provides some uniformity
with `contrib/azurerm`.

* Fix Terraform Exoscale test

* Fix Terraform 0.14 test
2021-02-03 07:32:28 -08:00
Florian Ruynat 9007d6621a
Update nginx, minor weave and misc CI tools (vagrant/terraform) (#7215) 2021-01-26 08:22:34 -08:00
Florian Ruynat 81b4ffa6b4
Add Fedora 33 CI, remove Fedora 31 (#7072) 2021-01-14 08:27:05 -08:00
Etienne Champetier 29f1c40580
Ignore all .git* for mardownlint (#7109)
Signed-off-by: Etienne Champetier <champetier.etienne@gmail.com>
2021-01-07 10:16:53 -08:00
Kenichi Omichi 398a995798
Fix markdownlint failures under ./roles/ (#7089)
This fixes markdownlint failures under roles/
2020-12-30 05:07:49 -08:00
Kenichi Omichi dc86b2063a
Fix markdown failures on contrib/terraform (#7082)
This fixes markdown failures on contrib/terraform.
2020-12-25 12:10:27 -08:00
Cristian Klein fd3ebc13f7
Fix terraform0.13 errors (#7077)
* [terraform/aws] Fix Terraform >=0.13 warnings

Terraform >=0.13 gives the following warning:

```
Warning: Interpolation-only expressions are deprecated
```

The fix was tested as follows:
```
rm -rf .terraform && terraform0.12.26 init && terraform0.12.26 validate
rm -rf .terraform && terraform0.13.5 init && terraform0.13.5 validate
rm -rf .terraform && terraform0.14.3 init && terraform0.14.3 validate
```
which gave no errors nor warnings.

* [terraform/openstack] Fixes for Terraform >=0.13

Terraform >=0.13 gives the following error:
```
Error: Failed to install providers
Could not find required providers, but found possible alternatives:
  hashicorp/openstack -> terraform-provider-openstack/openstack
```

This patch fixes these errors.

This fix was tested as follows:
```
rm -rf .terraform && terraform0.12.26 init && terraform0.12.26 validate
rm -rf .terraform && terraform0.13.5 init && terraform0.13.5 validate
rm -rf .terraform && terraform0.14.3 init && terraform0.14.3 validate
```
which gave no errors nor warnings for Terraform 0.13.5 and Terraform
0.14.3. Unfortunately, 0.12.x gives a harmless warning, but
with 0.14.3 out the door, I guess we need to move on.

* [terraform/packet] Fixes for Terraform >=0.13

This fix was tested as follows:
```
export PACKET_AUTH_TOKEN=blah-blah
rm -rf .terraform && terraform0.12.26 init && terraform0.12.26 validate
rm -rf .terraform && terraform0.13.5 init && terraform0.13.5 validate
rm -rf .terraform && terraform0.14.3 init && terraform0.14.3 validate
```

Errors are gone, but warnings still remain. It is impossible to please
all three versions of Terraform.

* Add tests for Terraform >=0.13
2020-12-23 05:08:26 -08:00
Kenichi Omichi 5b5726bdd4
Improve markdownlint for contrib/network-storage (#7079)
This fixes markdownlint failures under contrib/network-storage and
contrib/vault.
2020-12-23 00:00:26 -08:00
Kenichi Omichi 1347bb2e4b
Improve markdownlint coverage (#7075)
Now markdownlint covers ./README.md and md files under ./docs only.
However we have a lot of md files under different directories also.
This enables markdownlint for other md files also.
2020-12-22 04:44:26 -08:00
Kenichi Omichi 43dbff938e
Exclude .git/ from shellcheck (#7048)
If a branch name contains '.sh', current shellcheck checks the branch
file under .git/ and outputs error because the format is not shell
script one.
This makes shellcheck exclude files under .git/ to avoid this issue.
2020-12-16 15:51:09 -08:00
Florian Ruynat f6eed8091e
Remove contiv related files (#6964) 2020-11-30 06:48:50 -08:00
Florian Ruynat 947162452d
Forgotten debian10 test during nightly tests (#6769) 2020-09-30 07:19:26 -07:00
Florian Ruynat 49bcf91aaf
Allow period ci jobs to fail (#6737) 2020-09-24 09:22:06 -07:00
Maxime Guyot 09b8314057
Add support for periodic CI (#6715) 2020-09-18 08:08:46 -07:00
Florian Ruynat 705ad84ce7
Update third party librairies and tools (#6669) 2020-09-17 02:36:46 -07:00
Kuralamudhan Ramakrishnan 90e5f8ffe1
adding ovn4nfv in kubespray (#6381)
Signed-off-by: Kuralamudhan Ramakrishnan <kuralamudhan.ramakrishnan@intel.com>
2020-07-31 07:33:08 -07:00
Florian Ruynat bf6168fca8
Move fedora30 jobs to fedora32 (#6426) 2020-07-30 23:31:07 -07:00
Arthur Outhenin-Chalandre 3550e3c145
Adding kube-proxy-replacement support in cilium (#6334)
Signed-off-by: Arthur Outhenin-Chalandre <arthur@cri.epita.fr>
2020-07-30 02:46:31 -07:00
Lovro Seder 96a2b386f2
Fix shellcheck url (#6462) 2020-07-28 05:57:08 -07:00
Maxime Guyot b35e6558bc
Always enable GitLab CI artifacts for cluster-dump (#6412) 2020-07-16 13:45:00 -07:00
Arthur Outhenin-Chalandre abfa1636e4
Fix kube-proxy post deployment removal (#5554)
* Fix kube-proxy removal

* Fix unwanted skipped task for kube-proxy
* Fix kube_proxy_remove default

Signed-off-by: Arthur Outhenin-Chalandre <arthur@cri.epita.fr>

* Add test for kube-router svc proxy

Signed-off-by: Arthur Outhenin-Chalandre <arthur@cri.epita.fr>
2020-07-13 07:12:33 -07:00
Maxime Guyot 935c5093e2
Enable OVH CI (#6365) 2020-07-06 01:56:51 -07:00
Joel Seguillon 4c1e0b188d
Add .editorconfig file (#6307) 2020-06-29 12:39:59 -07:00
Maxime Guyot 69a48cbdd7
Add Vagrant CI for Ubuntu 20.04 (#6279) 2020-06-18 01:18:05 -07:00
Maxime Guyot c6588856c7
Add Ubuntu 20.04 support and use Python 3 (#6157) 2020-06-16 13:04:05 -07:00
Maxime Guyot 9afd3f0c32
Use a random subnet for elastx CI (#6232) 2020-06-06 12:11:45 -07:00
Maxime Guyot b98cb74f5e
Use 19.03.9 in localhost CI (#6201) 2020-06-04 08:59:14 -07:00
Maxime Guyot fd59556222
Add Elastx CI (#6127) 2020-06-03 00:00:17 -07:00
Maxime Guyot 83d945127f
Make vagrant CI normal (#6074) 2020-05-28 00:46:02 -07:00
Maxime Guyot 1617a6ea8e
CI upgrade from v2.13.1 (#6188) 2020-05-27 05:22:40 -07:00
Florian Ruynat e9ce7243b8
Match docker-cli version with docker-engine version (when available) (#6163) 2020-05-25 05:37:11 -07:00
Maxime Guyot 21ea079896
Disable OVH CI (#6114) 2020-05-09 15:19:50 -07:00
Florian Ruynat c44f13114f
Allow containerd runtime with fedora os (30/31) - add CI test (#6094) 2020-05-08 07:55:43 -07:00
Maxime Guyot 641a2a8bb4
Skip molecule tests for Ubuntu 18.04 (#6077) 2020-05-05 07:17:09 -07:00
Florian Ruynat 38daee41d5
Reorder tests in packet file (#6067) 2020-05-03 12:36:26 -07:00
Maxime Guyot 412d560bcf
Add CI for 16x ubuntu servers (#6040) 2020-04-29 07:14:24 -07:00
Sergey 69603aed34
add strategy mitogen_linear when installed mitogen (#5985)
* add strategy mitogen_linear when installed mitogen

* add small docs

Rename playbook file

The raw action executes as a regular Mitogen connection, which requires Python on the target, so add strategy: linear to bootstrap-os role playbook.

* add mitogen to  CI test
fix typo

* enable mitogen test on deploy-part1 tests
change version from master to release
download tar.gz archive

* run all CI tests with mitogen

* disable mitogen with upgrade CI tests

* enable mitogen on CI tests via env vars

* disable mitogen on CI test by default, enable on some different OS

* disable mitogen CI test on centos8
(get error  /usr/bin/python: No such file or directory)
2020-04-24 05:20:07 -07:00
Maxime Guyot 6674be2572
Cleanup Vagrant VMs before molecule and vagrant CI (#6009) 2020-04-24 01:30:07 -07:00
Maxime Guyot b44f7957d5
Update CI matrix (#6010) 2020-04-23 09:51:11 -07:00
Maxime Guyot 09bccc97ba
Add CRI-O CI (#5460) 2020-04-22 06:09:52 -07:00
Maxime Guyot 4d7eca7d2e
Add Dockerfile for vagrant image (#5977) 2020-04-18 13:53:36 -07:00
Maxime Guyot 3134dd4c0d
Drop support for Fedora 28 and add Fedora 30 and 31 (#5969) 2020-04-18 06:35:36 -07:00
Maxime Guyot 56a9c7a802
Add Vagrant CI (#5487) 2020-04-18 06:09:35 -07:00
Florian Ruynat 6a92e34994
Update tests names (#5904) 2020-04-15 09:24:03 -07:00
Florian Ruynat c929b5e82e
Upgrade kube-ovn to v1.1.0 and move test from centos7 to centos8 (#5852) 2020-04-15 03:10:03 -07:00