Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

libvirt: increase master memory to 3GB #408

Merged
merged 1 commit into from
Oct 3, 2018
Merged

libvirt: increase master memory to 3GB #408

merged 1 commit into from
Oct 3, 2018

Conversation

sjenning
Copy link
Contributor

@sjenning sjenning commented Oct 3, 2018

After CVO merged, 2GB is no longer sufficient for master operations. The VM gridlocks with kswapd and becomes slow to respond or completely unresponsive.

I bumped my master to 4GB to get this working, but 3GB should be sufficient for now.

[root@dev-master-0 ~]# echo 3 > /proc/sys/vm/drop_caches 
[root@dev-master-0 ~]# free -h
              total        used        free      shared  buff/cache   available
Mem:           3.9G        2.1G        1.1G        8.4M        679M        1.4G
Swap:            0B          0B          0B

@aveshagarwal @derekwaynecarr @rphillips @abhinavdahiya

@sjenning sjenning requested a review from crawford October 3, 2018 15:00
@openshift-ci-robot openshift-ci-robot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Oct 3, 2018
@sjenning sjenning requested a review from abhinavdahiya October 3, 2018 15:01
@wking
Copy link
Member

wking commented Oct 3, 2018

I bumped my master to 4GB to get this working, but 3GB should be sufficient for now.

Do we have time to dig into who's so hungry? Or are we just bumping this with the hope that we'll be able to circle back once things quiet down and tighten things up? Or is 3GB really reasonable for the sort of things the masters are doing?

@abhinavdahiya
Copy link
Contributor

abhinavdahiya commented Oct 3, 2018 •
edited
Loading

Haven't seen this hang.
We are double deploying stuff; kube-core-operators and all new operators. This should be fixed soon.
I'm hesitant upgrading the limit unless majority people are blocked on this.

@sjenning
Copy link
Contributor Author

sjenning commented Oct 3, 2018 •
edited
Loading

This should be fixed soon.

@abhinavdahiya who is doing this?

@abhinavdahiya
Copy link
Contributor

abhinavdahiya commented Oct 3, 2018 •
edited
Loading

#322
#381
are two.

@mrunalp
Copy link
Member

mrunalp commented Oct 3, 2018

I am seeing this as well on my master node.

@sjenning
Copy link
Contributor Author

sjenning commented Oct 3, 2018

@abhinavdahiya can we merge this if I promise to revert once we drop KCO? 👼

@abhinavdahiya
Copy link
Contributor

/approve

@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 3, 2018
@abhinavdahiya
Copy link
Contributor

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Oct 3, 2018
@openshift-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abhinavdahiya, sjenning

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-robot openshift-merge-robot merged commit bf36c90 into openshift:master Oct 3, 2018
@cgwalters cgwalters mentioned this pull request Dec 4, 2018
Merged
cgwalters added a commit to cgwalters/installer that referenced this pull request Dec 5, 2018
@cgwalters
libvirt: Add Terraform variables for memory/CPU, bump master to 4GiB
e59513f 
My main dev environment is a Lenovo P50 with 64GB of RAM - I got
it specifically to run some large VMs (and/or many VMs) specifically
with OpenShift in mind.

First, default masters to 4096 MiB since we are seeing a default
install be overloaded.

And for me, increasing RAM on my master to 8GB is a *very* noticeable speed improvement
and I think reliabilty; before I saw the apiserver be `OOMKilled`
sometimes, and `kswapd0` was constantly doing writeback.

These variables aren't bubbled all the way up to the documented
installer config, but one can now do e.g.:

```
$ env TF_VAR_libvirt_master_memory=8192 TF_VAR_libvirt_master_vcpu=4 ./bin/openshift-install create cluster --dir=osiris
```

Previously:

 - openshift#408
 - openshift#163
This was referenced Dec 5, 2018
Merged
Merged
Merged
Merged
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@crawford crawford Awaiting requested review from crawford

@abhinavdahiya abhinavdahiya Awaiting requested review from abhinavdahiya

@smarterclayton smarterclayton Awaiting requested review from smarterclayton

@yifan-gu yifan-gu Awaiting requested review from yifan-gu

Assignees

@abhinavdahiya abhinavdahiya

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@sjenning @wking @abhinavdahiya @mrunalp @openshift-ci-robot @openshift-merge-robot