CLOUDSTACK-10188 - Resource Accounting for primary storage is Broken when Domains are in use

[bwsw]

https://issues.apache.org/jira/browse/CLOUDSTACK-10188

During storage expunge domain resource statistics for primary storage space resource counter is not updated for domain. This leads to the situation when domain resource statistics for primary storage is overfilled (statistics only increase but not decrease).

Global scheduled task resourcecount.check.interval > 0 provides a workaround but not fixes the problem truly because when accounts inside domains use primary_storage allocation/deallocation intensively it leads to service block of operation.

NB: Unable to implement marvin tests because it (marvin) places in database weird primary storage volume size of 100 when creating VM from template. It might be a sign of opening a new issue for that bug.

[yadvr]

@bwsw can you fix the conflicts, thanks.

[yadvr]

ping @bwsw

[bwsw]

@rhtyd I'll do it in a couple of days.

[bwsw]

@rhtyd Do we have a chance including it to 4.11 yet? I overviewed Travis logs and only one test fails which seems to be a timeout reason, not a real failure. We already resolved conflicts twice against moving master. I don't even imaging how much we will be asked to do so if waiting 4.12))
And I suppose it's a bug which is vital for users who use domains, because It blocks creating new VMs.

[yadvr]

@bwsw sure, please fix the conflicts.
Ideally today we are supposed to freeze the branch to only accept critical/blocker fixes until the rc1 is put to vote. I'll start a discussion on dev@ to gather consensus. Meanwhile, if you've time/bandwidth do work on it.

[bwsw]

@rhtyd fixed. Conflict is not related to PR code, just code reformatting prevented automatic merge.

[borisstoyanov]

@blueorangutan package

[blueorangutan]

@borisstoyanov a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-1619

[borisstoyanov]

@blueorangutan test

[blueorangutan]

@borisstoyanov a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

[blueorangutan]

Trillian test result (tid-2073)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 31361 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2362-t2073-kvm-centos7.zip
Intermitten failure detected: /marvin/tests/smoke/test_routers_network_ops.py
Intermitten failure detected: /marvin/tests/smoke/test_volumes.py
Intermitten failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermitten failure detected: /marvin/tests/smoke/test_host_maintenance.py
Smoke tests completed. 66 look OK, 1 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_07_resize_fail	Failure	15.34	test_volumes.py

[bwsw]

@rhtyd @borisstoyanov
Same test [test_volumes.py/test_07_resize_fail] fails in previously merged #2088 which includes massive volume changes. I don't see anything related to my code yet, but suppose that PR [2088] could bring a regression. Before that merge I haven't seen that test to fail.

[bwsw]

Here #2379 (comment) it also fails. I suppose the regression is introduced in #2088

[yadvr]

Thanks @bwsw I'll investigate. meanwhile, rekick tests. However #2379 (comment) does not have resize related failure.
@blueorangutan package

[blueorangutan]

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

[yadvr]

Additional review requested - @DaanHoogland @borisstoyanov @wido @marcaurele @rafaelweingartner and others

[blueorangutan]

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-1633

[bwsw]

@rhtyd
Hm, then what does it mean (copied from there):

test_07_resize_fail | Failure | 15.42 | test_volumes.py

[yadvr]

@bwsw it means one of the test cases from test_volumes.py - test_07_resize_fail, you can analyze why it failed from the marvin test logs - Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2362-t2073-kvm-centos7.zip. It maybe most likely env related. I'll kick test again.
@blueorangutan test

[blueorangutan]

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

[borisstoyanov]

Yes @bwsw most likely this failure is not related to your changes, but since they are in the same domain lets wait until the results come out and make a decision then. If you're further interested you can investigate the failure and propose a fix to stabilize the tess. Thanks.

[bwsw]

@borisstoyanov, got it, not sure can find a time for additional research.

[blueorangutan]

Trillian test result (tid-2095)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 28771 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2362-t2095-kvm-centos7.zip
Intermitten failure detected: /marvin/tests/smoke/test_deploy_virtio_scsi_vm.py
Intermitten failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermitten failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermitten failure detected: /marvin/tests/smoke/test_host_maintenance.py
Intermitten failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 65 look OK, 2 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_05_rvpc_multi_tiers	Failure	310.72	test_vpc_redundant.py
test_05_rvpc_multi_tiers	Error	336.43	test_vpc_redundant.py
test_hostha_enable_ha_when_host_in_maintenance	Error	0.63	test_hostha_kvm.py

[borisstoyanov]

LGTM, I don't think these failures are related to the changes, travis failure was a timeout of the run

[yadvr]

Re-kicked Travis by closing/opening the PR.

[yadvr]

LGTM.

[yadvr]

Test LGTM, merging this based on code reviews and test results.