CMS class unloading took much time Announcing the arrival of Valued Associate #679: Cesar Manara Planned maintenance scheduled April 23, 2019 at 23:30 UTC (7:30pm US/Eastern) Data science time! April 2019 and salary with experience The Ask Question Wizard is Live!GC output clarificationMinor GC pause times are increasing. What are the possible reasons?JVM GC problemsGarbage collection doesnt seem to work in javaUnderstanding metaspace line in JVM heap printoutJava 8 Metaspace: OutOfMemoryError: Dealing with reflection InflationWhy does Eden memory keep increasing in 64-bit JVM, but not in 32-bit JVM?GC CMS Final Remark - taking too longjvm in kubernetes/docker running out of memory faster than standaloneHow to tuning JVM 8 G1 argument to avoid Full GC (Allocation Failure)

Understanding piped command in Gnu/Linux

Putting class ranking in CV, but against dept guidelines

Why shouldn't this prove the Prime Number Theorem?

How much damage would a cupful of neutron star matter do to the Earth?

Can gravitational waves pass through a black hole?

3D Masyu - A Die

First paper to introduce the "principal-agent problem"

How do you write "wild blueberries flavored"?

Why are current probes so expensive?

Why can't fire hurt Daenerys but it did to Jon Snow in season 1?

How to achieve cat-like agility?

Did John Wesley plagiarize Matthew Henry...?

How to evaluate this function?

Noise in Eigenvalues plot

Is there any significance to the prison numbers of the Beagle Boys starting with 176-?

Is Normal(mean, variance) mod x still a normal distribution?

Are there any irrational/transcendental numbers for which the distribution of decimal digits is not uniform?

Does the main washing effect of soap come from foam?

Fit odd number of triplets in a measure?

My mentor says to set image to Fine instead of RAW — how is this different from JPG?

Where and when has Thucydides been studied?

Why complex landing gears are used instead of simple, reliable and light weight muscle wire or shape memory alloys?

Short story about astronauts fertilizing soil with their own bodies

Sally's older brother



CMS class unloading took much time



Announcing the arrival of Valued Associate #679: Cesar Manara
Planned maintenance scheduled April 23, 2019 at 23:30 UTC (7:30pm US/Eastern)
Data science time! April 2019 and salary with experience
The Ask Question Wizard is Live!GC output clarificationMinor GC pause times are increasing. What are the possible reasons?JVM GC problemsGarbage collection doesnt seem to work in javaUnderstanding metaspace line in JVM heap printoutJava 8 Metaspace: OutOfMemoryError: Dealing with reflection InflationWhy does Eden memory keep increasing in 64-bit JVM, but not in 32-bit JVM?GC CMS Final Remark - taking too longjvm in kubernetes/docker running out of memory faster than standaloneHow to tuning JVM 8 G1 argument to avoid Full GC (Allocation Failure)



.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;








1















On large load was noticed large GC pause(400ms) for our application. During investigation it turns out, that pause happens on CMS Final Remark and class unloading phase took a lot more time, than other phases(10x-100X):



 (CMS Final Remark) 
[YG occupancy: 142247 K (294912 K)]
2019-03-13T07:38:30.656-0700: 24252.576:
[Rescan (parallel) , 0.0216770 secs]
2019-03-13T07:38:30.677-0700: 24252.598:
[weak refs processing, 0.0028353 secs]
2019-03-13T07:38:30.680-0700: 24252.601:
[class unloading, 0.3232543 secs]
2019-03-13T07:38:31.004-0700: 24252.924:
[scrub symbol table, 0.0371301 secs]
2019-03-13T07:38:31.041-0700: 24252.961:
[scrub string table, 0.0126352 secs]
[1 CMS-remark: 2062947K(4792320K)] 2205195K(5087232K), 0.3986822 secs]
[Times: user=0.63 sys=0.01, real=0.40 secs]

Total time for which application threads were stopped: 0.4156259 seconds,
Stopping threads took: 0.0014133 seconds


This pause always happens in first second of performance test, the duration of pause varies from 300ms to 400+ms.



Unfortunately, I have no access to server(it's under maintenance) and have only logs for several test runs. But when the server would available, I want to be prepared for further investigation, but I have no idea of what causes such a behavior.



My first thought was about Linux Huge pages, but we don't use them.



After more time in logs, I found following:



Heap after GC invocations=7969 (full 511):
par new generation total 294912K, used 23686K [0x0000000687800000, 0x000000069b800000, 0x000000069b800000)
eden space 262144K, 0% used [0x0000000687800000, 0x0000000687800000, 0x0000000697800000)
from space 32768K, 72% used [0x0000000699800000, 0x000000069af219b8, 0x000000069b800000)
to space 32768K, 0% used [0x0000000697800000, 0x0000000697800000, 0x0000000699800000)
concurrent mark-sweep generation total 4792320K, used 2062947K [0x000000069b800000, 0x00000007c0000000, 0x00000007c0000000)
Metaspace used 282286K, capacity 297017K, committed 309256K, reserved 1320960K
class space used 33038K, capacity 36852K, committed 38872K, reserved 1048576K
}


Heap after GC invocations=7970 (full 511):
par new generation total 294912K, used 27099K [0x0000000687800000, 0x000000069b800000, 0x000000069b800000)
eden space 262144K, 0% used [0x0000000687800000, 0x0000000687800000, 0x0000000697800000)
from space 32768K, 82% used [0x0000000697800000, 0x0000000699276df0, 0x0000000699800000)
to space 32768K, 0% used [0x0000000699800000, 0x0000000699800000, 0x000000069b800000)
concurrent mark-sweep generation total 4792320K, used 2066069K [0x000000069b800000, 0x00000007c0000000, 0x00000007c0000000)
Metaspace used 282303K, capacity 297017K, committed 309256K, reserved 1320960K
class space used 33038K, capacity 36852K, committed 38872K, reserved 1048576K
}


Investigating GC pause happens between GC invocations 7969 and 7970. And the amount of used space in meta space is almost the same(it's actually increased)



So, It looks like it's not actually some stall classes which aren't used anymore(since no space was cleared) and it's not safe point reaching issue - since blocking of threads took small time(0.0014133).



How to investigate such case and what diagnostic information is required for proper preparedness.



Technical details



Centos5 + JDK8 + CMS GC with args:



-XX:+CMSClassUnloadingEnabled 
-XX:CMSIncrementalDutyCycleMin=10
-XX:+CMSIncrementalPacing
-XX:CMSInitiatingOccupancyFraction=50
-XX:+CMSParallelRemarkEnabled
-XX:+DisableExplicitGC
-XX:InitialHeapSize=5242880000
-XX:MaxHeapSize=5242880000
-XX:MaxNewSize=335544320
-XX:MaxTenuringThreshold=6
-XX:NewSize=335544320
-XX:OldPLABSize=16
-XX:+UseCompressedClassPointers
-XX:+UseCompressedOops
-XX:+UseConcMarkSweepGC
-XX:+UseParNewGC









share|improve this question
























  • 400 ms is a lot for your application? if this is indeed the case, I wonder if you have JVM engineers on your team

    – Eugene
    Mar 22 at 12:56











  • Normally, the word “since” implies a causal relationship. In your sentence, “…since Metaspace is located in native memory, most of the time GC process was blocked by system and did not any useful work”, I can’t see that causal relationship.

    – Holger
    Mar 22 at 13:29











  • @Eugene, yes, we have performance engineers, but they are all busy right now.

    – Roman Makhlin
    Mar 22 at 14:10

















1















On large load was noticed large GC pause(400ms) for our application. During investigation it turns out, that pause happens on CMS Final Remark and class unloading phase took a lot more time, than other phases(10x-100X):



 (CMS Final Remark) 
[YG occupancy: 142247 K (294912 K)]
2019-03-13T07:38:30.656-0700: 24252.576:
[Rescan (parallel) , 0.0216770 secs]
2019-03-13T07:38:30.677-0700: 24252.598:
[weak refs processing, 0.0028353 secs]
2019-03-13T07:38:30.680-0700: 24252.601:
[class unloading, 0.3232543 secs]
2019-03-13T07:38:31.004-0700: 24252.924:
[scrub symbol table, 0.0371301 secs]
2019-03-13T07:38:31.041-0700: 24252.961:
[scrub string table, 0.0126352 secs]
[1 CMS-remark: 2062947K(4792320K)] 2205195K(5087232K), 0.3986822 secs]
[Times: user=0.63 sys=0.01, real=0.40 secs]

Total time for which application threads were stopped: 0.4156259 seconds,
Stopping threads took: 0.0014133 seconds


This pause always happens in first second of performance test, the duration of pause varies from 300ms to 400+ms.



Unfortunately, I have no access to server(it's under maintenance) and have only logs for several test runs. But when the server would available, I want to be prepared for further investigation, but I have no idea of what causes such a behavior.



My first thought was about Linux Huge pages, but we don't use them.



After more time in logs, I found following:



Heap after GC invocations=7969 (full 511):
par new generation total 294912K, used 23686K [0x0000000687800000, 0x000000069b800000, 0x000000069b800000)
eden space 262144K, 0% used [0x0000000687800000, 0x0000000687800000, 0x0000000697800000)
from space 32768K, 72% used [0x0000000699800000, 0x000000069af219b8, 0x000000069b800000)
to space 32768K, 0% used [0x0000000697800000, 0x0000000697800000, 0x0000000699800000)
concurrent mark-sweep generation total 4792320K, used 2062947K [0x000000069b800000, 0x00000007c0000000, 0x00000007c0000000)
Metaspace used 282286K, capacity 297017K, committed 309256K, reserved 1320960K
class space used 33038K, capacity 36852K, committed 38872K, reserved 1048576K
}


Heap after GC invocations=7970 (full 511):
par new generation total 294912K, used 27099K [0x0000000687800000, 0x000000069b800000, 0x000000069b800000)
eden space 262144K, 0% used [0x0000000687800000, 0x0000000687800000, 0x0000000697800000)
from space 32768K, 82% used [0x0000000697800000, 0x0000000699276df0, 0x0000000699800000)
to space 32768K, 0% used [0x0000000699800000, 0x0000000699800000, 0x000000069b800000)
concurrent mark-sweep generation total 4792320K, used 2066069K [0x000000069b800000, 0x00000007c0000000, 0x00000007c0000000)
Metaspace used 282303K, capacity 297017K, committed 309256K, reserved 1320960K
class space used 33038K, capacity 36852K, committed 38872K, reserved 1048576K
}


Investigating GC pause happens between GC invocations 7969 and 7970. And the amount of used space in meta space is almost the same(it's actually increased)



So, It looks like it's not actually some stall classes which aren't used anymore(since no space was cleared) and it's not safe point reaching issue - since blocking of threads took small time(0.0014133).



How to investigate such case and what diagnostic information is required for proper preparedness.



Technical details



Centos5 + JDK8 + CMS GC with args:



-XX:+CMSClassUnloadingEnabled 
-XX:CMSIncrementalDutyCycleMin=10
-XX:+CMSIncrementalPacing
-XX:CMSInitiatingOccupancyFraction=50
-XX:+CMSParallelRemarkEnabled
-XX:+DisableExplicitGC
-XX:InitialHeapSize=5242880000
-XX:MaxHeapSize=5242880000
-XX:MaxNewSize=335544320
-XX:MaxTenuringThreshold=6
-XX:NewSize=335544320
-XX:OldPLABSize=16
-XX:+UseCompressedClassPointers
-XX:+UseCompressedOops
-XX:+UseConcMarkSweepGC
-XX:+UseParNewGC









share|improve this question
























  • 400 ms is a lot for your application? if this is indeed the case, I wonder if you have JVM engineers on your team

    – Eugene
    Mar 22 at 12:56











  • Normally, the word “since” implies a causal relationship. In your sentence, “…since Metaspace is located in native memory, most of the time GC process was blocked by system and did not any useful work”, I can’t see that causal relationship.

    – Holger
    Mar 22 at 13:29











  • @Eugene, yes, we have performance engineers, but they are all busy right now.

    – Roman Makhlin
    Mar 22 at 14:10













1












1








1








On large load was noticed large GC pause(400ms) for our application. During investigation it turns out, that pause happens on CMS Final Remark and class unloading phase took a lot more time, than other phases(10x-100X):



 (CMS Final Remark) 
[YG occupancy: 142247 K (294912 K)]
2019-03-13T07:38:30.656-0700: 24252.576:
[Rescan (parallel) , 0.0216770 secs]
2019-03-13T07:38:30.677-0700: 24252.598:
[weak refs processing, 0.0028353 secs]
2019-03-13T07:38:30.680-0700: 24252.601:
[class unloading, 0.3232543 secs]
2019-03-13T07:38:31.004-0700: 24252.924:
[scrub symbol table, 0.0371301 secs]
2019-03-13T07:38:31.041-0700: 24252.961:
[scrub string table, 0.0126352 secs]
[1 CMS-remark: 2062947K(4792320K)] 2205195K(5087232K), 0.3986822 secs]
[Times: user=0.63 sys=0.01, real=0.40 secs]

Total time for which application threads were stopped: 0.4156259 seconds,
Stopping threads took: 0.0014133 seconds


This pause always happens in first second of performance test, the duration of pause varies from 300ms to 400+ms.



Unfortunately, I have no access to server(it's under maintenance) and have only logs for several test runs. But when the server would available, I want to be prepared for further investigation, but I have no idea of what causes such a behavior.



My first thought was about Linux Huge pages, but we don't use them.



After more time in logs, I found following:



Heap after GC invocations=7969 (full 511):
par new generation total 294912K, used 23686K [0x0000000687800000, 0x000000069b800000, 0x000000069b800000)
eden space 262144K, 0% used [0x0000000687800000, 0x0000000687800000, 0x0000000697800000)
from space 32768K, 72% used [0x0000000699800000, 0x000000069af219b8, 0x000000069b800000)
to space 32768K, 0% used [0x0000000697800000, 0x0000000697800000, 0x0000000699800000)
concurrent mark-sweep generation total 4792320K, used 2062947K [0x000000069b800000, 0x00000007c0000000, 0x00000007c0000000)
Metaspace used 282286K, capacity 297017K, committed 309256K, reserved 1320960K
class space used 33038K, capacity 36852K, committed 38872K, reserved 1048576K
}


Heap after GC invocations=7970 (full 511):
par new generation total 294912K, used 27099K [0x0000000687800000, 0x000000069b800000, 0x000000069b800000)
eden space 262144K, 0% used [0x0000000687800000, 0x0000000687800000, 0x0000000697800000)
from space 32768K, 82% used [0x0000000697800000, 0x0000000699276df0, 0x0000000699800000)
to space 32768K, 0% used [0x0000000699800000, 0x0000000699800000, 0x000000069b800000)
concurrent mark-sweep generation total 4792320K, used 2066069K [0x000000069b800000, 0x00000007c0000000, 0x00000007c0000000)
Metaspace used 282303K, capacity 297017K, committed 309256K, reserved 1320960K
class space used 33038K, capacity 36852K, committed 38872K, reserved 1048576K
}


Investigating GC pause happens between GC invocations 7969 and 7970. And the amount of used space in meta space is almost the same(it's actually increased)



So, It looks like it's not actually some stall classes which aren't used anymore(since no space was cleared) and it's not safe point reaching issue - since blocking of threads took small time(0.0014133).



How to investigate such case and what diagnostic information is required for proper preparedness.



Technical details



Centos5 + JDK8 + CMS GC with args:



-XX:+CMSClassUnloadingEnabled 
-XX:CMSIncrementalDutyCycleMin=10
-XX:+CMSIncrementalPacing
-XX:CMSInitiatingOccupancyFraction=50
-XX:+CMSParallelRemarkEnabled
-XX:+DisableExplicitGC
-XX:InitialHeapSize=5242880000
-XX:MaxHeapSize=5242880000
-XX:MaxNewSize=335544320
-XX:MaxTenuringThreshold=6
-XX:NewSize=335544320
-XX:OldPLABSize=16
-XX:+UseCompressedClassPointers
-XX:+UseCompressedOops
-XX:+UseConcMarkSweepGC
-XX:+UseParNewGC









share|improve this question
















On large load was noticed large GC pause(400ms) for our application. During investigation it turns out, that pause happens on CMS Final Remark and class unloading phase took a lot more time, than other phases(10x-100X):



 (CMS Final Remark) 
[YG occupancy: 142247 K (294912 K)]
2019-03-13T07:38:30.656-0700: 24252.576:
[Rescan (parallel) , 0.0216770 secs]
2019-03-13T07:38:30.677-0700: 24252.598:
[weak refs processing, 0.0028353 secs]
2019-03-13T07:38:30.680-0700: 24252.601:
[class unloading, 0.3232543 secs]
2019-03-13T07:38:31.004-0700: 24252.924:
[scrub symbol table, 0.0371301 secs]
2019-03-13T07:38:31.041-0700: 24252.961:
[scrub string table, 0.0126352 secs]
[1 CMS-remark: 2062947K(4792320K)] 2205195K(5087232K), 0.3986822 secs]
[Times: user=0.63 sys=0.01, real=0.40 secs]

Total time for which application threads were stopped: 0.4156259 seconds,
Stopping threads took: 0.0014133 seconds


This pause always happens in first second of performance test, the duration of pause varies from 300ms to 400+ms.



Unfortunately, I have no access to server(it's under maintenance) and have only logs for several test runs. But when the server would available, I want to be prepared for further investigation, but I have no idea of what causes such a behavior.



My first thought was about Linux Huge pages, but we don't use them.



After more time in logs, I found following:



Heap after GC invocations=7969 (full 511):
par new generation total 294912K, used 23686K [0x0000000687800000, 0x000000069b800000, 0x000000069b800000)
eden space 262144K, 0% used [0x0000000687800000, 0x0000000687800000, 0x0000000697800000)
from space 32768K, 72% used [0x0000000699800000, 0x000000069af219b8, 0x000000069b800000)
to space 32768K, 0% used [0x0000000697800000, 0x0000000697800000, 0x0000000699800000)
concurrent mark-sweep generation total 4792320K, used 2062947K [0x000000069b800000, 0x00000007c0000000, 0x00000007c0000000)
Metaspace used 282286K, capacity 297017K, committed 309256K, reserved 1320960K
class space used 33038K, capacity 36852K, committed 38872K, reserved 1048576K
}


Heap after GC invocations=7970 (full 511):
par new generation total 294912K, used 27099K [0x0000000687800000, 0x000000069b800000, 0x000000069b800000)
eden space 262144K, 0% used [0x0000000687800000, 0x0000000687800000, 0x0000000697800000)
from space 32768K, 82% used [0x0000000697800000, 0x0000000699276df0, 0x0000000699800000)
to space 32768K, 0% used [0x0000000699800000, 0x0000000699800000, 0x000000069b800000)
concurrent mark-sweep generation total 4792320K, used 2066069K [0x000000069b800000, 0x00000007c0000000, 0x00000007c0000000)
Metaspace used 282303K, capacity 297017K, committed 309256K, reserved 1320960K
class space used 33038K, capacity 36852K, committed 38872K, reserved 1048576K
}


Investigating GC pause happens between GC invocations 7969 and 7970. And the amount of used space in meta space is almost the same(it's actually increased)



So, It looks like it's not actually some stall classes which aren't used anymore(since no space was cleared) and it's not safe point reaching issue - since blocking of threads took small time(0.0014133).



How to investigate such case and what diagnostic information is required for proper preparedness.



Technical details



Centos5 + JDK8 + CMS GC with args:



-XX:+CMSClassUnloadingEnabled 
-XX:CMSIncrementalDutyCycleMin=10
-XX:+CMSIncrementalPacing
-XX:CMSInitiatingOccupancyFraction=50
-XX:+CMSParallelRemarkEnabled
-XX:+DisableExplicitGC
-XX:InitialHeapSize=5242880000
-XX:MaxHeapSize=5242880000
-XX:MaxNewSize=335544320
-XX:MaxTenuringThreshold=6
-XX:NewSize=335544320
-XX:OldPLABSize=16
-XX:+UseCompressedClassPointers
-XX:+UseCompressedOops
-XX:+UseConcMarkSweepGC
-XX:+UseParNewGC






java-8 garbage-collection jvm-hotspot concurrent-mark-sweep






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Mar 22 at 14:09







Roman Makhlin

















asked Mar 22 at 12:54









Roman MakhlinRoman Makhlin

6831924




6831924












  • 400 ms is a lot for your application? if this is indeed the case, I wonder if you have JVM engineers on your team

    – Eugene
    Mar 22 at 12:56











  • Normally, the word “since” implies a causal relationship. In your sentence, “…since Metaspace is located in native memory, most of the time GC process was blocked by system and did not any useful work”, I can’t see that causal relationship.

    – Holger
    Mar 22 at 13:29











  • @Eugene, yes, we have performance engineers, but they are all busy right now.

    – Roman Makhlin
    Mar 22 at 14:10

















  • 400 ms is a lot for your application? if this is indeed the case, I wonder if you have JVM engineers on your team

    – Eugene
    Mar 22 at 12:56











  • Normally, the word “since” implies a causal relationship. In your sentence, “…since Metaspace is located in native memory, most of the time GC process was blocked by system and did not any useful work”, I can’t see that causal relationship.

    – Holger
    Mar 22 at 13:29











  • @Eugene, yes, we have performance engineers, but they are all busy right now.

    – Roman Makhlin
    Mar 22 at 14:10
















400 ms is a lot for your application? if this is indeed the case, I wonder if you have JVM engineers on your team

– Eugene
Mar 22 at 12:56





400 ms is a lot for your application? if this is indeed the case, I wonder if you have JVM engineers on your team

– Eugene
Mar 22 at 12:56













Normally, the word “since” implies a causal relationship. In your sentence, “…since Metaspace is located in native memory, most of the time GC process was blocked by system and did not any useful work”, I can’t see that causal relationship.

– Holger
Mar 22 at 13:29





Normally, the word “since” implies a causal relationship. In your sentence, “…since Metaspace is located in native memory, most of the time GC process was blocked by system and did not any useful work”, I can’t see that causal relationship.

– Holger
Mar 22 at 13:29













@Eugene, yes, we have performance engineers, but they are all busy right now.

– Roman Makhlin
Mar 22 at 14:10





@Eugene, yes, we have performance engineers, but they are all busy right now.

– Roman Makhlin
Mar 22 at 14:10












0






active

oldest

votes












Your Answer






StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);













draft saved

draft discarded


















StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55300060%2fcms-class-unloading-took-much-time%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown

























0






active

oldest

votes








0






active

oldest

votes









active

oldest

votes






active

oldest

votes















draft saved

draft discarded
















































Thanks for contributing an answer to Stack Overflow!


  • Please be sure to answer the question. Provide details and share your research!

But avoid


  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55300060%2fcms-class-unloading-took-much-time%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

Obelisk of Theodosius Contents History Description Notes Bibliography Further reading External links Navigation menuAge of spirituality : late antique and early Christian art, third to seventh centuryOver 60 picturesObelisks of the World41°00′21.24″N 28°58′31.43″E / 41.0059000°N 28.9753972°E / 41.0059000; 28.97539727724550-7235741376235741376

밀양 대씨 역사 각주 함께 보기 둘러보기 메뉴밀양 대씨

1973년 목차 사건 문화 탄생 사망 노벨상 달력 둘러보기 메뉴