Aborted Hudson Jobs

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Aborted Hudson Jobs

Jayaprakash Arthanareeswaran
I have seen three of my patches suffered aborted Hudson Jobs yesterday. Anyone has any clue of what's going on? Is this about concurrent jobs? I don't see the option being enabled, though.
 
Regards,
Jay


_______________________________________________
jdt-core-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jdt-core-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Aborted Hudson Jobs

Stephan Herrmann-2
#2699 has this:

14:38:18  Build timed out (after 169 minutes). Marking the build as aborted.

this is from #2687:

08:00:32  Build timed out (after 156 minutes). Marking the build as aborted.

I don't see a reason in #2697, though.

best,
Stephan

On 13.04.2017 05:22, Jayaprakash Arthanareeswaran wrote:

> I have seen three of my patches suffered aborted Hudson Jobs yesterday. Anyone has any clue of what's going on? Is this about
> concurrent jobs? I don't see the option being enabled, though.
>
> Regards,
> Jay
>
>
>
> _______________________________________________
> jdt-core-dev mailing list
> [hidden email]
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> https://dev.eclipse.org/mailman/listinfo/jdt-core-dev
>

_______________________________________________
jdt-core-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jdt-core-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Aborted Hudson Jobs

Jayaprakash Arthanareeswaran
Hi Stephan,
 
I did see those time out messages, but not much explanation in the console though. I was wondering why it was taking 160+ minutes in the first place, but I can't figure out.
 
Jay
 
----- Original message -----
From: Stephan Herrmann <[hidden email]>
Sent by: [hidden email]
To: [hidden email]
Cc:
Subject: Re: [jdt-core-dev] Aborted Hudson Jobs
Date: Thu, Apr 13, 2017 3:43 PM
 
#2699 has this:

14:38:18  Build timed out (after 169 minutes). Marking the build as aborted.

this is from #2687:

08:00:32  Build timed out (after 156 minutes). Marking the build as aborted.

I don't see a reason in #2697, though.

best,
Stephan

On 13.04.2017 05:22, Jayaprakash Arthanareeswaran wrote:

> I have seen three of my patches suffered aborted Hudson Jobs yesterday. Anyone has any clue of what's going on? Is this about
> concurrent jobs? I don't see the option being enabled, though.
>
> Regards,
> Jay
>
>
>
> _______________________________________________
> jdt-core-dev mailing list
> [hidden email]
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> https://dev.eclipse.org/mailman/listinfo/jdt-core-dev
>

_______________________________________________
jdt-core-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jdt-core-dev

 
 


_______________________________________________
jdt-core-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jdt-core-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Aborted Hudson Jobs

Stephan Herrmann-2
On 13.04.2017 14:02, Jayaprakash Arthanareeswaran wrote:
> Hi Stephan,
>
> I did see those time out messages, but not much explanation in the console though. I was wondering why it was taking 160+ minutes in
> the first place, but I can't figure out.

Good question.

One more detail from #2699:
13:08:55  Running org.eclipse.jdt.core.tests.dom.RunAllTests
14:38:18  Build timed out (after 169 minutes). Marking the build as aborted.

=> dom tests had been running for 90 minutes without completion and no intermediate sign of life.

For #2687 it "froze" in the middle of model tests (80 min. of quiescence):

06:30:48  Running org.eclipse.jdt.core.tests.model.AllJavaModelTests
06:35:59  [Ljava/util/ArrayList;]
06:36:00  [Ljava/lang/reflect/Array;, Ljava/sql/Array;, Ljava/util/concurrent/ArrayBlockingQueue;, Ljava/util/ArrayDeque;,
Ljava/lang/ArrayIndexOutOfBoundsException;, Ljava/util/ArrayList;, Ljava/util/ArrayPrefixHelpers;, Ljava/util/Arrays;,
Ljava/util/ArraysParallelSortHelpers;, Ljava/lang/ArrayStoreException;]
06:36:00  [Ljava/util/concurrent/ThreadPoolExecutor$AbortPolicy;, Ljava/util/AbstractCollection;,
Ljava/util/Spliterators$AbstractDoubleSpliterator;, Ljava/util/concurrent/AbstractExecutorService;,
Ljava/nio/channels/spi/AbstractInterruptibleChannel;, Ljava/util/Spliterators$AbstractIntSpliterator;, Ljava/util/AbstractList;,
Ljava/util/Spliterators$AbstractLongSpliterator;, Ljava/util/AbstractMap;, Ljava/lang/AbstractMethodError;]
06:36:19  Encoding tests using Workspace charset: UTF-8 and VM charset: UTF-8
06:36:20  Bug 110576 is not tested under linux os...
06:37:41  ClasspathTests#testInvalidClasspath1() may generate an expected Fatal Error...
06:37:41  [Fatal Error] :3:43: Element type "classpathentry" must be followed by either attribute specifications, ">" or "/>".
06:40:19  testExternalJarChanged5
08:00:32  Build timed out (after 156 minutes). Marking the build as aborted.

So, we are not looking for general slowness, but more like a bad freeze / deadlockish kind of thing.

Not good.

Looking at the hudson config:

timeout is "elastic": 200% of last non-failing job or 180 min total. Looks good.

I was looking for more options on dumping diagnostics when stuck, but found nothing.
Was it an old PDE/Build option that allowed to do things like take a screenshot when it timeouts?
Thread dumps would be perfect, right? I just filed https://bugs.eclipse.org/515272

Next time it happens, I suggest to search the hudson workspaces in locations like
target/surefire-reports for any traces/dumps/logs. Unfortunately, that information
is lost as soon as the next job starts, so you have to be quick :)

Stephan

>
>     ----- Original message -----
>     From: Stephan Herrmann <[hidden email]>
>     Sent by: [hidden email]
>     To: [hidden email]
>     Cc:
>     Subject: Re: [jdt-core-dev] Aborted Hudson Jobs
>     Date: Thu, Apr 13, 2017 3:43 PM
>
>     #2699 has this:
>
>     14:38:18  Build timed out (after 169 minutes). Marking the build as aborted.
>
>     this is from #2687:
>
>     08:00:32  Build timed out (after 156 minutes). Marking the build as aborted.
>
>     I don't see a reason in #2697, though.
>
>     best,
>     Stephan
>
>     On 13.04.2017 05:22, Jayaprakash Arthanareeswaran wrote:
>     > I have seen three of my patches suffered aborted Hudson Jobs yesterday. Anyone has any clue of what's going on? Is this about
>     > concurrent jobs? I don't see the option being enabled, though.
>     >
>     > Regards,
>     > Jay
>     >
>     >
>     >
>     > _______________________________________________
>     > jdt-core-dev mailing list
>     > [hidden email]
>     > To change your delivery options, retrieve your password, or unsubscribe from this list, visit
>     > https://dev.eclipse.org/mailman/listinfo/jdt-core-dev
>     >
>
>     _______________________________________________
>     jdt-core-dev mailing list
>     [hidden email]
>     To change your delivery options, retrieve your password, or unsubscribe from this list, visit
>     https://dev.eclipse.org/mailman/listinfo/jdt-core-dev
>
>
>
>
>
>
> _______________________________________________
> jdt-core-dev mailing list
> [hidden email]
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> https://dev.eclipse.org/mailman/listinfo/jdt-core-dev
>

_______________________________________________
jdt-core-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jdt-core-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Aborted Hudson Jobs

Markus Keller
I've filed
Bug 515274: JDT Core test freezes in Gerrit Hudson job
and created
https://git.eclipse.org/r/#/c/95044/
to hopefully get more live insight about test execution.

Markus




From:        Stephan Herrmann <[hidden email]>
To:        [hidden email]
Date:        2017-04-14 00:26
Subject:        Re: [jdt-core-dev] Aborted Hudson Jobs
Sent by:        [hidden email]




On 13.04.2017 14:02, Jayaprakash Arthanareeswaran wrote:
> Hi Stephan,
>
> I did see those time out messages, but not much explanation in the console though. I was wondering why it was taking 160+ minutes in
> the first place, but I can't figure out.

Good question.

One more detail from #2699:
13:08:55  Running org.eclipse.jdt.core.tests.dom.RunAllTests
14:38:18  Build timed out (after 169 minutes). Marking the build as aborted.

=> dom tests had been running for 90 minutes without completion and no intermediate sign of life.

For #2687 it "froze" in the middle of model tests (80 min. of quiescence):

06:30:48  Running org.eclipse.jdt.core.tests.model.AllJavaModelTests
06:35:59  [Ljava/util/ArrayList;]
06:36:00  [Ljava/lang/reflect/Array;, Ljava/sql/Array;, Ljava/util/concurrent/ArrayBlockingQueue;, Ljava/util/ArrayDeque;,
Ljava/lang/ArrayIndexOutOfBoundsException;, Ljava/util/ArrayList;, Ljava/util/ArrayPrefixHelpers;, Ljava/util/Arrays;,
Ljava/util/ArraysParallelSortHelpers;, Ljava/lang/ArrayStoreException;]
06:36:00  [Ljava/util/concurrent/ThreadPoolExecutor$AbortPolicy;, Ljava/util/AbstractCollection;,
Ljava/util/Spliterators$AbstractDoubleSpliterator;, Ljava/util/concurrent/AbstractExecutorService;,
Ljava/nio/channels/spi/AbstractInterruptibleChannel;, Ljava/util/Spliterators$AbstractIntSpliterator;, Ljava/util/AbstractList;,
Ljava/util/Spliterators$AbstractLongSpliterator;, Ljava/util/AbstractMap;, Ljava/lang/AbstractMethodError;]
06:36:19  Encoding tests using Workspace charset: UTF-8 and VM charset: UTF-8
06:36:20  Bug 110576 is not tested under linux os...
06:37:41  ClasspathTests#testInvalidClasspath1() may generate an expected Fatal Error...
06:37:41  [Fatal Error] :3:43: Element type "classpathentry" must be followed by either attribute specifications, ">" or "/>".
06:40:19  testExternalJarChanged5
08:00:32  Build timed out (after 156 minutes). Marking the build as aborted.

So, we are not looking for general slowness, but more like a bad freeze / deadlockish kind of thing.

Not good.

Looking at the hudson config:

timeout is "elastic": 200% of last non-failing job or 180 min total. Looks good.

I was looking for more options on dumping diagnostics when stuck, but found nothing.
Was it an old PDE/Build option that allowed to do things like take a screenshot when it timeouts?
Thread dumps would be perfect, right? I just filed
https://bugs.eclipse.org/515272

Next time it happens, I suggest to search the hudson workspaces in locations like
target/surefire-reports for any traces/dumps/logs. Unfortunately, that information
is lost as soon as the next job starts, so you have to be quick :)

Stephan

>
>     ----- Original message -----
>     From: Stephan Herrmann <[hidden email]>
>     Sent by: [hidden email]
>     To: [hidden email]
>     Cc:
>     Subject: Re: [jdt-core-dev] Aborted Hudson Jobs
>     Date: Thu, Apr 13, 2017 3:43 PM
>
>     #2699 has this:
>
>     14:38:18  Build timed out (after 169 minutes). Marking the build as aborted.
>
>     this is from #2687:
>
>     08:00:32  Build timed out (after 156 minutes). Marking the build as aborted.
>
>     I don't see a reason in #2697, though.
>
>     best,
>     Stephan
>
>     On 13.04.2017 05:22, Jayaprakash Arthanareeswaran wrote:
>     > I have seen three of my patches suffered aborted Hudson Jobs yesterday. Anyone has any clue of what's going on? Is this about
>     > concurrent jobs? I don't see the option being enabled, though.
>     >
>     > Regards,
>     > Jay
>     >
>     >
>     >
>     > _______________________________________________
>     > jdt-core-dev mailing list
>     > [hidden email]
>     > To change your delivery options, retrieve your password, or unsubscribe from this list, visit
>     >
https://dev.eclipse.org/mailman/listinfo/jdt-core-dev
>     >
>
>     _______________________________________________
>     jdt-core-dev mailing list
>     [hidden email]
>     To change your delivery options, retrieve your password, or unsubscribe from this list, visit
>    
https://dev.eclipse.org/mailman/listinfo/jdt-core-dev
>
>
>
>
>
>
> _______________________________________________
> jdt-core-dev mailing list
> [hidden email]
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
>
https://dev.eclipse.org/mailman/listinfo/jdt-core-dev
>

_______________________________________________
jdt-core-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jdt-core-dev





_______________________________________________
jdt-core-dev mailing list
[hidden email]
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jdt-core-dev
Loading...