XCTests crashing user session back to login screen


Alex Zavatone
 

Has anyone seen cases where running XCTests will crash the logged in user session out to the user login screen?

Every time it happens, there are no logs that I can find explaining the cause. My suspicion is memory leaks in the iOS app I am testing and zombie pointers writing to memory outside of the simulator.

Thanks in advance.


 

On Oct 1, 2019, at 1:20 PM, Alex Zavatone via Groups.Io <zav=mac.com@groups.io> wrote:

Has anyone seen cases where running XCTests will crash the logged in user session out to the user login screen?
Yikes. File a bug report with Apple — that’s got to be a bug in a system framework or service.

Every time it happens, there are no logs that I can find explaining the cause. My suspicion is memory leaks in the iOS app I am testing and zombie pointers writing to memory outside of the simulator.
No, that hasn’t been possible since the OS 9 days. Every process is isolated in its own address space, and there should be no way for an application process to break other apps or your login session. What’s happening sounds like a crash of the top-level process of your login session (is it still called loginwindow?) or of the WindowServer.

—Jens


Alex Zavatone
 

What’s scary is that I have screenshots of my terminal session wondows getting corrupted and also safari menus turning into checkerboard patterns.

We had a metric shitton of zombie pointers that I mostly cleaned up, but it’s scary watching a 16 GB Mac either reboot or just kick the user back to the login screen while crashing out of the user session and with no logs at all.

On Oct 2, 2019, at 10:47 AM, Jens Alfke <jens@mooseyard.com> wrote:


On Oct 1, 2019, at 1:20 PM, Alex Zavatone via Groups.Io <zav=mac.com@groups.io> wrote:

Has anyone seen cases where running XCTests will crash the logged in user session out to the user login screen?
Yikes. File a bug report with Apple — that’s got to be a bug in a system framework or service.

Every time it happens, there are no logs that I can find explaining the cause. My suspicion is memory leaks in the iOS app I am testing and zombie pointers writing to memory outside of the simulator.
No, that hasn’t been possible since the OS 9 days. Every process is isolated in its own address space, and there should be no way for an application process to break other apps or your login session. What’s happening sounds like a crash of the top-level process of your login session (is it still called loginwindow?) or of the WindowServer.

—Jens



Jeremy Hughes
 

On 2 Oct 2019, at 16:47, Jens Alfke <jens@mooseyard.com> wrote:

Every time it happens, there are no logs that I can find explaining the cause. My suspicion is memory leaks in the iOS app I am testing and zombie pointers writing to memory outside of the simulator.
No, that hasn’t been possible since the OS 9 days. Every process is isolated in its own address space, and there should be no way for an application process to break other apps or your login session. What’s happening sounds like a crash of the top-level process of your login session (is it still called loginwindow?) or of the WindowServer.
Apple’s support page (https://support.apple.com/en-gb/HT200553) suggests that it is still possible for an application to bring down the system:

"If your Mac suspects that a particular app caused the restart, it might ask whether you would like to move the app to the Trash. Click Move to Trash, then contact the software developer to see if a software update is available.”

Jeremy


Alex Zavatone
 

Hahaha. I love that note from the support page. Move Xcode and the Simulator to the trash?

Honestly, I also thought that it was impossible for an app to write outside of its memory space as of OSX. With what I was/am seeing, it is worrisome.

FYI, for Jens, there were no crash logs in user or system space for the loginWindow or whatever that process is called.

And even more FYI, the UITest session just did the same thing of crashing the user session back to the login screen on MacOS 10.4.6 and Xcode 10.2.1 just an hour ago.

On Oct 3, 2019, at 6:46 AM, Jeremy Hughes via Groups.Io <moon.rabbit=virginmedia.com@groups.io> wrote:

On 2 Oct 2019, at 16:47, Jens Alfke <jens@mooseyard.com> wrote:

Every time it happens, there are no logs that I can find explaining the cause. My suspicion is memory leaks in the iOS app I am testing and zombie pointers writing to memory outside of the simulator.
No, that hasn’t been possible since the OS 9 days. Every process is isolated in its own address space, and there should be no way for an application process to break other apps or your login session. What’s happening sounds like a crash of the top-level process of your login session (is it still called loginwindow?) or of the WindowServer.
Apple’s support page (https://support.apple.com/en-gb/HT200553) suggests that it is still possible for an application to bring down the system:

"If your Mac suspects that a particular app caused the restart, it might ask whether you would like to move the app to the Trash. Click Move to Trash, then contact the software developer to see if a software update is available.”

Jeremy




 



On Oct 3, 2019, at 4:46 AM, Jeremy Hughes via Groups.Io <moon.rabbit@...> wrote:

Apple’s support page (https://support.apple.com/en-gb/HT200553) suggests that it is still possible for an application to bring down the system:

Only if it triggers some kind of bug in the kernel or in a system-level framework. Bugs do exist.


On Oct 3, 2019, at 10:36 AM, Alex Zavatone via Groups.Io <zav@...> wrote:

Honestly, I also thought that it was impossible for an app to write outside of its memory space as of OSX.  With what I was/am seeing, it is worrisome.

It _is_ impossible for an app to write outside its memory space. (For one thing, other processes' memory isn't even mapped into the address space, so it literally has no address to be written to.) This is true of any current OS, i.e. anything fancier than an Arduino.

What _is_ possible is for an OS bug to mess up or crash a shared system process like the WindowServer from a request made by an app. Or for a kernel bug to crash the OS inside a system call made by an app.

—Jens


Sak Wathanasin
 



On 3 Oct 2019, at 20:10, Jens Alfke <jens@...> wrote:



On Oct 3, 2019, at 4:46 AM, Jeremy Hughes via Groups.Io <moon.rabbit@...> wrote:

Apple’s support page (https://support.apple.com/en-gb/HT200553) suggests that it is still possible for an application to bring down the system:

Only if it triggers some kind of bug in the kernel or in a system-level framework. Bugs do exist.


3rd party KEXTs more likely to be the culprit; OSX kernel panics are thankfully rare these days.

What _is_ possible is for an OS bug to mess up or crash a shared system process like the WindowServer from a request made by an app. Or for a kernel bug to crash the OS inside a system call made by an app.

Or some kind of h/w issue in Alex's case trigeering a kernel bug? Maybe dead or dying video or system memory, disk? Does it only happen in a particular Mac or everywhere?

Are there any recent core dumps in /cores? Not sure if this dir is created by defaul these days - you may have to manually create it if you want to enable core dumps.

Sak


Alex Zavatone
 



On Oct 3, 2019, at 2:10 PM, Jens Alfke <jens@...> wrote:



On Oct 3, 2019, at 4:46 AM, Jeremy Hughes via Groups.Io <moon.rabbit@...> wrote:

Apple’s support page (https://support.apple.com/en-gb/HT200553) suggests that it is still possible for an application to bring down the system:

Only if it triggers some kind of bug in the kernel or in a system-level framework. Bugs do exist.


On Oct 3, 2019, at 10:36 AM, Alex Zavatone via Groups.Io <zav@...> wrote:

Honestly, I also thought that it was impossible for an app to write outside of its memory space as of OSX.  With what I was/am seeing, it is worrisome.

It _is_ impossible for an app to write outside its memory space. (For one thing, other processes' memory isn't even mapped into the address space, so it literally has no address to be written to.) This is true of any current OS, i.e. anything fancier than an Arduino.

What _is_ possible is for an OS bug to mess up or crash a shared system process like the WindowServer from a request made by an app. Or for a kernel bug to crash the OS inside a system call made by an app.

—Jens

Yeah, that was my understanding too.  I wonder if this also applies to the video memory?  I’ll post screenshots of Terminal window and Safari menu data corruption when I get into the office.  

TY, as always.




Alex Zavatone
 



On Oct 4, 2019, at 5:02 AM, Sak Wathanasin <sw@...> wrote:



On 3 Oct 2019, at 20:10, Jens Alfke <jens@...> wrote:



On Oct 3, 2019, at 4:46 AM, Jeremy Hughes via Groups.Io <moon.rabbit@...> wrote:

Apple’s support page (https://support.apple.com/en-gb/HT200553) suggests that it is still possible for an application to bring down the system:

Only if it triggers some kind of bug in the kernel or in a system-level framework. Bugs do exist.


3rd party KEXTs more likely to be the culprit; OSX kernel panics are thankfully rare these days.

What _is_ possible is for an OS bug to mess up or crash a shared system process like the WindowServer from a request made by an app. Or for a kernel bug to crash the OS inside a system call made by an app.

Or some kind of h/w issue in Alex's case trigeering a kernel bug? Maybe dead or dying video or system memory, disk? Does it only happen in a particular Mac or everywhere?

Reproduced on 4 systems.

Are there any recent core dumps in /cores? Not sure if this dir is created by defaul these days - you may have to manually create it if you want to enable core dumps.


AHA!  How might I do that, Sak?
 
I’m looking for any tips on tracking this down, because 1) It honestly scares me and 2) I’ve never seen any crash like this before.

Seriously.  XCUITests are running using KIF (https://github.com/kif-framework/KIF) and somewhere about 30 minutes into the tests, while I am watching the screen, the Mac either reboots or kicks the user out of their logged in session and the computer presents the login screen.

Just last night after I reported this, I subclassed invokeTest() and are actually logging to a file the memory used and the test being run, so at least I have a file on disk that indicates the tests completed and the last test being run when the crash happens. Now to see if it always crashes at the same test.  : /

What I have seen is that when running UITests in the Simulator from Xcode 10.2.1 and logging to the console, the console uses up an immense amount of memory.  When viewing the memory pane while running, simply by clearing the console output and then hiding the output pane, up to 4 GB of RAM was returned. There’s no way in hell we are outputting 4 GB of RAM in the process of running 278 tests.

I have seen that it appears that memory free seems to approach < 2 GB when I’ve actually witnesses the crash in front of my own eyes, but this has also happened on Macs with 16, 24, 32 and 64 GB of RAM.  

Would the amount of Mach ports being used make any difference?  I’ve seen up to 6,000+ judging by the Activity Monitor.

We do have a ton of retain cycles thanks to idiotic use of the VIPER pattern and only using strong properties and vars. (I demand that you shoot me now.) I’m in the middle of the hell of going through this.  A note to the reader: Object graphs should not be circular.  That is bad.

Thanks in advance on this, everyone.  


Sak



Alex Zavatone
 

At the time of one of the crashes, I did find a rather odd message in the logs.  Unrecognized MacService property?  Any ideas if thie even matters at all?

Oct  3 17:35:53 iOS-Test com.apple.xpc.launchd[1] (com.apple.xpc.launchd.user.domain.503.100454.Aqua): com.apple.ReportCrash (lint): Unrecognized MachService property: DrainMessagesOnCrash
Oct  3 17:35:53 iOS-Test com.apple.xpc.launchd[1] (com.apple.xpc.launchd.user.domain.503.100454.Aqua): com.apple.ReportCrash.Self (lint): Unrecognized MachService property: DrainMessagesOnCrash
Oct  3 17:36:24 iOS-Test ReportCrash[95007]: assertion failed: 18G95: libxpc.dylib + 23654 [7DEE2300-6D8E-3C00-9C63-E3E80D56B0C4]: 0xf
 
 
 
The logging that I added to the test runs indicates that the last entry was at 2019-10-03 17:35:17.251 before crashing the user session.  I was watching while this happened.  However, there are 36 seconds between the last entry in my log (indicating a test had started) and the log entry of `Unrecognized MachService property: DrainMessagesOnCrash`.
 
2019-10-03 17:34:04.655  ####█████ Physical memory used before starting test: 490.4609375 MB
2019-10-03 17:34:04.655  ####█████ Test: -[UserViewsLegacyLoanOverview testShowLoanSelector]
2019-10-03 17:34:04.657  ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
2019-10-03 17:34:36.650  ####█████ Physical memory used before starting test: 500.41015625 MB
2019-10-03 17:34:36.650  ####█████ Test: -[UserViewsLegacyLoanOverview testShowsLenderExpenses]
2019-10-03 17:34:36.652  ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
2019-10-03 17:34:41.820  ####█████ Physical memory used before starting test: 498.359375 MB
2019-10-03 17:34:41.820  ####█████ Test: -[UserViewsLegacyLoanOverview testShowsNoLenderExpenses]
2019-10-03 17:34:41.822  ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
2019-10-03 17:34:48.309  ####█████ Physical memory used before starting test: 497.9609375 MB
2019-10-03 17:34:48.310  ####█████ Test: -[UserViewsLegacyLoanOverview testTwoActiveLoanInMultiLoan]
2019-10-03 17:34:48.311  ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
2019-10-03 17:35:13.968  ####█████ Physical memory used before starting test: 496.8515625 MB
2019-10-03 17:35:13.970  ####█████ Test: -[UserViewsLegal testEstimatedHomeValueDisclosure]
2019-10-03 17:35:13.970  ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
2019-10-03 17:35:17.249  ####█████ Physical memory used before starting test: 499.82421875 MB
2019-10-03 17:35:17.249  ####█████ Test: -[UserViewsLegal testEstimatedHomeValueDisclosureForNoHomeSnapshot]
2019-10-03 17:35:17.251  ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––

On Oct 4, 2019, at 5:02 AM, Sak Wathanasin <sw@...> wrote:



On 3 Oct 2019, at 20:10, Jens Alfke <jens@...> wrote:



On Oct 3, 2019, at 4:46 AM, Jeremy Hughes via Groups.Io <moon.rabbit@...> wrote:

Apple’s support page (https://support.apple.com/en-gb/HT200553) suggests that it is still possible for an application to bring down the system:

Only if it triggers some kind of bug in the kernel or in a system-level framework. Bugs do exist.


3rd party KEXTs more likely to be the culprit; OSX kernel panics are thankfully rare these days.

What _is_ possible is for an OS bug to mess up or crash a shared system process like the WindowServer from a request made by an app. Or for a kernel bug to crash the OS inside a system call made by an app.

Or some kind of h/w issue in Alex's case trigeering a kernel bug? Maybe dead or dying video or system memory, disk? Does it only happen in a particular Mac or everywhere?

Are there any recent core dumps in /cores? Not sure if this dir is created by defaul these days - you may have to manually create it if you want to enable core dumps.

Sak