macOS Memory Management Issues - Memory Leaking Followed By Autorelease Crash

[Kelmon]

OK, this topic is a follow-on to the NSMutableArray problem that I was having earlier in order to try and keep the topic focused and, hopefully, keep things going to a solution.

For new readers to the topic the synopsis of my application is this:

"The application is intended to generate XML files based on a template XML (provided as a resource file to the application) using the Cocoa XML API (classes like NSXMLDocument, NSXMLNode and NSXMLElement). Each "primary" document will have a number of related sub XML documents created where the primary document references the sub documents. The files created are written into directories created in the local file system with a new directory created after each 1000 primary files has been created to try and maintain performance (OS X seems to struggle when a large number of files are present in a directory). The application should be able generate approximately 4,000 primary documents and 120,000 sub documents in a run."

The program is expected to operate as follows:

1. The user enters an integer value representing the number of primary "invoice" XML documents to create and another for the number of related "shipment" documents. The application, when it's working correctly, is expected to create 3 sub XML documents for each shipment requested. For example, if the user enters 10 as the number of invoice documents and 5 as the number of required shipment documents per invoice the application is expected to create 10 + ( 10 x 5 x 3 ) documents, or 160 XML files in total.

2. The IBAction method takes the user's input and, in a While Loop, creates the 10 primary XML documents (instances of MMTriggerXMLDoc). In the initWith... method for the primary XML document a number of arguments are passed, including an instance of NSDictionary and NSXMLDocument (the NSXMLDocument now being created in the delegate method of AppController). The MMTriggerXMLDoc class uses accessor methods to store these arguments as local variables and sends the received object instances a retain message. The exception to this is for the instance of NSXMLDocument and maybe this where my problem lies. Since the NSXMLDocument created during application initialisation is meant to be a template the instance of MMTriggerXMLDoc creates a copy of the NSXMLDocument using the -copyNSXMLDocument)aDocument method. A reasonable amount of the setup of the object is performed using the initWith method of the superclass to MMTriggerXMLDoc, MMXMLDoc. Once the basic setup of the instance of MMTrigerXMLDoc is complete the object changes a number of the NSXMLElement values that makes it unique using a utility method I created in the superclass, MMXMLDoc.

3. As part of the setup process for the primary XML document, the sub documents required for that particular primary document are created (instances of MMSubXMLDoc) and stored in a local instance of NSMutableArray. Again, much of the setup is performed by the superclass, MMXMLDoc, and the NSXMLDocument created is again a copy of the original NSXMLDocument object loaded by the application at launch.

4. Once setup is complete the AppController object (still in the IBAction method's While Loop) requests the instance of MMTriggerXMLDoc to write itself to the file system using the -writeXMLDocumentToFileInDirectoryNSString *)directoryName method). In MMTriggerXMLDoc this method is an overridden version of one defined in the superclass and it is used to loop through the instance of NSMutableArray that holds the sub XML documents and sends each the -writeXMLDocumentToFileInDirectoryNSString *)directoryName message. Once this loop is completed the method calls the superclass implementation in order to write itself out to the file system.

5. Once the sub documents and primary document have been written to the file system the application releases the instance of MMTriggerXMLDoc which in turn calls a release method for all its instance variables, including the instance of NSMutableArray that is holding the matching instances of MMSubXMLDoc. This release occurs after each iteration of the While Loop that creates and writes out the XML files.


At the present time the following problems exist with the application:

1. There is a memory leak somewhere that is causing the application to use a lot of memory while it is running. When requesting 1000 invoice documents each with 5 shipments (16,000 XML documents in total) the memory consumed started at around 14MB and eventually consumed around 150MB.

2. The application crashes due to the Autorelease mechanism sending a release message at the end of the While Loop (i.e. at the end of the event triggered by the user pressing the Generate button).

Now, as best as I can tell I have applied the correct memory management techniques that I have read about. Namely, I have explicitly retained any objects that I need to hang onto and then released them in the appropriate dealloc method. Objects that have been created using init and copy methods have been explicitly released when I am finished with them. Objects that have been passed to me by other methods (such as class methods like [NSNumber numberWithIntint)aInt]) are left to the Autorelease mechanism to release once the event has been completed. Evidentially, however, I am not releasing something that I should be else I should not be seeing a consistently increasing memory usage by the application. Further, I must be releasing something manually that I should not be else the Autorelease mechanism should be causing a crash once the main While Loop that generates and writes out the XML files completes.

The latest version of my project can be downloaded from my iDisk at the following URL:

http://idisk.mac.com/mpeaston-Public?view=web

Suggestions on what is causing the memory leak would be much appreciated since this is the most pressing problem. Ideally, each iteration of the main While Loop in the -createDocs: method of AppController should start with only the basic memory usage required by the application to initialise and release any memory used before the next iteration runs. The Autorelease problem is an annoyance but one what only occurs once the application has effectively finished running.

 

[caveman_uk]

There is a memory leak somewhere that is causing the application to use a lot of memory while it is running. When requesting 1000 invoice documents each with 5 shipments (16,000 XML documents in total) the memory consumed started at around 14MB and eventually consumed around 150MB.

1. You don't release newTriggerDoc if writeXMLDocumentToFileInDirectory: fails in createDocs:

2. The application crashes due to the Autorelease mechanism sending a release message at the end of the While Loop (i.e. at the end of the event triggered by the user pressing the Generate button).

You don't need the releases below as they are already autoreleased

		[sequenceNbr release];
		sequenceNbr = [NSNumber numberWithInt:nextSequenceNbr];
		
		// Update the invoice number for the next trigger doc
		int newInvoiceNbr = [nextInvNumber intValue] + 1;
		[nextInvNumber release];
		nextInvNumber = [NSNumber numberWithInt:newInvoiceNbr];

or here

			[outputSubDirectory release];
			outputSubDirectory = nil;
			outputSubDirectory = [NSNumber numberWithInt:nextSubDirectory];

There may be other stuff around but I'd watch out for trying to release stuff that's autoreleased already.

 

[Kelmon]

Thanks for the response.

I'll add in the code to release the newTriggerDoc if the writing of it to the file system is unsuccessful. I don't expect this to help much since, as yet, I've not had a failure but it would be good programming practice to at least include the release.

I'll delete the release code for the Autoreleased objects that you noted and maybe that'll stop the crash at the end of the event. Good call there, I think.

 

[Kelmon]

I'll delete the release code for the Autoreleased objects that you noted and maybe that'll stop the crash at the end of the event. Good call there, I think.

Well, it was a good catch but the application still crashes at the end when the Autorelease runs. I've been through my code a few times looking for something else that will be set to autorelease and that I am explicitly releasing but I'm damned if I can see something. My experienced eyes than mine are definitely required here...

I don't suppose there is a way to identify the object the object that the Autorelease was trying to release when it crashed based on the memory address it was attempting access, is there?

 

[caveman_uk]

BTW You leak memory here

	// Attempt to create new NSXMLDocument from template file
	[self setXmlTemplate:[[[NSXMLDocument alloc] initWithContentsOfURL:furl
															   options:(NSXMLNodePreserveWhitespace|NSXMLNodePreserveCDATA)
																 error:&err] autorelease]];
	
	// If load failed, try again with different options
	if ([self xmlTemplate] == nil) {
        [self setXmlTemplate:[[[NSXMLDocument alloc] initWithContentsOfURL:furl
																   options:NSXMLDocumentTidyXML
																	 error:&err] autorelease]];

As an aside, you seem too keen on alloc..init. Many newbies are. You should learn that there are many class methods that are available that give autoreleased objects.

 

[Kelmon]

BTW You leak memory here

	// Attempt to create new NSXMLDocument from template file
	[self setXmlTemplate:[[[NSXMLDocument alloc] initWithContentsOfURL:furl
															   options:(NSXMLNodePreserveWhitespace|NSXMLNodePreserveCDATA)
																 error:&err] autorelease]];
	
	// If load failed, try again with different options
	if ([self xmlTemplate] == nil) {
        [self setXmlTemplate:[[[NSXMLDocument alloc] initWithContentsOfURL:furl
																   options:NSXMLDocumentTidyXML
																	 error:&err] autorelease]];

As an aside, you seem too keen on alloc..init. Many newbies are. You should learn that there are many class methods that are available that give autoreleased objects.

Hmm, the irony here is that the leaking code comes directly from Apple's own documentation in Xcode and online. Anyway, how would you recommend that I avoid this leak? More importantly, why does this code leak?

With respect to autorelease, is there a way to "force" it to happen? As you may note from the introduction to my project, the application is expected to generate thousands of XML documents and, at present, this is all occurring during a single event. My understanding is that autorelease only happens at the end of the event so it doesn't look like it is going to help here since, by the end of the event, I'll have run out of memory that could have been released ages ago.

One thing that I am looking at the moment is spawning a new thread to perform the actual generation and output of the XML files. At present the application becomes unresponsive immediately and, while this isn't really a problem, I'm sure that it is bad programming practice. I am also wondering whether doing this can enable me to take better control of the memory management of the application, plus allow me to provide a "Cancel" operation that can terminate the generation process.

 

[caveman_uk]

Hmm, the irony here is that the leaking code comes directly from Apple's own documentation in Xcode and online. Anyway, how would you recommend that I avoid this leak? More importantly, why does this code leak?

Actually it doesn't. I didn't see the autorelease at the very very far right. Sorry

 

[Kelmon]

I think previous versions of the application explicitly released rather than autoreleased the template XML document when it was loaded. Still, the biggest problem, as best I can tell, is coming from the main event loop when the user clicks the Generate button. When watching the application in the Activity Monitor you can see the memory usage climbing away so something is evidentially not being released. What I can't understand is that I explicitly release all the instance variables that I am passed as arguments in the dealloc methods for my custom classes so I'm damned if I can see where the memory is going.

At this rate I think I will need to put in NSLog messages throughout the application to report the retaincount: for each variable at each stage of the application. That'll be fun...