Your computer has run out of memory - yikes!!!

[Simon Knight]

Hi,
This feature may be well known but it just caught me out causing my app to be stopped by Mac OS on the reasonable grounds that the computer had run out of memory. At the time of the error the application running in the IDE was using 69 Gbytes of memory.

So why was this ?

Here is the line that caused the problem:

Code: Select all

put BuildCommand (tFileSpec,tOldKeywordList,pModifiedKWs,tBatchArguments)

It is inside a loop structure

Code: Select all

repeat for each line tLine in pFiles
      put item 1 of tLine into tFileSpec
      put item 2 of tLine into tOldKeywordList  -- pipe delimited
      put BuildCommand (tFileSpec,tOldKeywordList,pModifiedKWs,tBatchArguments) 
   end repeat

Here is the header of the function being called which is fully functional and works as designed. Note that two parameters are passed as reference, they are marked with the @ prefix and one of these is key to the issue.

Code: Select all

Function BuildCommand pFileSpec, pFileKeywords, @pKeywordUpdates, @pBatchArguments
snip
return pBatchArguments

The code runs as written. The loop is building a list of commands based on the contents of two lists. The list is passed by reference so it is getting updated in memory and is available for the parent loop to keep on passing back to the function (I hope that makes sense). The issue is that the variable grows by five or so lines on each pass. On each pass the put command sends the whole variable to the message box which is unable to keep up and buffers all the inputs. After a while this buffering causes the memory issue.

The solution is to correct the error in the calling line by adding the missing words into tBatchArguments

Code: Select all

repeat for each line tLine in pFiles
      put item 1 of tLine into tFileSpec
      put item 2 of tLine into tOldKeywordList  -- pipe delimited
      put BuildCommand (tFileSpec,tOldKeywordList,pModifiedKWs,tBatchArguments) into tBatchArguments
   end repeat

Just a simple software fix

- Doh!

best wishes

Simon

[FourthWorld]

How large is the list, and if it's large what is gained by displaying it in the Message Box?

[Simon Knight]

How large is the list, and if it's large what is gained by displaying it in the Message Box?

Ah Richard,

I think you misunderstood my post - sorry: I messed up, both in my post and in my coding.

I never intended to display the results of the function call in the message box but because I failed to include the correct end to the line of the function call thats what happened. The line should have ended with "into tBatchArguments".

Normally this mistake would cause the code to run incorrectly as on each pass of the loop the function would be called with the empty variable tBatchArguments meaning that it would only ever have the text result of a single call to the function assigned. Unfortunately, because the variable was passed by reference the variable does get extended with each function call and looks o.k. when inspected in the variable inspector.

Initial tests were run with small data sets (collections of image files) with the result that the message box could keep up. Another contributory factor was that the message box was on my second screen so was easy to ignore.

With testing complete I tried to run it against my complete image collection of 100,000 image files. While not all images were processed many tens of thousands were and each imaged processed added a further six of so lines of text to the variable tBatchArguments. Each time the function returned it posted the variable to the message box with the result it had to buffer the tens of thousands very long text variables (30 Mbytes when complete). In a short while the memory filled up, all 69 Gbytes of it.

In summary two features of livecode, the put command defaulting to the message box and passing variables by reference came together along with my omission and a large dataset came together to break the IDE in a way that threw no livecode errors and was not immediately obvious.

My bad !
best wishes

Simon

[FourthWorld]

Ah, now I understand. Thank you. Good you got it sorted.

[bwmilby]

I would suggest changing "buildCommand" to a handler instead of a function. The additional churn of using both a by reference variable but then copying and assigning it probably isn't that good. Since the engine does use "copy on write", memory use may not be an issue but the constant adding/removing references to that variable probably isn't great for performance.

[Simon Knight]

#Brian,

Interesting point and one I had not considered.

I have just run a test :

Using function call version took 332 seconds.
Using command call version took 308 seconds.

Thats approximately a 7% reduction in processing time just as you suggested.

Mind you the output of the routine is passed to exiftool and the total execution time in something like five hours.

best wishes
Simon

[richmond62]

put command defaulting to the message box

This sometimes gets on my nerves, and I wonder if:

pseudo:

put XXX into messageBox

or, perhaps:

put XXX into msgBox


might not be better, so there is NO default.

[richmond62]

passing variables by reference

I would be extremely grateful if someone could explain what this means.

[bwmilby]

“By reference” means that instead of making a copy, that actual variable “reference” (pointer in other languages) is used. This allows changes to the variable inside the handler to be retained. The “@“ in the handler definition is what makes the variable “by reference”. See “pass by reference” in the LC dictionary.

[Simon Knight]

I would be extremely grateful if someone could explain what this means.

Ok I'll try.
When a variable is created and a value assigned the contents take up some of the available memory. When passing a variable to a handler in Livecode the engine defaults to making a copy of the variable passed for the handler to use. If however the @ character is used in the declaration of the handler the variable is passed by reference, it is in effect a pointer to the same block of memory. This code may help to explain the concept :

Code: Select all

put 2022 into tYear -- contents = 2022
-- call first function
put WhatIsNextYear(tYear) into tNextYear

-- Post call tYear will still be 2022 and tNextYear "elephant"

--Call the second function which uses by reference:
put WhatIsNextYearByRef(tYear) into tNextYear

-- Post call both tYear and tNextYear are both now set to "elephant"

-- its the @ character the causes the change of variable passing


......

Function WhatIsNextYear pYear
   -- pYear is independent of tYear in calling code
   put "elephant" into pYear
   return pYear
end WhatIsNextYear

Function WhatIsNextYearByRef @pYear
   -- pYear is NOT independent tYear in calling code, they both reference the same block of memory
   put "elephant" into pYear. -- this line changes both variables because they are one.
   return pYear
end WhatIsNextYear

All untested but you should get the idea. Other languages such as VBA default to always passing by reference.
Passing by reference makes a massive difference to code execution if the variable being passed contains a lot of data. For example in a much earlier post, circa 2012, I described a block of code that was taking fifty plus hours to parse some data which was stored in an array. I think it was Bernd who advised me to swap to passing by reference and the time to execute was cut to approximately 120 seconds.

best wishes
Simon

[Simon Knight]

My earlier post :
https://forums.livecode.com/viewtopic.p ... 180#p32182

It was 56 hours down to 70 seconds all with the addition of one "@" character.

best wishes

Simon

[richmond62]

It was 56 hours down to 70 seconds all with the addition of one "@" character.

Cripes! That difference is incredible.

[Simon Knight]

Cripes! That difference is incredible.

Yes it is. My code was passing a large array of data many many times and all that duplication of data caused a significant slowdown.

best wishes
Simon

[FourthWorld]

Fun and useful trivia about parameter passing: several versions ago Mark Waddingham changes param handling internally to use copy-on-write. Since then, all read operations on passed parameters are done on the original data, the pointer to it all that was passed to the handler. If the engine encounters a statement that would modify the variable, only then does it perform the expensive copy operation.

So in cases like this one, explicitly calling by reference is needed, since the data being worked on is intended to remain the original.

But in read-only cases, there used to be a performance boost from passing by reference but that's now happening across the board automatically for us, no changes to our scripts needed.